Your AI Center of Excellence needs to govern what agents can do, not just what teams can build.

This week in my diary, I’m revisiting the four-part AI CoE series I wrote last year. The question isn’t whether I’d throw it away. It’s what I’d make more explicit before giving an agent permission to change something in the business.

The original series was already about the agentic era. Its foundation was Platform + People + Patterns. I still believe in that foundation. This is a short update, not a replacement.

If you missed the originals, here are all four:

1. The overall model: Winning with AI CoEs in the Agentic Era

2. Platform: The Foundation of an AI CoE

3. People: The Force Multiplier of the AI CoE

4. Patterns: Your Real Competitive Advantage in AI

Think of that last pillar as the bridge to processes: reusable ways of building, releasing, operating, and retiring AI systems.

The moment the operating model changes

Imagine Maya, a customer-operations lead at a retailer. Her AI assistant explains the refund policy. Now she wants it to issue refunds, which means giving it permission to move company money.

The agent can recommend a refund, but the refund service must check whether the action is allowed and whether human approval is required. The agent cannot grant itself permission.

That is what I would add to the CoE’s responsibilities: define what agents can do, enforce those limits, and name the people accountable for their actions.

1. People: name the owner of the action

In the people edition, I proposed a simple division of responsibility: the CoE provides shared tools, rules, and support, while business teams remain responsible for how they use AI and the results it produces.

For every agent, I’d make three responsibilities clear:

• The business owner decides what the agent is allowed to do and when it must ask for help.

• The technical owner manages its access, tests its behavior, monitors it, and handles technical failures.

• The human decision-maker approves or rejects actions that require human judgment.

These do not have to be three different people. For Maya’s refund agent, Maya could own the refund rules, while an engineering lead maintains the controls. Everyone should know who can stop the agent if something goes wrong.

Before deciding how much authority to give an agent, I’d ask three questions from Anthropic’s architect course.

For Maya, drafting a recommendation can stay with the agent. Current order facts and refund rules belong in existing systems. High-stakes exceptions need an authorized person before execution.

Anthropic also discusses calibrated confidence: confidence scores that have been checked against actual results to see whether they are reliable. These scores can help prioritize human review, but the potential harm and difficulty of undoing an action matter more. High confidence alone is not permission to act.

When an action needs human approval, the system should show what the agent wants to do, the relevant facts, and why approval is needed. For Maya’s refund agent, the reviewer should check the order details, refund amount, and applicable policy rather than simply trust the agent’s explanation.

Not every routine response needs this level of review. Teams can spot-check lower-risk work where policy allows, but actions requiring approval must always wait for that approval.

2. Platforms: enforce limits outside the model

The platform edition already included gateways, budgets, approvals, and observability. I’d now be more precise about where enforcement happens.

Below is an AI landing Zone Architecture from Microsoft. I receommend checking out Agent Landing Zone.

The platform team owns the shared infrastructure and guardrails agents run on: identity, gateways, budgets, approval mechanisms, and monitoring. The application team wires these controls into Maya’s refund workflow, while the refund service enforces business permissions before money moves.

A model gateway manages requests to the model, but the company’s refund API, the backend that actually processes refunds, must independently check whether the agent is authorized to issue that refund.

For Maya’s refund agent, I’d require:

• Scoped access: Give the agent access only to the orders and refund functions/tools it needs, not a shared administrator account.

• Pre-action authorization: The refund service must check who is requesting the refund, which order it concerns, the amount, and the current rules. If permission is missing, it must not proceed.

• Bounded Approval: Human approval applies to a specific order and amount. If either changes, the agent must request approval again.

• Bounded execution: Limit how long the agent can run, how much it can spend, and how often it can retry. If a request times out, check whether the refund happened before trying again. Use the service’s duplicate-payment protection, keeping the same retry key within its supported time limit. If the outcome is still unclear, stop and let a person investigate.

• Traceability and recovery:: Record what was requested, why it was allowed or denied, who approved it, and what happened. Keep only necessary sensitive information and restrict access to it. Make sure the team can disable the agent and take over unresolved cases. Stopping the agent does not reverse money already refunded.

This is consistent with OWASP’s excessive-agency guidance: minimize functionality and permissions, require approval for high-impact actions, and enforce authorization downstream rather than trusting the LLM to decide.

3. Patterns and processes

In the patterns edition, I argued that reusable patterns should include guardrails and lifecycle discipline. I’d extend every golden path with an operating contract: allowed actions, owners, approval rules, evaluation cases, stop conditions, and a recovery procedure.

I’d first agree with the business team on what the agent must do, what it must never do, and how we’ll test both. We should define these requirements before writing production code, rather than change our definition of success to match whatever the prototype can already do.

I’d also correct my earlier claim that automated evaluations mean “no manual checks needed.” Tests help show whether an agent behaves as expected, but passing them does not give it permission to issue a refund, remove required human approval, or prove that the system meets regulatory requirements.

For Maya, I’d test unauthorized orders, malicious instructions in customer messages, changed amounts after approval, and ambiguous payment timeouts. Customer text is evidence to assess, not authority to override policy. (Prompt Injection Attacks)

For each guardrail, I’d record the control, owner, and evidence that it works. Suppose Maya’s policy requires human approval for refunds above $100. The refund service must block those payments until approval is recorded. An engineering lead owns that check, and the evidence includes a passing test showing that an unapproved refund is rejected.

I’d also decide who responds when something goes wrong. If the agent repeatedly attempts refunds outside its permissions, a named owner should investigate, suspend its access if necessary, and verify that the problem is resolved. Scheduled reviews should check that permissions and approval rules still match the current policy. A dashboard can flag a problem, but someone must own the response.

For Maya’s team, I’d expand the agent’s authority in stages:

• Read-only assistance: The agent explains the refund policy but cannot send money.

• Approval-required actions: The agent prepares a refund, but a person must approve it before payment.

• Limited automatic actions: If testing and observed results justify it, the agent handles routine refunds within explicitly permitted limits. Required approvals still apply.

If a fixed workflow can handle the task reliably, I’d use that instead of an autonomous agent.

Changes to the model, tools, permissions, or policy should trigger relevant tests again. For example, a shorter return window requires new eligibility tests. I’d measure correctly resolved cases, unauthorized-action attempts, staff corrections, and cost per resolved case.

Adding more agents is not progress if staff spend more time fixing their mistakes.

Final thoughts

I’d keep the people, platform, and patterns framework, but sharpen what the AI CoE provides: clear ownership rules, shared platform controls, and reusable patterns for testing, approval, and recovery.

The CoE’s mandate should be to set common rules and ensure they are enforced across business units, especially for high-stakes or hard-to-reverse agent actions. Each business unit should name who is accountable for those actions and when human approval is required.

Business teams would still own their workflows and outcomes. The CoE’s job is to make it easier for every team to deploy agents with clear permissions and accountability, without rebuilding those safeguards from scratch.

♻️ If this was useful, share it with someone building with AI.

✉️ Subscribe at newsletter.karuparti.com so you never miss an edition.

P.S. Want more? 👋

1/ My visual guide to agentic AI → Gumroad

2/ Daily deep dives on agentic AI architecture → LinkedIn

X, Instagram

References

• Anurag Karuparti, Winning with AI CoEs in the Agentic Era, October 17, 2025. Original three-pillar framework.

• Anurag Karuparti, Platform: The Foundation of an AI CoE, October 24, 2025. Shared platform, controls, and federated ownership; the automated-evaluation wording revisited above.

• Anurag Karuparti, People: The Force Multiplier of the AI CoE, October 31, 2025. Responsibilities and operating model.

• Anurag Karuparti, Patterns: Your Real Competitive Advantage in AI, November 7, 2025. Reuse, embedded controls, versioning, and retirement.

• OWASP, LLM06:2025 Excessive Agency. Least privilege, human approval, and downstream authorization.

• Anthropic, Building effective agents. Simplicity, environmental feedback, testing, human checkpoints, and stopping conditions. The refund operating contract is my proposed application, not a quoted standard.

• Anthropic, Claude Certified Architect Professional course. Module 1: delegation through reversibility, stakes, and accountability. Module 2: evaluations as acceptance criteria. Module 3: risk-based review, calibrated confidence, and controls with owners and evidence. Module 4: scheduled reviews and operational response. Paraphrased from the course material; lesson access may require registration.

Anu Karuparti Creator, Diary of an AI Architect How enterprises actually ship AI to production

Read by 3,000+ FDEs, AI Architects, and Engineering Leaders from Microsoft, Google, IBM, PwC and others.

Connect with me on LinkedIn.

Want to partner? Email me at anurag.karuparti@gmail.com.

Welcome back to Diary of an AI Architect. Today, we are stepping outside our usual solo deep dives to bring you a guest essay by my colleague Prosenjit Das, Global Black Belt for AI at Microsoft.

I’m genuinely excited to introduce Prosenjit to this community. I’ve personally learned a great deal from his expertise and guidance over the past few years at Microsoft. He works closely with Microsoft’s top enterprise customers, helping them translate Agentic AI innovation into measurable business value. He brings a unique perspective on bridging intelligent applications with real-world enterprise challenges.

My earlier posts on context engineering and decision quality and the three-layer Context Stack explored the governed data foundation agents need. Prosenjit makes that tactical in this essay, showing how Microsoft’s intelligence stack can address operational challenges.

This week in my diary, he connects Foundry IQ for document grounding, Fabric IQ for real-time and historical evidence, and Work IQ for authorized workplace context. The goal is a shared intelligence layer designed for relevant evidence, secure access and human control. Below are his words.

Table of Contents

1. Why build an Intelligence layer?

2. Who can use the Intelligence layer and how?

3. What are some industrial use cases and what does the Intelligence layer look like for them?

   • Connected Worker Safety

   • SportsIQ

4. How to build or measure one?

Below are the words of Prosenjit Das, GBB AI Apps at Microsoft | October 2026

1. Why build an Intelligence layer?

In my years of experience working as a Developer, then Enterprise Architect and currently in Microsoft and focused only on the top Global customers across several industries, I have always seen the information needed for one operational decision sits in several systems, siloed.

For example, in Construction, Manufacturing or Retail, current events arrive through cameras, sensors or transactions; history sits in analytical stores; procedures and approvals live elsewhere.

In sports, game moments must drive all decisions but key historical context lives elsewhere and company marketing strategy or policy guardrails at other places.

A human or an autonomous agent must relate these sources before it can give a useful and properly correlated answer. Repeating those joins, definitions and retrieval decisions inside each manual process or agent makes the system harder to maintain and its answers harder to explain. Also drives latency and cost considerably! Cost for siloed manual buildout or cost in terms of tokens.

Instead, imagine building an Intelligence layer to assemble context for a particular task that separates domain reasoning from retrieval orchestration. A question about one work zone in Construction or one game in Sports rarely needs the whole estate.

When a knowledge request reaches the knowledge layer, the knowledge base takes responsibility for retrieval orchestration. It can use an LLM to plan the request into focused subqueries, select the appropriate configured knowledge sources, execute searches in parallel using keyword, vector, or hybrid retrieval, semantically re-rank the results, and aggregate the most relevant evidence.

If the initial results do not meet relevance standards, it should perform iterative retrieval when configured with the appropriate reasoning effort. It then returns either the retrieved grounding content or an optionally synthesized answer, together with source citations, while enforcing the caller’s permissions.

The domain agent therefore does not need to know how individual knowledge sources should be searched. It receives the task-relevant evidence from Knowledge layer and applies domain-specific reasoning, business rules, confidence requirements, and actions to that evidence.

Source access should remain reusable and governed, so agents just call the knowledge layer using its own identity or the user’s identity and retrieve relevant information they are authorized to access rather than each rebuilding joins or getting tightly coupled with disparate knowledge sources and their drifts. The original systems still remain authoritative and control fine grained user access. This reduces avoidable context without discarding the basis for the answer.

2. Who can use the Intelligence layer and how?

Once built, the same intelligence layer can be used by different personas (and Agents) within the org.

• A safety officer,

• venue operator,

• support specialist or

• analyst

can use it through Teams or a Copilot experience. Domain agents can consume structured context through authorized APIs or Model Context Protocol (MCP) tools once they are triggered by events they have subscribed to.

They share entity definitions but receive different authorized slices and action rights. An event-triggered workflow still needs explicit delegation; the event itself cannot supply a user’s permissions.

Leveraging the intelligence layer must remain strictly decoupled from the channels where users or agents operate. But each builds their own scoped memory through each interaction with the same central knowledge layer.

Person A and Person B start with same knowledge base on day-1 but over time they build their own intelligence layer and memory scoped to each of their own.

As an example, in my IP that I built for the Sports industry, I map approved documents and policy grounding to Foundry IQ, and structured historical and real-time event data and business meaning to Fabric IQ.

Work IQ takes control in individual and collaboration context, ownership, hierarchy and prior decisions through an approved integration; Weather, traffic and schedules provide additional context and can be sourced from WebIQ or simply web search through Foundry IQ based on the exact need. The workload determines which capabilities are needed. A vector index alone cannot supply the entity relationships or action authority.

I keep the requesting person’s permissions attached to retrieval, including document access and source-level row or object restrictions. An agent identity cannot substitute for those rights. Checks on sensitive information must also follow it through retrieval, tools and the final response. The reader should be able to inspect the cited source and see where the evidence is strong and where not.

Once the Intelligence layer is built that each company can call their own IQ, they can point CoPilot or Cowork or Custom domain agents (or sub-agents) or coding agents at them to fetch the same clean and approved knowledge every single time, and over time build their own knowledge graph, hyper-personalized!

3. What are some industrial use cases and what does the Intelligence layer look like for them?

Connected Worker Safety

In the construction-safety architecture I developed earlier this year, a planner routes real-time, historical or mixed questions. The event path runs from camera capture to object storage, a function and a vision assessment. The resulting structured event enters Fabric Real-Time Intelligence and an agent workflow. The vision output is a typed risk assessment, allowing downstream components to work with explicit observations rather than repeatedly interpreting the same image.

For a zone alert, retrieval should cover environmental and equipment events for the relevant period, then use the dimensional Lakehouse for related incidents, certifications and corrective actions. External benchmarks can add perspective without becoming the authority for the response.

Real-time and historical agents supply evidence; a synthesizer applies the current standard operating procedure (SOP), and a communication agent routes the recommendation. Authorized task and certification records may be relevant, but worker identity or certification must not be inferred from camera images.

Historical analysis adds useful context but must not delay established alerts or stop-work procedures. Those paths must remain operational if an AI dependency fails.

Human judgement and stop-work authority remain in place; the AI cannot certify safety or automatically authorize work to resume. Ontology relationships can organize the evidence, but they do not establish causality; trained predictive models and ontology-based explanations remain future work.

SportsIQ

For an illustrative match-day offer, I retrieve weather, ticket inventory, point-of-sale activity and authorized loyalty context for the game and seat zone in question. A versioned model of Fan, Game, Offer, Seat Zone and Sponsor makes those relationships explicit.

Eligibility rules and budget, contact-fatigue, staffing and venue-capacity limits constrain the proposal. Changing inventory needs to be rechecked before execution: an offer can be commercially sensible but operationally unsuitable if the venue cannot support the additional demand.

I separate policy interpretation from enforcement. Deterministic middleware checks the rules, and human rejection or timeout means no write, and changed inputs require reapproval. Idempotency, concurrency checks, bounded retries and a stop switch address execution failures. Durable memory also needs authorized reads and writes, retention, deletion and revocation.

4. How to build or measure one?

The starting point is one task and its accountable owner, followed by a map of entities, authoritative sources, freshness limits and access boundaries. A read-only implementation provides a baseline for retrieval and reasoning before writes are introduced.

Specify what constitutes a correct answer, an appropriate abstention and a required escalation. This makes the evaluation reflect the operational task rather than only the fluency of the response.

The measurement approach I proposed in either implementation looks somewhat like this -

• Correlated traces capture model, retrieval, tool and handoff versions alongside latency and cost.

• Agent Insights groups recurring issues and representative traces, with affected versions and likely causes.

• A subject-matter expert should confirm or dismiss each suspected issue; the grouping helps investigation but is not a conclusive diagnosis.

Each evaluation case records whether the task was resolved, partly resolved or unresolved, whether the evidence supported the answer, and whether the right response was an answer, abstention or escalation. The reviewer identifies the failure layer, severity, confidence, and authoritative source. Reviewing a random control sample alongside selected failures helps reduce selection bias. An example enters the evaluation dataset only after explicit approval.

In this approach, the baseline is evaluated on the approved, versioned golden and regression dataset before optimization, and its manifest is frozen for reproducibility. Engineers can then change the model, prompt, knowledge, tool or agent topology and compare the candidate on the same tasks and datasets. Results need examination by case, domain and severity across quality, safety, outcome, latency and cost, because an overall improvement can hide a regression in a critical task.

A human release decision precedes shadow operation or a limited release ring, followed by monitoring and a decision to promote or roll back. Production must not automatically rewrite or promote itself. Domain experts own authoritative content, expected behavior and risk thresholds; engineers own the experiments. Managed platform tooling supports the work, while the organization retains the release decision.

A specialist agent is worth testing when traces reveal distinct tools or permissions, knowledge authority, state or memory, concentrated failures or costs, or different safety criteria. I would compare a single agent, a specialist and a stateless MCP tool under the same evidence and access requirements.

Dividing every business domain into another agent adds coordination work without necessarily improving the task. Also adds dynamic trajectory that makes evaluation harder.

For worker safety use case, I assess intent accuracy, query and data correctness, SOP alignment, relevance and actionability, using Open Telemetry traces to locate failures. Tests should include stale telemetry, noisy sensors, revoked access, missing procedures, cross-domain memory requests and attempts to bypass approval.

Rejected, expired or altered commands must not execute. Missed hazards and false positives need separate reporting because they have different operational consequences.

Context efficiency can be assessed through relevant retrieval, unnecessary prompt tokens, repeated tool calls, and latency and cost per successfully completed, authorized task.

Replaying cases with recorded source versions helps distinguish a context defect from a model change. I would accept lower token use only if the essential evidence and safe behavior are preserved, with the domain owner agreeing the trade-offs and release thresholds.

If a safety answer cites an obsolete procedure, I would investigate source selection and freshness before changing the model. If a SportsIQ offer fails because inventory changed, I would examine the execution-time check. This separation matters: the next experiment should address the component that failed, and the regression case should preserve enough evidence to show whether the change actually helped.

I am aware it’s a lot to achieve for organizations who have to responsibly manage real production systems but Foundry’s inbuilt Eval harness (Traces, Agent Insights, Evaluation and Optimizer) makes it a lot easier for organizations.

Across these implementations, my focus remains on giving people and agents relevant, current context within clear access and action boundaries. Operational evidence and domain review should guide how that context and the architecture supporting it improves over time.

Context is really the king as they say!!

About the guest

Editor’s bio: Prosenjit Das is a Global Black Belt for Azure Apps and AI at Microsoft. His work focuses on intelligent applications for enterprise customers, with interests spanning multi-agent architecture.

Connect with Prosenjit on LinkedIn.

♻️ If this was useful, share it with someone building with AI.

✉️ Subscribe at newsletter.karuparti.com so you never miss an edition.

P.S. Want more? 👋

1/ My visual guide to agentic AI → Gumroad

2/ Daily deep dives on agentic AI architecture → LinkedIn

5/ Visual frameworks and carousels → Instagram

6/ 60-second production lessons → TikTok

References

1. Microsoft: What is Foundry IQ?

2. Azure AI Search: Retrieval reasoning effort

3. Microsoft: Connect agents to Fabric IQ (preview)

4. Microsoft Fabric: Ontology as data-agent context

5. Microsoft: Work IQ API overview

6. Microsoft Foundry: Web search tool

7. Microsoft Foundry: Insights (preview)

8. Microsoft Foundry: Agent optimizer

9. OpenTelemetry: Traces

10. MCP authorization specification, 2025-06-18

11. Prosenjit Das: GitHub profile

12. Prosenjit Das: LinkedIn profile

13. Microsoft Entra: On-behalf-of delegation

14. Azure AI Search: Document-level access

Anu Karuparti Creator, Diary of an AI Architect How enterprises actually ship AI to production

Read by 3,000+ FDEs, AI Architects, and Engineering Leaders from Microsoft, Google, IBM, PwC and others.

Connect with me on LinkedIn.

Want to partner? Email me at anurag.karuparti@gmail.com.

Another interesting week in AI...

Gemini 4 Argon is pushing coding and enterprise workflows forward. Meanwhile, Anthropic’s Opus 5.5 announcement reports an early tester’s single-prompt game comparison where graphics and polish stood out. The demos are getting wild.

But a quieter architecture question has my attention.

For the past few years, we have been putting LLMs everywhere.

• Generate an email? LLM.

• Summarize a contract? LLM.

• Write code? LLM.

Need to decide whether a customer should receive a refund? Also LLM.

That last one is where things get interesting.

Generating an answer and making a decision are not necessarily the same problem.

This week in my diary, I have been looking at Jev, TypeSafe’s System One model. I think the bigger story is not just Jev. It is the idea of decision models as a distinct primitive inside software.

Instead of generating open-ended text, Jev answers bounded questions with typed, probabilistic results. There is no paragraph to parse, but that does not mean the answer is correct or the application should automatically act.

The surrounding software still owns the workflow.

If you learn better visually, check out my Visual AI Library on Gumroad. I use these cheat sheets myself before customer meetings to refresh key concepts and explain them more clearly. There is a lot of noise in AI. This library gives you the distilled version, so you can quickly review important concepts before interviews, exams, presentations, or customer conversations.

1. So what exactly is Jev?

TypeSafe introduced Jev in a post dated September 15, 2026. It describes the interface simply: supply state and typed questions, receive structured answers your code can use.

Traditional LLMs generate sequences of tokens. Jev takes a different approach. It gives up open-ended string generation and focuses on decisions among defined possibilities.

TypeSafe calls its training approach Reinforcement Learning for Calibrated Decisions (RLCD). That is a provider-described method, not proof that every domain already has perfectly calibrated predictions.

The simplest way I think about the distinction is:

LLMs generate. Decision models judge. Code decides what happens next.

For example, an LLM might write:

“Based on the customer’s cancellation date and billing history, I recommend a refund.”

A typed decision model approaches the same problem differently. Instead of generating freeform text, it returns a structured result such as a class, score, or probability that the application can use directly.

So Jev could answer a narrower question:

“Does this request fit the supplied refund policy?”

A Noul result might be 0.96, representing the probability of yes.

The application can then combine that probability with its own business rules to decide what happens next.

Jev currently handles text, including text inside JSON, rather than images, audio, or video. It also does not generate a reasoning explanation.

And importantly, modern LLMs can also return structured outputs. So the real question is not whether LLMs can do this.

It is whether a model specialized for narrow decisions can do it faster, cheaper, or more consistently than a general-purpose model.

2. Who actually asks Jev the question?

The customer is not asking Jev, “Should you refund me?”

Your application asks Jev.

Imagine Maya, an engineer building a support assistant. A customer writes: “I cancelled last week, but you charged me again. Please refund this.”

Maya’s application gathers permitted conversation history, account facts, recent charges, refund history and the relevant policy. Dates and charge calculations come from code and authoritative systems, not a model’s guess.

It sends the relevant state alongside predefined questions:

1. Intent: Refund, cancel, change plan or something else?

2. Policy fit: Does this request match the supplied refund policy?

3. Review: Does the message contain a reason to involve a person?

These can be independent questions about the same state. Their answers do not secretly become instructions for one another.

The application interprets the results. A strong policy-fit signal can propose a refund for backend checks; an unclear result can request evidence or review.

But eligibility, permissions, payment limits and required approvals still apply before any payout. A probability threshold cannot bypass them.

Now we have separated two responsibilities: Jev provides semantic judgment. Code controls what happens because of it.

3. Jev has three basic decision primitives

The product names are unfamiliar. The jobs are not: check, choose, score.

Noul returns a number between zero and one: the probability of yes. Your application decides how to interpret it.

Choice returns the selected option and probabilities across the choices. Score returns a numeric result for your ordered rubric, which can be fractional.

Choice and Score also provide confidence. Noul does not provide a separate confidence field.

Production workflows rarely depend on one giant decision. They depend on many small judgments that code combines.

4. Think of it as a smart if-statement

Traditional software might contain this illustrative Python:

if invoice.total_usd > 10_000:
    require_approval(invoice)

There is no reason to use AI to compare those numbers.

But consider: “If this invoice description appears inconsistent with the work we bought, route it for review.”

How do you express that as a simple string comparison?

“Cloud infrastructure consulting” and “Azure architecture advisory services” could describe the same work. Different wording does not necessarily mean a mismatch.

This is where a bounded semantic question helps. The model estimates whether the descriptions match; code uses that signal to continue checking or flag the invoice.

The signal is not payment approval.

I find TypeSafe’s “smart if-statements” framing useful: code handles exact rules, decision models handle bounded interpretation, and reasoning models handle open-ended complexity.

5. System One versus System Two AI

The name borrows Daniel Kahneman’s System 1 and System 2 framing in Thinking, Fast and Slow: quick judgments versus deliberate reasoning.

For AI, I treat this as a task analogy, not a claim about biological thinking or a guarantee that every LLM is slow.

Imagine asking your most senior architect hundreds of thousands of times: “Billing or technical support?”

They can answer. That does not mean they are the right resource for every repetition.

The same principle could apply to models. A reasoning model can plan and write, a decision model can handle known-option judgments, tools can perform scoped work, and people can handle consequential uncertainty.

That is a division of responsibilities, not a mandatory chain every request must traverse.

6. Where decision models fit in the enterprise

TypeSafe publishes four enterprise-style workflow evaluations, not production customer case studies. Here are simplified examples of Jev’s role in each

Customer service

A customer writes: “I cancelled last week, but you charged me again. Please refund this.”

The application already has the customer message, account details, cancellation date, billing history, and refund policy. Instead of asking Jev to write a response, it asks a narrow question:

“What does the customer want?”

Possible choices might be refund, cancel, change plan, or other.

Jev returns refund with a high confidence score.

That does not trigger an immediate payment. The application opens the refund workflow, where code checks the charge, eligibility, permissions, and any required approvals.

Jev interprets the intent. Code decides whether the refund is allowed. A language model can still generate the customer-facing reply.

Invoice processing

An invoice says “Azure architecture advisory.” The purchase order says “cloud consulting.”

The wording is different, but the underlying work may be the same.

The application can ask Jev:

“Do these descriptions refer to the same work?”

Jev might return a Noul probability such as 0.93 for yes.

That signal becomes one input into the invoice workflow. The application can combine it with deterministic checks for amounts, dates, vendor identity, duplicate invoices, contract terms, and approval requirements.

Jev handles the semantic comparison. It does not calculate totals or authorize payment.

Security incident response

An alert shows a server login followed by customer-data downloads. The maintenance record only authorizes a software update.

The application asks:

“Does this activity match the approved maintenance work?”

Possible choices might be matches, does not match, or unclear.

If Jev returns does_not_match, the application can raise the severity or route the incident for investigation.

That still does not mean Jev can shut down the server. Containment actions remain governed by predefined security policies, permissions, and approval rules.

Jev helps interpret the activity. Code determines what response is allowed.

Agent trace observability

A customer asks an agent to cancel a subscription.

The agent gives a helpful explanation of how to cancel, but never actually performs the cancellation.

The response sounds good, but the task failed.

The application can give Jev the customer request, conversation, tool calls, tool results, and final response, then ask:

“Was the customer’s task completed?”

Possible choices might be completed, not completed, or unclear.

If Jev returns not_completed, the application can flag the trace for review instead of treating a fluent answer as success.

This is an important distinction for agent evaluation. The question is no longer only whether the response was good. It is whether the system actually completed the task.

Across all four examples, the pattern is the same:

Context → bounded question → structured judgment → business rules → action

Jev handles the semantic judgment. The surrounding application decides what happens next.

AI judges. Code governs. Software acts.

7. Why probabilities matter

Suppose a Noul question about policy fit returns 0.51. Now compare that with 0.995.

If code turns both into “yes,” an important distinction disappears.

An uncertain result might justify more evidence, a reasoning model or human review. A stronger signal might justify less semantic investigation, not fewer mandatory controls.

Do not confuse a probability with confidence. Choice and Score provide a separate confidence statistic derived from their distributions. Noul provides its yes-probability without that extra field.

TypeSafe describes calibrated probabilities, but calibration concerns outcomes across groups of predictions. It does not guarantee one decision is correct.

There is no universal 95% or 98% threshold that makes a payment, security response or medical decision safe. Test thresholds on your data, with the cost of errors and actual review capacity in mind.

Autonomy is an application policy, not simply a property of how intelligent the model is.

8. Does typed output mean Jev cannot be wrong?

No.

Predefined outputs address a structural problem: the model is not supposed to invent a category outside the schema.

They do not solve every judgment problem.

Separate two questions:

• Structural reliability: Does the answer fit the expected type?

• Decision accuracy: Did it choose the right answer?

A typed wrong answer is still a wrong answer.

Malicious text can also influence Jev’s answers. Keep customer text and retrieved content on the untrusted side of the boundary. A “no attack detected” label does not make them safe instructions.

You still need evaluations, calibration testing, auditability, escalation rules and human review where risk requires it.

9. What about the speed and cost claims?

TypeSafe mentions an end-to-end response time of 70ms-500ms. This can range from 40x-200x faster for the same levels of frontier intelligence for System One shaped queries.

Test your workflow against the incumbent route, simple rules and a small structured-output LLM. Measure accuracy, calibration, end-to-end latency, cost, escalation workload, false positives, false negatives and the business outcome.

Vendor model latency does not include every retrieval step, retry, review or correction. Cheaper inference is not automatically cheaper operations.

The bigger question is not whether your improvement matches a headline multiplier.

Why spend frontier-model intelligence on every tiny decision when a narrower approach might do the job?

10. This is an intelligence-routing problem

I recently wrote about token maxing versus value maxing. The principle applies here too.

The goal is not merely “use fewer tokens.” It is to use the least expensive approach that reliably produces the required outcome.

These are resources to route between, not an escalation ladder where permission checks appear only at the end. Code-owned controls apply on every action path.

The mistake is sending every problem straight to the largest model.

11. Where I would test Jev first

I would start with a reversible, well-labeled task: intent classification, ticket routing, document categorization, retrieval relevance or trace-review prioritization.

Keep contradictory retrieval evidence when it matters; relevance filtering should not erase inconvenient facts.

Be more cautious about moving money, blocking critical systems, making medical decisions or taking legally consequential actions. In those workflows, a model judgment can be one signal, never the entire authorization mechanism.

Learn by doing! Here are some awesome list of demos and repositories on Github.

The bottom line

Pick one repeated judgment with known possible answers, test a decision model against your existing approach, and keep permissions, exact rules and required approvals in your software.

The smartest architecture may not use the smartest model everywhere. It may know exactly where intelligence is actually required.

♻️ If this was useful, share it with someone building with AI.

✉️ Subscribe at newsletter.karuparti.com so you never miss an edition.

Anu Karuparti Creator, Diary of an AI Architect How enterprises actually ship AI to production

Read by 3,000+ FDEs, AI Architects, and Engineering Leaders from Microsoft, Google, IBM, PwC and others.

Connect with me on LinkedIn.

Want to partner? Email me at anurag.karuparti@gmail.com.

References

• Google: Gemini 4 Argon. Coding and enterprise-workflow framing.

• Anthropic: Claude Opus 5.5. Attributed early tester game comparison, not an independent graphics benchmark.

• Anthropic: Structured outputs. A fair generative-model baseline.

• My previous edition: 10 ways to stop wasting tokens. Value-maxing framework and the Jev teaser.

• TypeSafe: Introducing System One models and Jev. September 15 article dateline, RLCD, workflow benchmark claims, reference-model methodology and stated limitations.

• TypeSafe: Introduction and System One. Model scope, typed decisions, independent questions and task analogy.

• TypeSafe: Quick start and Score. Primitive response shapes and fractional ordered scores.

• TypeSafe: Confidence. Distribution-derived confidence, Noul distinction and workflow-specific thresholds.

• TypeSafe: How to build with System One. Code-owned workflows, narrow questions, policy comparison and review signals.

• TypeSafe workflow evaluations: customer service, invoice processing, security incidents, and agent trace observability. Internally authored evaluation scenarios, not claimed customer deployments.

• TypeSafe: Jev 1.13 jaggedness. Exact arithmetic/date and adversarial-state limitations.

• TypeSafe: Classifying RAG passages. Relevance, evidence, contradictions and screening as separate signals.

• TypeSafe: Models and Legal. Version pinning, request limits and applicable provider terms.

Disclaimer: The stories and scenarios in this article are hypothetical, inspired by patterns observed across similar real-world experiences. They are used to convey key concepts more effectively and do not represent any specific individual or organization.

He also emphasized stronger engineering practices and independent testing, making safety something we build into systems, not just talk about.

Separately, I’ve been digging into Jev, TypeSafe AI’s new model, now in early access.
It’s built for bounded semantic decisions: choosing a category, scoring against a rubric, or assessing whether a statement holds, rather than generating paragraphs.

I can see the developer appeal: typed outputs, uncertainty signals, questions evaluated in parallel, and decision logic you compose in code.

That doesn’t make every judgment correct, but it gives us an interesting way to build around uncertainty.

I’m researching where it could genuinely help in production. More on Jev soon. Stay tuned.

This week in my diary, though, I’m exploring a related question: are we paying for useful decisions, or just more tokens?

Let’s get into 10 ways to move from token maxing to value maxing and build smarter AI workflows

• Token-maxxing asks: How do I throw the most intelligent model, the most context, and the most tokens at the problem?

• Value-maxxing asks: How do I use the right model, context, tools, and compute to produce a correct, useful, and trustworthy outcome at the lowest total cost?

That distinction matters.

• A frontier model is wasteful if a smaller model produces the same outcome.

• A cheaper model is not cheaper if it causes three retries.

• More context is not better if most of it is irrelevant.

• More reasoning is wasted compute if the task is straightforward.

• An autonomous agent is not productive if it creates more human cleanup.

The real optimization target is not tokens used or tokens saved.

It is total cost per successful task.

If you learn better visually, check out my Visual AI Library on Gumroad. I use these cheat sheets myself before customer meetings to refresh key concepts and explain them more clearly. There is a lot of noise in AI. This library gives you the distilled version, so you can quickly review important concepts before interviews, exams, presentations, or customer conversations.

1. Spray and pray → Intent first

Do not default every workload to the strongest model.

Match the model to the task. Use a mid-tier model option that passes your quality checks, and reserve stronger models for work that genuinely needs them.

This is the first step from model-centric optimization toward workload-centric optimization.

Architecture takeaway: Define acceptable errors and rollback conditions before comparing models. A good average can hide failures in one ticket category, so include edge cases rather than grading only easy requests.

Example: I’d start with Claude Sonnet on labeled billing, delivery, and technical tickets, then test Haiku on the same set. I’d switch only if Haiku meets the quality bar in each category at lower total cost. Unclear cases go to a person, not whichever model sounds most confident.

2. No tools → Scoped tools

Do not spend tokens asking a model to reason about information it does not have.

When a task needs live information, more reasoning cannot replace access to it. A Model Context Protocol, MCP, server can expose the specific tools the job needs, with permissions enforced outside the prompt.

This moves optimization from better prompting toward better system design.

Architecture takeaway: Split responsibilities. The model interprets the request. The system of record supplies current facts. Application code enforces access. Do not ask the model to perform all three roles.

Example: For “Where is order #1042?”, I’d expose a read-only order-status tool. The backend verifies that the signed-in customer owns the order before returning its carrier, tracking link, and latest status. No refund or payroll tools. A lookup timeout means “I couldn’t check,” not “your order doesn’t exist.”

3. Connector sprawl → One governed gateway

As your agent gains capabilities, integration architecture starts mattering as much as model architecture.

For a growing integration estate, a shared gateway can centralize authentication, policy checks, observability, and audit records.

MCP alone does not provide that governance, and adding a gateway does not automatically save tokens. The value comes from reducing duplication, inconsistent controls, and operational complexity.

Architecture takeaway: Centralize the integration, not everyone’s privileges. Shared infrastructure also concentrates failures, so name an operational owner and decide what stops safely when the gateway is unavailable.

Example: I’d put customer-record lookup and shipment lookup behind one gateway. A support rep can read assigned customer records. A sales rep cannot inherit that access just by using the same endpoint. Check identity and permissions per action, retain scoped downstream credentials, and run redundant gateway instances, not one shared admin login.

4. Dumping everything → Scoped, not starved

More context is not always better context.

Every token should earn its place, but aggressive trimming can make a system worse if you remove information that changes the answer.

The goal is contextual precision.

Architecture takeaway: Cut irrelevant history, not facts that could change the decision. Preserve exact identifiers, amounts, applicable policy versions, and where the evidence came from when shortening context.

Example: For a refund on order #1042, I’d send the purchase date, $79 paid, the damage report, and the applicable 30-day refund policy with its effective date and damaged-item exception. Leave out unrelated chats and card details.

5. Agentic by default → Match the pattern

Autonomy has a cost.

Every additional agent step can introduce tokens, latency, tool calls, retries, and new failure modes.

Use a deterministic workflow when the steps are known. Use an agentic loop when the next step genuinely depends on what the system discovers.

Architecture takeaway: Define “done” and “stuck” in observable terms before allowing a loop. A call limit bounds the work. A useful handoff preserves findings, attempted actions, and what remains unresolved.

Example: For invoices, I’d extract supplier, amount, and purchase-order number, validate them in code, then route exceptions to accounts payable. For a suspected duplicate charge, an agent could compare invoices and payment records until the discrepancy is verified or progress stalls. I’d also cap it at five tool calls, then hand off unresolved findings to a person.

6. Fix it later → Read, plan, act

Value-maxxing also means reducing the cost of mistakes.

For consequential changes, review a short action plan before execution. Plan approval is not a substitute for checking the result.

The more expensive or irreversible the action, the stronger the checkpoint should be.

Architecture takeaway: Place checkpoints where mistakes are costly or hard to reverse. Give the reviewer the supporting evidence and proposed changes, not just an “Approve” button and a confident summary.

Example: I’d show a preview for 20 CRM contacts: record ID, old value, proposed value. A person approves only that change set. The application rechecks permissions and record versions before writing, verifies saved values afterward, and requests fresh approval if the records or scope changed.

7. Tool bloat → Load on demand

Tools consume context too.

If every tool definition enters every prompt, your system pays for capabilities that may never be used.

Expose a small task-specific set, or use tool search to load definitions when discovered, before calling them.

Architecture takeaway: Context management and permission management are different jobs. Discovering a tool makes its instructions available. It does not authorize execution. Measure discovery overhead before adding another step.

Example: I’d give a meeting assistant calendar lookup and availability tools first. When someone asks, “Turn that action item into a ticket,” discover and load the ticket-creation definition, then check permissions and confirm the write before calling it. If those are the only three tools, I’d skip search and load them up front.

8. No visibility → Full transparency

You cannot value-max a system you cannot measure.

Make available permissions and actual tool calls visible. Track model usage, latency, retries, failures, human escalations, and successful outcomes without turning your logs into a copy of sensitive customer data.

The unit of optimization should move from cost per token to cost per completed outcome.

Architecture takeaway: Monitoring needs a response, not just a dashboard. For each important signal, define the trigger, the person responsible, and the corrective action, including quality declines that do not produce errors.

Example: Suppose a trace shows six repeated order lookups after a timeout. I’d return a typed timeout error marked retryable, allow one retry, then hand off if it fails again. Log tokens, latency, retries, and outcomes with redacted arguments and restricted access. Compare cost per resolved ticket and quality, not hidden chain-of-thought.

Subscribe now

9. Wasteful outputs → Right output

Generation itself can create downstream work.

Do not ask the model for a five-paragraph explanation when another system needs three fields.

Specify what the next person or system actually needs. A usable answer beats a long answer that needs another prompt to summarize, extract, or reformat it.

Architecture takeaway: Separate format checks from meaning checks. Code can reject missing fields or invalid dates. Checking whether a decision was actually agreed requires comparison with the source, sometimes by a person.

Example: I’d ask for one row per meeting decision: decision, owner, due date. If nobody committed to a deadline, write “not agreed” rather than invent Friday. For software, require those fields in a validated JSON schema, allow null for missing values, and check each row against the transcript. Valid JSON is not proof of truth.

10. Compute as a crutch → Targeted reasoning

The most reasoning is not always the best reasoning.

Spend additional compute where it materially improves the outcome. Where your model supports configurable effort, test lower settings for straightforward tasks instead of defaulting to maximum.

This is where token-maxxing and value-maxxing converge.

You reduce compute when it adds little value, and spend more when the additional reasoning prevents expensive mistakes.

Architecture takeaway: Diagnose the gap before buying more reasoning. Missing facts need retrieval. Unclear instructions need clarification. Extra computation earns its place only when evaluation shows it improves the task’s outcome.

Example: On a Sonnet version that supports effort controls, I’d test low effort for “Turn this verified shipment record into a two-sentence update.” I’d test high effort for “Compare these two migration plans for downtime and rollback risk.” Keep each setting only if representative tests meet the quality bar.

Final thoughts…

Value-maxxing means optimizing the entire workflow, not one model call.

Measure total cost per successful task, including model usage, tools, retries, latency, human review, and rework.

Pick one recurring workflow, establish a baseline, change one variable, and measure the impact.

The goal is simple: spend the right amount of compute and effort to produce a result you can trust.

♻️ If this was useful, share it with someone building with AI.

✉️ Subscribe at newsletter.karuparti.com so you never miss an edition.

Anu Karuparti Creator, Diary of an AI Architect How enterprises actually ship AI to production

Read by 3,000+ FDEs, AI Architects, and Engineering Leaders from Microsoft, Google, IBM, PwC and others.

Connect with me on LinkedIn.

Want to partner? Email me at anurag.karuparti@gmail.com.

P.S. Want more? 👋

1/ My visual guide to agentic AI → Gumroad

2/ Daily deep dives on agentic AI architecture → LinkedIn

3/ Real-time takes on breaking AI news → X

4/ Casual hot takes and community → Threads

5/ Visual frameworks and carousels → Instagram

6/ 60-second production lessons → TikTok

7/ The full newsletter, free → newsletter.karuparti.com

This week in my diary, I’m bringing those questions back to the work: building AI systems that are useful, reliable, and stay within the boundaries we set.

As AI architects, we have an increasingly important role in shaping how these systems are designed, deployed, and used across society, while understanding and mitigating the risks they may pose to humanity.

On a personal note, I was promoted to Principal AI Apps Architect (Director) at Microsoft. It’s been an incredible journey, and I’m grateful for the people who have helped me grow.

I also earned Anthropic’s Claude Certified Architect - Professional certification. I took it to sharpen my architecture judgment when building multi-agent systems, not just learn another product.

What I valued most was the mix of technical principles and broader architectural decisions: what to automate, how to measure success, enterprise integration, and where human oversight belongs, GTM strategies and stakeholder enablement.

These are principles I can apply well beyond Claude, across the models and AI systems I work with, when enabling my customers.

In my next post, I’ll share how I cleared the certification and why I liked it so much.

For now: GPT-6 Astra or Claude Fable 5.1? Let’s look at execution, building, time, and cost to find the right fit for your business. The evidence comes from published evaluations, my first-hand research playing with both models.

I created a YouTube Short too on this. Check it out here!

The architect asked to pick one winner

Imagine Priya, a staff architect at a national retailer. Her finance team wants an agent to open a sales spreadsheet, reconcile the numbers, build a chart, update a presentation, and save the accompanying document.

Her engineering team wants something different: investigate a recurring inventory-service bug, trace dependencies, propose a fix, and verify it without breaking another service. Procurement asks Priya to standardize on one model for both.

The spreadsheet workflow fails if the agent updates the wrong file. The debugging workflow fails if a plausible patch hides the underlying defect. A quick answer is not the definition of done in either case.

Priya is not choosing the smartest colleague in the room. She is assigning responsibilities, tools, and acceptance criteria to two different jobs.

If you enjoy visuals like these and learn better visually, check out my Visual Library on Gumroad.

The framework: model loyalty versus workload fit

I use a four-decision routing test:

Reflex: Pick a model -> give it every task -> optimize the bill later
Fix:    Define the job -> test the model + tools -> verify the outcome

• Execution: Can it complete the required actions across your tools?

• Building: Can it understand the problem and produce a correct, maintainable artifact?

• Time: Does it meet the deadline for an accepted result, not just the first answer?

• Cost: What does that accepted result cost, including cache writes, retries, and review?

I use benchmark evaluations to decide which model to start with for each enterprise workload. I look at what each test actually measures, then connect that strength to the work our teams need done.

Astra

• Astra feels like an operator, the get-things-done model. It outperformed Fable 5.1 on AutomationBench-AA, which tests completing multi-step business workflows across SaaS applications through APIs while following business rules.

• That makes it my starting choice for updating CRM records, coordinating customer follow-ups, processing support escalations, employee onboarding workflows, and checking expenses against budget rules. ^[1][2]

• Its stronger Terminal-Bench 4.0 results point toward command-line engineering and operations work: troubleshooting failed builds, fixing dependency issues, debugging CI/CD pipelines, diagnosing data or ML jobs, and running scripts and automated tests. These tasks require the model to actually operate tools and configure environments, not simply answer coding questions. ^[1][2]

• For browser and desktop automation, I’d also prioritize Astra: retrieve supplier invoices, fill portal forms, update spreadsheets, perform frontend QA, or move information between enterprise applications.

• OSWorld 2.0 tests operating that software, and OpenAI reports Astra ahead of GPT-5.6 Sol and Claude Opus 5 on its offline, partial-credit comparison. That supports my evaluation too, although the published setup does not establish a matched comparison against Fable 5.1. ^[3][4]

Fable

• Fable feels more like an analytical builder. It outperformed Astra on SciCode, which tests scientific programming rather than software development broadly. I’d start with it for engineering simulations, numerical calculations, scientific data analysis, and research-oriented code where domain reasoning and code generation have to work together. [^1][2]

• It also leads Astra on AA-LCR v1.1, which tests reasoning across long documents. I’d apply that strength to comparing supplier contracts, analyzing requests for proposals (RFPs), reviewing financial reports, finding conflicting policies, mapping regulatory requirements, and checking requirements against technical documentation. ^[1][2]

• Then I check the economics. Astra’s lower cost per task in the cited evaluation and Fable’s cheaper cached-input pricing create different opportunities.

• Fable becomes particularly interesting when the same large context, such as policies, codebases, contracts, or documentation, is reused repeatedly. I’d compare reasoning effort, eligible cache reuse, latency, failure rates, and human-review costs.

• The final choice depends on correct results, human effort saved, and total cost per completed workflow, not token price alone. ^{[1][8][9][10]}

• The independent comparisons use Artificial Analysis’s September 15 snapshot at maximum reasoning effort, with Fable’s default fallback enabled, meaning another model may handle some requests.

• These are workloads I would evaluate based on benchmark signals, not measured customer outcomes. ^[1][2]

• Below, I unpack all four decisions, explain where smaller models fit, and show the handoff I would use to test a Fable-plans/Astra-executes workflow.

  You do not need a frontier model for every task.

  When I design an enterprise AI workload, I increasingly think about four decisions: what role the model needs to play, how much intelligence the task actually requires, how much autonomy I am willing to give it, and what it costs to produce a correct outcome.

  The goal is not to find the single best model. The goal is to use the right amount of intelligence at each step of the workflow.

Decision 1: Is the model planning or executing?

Read more

Welcome back to Diary of an AI Architect. Today, we are stepping outside our usual solo deep dives to bring you a guest essay by Rohan Kodialam, co-founder and CEO of Sphinx AI, a company building a knowledge and reliability layer for enterprise AI. Before Sphinx, Rohan was an AI research leader at Citadel.

This week in my diary, I’m returning to a question behind my earlier posts on context engineering and decision quality and the three-layer context stack: what happens to that investment when you change models? Rohan brings a founder’s perspective on business context to an uncomfortable argument: access to another model is not the same as the ability to move your business knowledge to it. Read this with your own model-switching plan in mind. Below are his words.

Enterprise AI teams like to talk about model optionality. The idea is straightforward: don’t become dependent on OpenAI, Anthropic, Google, or any single provider. Build your stack so you can use whichever model offers the best combination of intelligence, cost, latency, security, and reliability. The logic is increasingly difficult to argue with. Menlo Ventures estimates that Anthropic captured 40% of enterprise LLM spending in 2025, up from 24% a year earlier, while OpenAI accounted for 27% and Google reached 21%. In a market moving this quickly, betting your enterprise architecture on one permanent winner makes little sense.

But there is a problem with how we think about model optionality. Changing which API processes a request is relatively easy. Moving everything your AI system has learned about your business is much harder. An enterprise AI system needs to know far more than what exists in a database or document repository.

It needs to understand that finance and sales may define the same metric differently, that one system should take precedence when two sources disagree, that a particular field has an unusual historical meaning, or that a workflow contains exceptions understood by employees but poorly documented anywhere else.

If that knowledge becomes embedded in model specific prompts, fine tuning, memory systems, or application logic, the enterprise may technically be able to switch models while remaining practically locked into the system built around the old one.

That matters because the future of enterprise AI will almost certainly be multi model. IDC argues that enterprises should design around portfolios of models rather than a single stack, continuously choosing among models based on performance, cost, latency, and the use case.

Gartner has similarly warned that relying on a single AI model is a common enterprise pitfall and recommends composite approaches that combine multiple models and techniques. One model may be best for complex reasoning, another for coding, another for high volume repetitive tasks, and a smaller or open weight model may make more sense when privacy, cost, or control is paramount.

The architecture therefore needs a clean separation between what is replaceable and what is durable. Models are replaceable. A company’s institutional knowledge is not. Definitions, relationships, business rules, permissions, historical decisions, and the knowledge required to interpret internal data should belong to the enterprise rather than to whichever model happens to be reasoning over them.

Think of the model as an intelligence engine. You should be able to replace that engine without rebuilding everything the system knows about the organization around it. When a new model appears, the question should be, “How well does this model perform using our existing business context?” rather than, “How do we teach another model how our company works?”

There is an economic argument for this architecture as well. Enterprise generative AI spending reached an estimated $37 billion in 2025, more than triple the $11.5 billion spent in 2024. As that number grows, companies will increasingly care about routing workloads intelligently. Using the most capable frontier model for every request may make little sense if a smaller model can handle repetitive tasks at a fraction of the cost.

Optionality also creates negotiating leverage and resilience. Providers can change prices, experience outages, alter policies, or fall behind competitors. The ability to move meaningful workloads elsewhere is valuable only if switching does not require reconstructing the institutional knowledge those workloads depend on.

There is also a broader constraint that enterprise architecture discussions often overlook: model choice is not purely an individual company decision. At global scale, choice is constrained by where compute capacity exists and which models can actually serve enormous volumes of production demand. An open model may look compelling on benchmarks, but the entire market cannot necessarily migrate to it tomorrow. The combination of rapidly changing model quality, economics, and available compute makes predicting the dominant provider several years from now nearly impossible.

That is why enterprises should architect for a model market they cannot predict. The goal is not simply to avoid committing to one API. It is to ensure the most valuable part of the AI stack, the accumulated understanding of how your business actually works, remains yours regardless of which model comes next. True model optionality means being able to change the intelligence engine without forcing the new engine to relearn your business.

Connect with Rohan on LinkedIn.

From my diary: test the portability, not just the API

Editor’s closing note, Anurag Karuparti

My earlier context-engineering editions asked what an agent needs to know and how we govern the data that supplies it. Rohan adds a useful ownership test: can that context survive the next model choice?

Here is the exercise I would take into the next architecture review. Pick one workflow and identify:

• Meaning: Where do we record what metrics mean, which sources to trust, and any exceptions?

• Control: Can we move that knowledge, track changes, and keep the same access rules?

• Evidence: Can another model pass the same tests, including handling conflicting sources and blocking unauthorized access?

Portable context does not guarantee identical behavior. Test the replacement model’s tool calls, outputs, and permission boundaries before moving a production workload. Keep the knowledge reusable; earn confidence in each new engine.

♻️ If this was useful, share it with someone building with AI.

✉️ Subscribe at newsletter.karuparti.com so you never miss an edition.

Share

Anu Karuparti Creator, Diary of an AI Architect How enterprises actually ship AI to production

Read by 3,000+ FDEs, AI Architects, and Engineering Leaders from Microsoft, Google, IBM, PwC and others.

Connect with me on LinkedIn.

Want to partner? Email me at anurag.karuparti@gmail.com.

P.S. Want more? 👋

1/ My visual guide to agentic AI → Gumroad

2/ Daily deep dives on agentic AI architecture → LinkedIn

3/ Visual frameworks and carousels → Instagram

4/ 60-second production lessons → TikTok

References

1. Menlo Ventures, 2025: The State of Generative AI in the Enterprise, December 9, 2025. Source for the guest’s spending and provider-share estimates. These are Menlo’s 2025 estimates, not current September 2026 market shares. The study surveyed 495 U.S. enterprise AI decision-makers; share estimates approximate spending from production API usage. The $37 billion estimate excludes chips, inference/model serving, and AI features embedded in existing software. Its $11.5 billion 2024 comparator is restated on the same inference-excluding basis.

2. Tim Law and Zhenshan Zhong, IDC, Beyond LLMs: Why AI Strategy Now Requires Multi-Model, Multimodal, and Multi-Agent Architectures, April 13, 2026. Supports model portfolios, use-case matching, and cost/performance/latency evaluation.

3. Ben Yan, Pieter den Hamer, Deepak Seth, and Joe Antelmi, Gartner, Go Beyond a Single Model and Use Composite AI to Enhance Enterprise AI Adoption, July 2, 2025. The public abstract supports the guest’s attribution; the full research requires access. Composite AI includes different techniques, not only switching LLM providers.

4. Sphinx, Sphinx launches with $9.5M to redefine how AI works with data, September 9, 2025, and the company’s current product description. Sources for the editor’s short bio: co-founder/CEO, prior Citadel AI research role, and company focus. Product positioning is the company’s description, not an independent performance endorsement.

5. Anurag Karuparti, How context engineering can affect your organization’s decision quality, March 6, 2026. Earlier discussion of instructions, memory, retrieval, tools, and structured outputs as components of enterprise context.

6. Anurag Karuparti, How to build a strong enterprise AI moat with context engineering and data estate, April 24, 2026. The three-layer context stack: unified data foundation, retrieval architecture, and governance/documentation. Public preview available; full edition is subscriber-restricted.

This week in my diary, I am working through the missing outcome-translation layer between an agent trace and the KPI a business owner is actually paid to move.

Observability tells me how the system behaved. It can show model latency, token usage, errors, tool calls, workflow duration, and agent trajectories. Those signals are necessary for operating production AI.

They are not the same as business outcomes.

The dashboard that answered the wrong question

Picture Maya, the VP of Customer Service at a national retailer. Her team deploys an AI agent to help customers resolve order and return questions.

Two weeks later, the engineering review looks excellent. The dashboard shows 1.8-second latency, a 92% task success rate, and a 1.4% error rate. Every figure in this scenario is illustrative.

Maya asks three questions:

• Did average handle time (AHT) fall?

• Did first-contact resolution (FCR) improve?

• Did customer satisfaction (CSAT) move?

Nobody can answer. The agent telemetry lives in the observability platform. The business outcomes live in the contact-center and case-management systems. The two datasets do not share a governed measurement contract.

The team has built a beautiful speedometer without connecting it to the destination.

All values shown in this supplied illustration, like the scenario figures in this article, are illustrative.

The framework: observability versus outcome translation

I call the missing architecture the Outcome Translation Layer. It connects AI telemetry to business records and asks one simple question: when this technical signal moves, does a business result move too?

Technical question: Is the agent working well?
  -> latency, errors, task success, eval scores

Business question: Did the customer or business improve?
  -> handle time, resolution, support cost, satisfaction

Simple measurement plan (depends on your use case):
  -> same case + groups to compare + time to wait
  -> what must not get worse + decision owner

This is not a proof machine. Lower latency does not automatically cause lower handle time.

A higher task-success score does not automatically mean more customers resolved their issue.

It is a shared plan for testing whether a technical improvement leads to a business improvement.

You may hear analysts call the groups a cohort, the time to wait a window, the customer type a segment, and the must-not-get-worse checks guardrails. I will use the plain-English terms below.

The six questions that matter

A useful measurement plan does not need analytics jargon. Engineering and the business owner only need to agree on six questions.

1. What business result matters? Name the number the business wants to improve.

2. Which technical signal may influence it? Pick one early signal, such as task success or latency.

3. What shared ID connects the records? Use a real conversation or case ID that exists in both systems.

4. Which groups will we compare, and for how long? Decide who saw each agent version and how long to wait for the result.

5. What must not get worse? Protect quality, safety, customer satisfaction, and access.

6. Who makes the decision? Name the person who can scale, investigate, or stop the rollout.

Maya’s returns example

Suppose Maya wants fewer customers to contact support twice about the same return. Her business result is first-contact resolution: did the customer’s issue stay resolved after the first conversation?

Engineering chooses agent task success as the technical signal. For this workflow, success does not mean the agent produced an answer. It means the agent checked the order, applied the return policy correctly, and created the return without a human fixing it later.

Every conversation already has a support case number. Engineering adds that same case number to the agent trace. Now the team can connect what the agent did with what happened to the customer afterward.

The team sends one group of return requests to the current agent and another comparable group to the new version. They wait 24 hours to see whether either group contacts support again about the same case.

They also agree that customer satisfaction, transfer rates, and policy violations must not get worse. If the new agent resolves more cases but creates more policy mistakes, it is not a win.

At the end of the test, Maya and the AI platform owner review the results together. They decide whether to expand the new version, investigate a problem, or stop the rollout.

That is the Outcome Translation Layer in plain English. The agent trace says, “the return task completed successfully.” The support record says, “the customer did not come back within 24 hours.” The shared case number connects those two statements.

The comparison groups, waiting period, quality checks, and named owners turn that connection into evidence the business can use.

Translation 1: latency to average handle time

Latency is an operational measurement. Average handle time, by contrast, covers the customer interaction from start to finish. Faster model responses may reduce it, but only if they reduce the whole interaction.

An agent can answer quickly and still increase handle time through extra clarifying turns, incorrect actions, human transfers, or reopened cases. That is why I would join model and workflow timing to a real conversation or case, then compare total handle time across agent versions and customer intents.

For this contract, the measurement question becomes: when agent latency changes, does end-to-end handle time change in the same cohort, without increasing transfers, reopens, or abandonment?

Translation 2: task success to first-contact resolution

Task success is only valuable if the task definition matches the customer’s job.

A tool call returning HTTP 200 is not a resolved return. A workflow reaching its terminal node is not proof that the customer received the right outcome.

I define application-level task success against a business acceptance condition: the refund was correctly issued, the replacement was created, or the policy question was answered without another contact.

Then I join that event to the case record and observe whether the case reopened inside the agreed window.

This is where my earlier edition on the three-layer Eval Stack connects to the Outcome Translation Layer.

Evals tell us whether the agent’s behavior met a quality contract. Outcome translation tells us whether that behavior changed what happened to the business process.

“Task success improved from 88% to 94%” is an engineering result. “First-contact resolution improved from 71% to 78%” would be a business result, but both ranges here are illustrative, and the second still needs a valid cohort, window, and confounder review before anyone claims causation.

Translation 3: failed tasks and escalations to support cost

Failure counts are not costs. Two failed tasks can have radically different consequences: one triggers a five-minute correction, while another creates a regulatory escalation and hours of specialist work.

This makes the trade-off visible. A more capable model may cost more per call but reduce rework. A cheaper model may lower inference spend while creating enough escalations to increase total support cost.

The right question is not, “Did AI cost go down?” It is, “Did total cost per correctly resolved case go down, with quality and access protected?”

Translation 4: response quality to customer satisfaction

Response quality is usually measured with rules, human review, or an LLM judge. Customer satisfaction is reported by the customer. They are related signals, but they come from different observers.

I would keep both. The quality evaluation catches policy, clarity, groundedness, and tone problems even when a survey is missing. The customer score reveals whether technically acceptable responses actually felt helpful.

The join must be handled carefully. OpenTelemetry notes that captured prompts, system instructions, and model outputs can contain sensitive or personally identifiable data, and content capture is opt-in. The Outcome Translation Layer should prefer stable identifiers and derived scores over copying raw customer text into every analytics system.

NIST’s AI Risk Management Framework Measure guidance reinforces this broader approach: evaluate under deployment-like conditions, monitor in production, use field data and affected-user feedback, involve domain experts, and document measurable improvements or declines. A model-quality score alone cannot do that job.

The architecture: two lanes and one governed join

I think about this as two event lanes that meet in a governed analysis layer.

The telemetry lane contains traces, metrics, agent version, tool calls, application-defined task outcomes, and eval scores.

The business lane contains case status, interaction duration, transfer events, reopen events, survey results, and cost records.

The Outcome Translation Layer joins only what the measurement contract permits. It applies the agreed window, cohort, segment, and guardrails before producing evidence for a decision.

This design also creates a useful ownership boundary. Platform teams own reliable telemetry. Business teams own outcome definitions. A shared measurement owner owns the contract and the decision rule.

Correlation is the start, not the verdict

Joining two datasets does not prove the agent caused the movement. Seasonality, routing changes, policy changes, staffing, promotions, and customer mix can all move the same KPI.

Start with correlation to find a plausible relationship. Then use a staged rollout, holdout, or controlled cohort where the risk allows it. Record the agent version and experiment assignment in the contract so the comparison survives a deployment.

I also pair every target metric with a guardrail. Zendesk’s current guidance on average handle time makes the reason concrete: a low AHT can indicate efficient service, but it can also reflect rushed or unresolved interactions. AHT should be read alongside customer satisfaction, first-contact resolution, and customer effort.

This prevents Goodhart’s Law from turning the Outcome Translation Layer into a new dashboard theater.

When a measure becomes the target, teams will optimize it. Guardrails make the optimization honest.

How I would apply this with Maya’s team

I would not begin by connecting every trace to every enterprise system. I would choose one high-volume intent, one business outcome, and one accountable owner.

For a returns workflow, the first contract might target first-contact resolution. Engineering would emit genuine conversation_id, case_id, agent_version, task acceptance, latency percentiles, and eval scores.

Customer Service would define resolution, the reopen window, eligible channels, excluded cases, and the minimum CSAT and policy guardrails.

The team would then run a staged cohort. If task success improves but first-contact resolution stays flat, the translation failed and the agent may be optimizing the wrong task.

If both move while policy violations or transfers worsen, the deployment still fails the contract.

If first-contact resolution improves in the exposed cohort, holds across priority segments, and survives the guardrails, Maya finally has evidence that belongs in a business review. The green dashboard remains useful, but it is no longer the destination.

The bottom line

Production AI is not successful because the dashboard is green. It is successful when a governed measurement contract shows that the business metric moved without sacrificing quality, safety, or access.

Observability explains the system. The Outcome Translation Layer explains whether the system was worth operating.

♻️ If this was useful, share it with someone building with AI.

Share

✉️ Subscribe at newsletter.karuparti.com so you never miss an edition.

Anu Karuparti

Creator, Diary of an AI Architect

How enterprises actually ship AI to production

Read by 3,000+ FDEs, AI Architects, and Engineering Leaders from Microsoft, Google, IBM, PwC and others.

Connect with me on LinkedIn.

Want to partner? Email me at anurag.karuparti@gmail.com.

P.S. Want more? 👋

1/ My visual guide to agentic AI → Gumroad

2/ Daily deep dives on agentic AI architecture → LinkedIn

3/ Real-time takes on breaking AI news → X

4/ Casual hot takes and community → Threads

5/ Visual frameworks and carousels → Instagram

6/ 60-second production lessons → TikTok

7/ The full newsletter, free → newsletter.karuparti.com

This week in my diary, I am documenting the system itself: the agent pipeline that scouts trends, manages my content backlog, and edits my videos while I do the only part that actually needs me.

By day, I am an applied AI architect. I build production-grade AI agents for Fortune 500 companies.

On nights and weekends, I am a part-time content creator. I am also a dad to a one-year-old.

Finding time is not easy. So I build systems, with agents of course, that do most of the heavy lifting and get me to the outcome faster. In this case, the outcome is filming a script for my YouTube, Instagram, and TikTok channels.

In this article I will deep dive into my workflow, stage by stage, so you can replicate it if you are building your own content creation system.

This edition is sponsored by Runner. It also happens to be the tool at the center of the workflow you are about to read.

Runner is an AI Chief of Staff that connects to your tools and gets work done:

• Automate work: Inbox triage, meeting prep, research, outreach, follow-ups, and content repurposing.

• Connect your apps: Gmail, Slack, Calendar, Drive, Docs, Sheets, Notion, HubSpot, GitHub, Linear, and more.

• Save Workflows: Reuse your playbooks for reviews, lead qualification, client updates, audits, and content calendars.

• Schedule Runners: Run briefings, pipeline checks, trend scans, and follow-ups without prompting.

• Keep control: Use memory for your preferences and voice, with sensitive actions held for review.

  TRY RUNNER NOW

(Thanks to Runner for partnering on this newsletter and sharing the ultimate agentic workflow guide for automations)

The math that forced the system

Part-time means the math is brutal. I get maybe an hour a day, if I am lucky, to create.

For a long time that hour got eaten by everything except creating: scrolling for trends, re-finding the script I started, figuring out where I left off, deciding what to post. By the time I sat down to actually make something, the window was gone.

So, I stopped trying to find more time and started building a system that makes the hour I have actually count.

Here is the honest truth about being a part-time creator: anything that is not seamless does not get done.

If an idea requires me to open six tabs, copy something from one place to another, and remember where I put the draft, it dies. Every time.

Not because I am lazy. Because the friction is bigger than the time I have.

The framework: find more time versus remove the friction

Reflex: "I need to find more time"
  -> scroll for trends at 11 pm
  -> lose the hour to searching and setup
  -> post on adrenaline, burn out by month four

Fix: "I need to remove the friction"
  -> agents do the noticing and the logging
  -> one pipeline holds every idea, script, and status
  -> the hour goes to the one part that needs me

So, I am automating the parts that do not require me and structuring the parts that do.

The whole thing is designed around one question: when I sit down to create, how fast can I go from “open laptop” to “filming a script”?

The pipeline has five stages:

1. Trend scan: a scheduled Runner agent scans TikTok, Instagram Reels, X, and YouTube and writes ideas into a Google Sheet.

2. Curation: I pick the winners, then Runner moves the selected rows from the Sheet into my Notion content calendar.

3. Scripting: I write the script in Google Docs, with Runner as a brainstorming partner when I am stuck.

4. Filming: I record the video. I keep this part human because my face, voice, and judgment are the product.

5. Editing: a Runner Workflow runs my Remotion skill and turns the raw clip into a finished vertical short.

Stage 1: the trend scan that runs without me

Every morning, a Runner scan runs across TikTok, Instagram Reels, X, and YouTube. It pulls what is actually trending in AI: sounds, formats, viral posts, hot takes. Then it writes 20+ content ideas into a Google Sheet.

Each row has the trend, a hook, a content angle, a caption, hashtags, format notes, and the best posting time.

I do not scroll for trends anymore. I wake up and the ideas are already sitting in a spreadsheet. That is the noticing, done for me, every day, on a schedule.

Here is the system in practice: the morning scan summary on top, and the Sheet it keeps building over time below. I treat the output as a queue of research leads, not publication-ready facts.

Stages 2 and 3: the part that actually needs me

The Sheet is the raw idea bank. Append-only. I never curate it in place. When I sit down to create, I pick one or two rows that fit my angle and drop them into a Notion pipeline.

Each piece of content is one item that moves through statuses: Idea → Scripting → Filming → Scheduled → Published. One glance tells me exactly what is in scripting, what is waiting to be filmed, and what ships next.

Then I write the script. That is the part that needs me: the thinking, the take, the hook. The script lives in a Google Doc the system already created and linked to the Notion item, so I open it and start typing.

Could an agent draft the script too? Probably. But the point of the system is not to remove me from the work that is mine. When I am stuck, Runner is a good brainstorming partner because it already has the trend row and my workspace as context.

The system’s real job is to make sure that by the time I am writing, everything else is handled: the idea is captured, the task exists, the page is open and waiting.

Stages 4 and 5: filming, then a Runner Workflow edits the video

I film the script. That is the whole stage. The system cannot supply my point of view or presence; that part stays human.

Editing used to be the second biggest time sink after trend scouting. Now I hand the raw clip to a Runner Workflow that runs my Remotion skill (ported from my Claude Skill).

It cuts the clip into a vertical 1080x1920 short, adds word-synced captions, drops in motion graphics and brand logos, and renders a thumbnail with a title you can read on a phone.

The editing logic started as a Claude skill. I did not throw that work away. I brought the workflow logic into Runner and saved it as a reusable Workflow.

Runner’s own model is simple: a Workflow says how, a Runner owns the job, and the schedule says when.

If you already have Claude skills, you do not have to rebuild from a blank page. Bring the instructions, checklists, and decision rules you already encoded into a Runner Workflow, then let a Runner invoke that playbook on demand or on a schedule.

For me, that meant moving my caption style, safe zones, font sizes, brand colors, and render steps into the Remotion workflow. Every edit is still a prompt, not an evening, but now it runs inside the same system as the rest of the pipeline.

Why it’s all in one place now.

I used to run this through an OpenClaw agent linked to my Telegram. The idea was smart: trends land in a chat, I react, things get logged. I wrote about that architecture in my OpenClaw and Hermes edition.

But the UX was brutal. Everything was a flat scroll of messages. No structure, no views, no way to see what is in scripting versus what is scheduled.

My ideas, my scripts, and my statuses were all mashed into one thread, and I would spend my hour just searching for what I wrote yesterday.

What I wanted was something that felt like the systems I build at work: structured, connected, personal, but with a web UX that actually worked.

That is what made Runner click for me. My Notion and Google Drive are connected, so it reads my trend Sheet, files selected ideas in my Notion pipeline, and creates Docs in my Drive, all from one place.

It also remembers how I work: the topics I cover, the systems I use, and the context behind my projects.

I do not have to introduce myself or reconstruct the workflow every time I start a session. It feels less like prompting a blank chatbot and more like working with a collaborator who already knows the setup.

I am not copy-pasting between apps or rebuilding context anymore. The apps talk to each other, Runner keeps the thread, and I focus on the judgment.

The part-time creator’s real advantage

People assume part-time creators are at a disadvantage because they post less. Here is the flip side: you are forced to build a system run by agents, because you have no other choice. And a system compounds in a way that willpower does not.

The creators posting every day on adrenaline and caffeine will burn out by month four. The ones with a pipeline (automated intake, a place for the script, a way to review what worked) just keep going.

And in content, the people who keep going are the ones who win.

Thank you for reading Diary of an AI Architect

Anu Karuparti Creator, Diary of an AI Architect

How enterprises actually ship AI to production

Read by 3,000+ FDEs, AI Architects, and Engineering Leaders from Microsoft, Google, IBM, PwC and others.

Connect with me on LinkedIn.

Want to partner? Email me at anurag.karuparti@gmail.com.

Graph engineering determines when agents work sequentially or concurrently, where evaluators and humans enter the loop, and how routing and handoffs lead to a final business outcome.

This week in my diary, I am breaking that system into five control layers: prompt, context, harness, loop, and graph.

I have helped ship more than 50 production generative AI systems. The teams that struggle are rarely blocked by the wording of one prompt. They are struggling with what reaches the model, what the model can touch, how the agent recovers, and how the work moves across a larger system.

The graph layer in this framework was sharpened by Greg Isenberg’s recent explanation of graph engineering. His video is a useful companion if you want the outermost layer in plain English.

Here is the mental model I now use with enterprise teams: prompts steer, context informs, harnesses constrain, loops persist, and graphs coordinate.

My YouTube short explains the key differences between these five layers in 90 seconds.

Priya’s agent passed the demo and failed the system

Imagine Priya, a staff AI architect at a regional bank. Her team builds a coding agent to fix low-risk defects in an internal payments service.

The demo looks excellent. The prompt tells the agent to inspect the issue, avoid unrelated changes, run the tests, and return a pull request summary. On a clean sample repository, it finds the bug and produces a reasonable patch.

Then the pilot reaches a real repository. The agent misses an architecture decision buried in the docs, selects a shell command with a broader scope than intended, retries the same failing test without changing its hypothesis, and sends the pull request to the wrong review path.

The team asks, “How should we improve the prompt?” Priya asks a better question: which control layer failed? The instruction was only one part of the system, and it was not the part that caused most of the damage.

The framework: one model call versus a controlled system

Think of the five concerns as control layers around the model:

MODEL CALL = prompt + context
AGENT      = model call + harness + loop
SYSTEM     = agents + deterministic steps + humans connected by a graph
EVALS      = evidence that every layer works

Each layer answers a different architecture question.

Prompt and context operate closest to the model. Harness and loop turn the model into an agent. Graph engineering turns agents, functions, and human decisions into a system.

Layer 1: prompt engineering

Prompt engineering controls the instruction: role, goal, constraints, examples, output contract, and success criteria.

Priya’s coding agent might receive this:

Analyze the reported defect and propose the smallest safe fix.
Do not change unrelated behavior.
Return the root cause, files changed, test evidence, and residual risk.
Stop and ask for approval if the fix changes an external contract.

The unit being optimized is one model interaction. A stronger prompt reduces ambiguity and makes the expected output inspectable, but it cannot supply a missing design document, restrict a dangerous tool, or decide which reviewer receives the result.

Prompts still matter. In an agent system, the prompt is the steering wheel, not the car.

Layer 2: context engineering

Context engineering controls what reaches the context window: retrieved documents, conversation history, tool results, code files, policies, summaries, and the information deliberately excluded.

Ask the same model to summarize the risks in an 80-page contract.

Setup A dumps the document into the window with no retrieval plan.

Setup B retrieves the liability, indemnification, termination, and data-use clauses, includes the company’s risk policy, preserves linked definitions, and allows the agent to pull the full text when a cross-reference matters.

The prompt did not change. The evidence available to answer it did.

Setup B is designed to produce a more useful risk review, provided the retrieval layer is tested for omissions.

This connects to my earlier edition, “Context engineering is about to become your organization’s biggest moat.” A brilliant instruction with a poor working set still produces a poor decision.

For Priya, the missing architecture decision is a context failure. Rewording the prompt may hide the symptom for one test case, but fixing retrieval and context assembly repairs the class of failure.

Layer 3: harness engineering

I use harness engineering as shorthand for the runtime envelope around the model: tools, file access, shell access, MCP connections, sandboxing, permissions, logging, approval boundaries, timeouts, and error surfaces.

The model decides, “I need to run the tests.” The harness decides whether test execution is available, which commands are allowed, which directory is visible, how long the process can run, what output returns, and what gets recorded.

MCP can standardize how an AI application connects to external tools, resources, and prompts. It does not decide that an agent deserves production write access. Identity, authorization, least privilege, and approval policy still belong in the host and its surrounding control plane.

This is where capability meets control. In regulated environments, this is often the layer the security team cares about first.

For Priya, the broad shell command is a harness failure. The fix is not “please be careful” in the prompt. The fix is a narrower tool, an allowlist, a sandbox, and an approval gate for commands with a larger blast radius.

Learn how to implement Agent Harness with Microsoft Agent Framework here.

Layer 4: loop engineering

Loop engineering controls the cycle: act, observe, evaluate, adjust, and repeat. It owns retry policy, validators, completion criteria, budgets, stop conditions, and escalation.

The distinction from the harness matters:

• Harness: Can the agent execute the test, in which sandbox, with what permission and timeout?

• Loop: Does a failed test trigger another attempt, what must change before retrying, how many attempts are allowed, and what counts as done?

You can have a good harness and a terrible loop. An agent can be perfectly sandboxed, fully logged, and still repeat the same failed fix until it exhausts its budget.

This is the layer I explored in my earlier edition on loop engineering. Priya’s repeated test failure needs a new hypothesis requirement, a retry cap, and an escalation rule, not broader filesystem access.

Layer 5: graph engineering

Graph engineering controls the topology of the whole workflow. Nodes may be agents, deterministic functions, data transformations, evaluators, or human gates. Edges define sequencing, routing, parallel branches, recovery paths, and where loops live.

Loop asks, “How should this agent keep working?” Graph asks, “Which component works next, and how does the system coordinate?”

LangGraph describes itself as a low-level orchestration runtime for long-running, stateful workflows. Its useful design idea is not simply drawing boxes. It is mixing deterministic steps with model-driven steps while preserving state, durable execution, and human interrupts.

Learn how to build agent graphs with Microsoft Agent Framework here.

Greg Isenberg’s graph engineering framing makes this layer easy to see: stop asking one chat to plan, research, write, and approve its own work. Split responsibilities, draw the real dependencies, and keep the human where consequences become expensive.

The evidence also argues against graph maximalism. Anthropic reported that its multi-agent research system outperformed a single-agent setup by 90.2% on an internal breadth-first research evaluation, but its multi-agent runs used about 15 times as many tokens as chat interactions. That result is specific to Anthropic’s research workload, not a universal multiplier, and it captures the trade-off perfectly: graphs earn their complexity only when the task value and parallelism justify the bill.

For Priya, the wrong review path is a graph failure. The system needs an explicit route from code changes to tests, security review, contract review when needed, and human approval before merge.

All five layers at once

The cleanest way to understand the model is one coding workflow using every layer simultaneously.

• Prompt: “Fix this defect without changing unrelated behavior.”

• Context: Issue, repository files, architecture decisions, test failures, ownership metadata, and relevant git history.

• Harness: Scoped terminal, filesystem tools, git operations, sandbox, permissions, secrets isolation, and traces.

• Loop: Inspect, form a hypothesis, modify, test, diagnose, retest, and stop or escalate against explicit criteria.

• Graph: Triage, planner, coding agent, deterministic tests, security reviewer, human approval, and merge.

These layers do not replace one another. A production agent uses all five at the same time, and the weakest control often sets the reliability ceiling for the whole system.

Three honest caveats

I want you to use this framework, but I also want you to use it accurately.

The terms are not equally mature. Prompt engineering and context engineering are established industry language. Agent harness is a real category, while loop engineering and graph engineering are newer labels for concerns practitioners also call agent loops, workflows, and orchestration.

The boundaries leak. Memory can belong to context, harness, or runtime state. Verification can sit inside a tool boundary, a retry loop, or a separate graph node. These are five concerns, not five perfectly isolated software components.

The layers are not a build order. I often sketch the graph first (workflow), define the loops and control boundaries next, then tune prompts last. Think concentric controls around the model, not a waterfall pipeline.

The value of the framework is diagnostic clarity. It gives the team a better answer than “the AI was weird.”

The concern that spans all five: evals

There is a sixth concern that cuts across every layer: evals.

How do you know the prompt follows instructions? Did context retrieval include the decisive evidence? Did the harness allow the right tool and deny the unsafe one? Did the loop stop for the right reason? Did the graph route the risky case to a human?

Measure each layer with evidence that matches its job:

OpenAI’s evaluation guidance starts with the same discipline: define the desired behavior, run representative test inputs against explicit criteria, analyze the results, and iterate. Evals are not a sixth ring outside the architecture. They are the measurement system running through all five.

The bottom line

If your team is still treating prompt engineering as the whole job, you are staffing for one fifth of the architecture. Prompts steer, context informs, harnesses constrain, loops persist, and graphs coordinate.

Build all five. Measure all five. Then ask your team which layer is weakest before you rewrite another prompt.

♻️ If this was useful, share it with someone building with AI.

✉️ Subscribe at newsletter.karuparti.com so you never miss an edition.

Thank you for reading my newsletter and supporting my work. I truly appreciate it.

Anu Karuparti Creator, Diary of an AI Architect How enterprises actually ship AI to production

Read by 3,000+ FDEs, AI Architects, and Engineering Leaders from Microsoft, Google, IBM, PwC and others.

Connect with me on LinkedIn.

Want to partner? Email me at anurag.karuparti@gmail.com.

P.S. Want more? 👋

1/ My visual guide to agentic AI → Gumroad

2/ Daily deep dives on agentic AI architecture → LinkedIn

3/ Real-time takes on breaking AI news → X

4/ Casual hot takes and community → Threads

5/ Visual frameworks and carousels → Instagram

6/ 60-second production lessons → TikTok

7/ The full newsletter, free → newsletter.karuparti.com

References

1. Greg Isenberg. “Why Graph Engineering will 10x your Claude/Codex.” YouTube. Source for the graph-engineering framing and the shift from one overloaded chat to designed handoffs.

2. Anthropic. “Building effective agents.” Definitions of workflows and agents, the augmented LLM, agent loops, parallelization, evaluator-optimizer, human checkpoints, and the recommendation to add complexity only when it improves outcomes. https://www.anthropic.com/engineering/building-effective-agents

3. Anthropic. “How we built our multi-agent research system.” Source for the 90.2% internal research-eval result, approximately 15x token-use figure, orchestration pattern, evaluation lessons, and production reliability trade-offs. https://www.anthropic.com/engineering/multi-agent-research-system

4. LangChain. “Runtimes, frameworks, and harnesses.” Current distinctions among agent frameworks, runtimes, and agent harnesses. https://docs.langchain.com/oss/python/concepts/products

5. LangChain. “LangGraph overview.” Low-level orchestration, deterministic and agentic steps, persistence, durable execution, and human-in-the-loop controls. https://docs.langchain.com/oss/python/langgraph/overview

6. Model Context Protocol. “What is MCP?” MCP as an open standard for connecting AI applications to external data, tools, and workflows. https://modelcontextprotocol.io/docs/getting-started/intro

7. Model Context Protocol. “Architecture overview.” Host-client-server responsibilities, tools and resources, transports, and authorization boundaries. https://modelcontextprotocol.io/docs/2026-07-28/learn/architecture

8. OpenAI. “Working with evals.” Defining desired behavior, representative test data, explicit grading criteria, and iterative analysis. https://developers.openai.com/api/docs/guides/evals

Disclaimer: The stories and scenarios in this article are hypothetical, inspired by patterns observed across similar real-world experiences. They are used to convey key concepts more effectively and do not represent any specific individual or organization.

The most important question about Kimi K3 is not whether it beats the best closed model. It is what happens to the rest of the AI economy when frontier-level intelligence becomes easier to download, customize, and distribute.

That question is the starting point for Part 2 of my Kimi K3 analysis.

In Part 1, I wrote from a builder/architect’s perspective. I explained why architects should pay attention to Kimi K3 and how open-weight models change the way we design AI systems.

This time, I am looking at the same shift as an investor.

That is different from my usual technical deep dives. Promise that I will have more technical deep dives in the coming days.

I also invest across the AI ecosystem, and use my technical background to understand which trends could create durable businesses, which could compress margins, and where value might move next. Below is my analysis, not investment advice.

One post from investor Gavin Baker made me stop and think. I read it several times because it offered a useful explanation for the market’s reaction to K3. His argument was that open-weight models could pressure the economics of leading closed-model companies while benefiting much of the surrounding AI stack.

That leads to the bigger question now worrying Wall Street:

If intelligence becomes cheaper, are the enormous investments in chips, power, data centers, and cloud infrastructure still justified?

To answer that, we first need to understand two competing strategies in the global AI race.

Leading US frontier labs such as OpenAI and Anthropic primarily monetize access to proprietary models.

Chinese labs such as Moonshot, DeepSeek, Qwen, and Z.ai increasingly use open-weight releases to win distribution, attract developers, and accelerate ecosystem adoption.

This is not a perfect US-versus-China divide. Both countries have open and closed models. But their current centers of gravity are different.

We have seen versions of this competition before: proprietary operating systems versus Linux, iOS versus Android, and closed platforms versus open ecosystems.

Open alternatives do not necessarily destroy proprietary products. Android did not eliminate iOS, and Linux did not eliminate commercial operating systems.

But they changed pricing power, distribution, and where profits accumulated.

That is the question I want to explore:

If open weights compress margins at the model layer, does economic value disappear, or does it migrate toward chips, clouds, inference platforms, and applications? And why is Nvidia betting on the second outcome?

Moonshot AI released K3’s full weights on July 27 under a custom license. The model has 2.8 trillion total parameters, activates 104 billion per token, and supports a one-million-token context window ^[1]. Its official Hugging Face checkpoint recorded more than 1.1 million downloads in the last month as of August 4 ^[2].

K3 is not free compute. Moonshot recommends supernode configurations with 64 or more accelerators for deployment ^[1]. What became freely obtainable was the model artifact. The expensive work of serving it moved elsewhere.

That distinction is the investment story.

The Story…

Rina is a portfolio manager covering AI infrastructure and software. On Monday morning, her team puts two charts on the screen.

The first shows K3’s release and the sharp declines in two Chinese AI competitors. Reuters reported that Zhipu and MiniMax shares were down 27.7% and 16.5%, respectively, near the close on launch day ^[3]. The second shows the familiar AI stack: power, accelerators, networking, clouds, model labs, and applications.

Her analyst asks the obvious question: “If models become cheaper and more open, is that bearish for the companies funding the compute buildout?”

The answer depends on which layer you are analyzing.

Cheaper intelligence can hurt a company whose advantage is scarcity at the model layer. The same price decline can help an application company that now buys more capability for every dollar.

It can help an inference provider that serves many open models. It can help a chip vendor if lower prices unlock enough additional workloads.

One event can be deflationary for one layer and expansionary for another.

The five-layer value migration map

I use a five-layer framework to analyze this shift:

1. Model margins: Does open competition reduce scarcity and pricing power?

2. Compute volume: Does cheaper intelligence create enough new usage to offset efficiency?

3. Distribution power: Which model ecosystem becomes the default starting point for builders?

4. Value capture: Which infrastructure and application layers gain leverage?

5. Thesis breakers: What evidence would prove the bullish interpretation wrong?

This is not a forecast for any individual stock. It is a way to reason about how one technical shift can change the economics of the whole stack.

Layer 1: Model margins face a new ceiling

Investor Gavin Baker captured the sharpest version of the argument in a July 17 post. He called K3 a potential inflection point that could be negative for Anthropic and OpenAI while benefiting almost every other company ^[4]

His hypothetical is a world where two or three frontier labs sustain 90% inference margins. In that world, those labs gain extraordinary bargaining power over power suppliers, data centers, semiconductor vendors, hyperscalers, and application companies. They can also integrate downward into infrastructure and upward into software.

Open models put a ceiling on that scenario. If a credible alternative can be downloaded, modified, and served by several providers, the model layer becomes harder to monopolize. Customers gain a fallback. Inference companies gain inventory. Application builders gain negotiating leverage.

K3 does not prove proprietary models lose. Baker explicitly says the threat is only potential. Product experience, agent harnesses, reliability, and stronger internal checkpoints may protect the leading closed labs ^[4]. A weight file is not a complete product.

K3’s weights are available to download, modify, and deploy, including for many commercial uses. But the license adds conditions for companies operating model-as-a-service businesses and for products above certain scale thresholds. Large model-hosting businesses may need a separate agreement with Moonshot, while very large products may need to display Kimi K3 attribution ^[5].

In other words, the weights are open, but the commercial license is not unrestricted. That is why I call K3 an open-weight model, not unconditionally open-source software.

Layer 2: Jensen Huang is betting on elasticity

Nvidia CEO Jensen Huang’s response to K3 was unusually direct. “Free AI should be great for hardware. Free AI should be great for chips. Free AI should be great for data centers,” he told Axios ^[6].

That is an elasticity argument.

If the price of useful intelligence falls, more developers experiment, more companies deploy, and existing applications call models more often. Each model invocation still requires compute somewhere, even when the weights cost nothing to acquire.

The arithmetic is simple. Total inference demand depends on three moving variables:

# Illustrative example.
task_growth = 4.0              # Companies run four times as many AI tasks.
tokens_per_task = 0.60         # Each task uses 40% fewer tokens.
compute_per_token = 0.80       # Each token needs 20% less compute.

new_compute_demand = task_growth * tokens_per_task * compute_per_token
print(new_compute_demand)      # 1.92x the original compute demand

Start with 100 units of compute demand. Running four times as many tasks would raise demand to 400 units. Using 40% fewer tokens per task reduces it to 240 units, and using 20% less compute per token reduces it again to 192 units.

The final result is still 192 units versus the original 100, so total compute demand rises 92%. Usage grew faster than the combined efficiency gains.

That break-even point is the entire Nvidia debate. In this example, each task needs only 48% as much compute as before. Therefore, companies must run slightly more than twice as many tasks, about 2.1 times as many, before total compute demand increases.

If the above math doesn’t make sense, please see below

Before:

1,000 tokens per task
× 1 compute unit per token
= 1,000 compute units per task

After:

600 tokens per task
× 0.8 compute units per token
= 480 compute units per task

480 / 1,000 = 48%

So each task is 52% cheaper in compute terms.

If companies run 4× as many tasks:

4 × 48% = 192%

So total compute consumption becomes 1.92× the original amount, even though every individual task became dramatically more efficient.

The break-even point is:

1 ÷ 0.48 = 2.08×

So:

Below ~2.08× task growth → total compute demand falls.
At ~2.08× task growth → compute demand stays roughly flat.
Above ~2.08× task growth → total compute demand rises.

K3 itself is not yet the clean proof. Baker’s caveat is that K3 is less token efficient than leading closed alternatives on Artificial Analysis data, so the true “Sputnik moment” would be an open frontier model that also leads on intelligence per dollar ^[4].

A computationally hungry open model may increase infrastructure demand, but customers ultimately optimize for completed work, not raw tokens.

Layer 3: Distribution is becoming strategic power

China’s open-weight strategy is not only about model revenue. It is also about becoming the foundation other people build on.

Hugging Face reported that Chinese models accounted for 41% of downloads over the prior year, the largest national share on its platform ^[7]. The report also found that Qwen had more than 113,000 derivative models using it as a base, and more than 200,000 repositories tagging Qwen ^[7].

Those numbers are platform-specific, not a measure of the entire AI market. They still reveal a powerful distribution flywheel:

Open weights
  -> more downloads and experiments
  -> more fine-tunes, quantizations, and serving support
  -> lower adoption friction
  -> more applications and developer familiarity
  -> stronger ecosystem influence

The contrast with the dominant US frontier strategy is useful, but it is not a clean national binary. OpenAI and Anthropic emphasize proprietary frontier services. China has made frequent open releases from DeepSeek, Qwen, Z.ai, MiniMax, and Moonshot a competitive norm. At the same time, the US has open efforts such as GPT-OSS, Gemma, Llama, and Nvidia’s Nemotron, while Chinese companies also sell closed products and APIs.

The strategic difference is emphasis. Many US frontier companies monetize scarce access to their best systems. Chinese labs are increasingly using open artifacts to win distribution, accelerate derivatives, support domestic hardware, and pull developers into their ecosystems ^[7][8].

Stanford’s 2026 AI Index says the US-China model performance gap had narrowed to 2.7% as of March, even as the US retained advantages in top-tier model production and private investment ^[9]. When capability gaps shrink, distribution matters more.

Layer 4: Value does not disappear, it moves

Vercel’s June production data offers an early picture of that migration. Open-weight models ran 29% of tokens through its AI Gateway on less than 4% of spend. Meanwhile, the top four US frontier labs captured 95% of spend, and Anthropic alone captured 61% of spend on 32% of tokens ^[10].

This is one gateway’s routing data, not the whole market. But it supports a two-tier structure already forming in production:

The strongest application companies may treat models the way modern software treats databases. The underlying engine matters, but customer value lives in workflow design, proprietary context, evaluation, permissions, integration, and distribution.

Nvidia is already acting on this thesis. Its Nemotron program publishes open weights, training data, and recipes. Its Nemotron Coalition is building an open frontier model on Nvidia infrastructure, and Nvidia says open models create more startups, more builders, and lower entry barriers ^[11][12].

There is a counter-risk for Nvidia. Chinese labs are also optimizing models for Huawei Ascend, Cambricon, and other domestic accelerators, while open software is reducing dependence on CUDA-specific deployment paths ^[8]. Open-model growth can expand global compute demand without guaranteeing that every new dollar lands on Nvidia hardware.

That nuance makes Huang’s position more interesting, not less. He is defending an expanding market while competing to keep Nvidia at the center of it.

Layer 5: What would break the thesis

A serious investment framework needs disconfirming evidence. I would watch five failure modes.

1. Efficiency outruns usage. If task volume grows slower than reductions in tokens per task and compute per token, total accelerator demand falls.

2. Products remain the moat. If users consistently pay for ChatGPT, Claude, and their agent harnesses regardless of model availability, value stays with closed platforms.

3. Open models fail high-stakes work. Vercel found that expensive, consequential workloads still concentrated in frontier closed models ^[10]. If that persists, open weights own volume but not profit pools.

4. Policy restricts distribution. Security reviews, sanctions, procurement rules, or data-governance concerns could slow adoption of Chinese weights in Western enterprises.

5. Alternative silicon captures the growth. If the Chinese open ecosystem increasingly trains and serves on domestic accelerators (compute), the open-model boom may weaken Nvidia’s geographic share even while total compute expands.

I would add one more operating test: cost per accepted result. Open weights win economically only when inference, retries, latency, engineering, governance, and human correction together beat the alternative. A low token price is not enough.

The bottom line

Kimi K3 is not proof that open models win every workload or that proprietary labs lose their businesses. It is evidence that frontier intelligence is becoming more contestable, and contestability changes who holds leverage across the stack.

The opportunity is not simply in owning the smartest model. It is in understanding where value moves when intelligence gets cheaper.

♻️ If this was useful, share it with someone building with AI.

✉️ Subscribe at newsletter.karuparti.com so you never miss an edition.

P.S. Want more? 👋

1/ My visual guide to agentic AI → Gumroad

2/ Daily deep dives on agentic AI architecture → LinkedIn

3/ Real-time takes on breaking AI news → X

4/ Casual hot takes and community → Threads

5/ Visual frameworks and carousels → Instagram

6/ 60-second production lessons → TikTok

7/ The full newsletter, free → newsletter.karuparti.com

References

1. Kimi K3: Open Frontier Intelligence - Moonshot AI

2. Kimi K3 model card and usage data - Moonshot AI and Hugging Face

3. China’s Moonshot unveils world’s largest open AI model - Reuters

4. Gavin Baker on Kimi K3 and AI value migration - X

5. Kimi K3 License - Moonshot AI

6. Nvidia’s Jensen Huang defends Chinese AI amid Kimi panic - Axios

7. State of Open Source on Hugging Face: Spring 2026 - Hugging Face

8. On the Shifting Global Compute Landscape - Hugging Face

9. The 2026 AI Index Report - Stanford HAI

10. AI Gateway Production Index, July 2026 - Vercel

11. NVIDIA Nemotron - Nvidia

12. NVIDIA launches the Nemotron Coalition - Nvidia

Anu Karuparti Creator, Diary of an AI Architect How enterprises actually ship AI to production

Read by 3,000+ FDEs, AI Architects, and Engineering Leaders from Microsoft, Google, IBM, PwC and others.

Connect with me on LinkedIn.

Want to partner? Email me at anurag.karuparti@gmail.com.

Investment note: This article is an educational analysis of AI industry economics, not investment advice or a recommendation to buy or sell any security.

Disclaimer: The stories and scenarios in this article are hypothetical, inspired by patterns observed across similar real-world experiences. They are used to convey key concepts more effectively and do not represent any specific individual or organization.

This week in my diary, I am working through a question I expect more AI architects to face: when should an enterprise use an open-weight model, and when is a closed model still the better architectural choice? And does Kimi K3 makes that question urgent.

Moonshot AI introduced Kimi K3, a 2.8-trillion-parameter model with native multimodality and a one-million-token context window.

Moonshot calls it the first open 3T-class model and says the full weights will be released by July 27, 2026¹. As I publish this on July 24, K3 is available through Kimi’s products and API, but the full weights are not yet downloadable.

The timing matters because K3 is not a small open model nibbling at the edges. It scores 57 on the Artificial Analysis Intelligence Index, three points behind Claude Fable 5 at 60 and two behind GPT-5.6 Sol at 59 ^[3][4][5]. Reuters also reported that Arena.ai ranked K3 first on its benchmark for building web interfaces at launch ^[2].

Then the market noticed. Reuters reported that shares of Chinese AI competitors Zhipu and MiniMax fell 27.7% and 16.5%, respectively, after the announcement ^[2]. I would not claim that K3 caused every wobble in US technology stocks, but those moves show that investors understood the threat: frontier capability may no longer belong only to labs that keep their best models behind an API.

I am not writing this from a geopolitical angle. I am writing it from a builder and architect’s angle: what does a competitive open-weights model like Kimi K3 actually mean for your AI solutions, and how do you get ahead of it.

Not whether open weights beat closed models everywhere, but how you build agents and orchestration that really harness this shift instead of getting caught flat-footed by it.

The framework: design for model optionality

Reflex: hardwire the newest model into every agent
  -> optimize for one leaderboard
  -> inherit one vendor's economics and constraints
  -> migrate the whole system when the frontier moves

Fix: separate orchestration from model execution
  -> classify the workload's constraint
  -> route it to open or closed execution
  -> measure quality, latency, and cost per completed task

The reflex is understandable. Leaderboards are legible and screenshots travel fast. But a model is one component in an agent system, not the architecture itself. Here are the four questions I would ask before adding K3 to an enterprise model portfolio:

• Move 1: Identify the constraint that open weights remove.

• Move 2: Decide which control you are willing to own.

• Move 3: Route workloads instead of choosing one winner.

• Move 4: Measure economics per completed task.

Move 1: Identify the constraint that open weights remove

Open weights are valuable when they remove a real constraint. They let you deploy inside a controlled environment, adapt the model, inspect artifacts, choose an inference provider, and avoid making one API the permanent center of your system. Chinese labs are gaining influence through this strategy, with MIT Technology Review reporting that Chinese open-weight models narrowly surpassed US models in global downloads over the year ending August 2025 ^[7].

For most teams, the constraint is data residency. The compliance team cares more about whether patient data leaves an approved boundary than a three-point benchmark gap. For some team, the constraint may be customization, offline operation, regional availability, or leverage in a vendor negotiation.

Open weights do not automatically mean easy self-hosting. Reuters noted that K3’s scale means few users are likely to host it themselves, and Moonshot recommends supernode configurations with 64 or more accelerators ^[1][2]. For many teams, “open” will mean using a managed inference partner while retaining the option to move, not putting K3 under someone’s desk.

Move 2: Decide which control you are willing to own

Open weights exchange vendor control for operator responsibility. You gain deployment and adaptation control, but you also own serving reliability, capacity planning, model patching, evaluation, safety controls, and incident response. Downloading weights is free. Operating frontier inference is not.

Closed models make a different trade. You give the provider more control over the model, roadmap, pricing, and availability. In return, enterprise contracts can provide service commitments, certifications, abuse monitoring, support, and a clearer party to hold accountable.

Heavily regulated industries do not universally prefer one side. A bank may prefer open weights for sovereign deployment and closed models for workloads where vendor assurance and indemnification matter more. The architecture should let policy choose the execution path at runtime.

K3 adds two model-specific risks. Moonshot says it can become unstable if an agent harness does not preserve its thinking history, and that its long-horizon training can make it excessively proactive on ambiguous tasks ^[1]. Those are not reasons to reject it. They are reasons to test harness compatibility and enforce explicit action boundaries before production.

The chart shows why routing matters. K3 sits close to the intelligence frontier, but its serving profile differs materially from the two closed models.

Move 3: Route workloads instead of choosing one winner

The winning design is often a portfolio. Put a policy-aware model router behind a stable orchestration contract.

• Send sensitive, customizable, or offline tasks toward an approved open-weight deployment.

• Send high-stakes tasks that need maximum capability or contractual assurance toward a closed provider. Keep deterministic checks outside both models.

K3’s web-interface result makes it especially interesting for visual coding agents, front-end prototyping, long-horizon engineering, and multimodal creation. Reuters reported that it ranked first on Arena.ai’s web-interface benchmark at launch ^[2]. That does not make it the automatic choice for every coding agent, but it earns a place in the evaluation set.

Move 4: Measure economics per completed task

Here is K3’s caveat. On the Artificial Analysis Intelligence Index v4.1 evaluation, K3 generated 130 million output tokens, compared with 87 million for Fable 5 and 70 million for GPT-5.6 Sol ^[3][4][5][6]. That is 1.86 times GPT-5.6 Sol’s output-token total for the same evaluation suite. K3 was also slower in the reported measurements, at 34.2 output tokens per second versus 73.1 for Fable 5 and 63.5 for GPT-5.6 Sol.

Top AI Investor Gavin Baker called K3 a “token wastrel,” arguing that cost per token hides its higher cost per completed task ^[8]. The phrase is provocative, but the architecture lesson is sound: do not compare model economics with list price alone.

This does not prove that open-weight models as a category are token inefficient. It proves that this version of K3 used more output tokens on this evaluation.

Your production metric should be cost per successful task, including retries, latency, infrastructure utilization, human review, and failure recovery.

That is the unit-economics discipline I wrote about in my earlier editions on tokenomics and model routing. A cheaper token can still produce a more expensive outcome.

What I will be watching, and how I will architect for it

I will not replace every model call with K3 after July 27. I will add it as a new execution target behind an existing model gateway, then run a controlled evaluation on representative traces. This is the part I am watching most closely: not the leaderboard, but how cleanly K3 slots into an architecture built for optionality.

I would watch five production metrics:

1. Successful task cost, not token price.

2. Tokens per accepted result, including retries.

3. Latency to useful completion, not first-token speed.

4. Human correction rate, especially for visual coding.

5. Boundary violations, because proactive agents (open-weight) need tight permissions.

Start with a narrow workload where K3 has a plausible advantage, such as front-end generation from screenshots. Shadow it against the incumbent model. Promote it only when quality and unit economics clear predefined gates.

The bottom line

Kimi K3 is a genuine open-weight frontier challenge, even though the full weights are scheduled for release after this article publishes. Its intelligence and visual-coding results deserve attention. Its scale, harness requirements, and token appetite demand architecture discipline.

Next week, in Part 2, I want to focus from an economy standpoint: what does an open-weights model like Kimi K3 mean for the entire AI stack? Stanford’s 2026 AI Index says the US-China performance gap has effectively closed, while the US still produces more top-tier models overall ^[9]. My thesis is that open-weights competition helps spread value across the full stack, chips, power, clouds, inference providers, and application builders, instead of letting a few leading labs with closed frontier models dominate demand. K3 gives us an early test of that thesis.

♻️ If this was useful, share it with someone building with AI.

✉️ Subscribe at newsletter.karuparti.com so you never miss an edition.

Thank you for reading my newsletter!

Anu Karuparti

Creator, Diary of an AI Architect

How enterprises actually ship AI to production

Read by 3,000+ FDEs, AI Architects, and Engineering Leaders from Microsoft, Google, IBM, PwC and others.

Connect with me on LinkedIn.

Want to partner? Email me at anurag.karuparti@gmail.com.

P.S. Want more? 👋

1/ My visual guide to agentic AI → Gumroad

2/ Daily deep dives on agentic AI architecture → LinkedIn

3/ Real-time takes on breaking AI news → X

4/ Casual hot takes and community → Threads

5/ Visual frameworks and carousels → Instagram

6/ 60-second production lessons → TikTok

7/ The full newsletter, free → newsletter.karuparti.com

References

1. Kimi K3: Open Frontier Intelligence - Moonshot AI

2. China’s Moonshot unveils world’s largest open AI model, closing in on US rivals - Reuters

3. Kimi K3: Intelligence, Performance & Price Analysis - Artificial Analysis

4. Claude Fable 5: Intelligence, Performance & Price Analysis - Artificial Analysis

5. GPT-5.6 Sol (max): Intelligence, Performance & Price Analysis - Artificial Analysis

6. Artificial Analysis Intelligence Index methodology - Artificial Analysis

7. China’s open-source bet - MIT Technology Review

8. Gavin Baker on Kimi K3’s token economics - X

9. The 2026 AI Index Report - Stanford HAI

Ramana makes the argument most teams learn the hard way: an LLM-based agent is not a bigger, smarter version of the traditional chatbot you already built, and treating the shift as an upgrade, instead of a paradigm change, is exactly how the engineering and governance model goes wrong. He has shipped production systems on every side of that line, from fraud-hunting clustering models to intent bots to agents that turn plain English into working data pipelines, and he recently formalized the transition in a research paper. This is the practitioner version. Below are his words. They are worth reading twice.

Table of Contents

1. The traditional ML era: when the question is “what pattern is hiding in this data?”

2. The bridge: intent-based chatbots, and why they had a ceiling

3. Three discontinuities, not one upgrade

4. RAG isn’t an enhancement - it’s a prerequisite

5. The part that actually requires new discipline

6. So which one do you actually need? Traditional ML vs LLM-Based Agent

7. The uncomfortable part nobody likes to admit

   Diary of an AI Architect is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

I’ve spent 18+ years building enterprise systems, and the last seven specifically on what feels like both sides of an industry line that a lot of people are only just starting to talk about: traditional machine learning versus LLM-based agents. I didn’t plan it that way. I moved from building unsupervised clustering models that hunt for fraud in claims data to building intent-based chatbots for enterprise support to architecting retrieval-augmented systems and generative AI agents that turn natural language requests into working data pipelines.

I got interested enough in the “why” behind that shift that I recently wrote a research paper on it - specifically on the transition from intent-based chatbots to LLM-powered agents. The conclusion I landed on, after digging through the architecture literature and lining it up against what I’d actually shipped in production: this isn’t an incremental upgrade. It’s a paradigm shift. Treating it like a bigger, smarter version of the old thing is exactly how teams get the engineering and governance model wrong.

Here’s what seven years of building on both sides of that line - and then formalizing it on paper - taught me.

The traditional ML era: when the question is “what pattern is hiding in this data?”

My earlier work was almost entirely about pattern discovery in structured, historical data. The clearest example: building an unsupervised fraud detection system for a healthcare payer, hunting for anomalous claim behavior across a huge volume of pharmacy transactions.

There was no labeled “this is fraud” dataset to train against; that’s exactly why it was a clustering problem, not a classification problem. I used K-Means and HDBSCAN together, layered with PCA for dimensionality reduction, to group pharmacy claims by behavioral similarity and then profile which clusters looked statistically abnormal. The output wasn’t a yes/no prediction; it was a risk score that told investigators where to look first.

A parallel project used the same mindset for a different question: predicting which website visitors during an open enrollment period were first-time shoppers versus returning members, based on navigation patterns and time-on-page. No LLM in sight, just feature engineering, a classification model trained on historical visit data, and a deployment pipeline that scored behavior in near-real time.

That’s still, in my experience, the right toolkit for a huge chunk of real business problems: anomaly detection, propensity scoring, churn prediction. If your problem is “find the needle in a structured haystack,” an LLM is usually the wrong needle-finder.

The bridge: intent-based chatbots, and why they had a ceiling

Before generative AI, I spent years building conversational systems the old way: intent classification, entity extraction, and deterministic dialogue state machines, the architecture that platforms like Dialogflow, LUIS, and Lex industrialized through the 2010s. I built bots for sales support, supply chain order tracking, and IT service desk requests, and within their design envelope, they worked. Deterministic flows are auditable. A state machine can be certified for compliance review in a way a language model can’t. For high-volume, bounded queries, that architecture reliably automated 60-70% of tier-1 support volume; there’s real published evidence behind that number.

But the ceiling was structural, not a matter of more engineering effort. I frame it in the paper as three inherent constraints:

• Intent coverage decay. Every time a user phrased something outside the enumerated intent taxonomy, the fix was weeks of retraining, not a quick patch.

• Contextual fragility. The dialogue layer tracked intent, entities, and phase, not the actual conversation. Multi-turn exchanges that needed real memory produced what felt to users like conversational amnesia.

• Fallback brittleness. Out-of-scope queries had exactly one destination: an apology and a menu. There was no mechanism for handling what the system wasn’t explicitly built to handle.

Those three limits trace directly back to the architecture’s core assumptions - a closed intent ontology, a deterministic state graph, and template-bound output. You couldn’t engineer your way out of them. You had to replace the assumptions.

Three discontinuities, not one upgrade

This is the part of my research I think is most useful for anyone evaluating this shift right now: it isn’t one change, it’s three separate structural breaks, each replacing a different layer of the old architecture.

I built the diagram below the same way I built it for the paper, because seeing the three discontinuities side by side is what made the “kind, not degree” argument click for me:

• First: ontology-bound classification gives way to open-vocabulary semantic inference. An LLM doesn’t sort an utterance into a predefined bucket, it infers meaning from context, history, and retrieved evidence, with no ceiling from an enumerated taxonomy. I felt this directly building a retrieval-augmented search system that let finance, HR, and sales teams ask plain-language questions and get answers pulled straight from internal documents, instead of filing a ticket or digging through folders.

• Second: handcrafted state machines give way to prompt-governed generation. Behavioral specification stops being a transition graph that takes weeks to modify and becomes natural-language instructions in a system prompt, a change measured in hours.

• Third: reactive single-turn response becomes autonomous multi-step execution. This is the one I think gets underrated. I recently led the architecture for a generative AI assistant that sits on top of a legacy data platform, a business user types a request in plain English, the system interprets intent, generates the right structured query, and returns usable data with no SQL knowledge required. That’s not a chatbot answering a question. That’s an agent planning, acting, and adjusting across steps.

None of these three are differences of degree. They’re differences of kind, and the research bears that out. ReAct-style agentic frameworks that interleave reasoning with tool use improved multi-hop task accuracy by 38 percentage points over non-agentic baselines on benchmark evaluation.

Retrieval grounding cut hallucination rates by roughly 21 percentage points relative to ungrounded generation. Instruction-tuned models were preferred by human evaluators over base model output 71% of the time. Those aren’t marginal gains, they’re evidence the underlying architecture changed, not just the size of the model.

RAG isn’t an enhancement - it’s a prerequisite

If there’s one thing I’d correct in how people talk about enterprise LLM deployment, it’s this: retrieval-augmented generation gets described as an optional add-on. In production, it’s not optional. Any system that needs to answer questions about policy documents, product specs, or internal knowledge bases - anything not in the model’s training data - will hallucinate with non-negligible frequency answering from parametric memory alone.

Every RAG system I’ve built follows the same shape: embed the query, retrieve the top-k relevant passages from a vector index, assemble that context, and condition generation on the retrieved evidence rather than the model’s own memory. The grounding is the architecture decision, not a prompting trick layered on afterward.

The part that actually requires new discipline

Traditional ML models are deterministic: same input, same output, every time. That’s what makes them auditable and regulator-friendly. LLM-based agents trade that determinism for flexibility, and that trade demands different engineering discipline than a classification model ever needed:

• Guardrails replace feature engineering. In traditional ML, most effort goes into deciding what signal to feed the model. In LLM systems, comparable effort goes into deciding what the model is not allowed to do - validating generated queries before they execute, constraining outputs to expected formats, catching a hallucinated field before it reaches a downstream system.

• Evaluation looks completely different. A classification model has a confusion matrix. An agent answering open-ended requests needs phased rollouts, starting simple before tackling complexity, and real humility about what “good enough” means before you widen scope.

• The audit trail gets harder to read. An intent-based system’s trail is a clean sequence: intent classified, state transitioned, template selected. An agentic system’s trail spans multiple inference calls, tool invocations, and reasoning steps - a harder record to inspect, and one the field hasn’t fully standardized governance for yet.

• Memory becomes a design choice, not a side effect. Old systems held session-scoped state and discarded it at close. LLM agents can carry short-term context, long-term vector-store memory, episodic recall, and procedural rules - turning stateless exchanges into something closer to an ongoing relationship. That’s powerful, and it’s also a governance decision every team underestimates going in.

So which one do you actually need?

• Reach for traditional ML when your data is structured and historical, the question is fundamentally statistical, you need guaranteed explainability for a regulator or auditor, and cost/latency at scale matter more than flexibility.

• Reach for an LLM-based agent when the bottleneck is the interface between a human and a system rather than the underlying data pattern, requests are open-ended or require synthesizing across unstructured sources, and you’re willing to invest in grounding and guardrails instead of expecting the model to be right out of the gate.

And increasingly, the best systems use both. A clustering model surfaces which cases need attention; an agent is the interface that lets a human ask follow-up questions about why. Traditional ML finds the signal. LLM agents make the signal usable by someone who isn’t a data scientist.

The uncomfortable part nobody likes to admit

The industry conversation treats LLM agents as the obvious upgrade path from “boring” traditional ML - and separately, treats today’s agents as a bigger version of yesterday’s chatbots. I don’t think either framing survives contact with the actual architecture. I’ve watched teams bolt an LLM onto a problem a simple classifier would have solved more cheaply and more reliably. And I’ve watched teams deploy an “agent” that’s really just a chatbot with a better vocabulary, without the guardrails, evaluation discipline, or audit logging that autonomous multi-step execution actually demands.

The skill that matters isn’t “know-how to build an agent.” It’s knowing which problem you’re looking at, and which of the three genuinely different architectural paradigms it calls for, before you pick a tool. That’s what seven years of building across all of them - on top of a much longer career in enterprise systems- made unavoidably clear to me.

Thank you for reading my newsletter!

Anu Karuparti
Creator, Diary of an AI Architect
How enterprises actually ship AI to production

Read by 3,000+ FDEs, AI Architects, and Engineering Leaders from Microsoft, Google, IBM, PwC and others.

Connect with me on LinkedIn.

Want to partner? Email me at anurag.karuparti@gmail.com.

P.S. Want more? 👋

1/ My visual guide to agentic AI → Gumroad

2/ Daily deep dives on agentic AI architecture → LinkedIn

3/ Real-time takes on breaking AI news → X

4/ Casual hot takes and community → Threads

5/ Visual frameworks and carousels → Instagram

6/ 60-second production lessons → TikTok

7/ The full newsletter, free → newsletter.karuparti.com

Coinbase cut its AI bill in half while token usage went up. Costs went down, consumption went up, and that combination is not luck. It is engineering.

On June 28, Brian Armstrong posted the exact playbook his team used. Five moves. Most teams are running none of them, which is why most teams watch their AI spend climb while their output barely moves.

I am an applied AI architect at Microsoft, and I have put production-grade agents inside Fortune 500 environments. So when I say Coinbase's playbook is smart, I mean it: it is exactly the kind of move the recent progress in open-weight models now makes possible.

The reflex answer most teams reach for is the wrong one. They renegotiate the contract, throttle usage, or wait for prices to drop. Coinbase did something different. They left the models alone and re-engineered the system around them. The bill fell by half and usage went up, because lower cost per task meant teams could afford to point AI at more tasks

This week’s edition breaks down those five moves, why each one works, and how they compound into a system instead of a list of tips.

Prefer to watch first? Here is the 80-second version:

The team bleeding tokens

Picture the head of engineering at a Series B fintech. Call her Anjali. Her org adopted an AI coding assistant and a handful of internal agents last year, and the productivity gains were real enough that adoption spread fast.

So did the bill. It tripled in a single quarter. The CFO started asking pointed questions, and the easy political move was on the table: cap everyone’s usage, or pull the most expensive tool entirely.

That move would have been a mistake, and Anjali knew it. The usage was creating value. The problem was not that her engineers were using AI too much. The problem was that every token was being spent at the worst possible price.

When her platform team finally looked under the hood, the pattern was almost comical. Routine code edits were being sent to the most expensive frontier model by default. The cache hit rate was close to zero, so the system re-paid for the same context on every single call. Agent sessions stayed open for hours, dragging a bloated context window into every request. And no engineer could see their own spend, so no one had any reason to care.

Anjali’s team did not have a model problem. They had a systems problem. And a systems problem has a systems fix.

The framework: spend on models versus engineer around them

There are two ways to respond to a high AI bill.

The reflex:   Bigger bill  ->  pay more / throttle / wait for prices to drop  ->  bigger bill
The fix:      Bigger bill  ->  treat the model as one component you control    ->  smaller bill, more usage

The reflex treats the model as a fixed cost you are at the mercy of. The fix treats the model as one part inside a system you design, where the default, routing, caching, context, and visibility are all knobs you own.

That reframe is the whole game. Once you stop seeing the model as the system and start seeing it as a component in the system, the cost stops being something that happens to you and becomes something you engineer.

The graph below, spend vs. token usage at Coinbase, proves it. So how did they pull it off?

The five moves are the knobs:

1. Change the default model: stop paying frontier prices for routine work

2. Build a routing layer: let the system pick the model, not the human

3. Fix your cache: the single biggest cost driver almost nobody looks at

4. Keep context lean: wasted tokens are a silent tax on every call

5. Make spend visible: align incentives instead of capping behavior

Let me go through each one.

Move 1: Change the default model

The first move is the one that sounds too simple to matter and matters more than anything else.

Coinbase switched its engineers to open-weight models like GLM 5.2 and Kimi 2.7 by default. The frontier model did not go away. It just stopped being the lazy default for every keystroke.

Here is why this works. The gap between open-weight models and the closed frontier has narrowed dramatically over the last two years. For a huge share of real engineering work, code completion, refactors, test generation, routine reasoning, the open-weight model is already good enough. You were paying a premium for capability you were not using.

The default is a pricing decision disguised as a quality decision. Nobody chose to send every routine prompt to the most expensive model. It was just the value that shipped in the box, and inertia did the rest. Flip the default, and the small fraction of work that genuinely needs the frontier can still reach for it on demand.

For Anjali’s team, this one change alone could move the needle more than any contract renegotiation could have, because it attacked the volume where the volume actually was.

Move 2: Build a routing layer

Changing the default gets you most of the way. A routing layer gets you the rest, and it does it without asking a single human to think about pricing.

The idea is to route every prompt by task difficulty and price, automatically. Hard, ambiguous, plan-this-whole-feature work goes to the frontier. Routine, well-scoped execution goes to the cheaper open-weight model. The system makes the call, in real time, on every request.

This is a known pattern with real research behind it. Model routing, formalized in work like RouteLLM, learns to predict whether a cheaper model will be good enough for a given prompt and only escalates when it will not. The RouteLLM authors reported cutting cost by more than two times in some cases without dropping response quality. Think of it as a load balancer, except instead of balancing for traffic, it balances for the cheapest model that can still do the job.

The architectural caution is the classifier. A router is only as good as its difficulty estimate, so you need a fallback path: if the cheap model fails or the confidence is low, escalate automatically and log it. The goal is not to be cheap on every prompt. The goal is to be exactly as expensive as each prompt requires, and no more.

The human picking the model is the bottleneck and the leak. Take the human out of the pricing decision and the system optimizes itself thousands of times a day.

Move 3: Fix your cache

This is the move that almost nobody looks at, and at Coinbase it was the single biggest cost driver.

Their cache hit rate went from 5% to 60%. Sit with that number, because it is enormous. On Anthropic’s published pricing, a cached input token is billed at one-tenth the price of a fresh one. OpenAI and Google pass through similar discounts. Going from 5% to 60% cache hits means the majority of your input tokens suddenly cost a fraction of what they did, on the exact same workload.

Prompt caching is a real, documented feature across Anthropic, OpenAI, and Google. The mechanism rewards a specific structure: put your stable content first, the system prompt, the tool definitions, the long reference docs, and put the volatile, request-specific content last. When the stable prefix does not change between calls, the provider serves it from cache instead of reprocessing it.

Most teams never structure prompts this way, so their cache hit rate sits near that 5% floor by accident. They are paying full freight to re-send the same system prompt and the same documentation on every call, thousands of times a day, and the line item never gets questioned because nobody is watching that number.

This is the highest-ROI move on the entire list. It requires no model change, no new vendor, and no behavior change from your engineers. It is pure architecture, and it is sitting unclaimed in almost every AI system I review.

Move 4: Keep context lean

If the cache is the token you should not re-pay for, context bloat is the token you should never have sent in the first place.

The discipline here is unglamorous and powerful. Start fresh sessions when you switch tasks instead of dragging an hour of irrelevant history into a new problem. Narrow the file scope so the agent sees the three files it needs, not the whole repository. Disconnect the tools you are not using, because every connected tool spends tokens describing itself in the context window.

Every token in the context window is paid for on every single call. A bloated window is not a one-time cost, it is a tax that compounds across every request in that session. The session that carries forty thousand tokens of stale context pays for those forty thousand tokens again and again until someone closes it.

This connects directly to my earlier edition on context engineering, where I argued that managing the context window is the core craft of building reliable agents. Here is the financial corollary: the same lean context that makes an agent more accurate also makes it dramatically cheaper. Precision and economy are the same discipline. You do not have to trade one for the other.

Wasted tokens are a silent tax precisely because they are invisible. Nobody sees the bloated window. They just see the bill, and they blame the model.

Move 5: Make spend visible

The last move is the one that holds the other four together, and it is not a technical move at all. It is an incentive move.

Coinbase did not impose hard caps. Instead, every engineer can see their own usage, and the message attached to it is explicit: more spend is fine, as long as it means more impact. Spend is not a sin to be punished. It is a resource to be deployed well, and visibility is what makes deploying it well possible.

Hard caps are the intuitive answer and the wrong one. A cap punishes the engineer doing high-leverage work that genuinely needs more tokens, while doing nothing to teach the engineer who is wasting them. Caps optimize for staying under a number, not for getting value per token.

Visibility flips the incentive. The same way you would not cap an engineer’s cloud spend but you would absolutely make it visible and tie it to outcomes, AI spend should be a number every engineer can see and feel ownership of. This is FinOps applied to tokens. When people can see what they spend and understand that spend is judged against impact, they self-correct without anyone writing a policy.

For Anjali’s team, this was the cultural unlock. The moment her engineers could see their own token spend on a dashboard, the wasteful patterns started disappearing on their own, because for the first time the cost was attached to a name instead of hidden in an aggregate invoice.

How the five moves compound

The reason this works as a system and not a checklist is that the moves hand off to each other. The default change shrinks the volume at the top. The router catches what the default misses. The cache discounts what survives. Lean context shrinks what gets cached and sent. Visibility keeps the whole thing honest.

Notice that no single move is the whole answer. A cheaper default with a five percent cache rate still leaks. A perfect cache behind a bloated context still overpays. Visibility with no routing just shows people a big number they cannot do anything about. The savings compound only when the moves run together, each one cutting cost at a different layer of the same request.

That is why teams who try just one of these are disappointed. They switch the default model, see a modest dip, and conclude the hype was overblown. They were missing the other four layers.

How I think about applying this

When I review an AI system’s cost now, I do not start by asking which model they are on or what they are paying per token. I ask one question first: at how many points in the request can you choose to spend less without spending worse?

If the answer is “the model picks itself, the cache is whatever it is, the context is whatever accumulated, and nobody sees their spend,” then I know exactly why the bill is what it is. There are five control points sitting unused, and no model negotiation will recover what those five points are leaking.

For Anjali’s team, the fix was not a cheaper contract. It was the control points they had never touched: a default flip, a router, a restructured prompt for cache reuse, a leaner context discipline, and a spend dashboard. The bill came down by more than half within two months, and adoption actually went up, because once each task got cheaper, AI became affordable for work that was previously too expensive to automate. This is the same lesson I kept hitting in my edition on LLM cost optimization: the cheapest token is the one you engineered your way out of sending.

The bottom line

Coinbase cut its AI bill in half while usage went up, and they did it without a single model breakthrough. They changed the default model, built a routing layer, fixed their cache, kept context lean, and made spend visible. Five moves, all architecture and discipline, none of them a new model.

Before you negotiate your next AI contract or throttle your team, audit those five control points. The savings are almost never in finding a cheaper model. They are in engineering the system around the model you already have.

Thank you for reading my newsletter!

Anu Karuparti
Creator, Diary of an AI Architect
How enterprises actually ship AI to production

Read by 3,000+ FDEs, AI Architects, and Engineering Leaders from Microsoft, Google, IBM, PwC and others.

Connect with me on LinkedIn.

Want to partner? Email me at anurag.karuparti@gmail.com.

Disclaimer: The Coinbase figures in this article are drawn from a public post by Brian Armstrong and are attributed as such. The story of Anjali and her team is hypothetical, inspired by patterns observed across similar real-world experiences. It is used to convey key concepts more effectively and does not represent any specific individual or organization.

P.S. Want more? 👋

1/ My visual guide to agentic AI → Gumroad

2/ Daily deep dives on agentic AI architecture → LinkedIn

3/ Real-time takes on breaking AI news → X

4/ Casual hot takes and community → Threads

5/ Visual frameworks and carousels → Instagram

6/ 60-second production lessons → TikTok

7/ The full newsletter, free → newsletter.karuparti.com

There is a term spreading through the agent-building community right now: loop engineering. The internet is buzzing about it, and like most things that go viral fast. So, I spent the last two weeks pulling it apart to figure out what is real, what is hype, and what actually matters for teams shipping agents into production.

Here is what I found. Loop engineering is not a new model. It is not a framework you install. It is a way of thinking about the architecture of long-running, self-improving systems.

The term was popularized by popularized by top engineering leaders like Boris Cherny (Head of Claude Code), Addy Osmani (Google’s Engineering Leader) and Peter Steinbeger (Creator of OpenClaw)

This week’s edition is about that difference, and the six engineering layers that make it possible.

The enterprise pain points

Imagine you are the platform lead at a mid-sized insurance company. Your team has built a claims-triage agent that reads incoming claims, classifies them, pulls the relevant policy, and drafts a recommendation for a human adjuster.

It works. On launch day it handles 60% of routine claims without escalation, and leadership is thrilled.

Then it stops getting better. Six weeks later it is still at 60%. The same edge cases fail the same way. The agent misreads the same ambiguous policy clauses on Monday that it misread the previous Monday. Adjusters start double-checking everything, which erases most of the time savings.

The diagnosis is simple. They built an open-loop system. A request comes in, the agent processes it, an answer comes out, and that is the end of the line. Nothing flows back. The agent has no way to notice it was wrong, no way to learn from the adjuster’s correction, no memory of last week’s mistake. It is a very expensive function call.

Your team built an agent. But what is needed is a loop.

The framework: open loop versus closed loop

Most AI systems today look like this:

Input → Process → Output → (stop)

A self-improving system looks like this:

Input → Process → Output → Feedback → Improve → back to Input

That second arrow, the one that bends back to the beginning, is the entire game. It is the difference between a system that is as smart as the day you shipped it and a system that compounds.

But that loop is not magic, and this is where most of the loop engineering hype falls apart. People talk about self-improving agents like you sprinkle a feedback step on top and intelligence emerges. It does not work that way.

The loop is held together by six engineering layers. Each one is a real architectural decision with real trade-offs. Get all six working together and the loop closes.

The six layers are:

1. Automations: the trigger layer

2. Worktrees: the parallel execution layer

3. Skills: the procedural memory layer

4. Connectors: the real-world links layer

5. Sub-agents: the validation layer

6. Memory: the state layer

Let’s go through each one.

Layer 1: Automations, the trigger layer

The first thing a self-improving system needs is a way to start without you.

Automations initiate workflows based on time, events, or system conditions. An email arrives. A database row updates. A calendar event fires. A claim lands in the queue. The trigger launches the full pipeline so execution is structured, not ad hoc.

This is the layer that moves an agent from “I asked it” to “it noticed and started on its own.” That shift sounds small. It is not. An agent you have to invoke is a tool. An agent that responds to the world is a system.

The architectural caution: triggers are where runaway loops are born. An event-driven agent that triggers another event can cascade. Before you wire automations, define the kill switch and the rate limit. A loop that starts itself must also be a loop you can stop.

Layer 2: Worktrees, the parallel execution layer

Read more

Vishal makes the uncomfortable argument most of the industry is still avoiding: your data stack isn't underperforming for AI because it's badly built, but because it was designed for a human reader and not an autonomous one. It's one of the sharpest takes I've read this year, and it comes from someone who has built in exactly the places where complexity, trust, and scale collide. Read this one twice.

Table of Contents

1. The Data Stack Was Built for a Human Reader

2. Why AI Exposes the Gaps in Enterprise Data

3. Governance Without Context Is Not Governance

4. The Enterprise Memory Gap

5. From Data-Driven to Context-Driven

   ---

1. The Data Stack Was Built for a Human Reader

For the last decade, the work of data teams was designed around a human reader. Pipelines moved data. Warehouses stored it. Transformation layers shaped it. Dashboards displayed it. The architecture assumed that a person would sit in front of a screen, interpret what the numbers meant, and decide what to do next.

That model worked because the human supplied what the stack did not: business context, judgment, memory, and an understanding of exceptions. An analyst knew why a metric moved, which customer was strategic, which policy could bend, and which action would be technically possible but organizationally wrong. Much of that knowledge never lived in a warehouse, BI tool, or transformation layer. It lived in people, meetings, Slack threads, email, documents, and the unwritten operating memory of the business.

AI changes that assumption. The reader is no longer only human. AI agents do not just consume reports; they reason over data, retrieve context, make recommendations, trigger workflows, and increasingly take action. That means the architecture can no longer depend on a human to fill the gaps between systems. The gaps have become the problem.

2. Why AI Exposes the Gaps in Enterprise Data

Most enterprises now operate a collection of specialized tools across ingestion, storage, transformation, metrics, cataloging, governance, observability, and BI. Each tool may be excellent at its function, but together they create seams. Humans have learned to work across those seams. They know which field is trustworthy, which dashboard is canonical, and which definition has changed. Agents do not inherit that tacit map. When meaning is distributed across tools, tickets, conversations, and tribal knowledge, the agent sees data without the operational logic that makes the data safe to use.

3. Governance Without Context Is Not Governance

Furthermore, traditional stacks often treat governance as a layer: permissions in one place, lineage in another, business definitions somewhere else, and policy documents outside the flow of work. That is not enough for AI. Governance is not only a rule; it is the act of deciding how a rule applies in context. A pricing policy may hold until a strategic account, a regulatory constraint, or an executive decision makes the standard rule inappropriate. Those exceptions are not edge cases. They are where much of the organization’s real judgment and authority reside. If that judgment is not captured where the work happens, an agent can follow the rule and still make the wrong decision.

4. The Enterprise Memory Gap

The modern data stack is good at storing artifacts: tables, dashboards, logs, and metrics. It is much weaker at preserving the reasoning that produced them. Why was the definition changed? Who approved an exception? What happened after a prior decision? Which tradeoff was accepted and why?

Today, that memory is scattered across unstructured sources and human recollection. For agents, retrieving fragments from those sources is possible, but it is brittle. Without a structured memory layer, an AI system may be technically connected to information yet still lack the continuity required to reason responsibly.

5. From Data-Driven to Context-Driven

Dashboards were built for interpretation, not autonomous action. A human can look at a chart and ask follow-up questions. An agent needs live business context on the same surface where it reasons and acts: definitions, permissions, lineage, exceptions, policies, semantic meaning, and memory. A context layer bolted onto the side of an existing stack may help with search, but it does not solve the architectural issue. Context has to be produced, governed, and updated as part of the operating system of the business.

This is why the traditional data stack did not fail because it was badly designed. It failed for AI because it was designed for a different reader. It is optimized for human interpretation. AI requires an architecture for governed reasoning and action.

The next generation of data architecture will need to bring the data plane, control plane, and context plane closer together. Ingestion, semantic modeling, governance, memory, and agent orchestration cannot remain isolated functions connected only through pipelines and human coordination.

They need to operate on a shared substrate where context is created as work happens, where rules and exceptions live together, and where agents can reason with the same constraints that the organization expects people to follow.

This does not mean every enterprise should discard its existing investments. The practical path is more likely to be evolutionary: identify the workflows where agents are expected to act, map the context and governance those workflows require, and build an AI-ready layer that can unify structured data, unstructured knowledge, policies, memory, and orchestration.

The goal is not another dashboard. The goal is an environment where humans and agents can operate from the same governed understanding of the business.

An AI-native data systems need to treat context as a first-class architectural concern, not as an afterthought. The broader shift is bigger than any one vendor. As AI agents become participants in business operations, organizations will need architectures that capture not only data, but also meaning, memory, and judgment. The companies that solve that problem will move beyond being data-driven. They will become context-driven.

About Vishal Singh: Vishal Singh is the Founder and CEO of DataGOL, an Agentic Data and AI Agents platform built to make enterprise data AI-ready for the next generation of autonomous systems. A repeat founder and enterprise software operator, Vishal previously co-founded Link3D, where he helped build additive workflow and digital manufacturing software for complex industrial environments before the company was acquired by Materialise NV, listed on Nasdaq under MTLS. He also co-founded Termgrid, a SaaS platform modernizing private capital markets workflows where he leveraged network effects and data moat

Across SaaS, network effects, data platforms, AI, and regulated industries, Vishal has repeatedly built software where complexity, trust, and scale collide. DataGOL emerged from his conviction that AI agents cannot transform the enterprise unless they are grounded in governed, reusable, AI-ready data. His mission is to collapse fragmented data stacks into intelligent infrastructure that lets enterprises move faster, spend less, and deploy AI with confidence.

Forward-Deployed Engineer (FDE) is becoming the absolute hottest job in AI right now, and there is a massive reason for it.

I’m an applied AI architect at Microsoft, and I’ve shipped AI agents into Fortune 500 production. I have seen personally how challenging it is to integrate AI into established organizational workflows.

The people actually winning in AI right now are FDEs, because they focus on this exact last-mile integration problem. Companies are drowning in daily AI updates, trying to force them into their existing workflows with years of messy data.

So what is the role of an FDE, and why are they suddenly commanding half-a-million-dollar salaries?

The Origin: Who Coined the Term?

The FDE isn’t new. Palantir coined “forward-deployed engineer” more than fifteen years ago. By 2009 the company already had around 120 forward-deployed engineers sitting inside customer sites, wiring messy government and bank data into something usable.

The role was modeled after high-end French restaurant kitchens, where the front-of-house staff is deeply integrated with the chefs and empowered to tell the customer when they are ordering wrong.

The idea was radical for its time: don’t ship software over the wall and hope. Send your best engineers to live next to the customer, watch them struggle, and bend the product to reality.

Palantir’s CTO Shyam Sankar has spent years evangelizing this model, and the “Palantir mafia” carried it into the next generation of startups.

Now the rest of the industry has caught up. In 2025, OpenAI publicly posted Forward-Deployed Engineer roles to help enterprises actually deploy GPT-class systems. Anthropic, Sierra, and a wave of AI startups followed.

The reason is simple: in the generative AI era, the bottleneck moved. It’s no longer “can we build a capable model?” It’s “can we make this model create real value inside one specific, messy organization?”

Why is it such a challenging job?

An FDE is part engineer, part consultant. They sit directly next to the customer and make the AI actually work in the real world.

Technical mastery is just the baseline. The real superpower is problem-solving under massive ambiguity. You have to understand both the code and the business, bridging the gap between internal engineering teams and the end user, and adapting software to real-world constraints like legacy security protocols, fragmented data, and executive politics.

Because of that, three things make FDEs worth a fortune:

1. They live in the problem. They watch real users hit real failures, so they build for what is actually breaking, not what looks good in a demo.

2. They own the whole stack. Model, harness, data pipelines, deployment. They don’t just hand off work after a prototype. They take a vague business problem and make AI production-ready.

3. They close the loop fast. They ship, they watch it break in production, they monitor, optimize, and fix it. Speed and time-to-value are their entire advantage.

What Are FDEs earning today?

Because this combination of deep AI technical skills and client-facing empathy is incredibly scarce, compensation for FDEs has skyrocketed.

Job postings for this role surged 800% over the last year as companies realized that while demos close deals, deployments keep them.

Today, the FDE is arguably the highest-paid generalist role in tech. Total compensation (TC) varies wildly based on where you are deploying the AI:

Note: The massive numbers at the Frontier Labs are heavily driven by equity, which now makes up 60–70% of an FDE’s compensation at those companies.

Why so high? Because the FDE is the person standing between an expensive AI investment and zero ROI. When a single deployment can unlock millions in value, or save a multi-million-dollar contract, the engineer who makes it actually work is not a cost center. They’re the deal.

Compensation follows leverage. And right now, the FDE has the most leverage in the building.

To sum it all up, the hottest job in AI right now is the person who can make it work for a customer in production and drive real value. That is the job that is currently worth a fortune.

P.S. Want more? 👋

1/ My visual guide to agentic AI → Gumroad

2/ Deep dives on agentic AI architecture → LinkedIn

3/ Real-time takes on breaking AI news → X

4/ Casual hot takes and community → Threads

5/ Visual frameworks and carousels → Instagram

6/ 60-second production lessons → TikTok

7/ The full newsletter, free → newsletter.karuparti.com

Share Diary of an AI Architect

References

• Palantir Technologies — origin of the “forward-deployed engineer” term and the ~120 FDEs by 2009: Wikipedia: Palantir Technologies (Palantir Metropolis section, citing Bloomberg).

• Shyam Sankar (Palantir CTO) on the forward-deployed model: Palantir Newsroom / Letters.

• OpenAI Forward-Deployed Engineer role: OpenAI Careers (FDE roles posted in 2025).

• Compensation ranges are directional, drawn from public reporting and crowd-sourced data: levels.fyi (Palantir and OpenAI compensation pages).

Two open-source projects just redrew the map for what multi-agent systems will look like running organizations in the next two years.

They both message you on WhatsApp,Telegram and other channels. They both read your files, run your shell, and complete real tasks while you sleep. And they have almost nothing in common under the hood.

That gap is not a feature gap. It is an architecture decision, and it is the same decision every enterprise team will face in 2027 when they stop buying chatbots and start running agents.

I put together a short video comparing the two this week. But before you watch it, I want to give you the mental model that makes the comparison actually land.

OpenClaw and Hermes represent two genuinely different bets on how intelligent work gets organized: one optimizes for breadth across many domains, the other for depth within a single reasoning loop.

The architecture below represents my perspective on how the multi-agent enterprise will evolve with these two frameworks. It brings together the two architectural philosophies discussed in this article. Before exploring the architecture itself, let's examine each philosophy.

The framework: orchestrated breadth vs compounding depth

OpenClaw for Orchestrated Breadth

• OpenClaw is a gateway-first system. Its core bet is that the hard problem is routing and control - who can reach your agent, from which channels, under what permissions, with what downstream handoffs.

• It is hub-centric by design: a central orchestrator managing multiple agents across diverse messaging platforms, with a large pre-built skill ecosystem and explicit multi-agent coordination.

• The framework is opinionated about surfaces and flexible about everything else. If your problem is communication complexity across a team or organization, OpenClaw is built for that surface area.

Hermes for Compounding depth

• Hermes takes the opposite bet. The hard problem, in its view, is memory and self-improvement.

• Hermes is a runtime-centric agent that writes and refines its own skills, retains context across sessions, and optimizes for repeated workflows over time. It is not trying to be everywhere.

• It is trying to become indispensable, increasingly useful for the specific work you do, across weeks and months of interaction.

• Neither is more advanced. They are different wagers on where the value actually lives.

OpenClaw: the leverage is in the routing layer.

Hermes: the leverage is in the learning loop.

OpenClaw: the orchestration philosophy

OpenClaw started life in November 2025 as Clawdbot, built by Peter Steinberger. After a trademark scuffle with Anthropic over the “Clawd” name and a fast double rename, it landed as OpenClaw and became one of the fastest projects in GitHub history to cross 100,000 stars. The design center is clear once you look at the pieces.

A heartbeat, not a prompt

A normal chatbot waits for you to type. OpenClaw runs a heartbeat: every 30 minutes it wakes up, checks for pending tasks, and acts without being asked. That one decision turns a question-answering tool into a proactive operator.

It is the difference between “AI that answers” and “AI that monitors, then acts, then messages you.”

Markdown memory you can read

Memory is not a vector database you have to trust. It is plain Markdown files on your disk: a curated MEMORY.md for long-term knowledge plus dated daily logs the agent reads at the start of each session.

Skills as a marketplace

Capabilities come from modular plugins pulled from the ClawHub marketplace. Over 5,700 community skills cover everything from email triage and calendar sync to browser automation and CI/CD monitoring.

The mental model is an app store for your agent: want a new behavior, install a skill.

Breadth as the product

OpenClaw speaks 50-plus messaging platforms, is model-agnostic across Claude, GPT, and local models, and treats its gateway as a thin delivery layer.

The philosophy in one line: value comes from how many things you can wire together.

Access is key here.

The risk in one line: the same breadth and system access got it called a “security nightmare” by Cisco, with malicious skills and exposed instances showing up in the wild.

Orchestrated breadth is powerful and it is also a wide attack surface.

Hermes: the learning philosophy

Hermes Agent, from Nous Research, makes the opposite bet. Where OpenClaw orchestrates many agents and many skills, Hermes is a single agent that gets more capable the longer it runs, not through config changes but through actual use.

The architecture is built around one idea: close the loop between what happened and what worked.

The learning loop

This is the heart of it. As a session runs, Hermes gets periodic nudges, internal prompts that ask the agent to look back and decide what is worth keeping.

When it finishes a task that took real work (five or more tool calls, a recovery from an error, a user correction, a non-obvious workflow), it writes a new skill file to disk. Not a log of what happened, a reusable procedure for next time.

And it keeps editing those skills, preferring small patches over rewrites so it never breaks what already works.

Four layers of memory, separated on purpose

Most agent memory rots because everything gets dumped into one store. Hermes splits it into four:

1. Prompt memory (MEMORY.md + USER.md): always-on, injected into every system prompt, deliberately capped near 3,500 characters to force curation over hoarding.

2. Session search (SQLite + FTS5): every session is archived and full-text indexed, retrieved on demand and summarized before it re-enters context.

3. Skills (procedural memory): how-to knowledge, loaded on demand so a large skill library costs far less context than loading everything up front.

4. Honcho (passive user modeling): It is an AI-native memory backend that adds dialectic reasoning to Hermes. Instead of key-value storage, it keeps a running model of your preferences, style, goals, and patterns by reasoning about conversations after they happen.

The distinction that matters: episodic memory (”what happened”) is kept separate from procedural memory (”how to do it”). That separation is why the system stays reliable as it grows.

The loop is the gateway

In OpenClaw the gateway just delivers messages. In Hermes the gateway is part of the same loop: an incoming message can trigger skill creation, a scheduled job writes its result back through the same path, and a conversation started on Telegram continues in your terminal because the session is tied to an ID, not a platform. Token caching, automatic provider fallback, and compression-as-consolidation round it out. The philosophy in one line: value comes from how much better the agent gets at your work over time.

When to use one over the other: enterprise use cases

The fastest way to make this concrete is to stop thinking about features and start thinking about the job.

Hermes fits enterprise workloads that demand deep, adaptive intelligence over a focused set of specialized internal data.

OpenClaw fits workloads that demand broad, multi-channel connectivity across many systems, with governance enforced as a guardrail you wrap around it rather than a property you get for free.

Here is where each philosophy actually wins inside a large organization.

Enterprise use cases for Hermes (narrow and deep):

• Legacy code modernization. Hermes analyzes an outdated, poorly documented repository, maps dependencies, and learns to generate compliant migrations to modern frameworks, accumulating context on the internal code quirks with every pass so the tenth module is migrated better than the first.

• Complex R&D knowledge synthesis. Clinical researchers or patent attorneys point Hermes at a growing library of highly technical documents. The moat is not bulk indexing, it is the custom analytical skills the agent writes to surface non-obvious cross-references, skills that get sharper with every paper it works through.

• Hyper-personalized customer escalations. For tier-3 support, Hermes learns individual enterprise-client SLA preferences, past resolution history, and technical environments, then drafts uniquely tailored fixes for the specific, recurring glitches each account hits.

Enterprise use cases for OpenClaw (wide and shallow):

• Omnichannel incident response. When a system outage hits, OpenClaw orchestrates the whole response at once: pull logs from AWS, open a Jira ticket, alert engineers in Slack, and broadcast status to clients over WhatsApp and email, concurrently.

• Cross-department employee onboarding. OpenClaw handles the diverse one-off tasks of hiring: provision a Google Workspace account, trigger a Slack invite, request equipment through an internal procurement API, and track completion inside Workday.

The honest enterprise answer is rarely all-or-nothing. Use breadth for the long tail of integrations and cross-system workflows, and depth for the handful of specialized, recurring workloads where the agent compounds context.

Match the architecture to the shape of the work, not the other way around.

The convergence of two frameworks

Here is the part most people miss when they treat this as a versus. These two philosophies are not drifting apart, they are converging.

The clearest signal is that Hermes ships a hermes claw migrate command to import an OpenClaw setup wholesale. Nous is not ignoring breadth, it is absorbing it: take everything you wired together in the orchestration world, then wrap it in a learning loop so it compounds. Depth is positioning itself as the next layer on top of breadth, not a replacement for it.

The reverse pressure is just as real. OpenClaw’s marketplace is full of skills, but the obvious next step for any orchestration platform is letting the agent author and refine its own skills instead of only installing them. The moment breadth platforms add a learning loop, the line between the two blurs.

You can already see where this lands. The agent stack for 2027 is not “breadth OR depth,” it is a single system that orchestrates many skills and channels (the OpenClaw bet) while continuously learning from its own work (the Hermes bet).

Connect-everything becomes the substrate; get-better-over-time becomes the engine that runs on top of it.

So if you are choosing today, choose for where you are now: breadth if you are exploring and need reach, depth if your workflows have stabilized and you want compounding returns.

But architect for convergence. Keep your memory, skills, orchestration, and identity as separate, swappable concerns, and the philosophy you pick today becomes an implementation detail you can upgrade tomorrow, not a wall you build around yourself.

The bottom line

OpenClaw and Hermes are not competing on features, they are competing on philosophy: orchestrated breadth that does more today, versus compounding depth that gets better over time.

The teams that win in 2027 will not be the ones with the most skills installed, they will be the ones who matched the architecture to the shape of the work.

References

• Trilogy

• OpenClaw documentation, “Introduction to OpenClaw” (heartbeat, Markdown memory, history, security): clawdocs.org

• OpenClaw ecosystem hub (AgentSkills, ClawHub, origin as Clawdbot, GitHub traction): openclaw-ai.dev

• Mr. Ånand, “Inside Hermes Agent: How a Self-Improving AI Agent Actually Works” (learning loop, four-layer memory, gateway, agent loop): mranand.substack.com

• Hermes Agent community site (capabilities, platforms, sandbox backends, OpenClaw migration): hermesagents.net

• Nous Research, Hermes Agent: hermes-agent.nousresearch.com

• agentskills.io open skill specification: agentskills.io

His team shipped a great feature.

Users loved it.

Then the invoice landed.

Same model. Same product. The cost had quietly tripled in six weeks.

Nobody did anything wrong. Nobody did anything on purpose either. That was the problem.

Here is what I told him. And here is what I tell every team that hits this wall.

You do not have a model problem. You have a habits problem.

LLM cost is not one big leak. It is twelve small ones.

Let me walk you through all twelve.

First, why this matters now

Token prices keep falling.

Yet enterprise AI bills keep climbing.

How? Because volume grows faster than price drops.

Google now processes over a quadrillion tokens a month. Deloitte’s 2026 CFO guidance calls AI the fastest-growing line item in tech budgets.

Translation: your bill is going up, even when the price per token goes down.

The teams that survive are not the ones with the cheapest model. They are the ones with the cleanest habits.

So let us fix the habits.

1. Choose the right model

Most teams reach for the biggest model by default.

Big mistake.

The biggest model is rarely the right one. It is just the safest-feeling one.

Pick the smallest, cheapest model that still clears your quality bar.

A simple rule I use:

• - Routing, classifying, extracting? Small model. (GPT-5.4-mini, Claude Haiku 4.5)

• - Drafting, summarizing? Mid model. (GPT-5.4, Claude Sonnet 4.6)

• - Hard reasoning, final answers? Large model. (GPT-5.5, Claude Opus 4.8)

Most workloads are small-model work wearing a large-model price tag.

2. Reduce input tokens

You pay for every word you send. Not just the words you get back.

So stop sending junk.

Cut the repeated examples. Cut the stale instructions. Cut the giant block of context you pasted “just in case.”

A 2,000-token system prompt on every call is a tax. You pay it forever.

Write tight. Send less. The model does not need your life story.

3. Limit output tokens

Long answers feel impressive. They are also expensive.

Set a hard cap on output length.

Ask for bullet points. Ask for structured responses. Ask for “answer in 3 sentences.”

You will be shocked how often a 50-token answer beats a 500-token one. For the user and the bill.

4. Use caching

Your users ask the same things over and over.

So stop re-generating the same answers.

Cache repeated prompts and responses. Use embedding similarity to catch the near-duplicates too.

Many providers now offer prompt caching for the static part of your prompt. That alone can cut input cost on long system prompts by half or more.

Free money. Take it.

5. RAG done right

Bad RAG is a silent budget killer.

People dump huge chunks into context and hope the model sorts it out.

The model does sort it out. And it charges you for every token while it does.

Retrieve only what the current question needs. Keep chunks small. Trim before you send.

Good RAG is not “more context.” It is less, but exactly right.

6. Batch requests

One-by-one API calls add up. Each one carries overhead.

If you are processing many items, batch them.

Group prompts into a single efficient call where the task allows it. Many providers also offer a batch tier at a steep discount for non-urgent jobs.

Overnight work does not need real-time pricing.

7. Function calling and structured output

Free-text answers are hard for code to use. So you ask the model to “explain,” then you parse the mess.

That explanation costs tokens. The re-asks cost more.

Use function calling and structured output instead.

Ask for clean JSON. Define a schema. Skip the verbose natural-language wrapper.

Fewer tokens. Fewer retries. Less glue code.

8. Prompt reuse and templates

Every team rewrites the same prompt five slightly different ways.

Each version drifts. Each one gets a little longer. Nobody owns it.

Build a library of proven, parameterized templates instead.

Write the prompt once. Tune it once. Reuse it everywhere.

This is not just a cost win. It is a quality win and a sanity win.

9. Tiered, two-step approach

Not every request deserves your most expensive model.

So filter first.

Use a cheap model for triage: sort, draft, simplify. Send only the hard cases up to the expensive model.

Think of it like a hospital. A nurse sees everyone. The surgeon sees the few who actually need surgery.

Most of your traffic never needs the surgeon.

10. Monitor, measure, optimize

You cannot fix what you cannot see.

Most teams have no idea which feature burns the most tokens. So they guess. And they guess wrong.

Track token spend by feature, by user, by request type.

Find the top three offenders. Fix those first.

A dashboard you check weekly beats a heroic cleanup after the bill explodes.

11. Use the right pricing and providers

Provider pricing is not flat. It changes by model, by region, by commitment.

Most teams pick a provider once and never look again.

Compare before you commit to production volume.

Look at reserved capacity. Look at committed-use discounts. Look at whether a different provider serves your specific workload cheaper.

This is a procurement decision, not just an engineering one. Treat it like one.

12. Optimize embeddings usage

Embeddings feel cheap. At scale, they are not.

Teams re-embed the same content over and over. They embed near-duplicates. They batch nothing.

Fix three things:

• Batch your embedding requests.

• Skip duplicate and near-identical content.

• Cache embeddings you already have.

Small per-call cost. Massive total cost. Worth the cleanup.

The pattern under all twelve

Read the list again. Notice something?

None of these are clever.

There is no secret model. No magic flag. No vendor trick.

Every single one is the same move: send fewer tokens, and only the ones that matter.

That is the whole game.

LLM cost optimization is not a one-time project. It is a discipline you bake into how your team builds.

How this maps to Azure

If you are building on Azure, you do not have to wire all twelve by hand. Most of them have a home in the platform.

Here is how I map them:

• Choose the right model + tiered routing: Use Microsoft Foundry model catalog to deploy a mix of small and large models, and route between them. Cheap model first, expensive model only when needed.

• Caching + reduce input tokens: Put Azure API Management in front of your models as an AI gateway. It supports semantic caching, token limits, and token metrics out of the box.

• RAG done right: Use Azure AI Search with tight chunking and hybrid retrieval, so you send the model less context, not more.

• Monitor, measure, optimize: Use Azure Monitor and the token metrics from API Management to see spend by feature and catch spikes before finance does.

• Embeddings: Cache embeddings in Azure Cosmos DB or your vector store, and batch generation jobs instead of embedding one item at a time.

Here is the shape of it:

The gateway is the key piece. It is where caching, token limits, and measurement all live in one spot. Most of your savings flow through that single layer.

Microsoft Learn references:

• Azure API Management as an AI gateway

• Azure AI Foundry model catalog

• Azure AI Search and retrieval-augmented generation

The bottom line

Your LLM bill is not one big leak. It is twelve small ones, and most teams never plug any of them until the CFO asks why the number tripled.

Start with three this week: pick the right model, cap your output, and turn on caching. Those three alone usually cut the bill by a third.

Then build the habit. Cheap tokens are not an accident. They are a design choice you make on every call.

♻️ If you found this valuable, share it with your network. ✉️ Subscribe for more: newsletter.karuparti.com

Disclaimer: The stories and scenarios in this article are hypothetical, inspired by patterns observed across similar real-world experiences. They are used to convey key concepts more effectively and do not represent any specific individual or organization.

References

• Google processing over a quadrillion tokens per month (Google AI blog)

• Deloitte 2026 CFO technology spend guidance

• Azure API Management AI gateway capabilities

• Azure AI Foundry models overview

• Azure AI Search retrieval-augmented generation

It is a health tracker for pet parents.

Log your pet’s food, symptoms, behaviors, and activities in under 5 seconds. The app finds patterns and potential triggers automatically. Track reminders, share timeline of activities and symptoms with vets and dog sitters, get a wellness score for your pets.

I am not a mobile developer.

I have never written Swift in production.

Yet here we are.

How it started

Back in December, I was experimenting with Claude Code, Anthropic’s AI coding agent.

I started by running /plan, a command that forces the AI to think before it builds.

That one habit changed everything.

Instead of generating code immediately, Claude mapped out the architecture first. Features. Data models. Edge cases. All before a single file was created.

Finding the problem worth solving

I did not start with an idea.

I started with a question.

I asked an AI agent to research Reddit. Specifically posts where people were frustrated, overwhelmed, or underserved.

One theme kept appearing: anxious pet parents.

People tracking their dog’s seizures in spreadsheets. Cat owners with pages of handwritten notes before every vet visit. No good tool to spot patterns over time.

I validated it by talking to real pet parents.

The pain was real.

How I built it

Knowing what to build took the most time. This was a side project, built evenings and weekends, so it took longer than it might have.

But once the direction was clear, implementation was fast. Coding agents are not the bottleneck anymore. Clarity is.

The entire app was built using Claude Code, Github Copilot and Codex with the Opus, Sonnet and GPT 5.x model.

My workflow looked like this:

1. /plan to describe the feature and let the AI think first

2. Review the plan, push back on anything that felt wrong

3. Let the agent build, test, and iterate

4. Use Codex to review the code for quality and correctness

5. Repeat

I acted more like a product manager than a developer.

I described what I wanted. Claude figured out how to build it.

The tool that made App Store previews easy

One thing I leaned on heavily: skill.md files.

A skill file is a procedural guide written in plain language.

You describe a task step by step. An AI agent executes it end-to-end.

I used a skill file for generating App Store screenshots.

The agent analyzed the app, identified the core benefits, picked the best simulator screenshots, and produced the final preview images. All from one command.

Skill files are becoming one of the most powerful patterns in AI-assisted building.

If you are building with agents, learn this pattern early.

The tech stack

Revenuecat for subscriptions.

No custom backend. No server to manage.

Firebase handled auth and data. Expo handled the native app layer. Claude Code handled the code.

What the app actually does

Paws & Patterns helps pet owners track everything in one place. Reduces manual journal entries.

• Log food, activity, symptoms, medications, vaccines, vet visits, and more

• Detect correlations between symptoms and potential triggers using vet-informed timing windows

• Clean timeline to track activities

• Share timeline with dog sitters when on vacation

• Set reminders for doctor appointments, medications, vaccines, etc.

• Generate PDF reports for your vet

• Track logging streaks to build a daily habit

• Support multiple pets in the household

If your dog keeps scratching and you have no idea why, this app helps you find the pattern.

The part AI could not do for me

Not everything was automated.

The final step, submitting to Apple’s App Store, is fully manual.

You write the app description. You add screenshots. You fill in privacy details. You answer Apple’s review questions.

I submitted on April 22.

Apple rejected it.

“Changes needed,” the email said.

I went back into App Store Connect, read the feedback, made the changes, and resubmitted on May 8.

On May 18, the email came: “Congratulations. Paws and Patterns has been approved for distribution.”

That felt real.

AI built the app. But Apple’s review team still had to sign off.

That final human gate reminded me: shipping software is not just about building. It is about meeting standards. Privacy, safety, quality.

I think the above part can also be automated in the future with strong browser automations or if Apple provides API access.

It took three weeks from first submission to approval.

Worth every day.

What I learned

You do not need to write code anymore to ship software.

You need to know what problem you are solving.

You need to think clearly about the user.

You need to be willing to iterate. And to handle the parts that are still manual.

The barrier to building is lower than it has ever been.

If you have an idea that solves a real problem, there is no excuse left.

Try it

Paws & Patterns is free for the first 7 days.

No commitment. Just try it with your pet for one week.

Website: pawsandpatterns.app

Download on the App Store →

Why I am doing this

I write a lot about AI Agents.

But I wanted to go beyond writing about it.

I wanted to use it, to build something useful for the society.

Paws & Patterns is the first of many consumer apps I plan to build with Coding Agents.

My goal is simple: find real problems that real people have. Niche problems. Underserved problems. The kind that never get a well-funded startup because the market looks too small.

Then build something useful for those people.

Fast. With AI. Without a team.

This is what the new era of building looks like.

What’s next

I am looking to integrate Paws & Patterns with other pet health trackers in the future.

If you have pets and have suggestions, apps you already use or features you wish existed, I would love to hear from you.

And if you have a problem you think deserves an app, something niche, something overlooked, send it my way.

I read every email: admin@agentchainai.com or anurag.karuparti@gmail.com

♻️ If this was useful, share it with a pet owner or a builder.

✉️ Subscribe at newsletter.karuparti.com so you never miss an edition.

P.S. Want more? 👋

1/ My visual guide to agentic AI → Gumroad

2/ Deep dives on agentic AI architecture → LinkedIn

3/ Real-time takes on breaking AI news → X

4/ Casual hot takes and community → Threads

5/ Visual frameworks and carousels → Instagram

6/ The full newsletter, free → newsletter.karuparti.com

Claude recently released finance-focused agents. Also agents for small businesses.

That caught my attention because it signals something bigger than one product launch: agents are starting to specialize by vertical.

The more I look at this shift, the more I keep coming back to one question:

Finance agents. Coding agents. Research agents. Customer support agents. Healthcare agents. Commerce agents.

What happens when AI agents stop being general assistants and start becoming domain specialists that make decisions on our behalf

This week, I wanted to dig into one vertical where that change could be massive: retail.

Because retail has always been designed around human behavior.

Clicks. Search bars. Landing pages. Promotions. Reviews. Checkout flows. Loyalty points. Retargeting ads.

But what happens when the customer is no longer the one doing the browsing?

What happens when the customer simply says, “Find the best option for me,” and an AI agent does the comparing, negotiating, purchasing, tracking, and returning?

That is the question I have been trying to understand:

**How will customers shop when AI agents do the buying?**

Most businesses are still optimizing for human shoppers.

Better landing pages. Better funnels. Better ads. Better checkout buttons. Better retargeting.

But the next commerce interface may not be a person clicking through your website.

It may be an AI agent acting on behalf of that person.

That sounds futuristic until you realize the building blocks are already here: computer-use agents, agent communication protocols, API-first commerce platforms, AI search, payment orchestration, fraud detection, and identity systems.

The real question is not whether AI agents will influence shopping.

The real question is whether your business is ready to be understood, compared, trusted, and transacted with by agents.

The shopping journey is about to change

Picture Sarah, a VP of Product at a healthcare company.

She needs to book a business trip to Chicago. Today, that means opening six tabs: airline, hotel, credit card rewards, maps, restaurant reservations, and the company travel policy.

She compares prices. Checks points. Reads reviews. Looks for cancellation rules. Books dinner. Saves receipts.

Now imagine Sarah saying this instead:

Book my Chicago trip. Keep it under policy. Prioritize hotels near the board office. Use loyalty points only if the redemption value is good. Book dinner for four people on Tuesday. Ask me before final purchase.

Her AI agent handles the work.

It checks travel policy. Compares hotels. Reviews loyalty benefits. Looks at cancellation terms. Books dinner. Creates the itinerary. Then asks Sarah to approve the final transaction.

Sarah does not browse.

Sarah delegates intent.

That is the shift.

When customers delegate intent, commerce moves from human navigation to agent execution.

And that changes the architecture.

If this mental model above gave you clarity on Agentic concepts,

I built a 74 visual AI library with animated guides that take you from agent basics to production grade architectures.

Agents. Agentic RAG.Multi agent orchestration.Enterprise governance. Production patterns.

Used by architects, engineers, and executives. 3 million plus impressions on Linkedin.

Built for fast clarity before meetings, interviews, and real world builds.

Get the full AI Visual Library here.

The six building blocks accelerating agentic commerce

The rapid pace of innovation in agentic AI is not coming from one model release.

It is coming from a stack of tools and protocols that make agents more useful, interoperable, and commercially actionable.

Here are the six developments I am watching most closely.

1. Model Context Protocol, or MCP, matters because agents need more than prompts.

   They need a standardized way to connect to tools, retrieve context, call functions, and operate across systems. In commerce, that means an agent can check inventory, apply a loyalty rule, retrieve a return policy, or create a support case without every team building a custom integration from scratch.

2. Agent-to-Agent Protocol, or A2A, matters because commerce will not be one agent talking to one website forever.

   Customer agents will negotiate with merchant agents. Broker agents will compare options across many providers. Partner agents will coordinate fulfillment, payment, and service recovery. For that to work, agents need a shared interaction model for exchanging capabilities, task status, context, and multimodal outputs across vendors and environments.

3. Agent Payments Protocol, or AP2, is one of the clearest signs that agentic commerce is moving from demo to transaction layer.

   The important idea is the mandate. A mandate connects user intent, cart details, and payment authorization into a verifiable audit trail. That matters because autonomous purchases need accountability. Merchants need to know the agent had authority. Users need limits and revocation. Payment networks need fraud controls. Enterprises need evidence.

4. Computer-use agents fill the gap when APIs are missing.

   An agent that can use a browser, click buttons, fill forms, and navigate a user interface can automate workflows that were never designed for agent access. This will be messy, fragile, and hard to govern at scale, but it is useful in niche domains where building a formal API is not economical.

5. Contextual AI-driven personalization changes recommendations from static search results to adaptive intent handling.

   Instead of asking, “What did this user click last week?” the system starts asking, “What is this person trying to accomplish right now, given their history, constraints, preferences, and current context?” That is where memory architectures, preference stores, vector search, and policy-aware personalization become important.

6. Dynamic planning with real-time adjustment is what makes the Sarah travel example possible.

   The agent does not just book a flight. It coordinates flights, hotels, restaurants, calendar constraints, loyalty programs, approvals, expense systems, and backup options. If the flight changes, the itinerary changes. If the hotel sells out, the agent replans. If an expense crosses policy, the workflow asks for approval.

That is the big shift.

Agentic commerce is not only about smarter recommendations. It is about systems that can understand intent, coordinate tools, transact safely, and adjust plans in real time.

The three ways agents will interact with your business

I think about agentic commerce through three interaction models.

1. Agent-to-site

This is the simplest model.

A customer’s AI agent visits your website, scans your catalog, compares prices, checks policies, and completes checkout.

This may happen through browser automation, public APIs, structured product data, or a combination of all three.

The key point: your website is no longer only a human experience. It becomes a machine-readable decision surface.

If your inventory, pricing, cancellation rules, loyalty benefits, and delivery promises are hard for an agent to interpret, you are already behind.

2. Agent-to-agent

This is where commerce gets more interesting.

Instead of a customer’s agent scraping your website, it talks directly to your business’s agent.

The customer agent says:

I need a hotel near downtown Chicago for three nights. My traveler has Gold status. Can you improve the offer if they add breakfast?

Your business agent responds:

Yes. I can offer 12% off if breakfast is bundled and the reservation is prepaid.

That is not a chatbot. That is a commerce negotiation layer.

It requires access to inventory, pricing, loyalty, margin rules, fraud signals, payment authorization, and business policy.

3. Brokered agent-to-site

This is the model every executive should pay attention to.

A broker agent sits between the customer and many businesses. It compares options, ranks them, and recommends a purchase.

Think OpenTable, Expedia, or Instacart, but generalized across more categories and powered by agents.

If the broker owns the customer interface, your business becomes one option inside someone else’s ranking system.

That ranking system may care about price, availability, reliability, refund flexibility, fulfillment confidence, and API quality.

What businesses need to build

Agentic commerce is not just a marketing channel.

It is an infrastructure shift.

Here are the six domains I would audit first.

1. Agent discoverability

Your products and services need to be discoverable by agents, not just humans.

That means structured catalogs, clear metadata, real-time availability, readable policies, and APIs, MCPs that expose what agents need to evaluate your business.

SEO does not disappear, but it changes.

The question becomes: can an agent confidently understand when your offer is the best fit for a customer’s intent?

2. Loyalty and clienteling

Most loyalty systems were built for humans checking points in an app.

Agents need something different.

They need to query loyalty status, unused rewards, customer preferences, tier benefits, personalized offers, and redemption rules.

If your loyalty program is trapped behind a human-only mobile interface, agents will ignore it.

If your loyalty system exposes secure, permissioned APIs, agents can use it to make better decisions for the customer.

3. Secure payments and fraud defense

This is the hardest part.

It is one thing for an agent to recommend a product. It is another thing for that agent to spend money.

Agentic commerce needs payment mandates that answer:

• Who authorized this agent?

• What is the spending limit?

• What categories are allowed?

• When is human approval required?

• How does the merchant verify authority?

• What happens if the agent makes a mistake?

A simple mandate might look like this:

{
  "agent_id": "travel_agent_789",
  "customer_id": "customer_123",
  "allowed_category": "business_travel",
  "max_transaction_amount": 1200,
  "approval_required_above": 500,
  "valid_until": "2026-06-30",
  "purpose": "Chicago board meeting"
}

That object is not just payment data.

It is identity, consent, policy, and fraud control in one package.

4. API-first commerce platforms

If core commerce actions are trapped inside a front end, agents will struggle.

Search, cart creation, pricing, promotions, checkout, order status, returns, and support escalation need clean APIs.

The architectural shift is from website-first commerce to capability-first commerce.

The website becomes one client.

Agents become another.

5. Connected in-store POS

Agentic commerce will not stay online.

A customer agent may reserve a product, apply a loyalty benefit, pre-authorize payment, and schedule pickup.

When the customer arrives in store, the POS needs to understand that journey.

If the online agent journey and the physical store system are disconnected, the experience breaks at the handoff.

6. Real-time fulfillment and returns

Agents will not only buy. They will track, change, cancel, return, and escalate.

That requires real-time visibility into orders, inventory, shipping, store availability, replacement options, and return eligibility.

In agentic commerce, operations become part of the product.

A bad fulfillment API can cost you the sale before a human ever sees your brand.

Highly recommend reading this Mckinsey paper on Agentic Commerce

The bottom line

Agentic commerce changes the buyer interface from clicks to delegated intent.

Your next customer may still be human, but the thing doing the shopping may be an agent.

The brands that win will not only have better marketing. They will have APIs, payments, loyalty systems, inventory, fraud controls, and fulfillment workflows that speak fluent agent.

♻️ If this was useful, share it with someone building with AI.

Share

✉️ Subscribe at newsletter.karuparti.com so you never miss an edition.

P.S. Want more? 👋

1/ My visual guide to agentic AI → Gumroad

2/ Deep dives on agentic AI architecture → LinkedIn

3/ Real-time takes on breaking AI news → X

4/ Casual hot takes and community → Threads

5/ Visual frameworks and carousels → Instagram

6/ 60-second production lessons → TikTok

7/ The full newsletter, free → newsletter.karuparti.com

References

• McKinsey - https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-agentic-commerce-opportunity-how-ai-agents-are-ushering-in-a-new-era-for-consumers-and-merchants

• Google Agent2Agent protocol announcement: https://developers.googleblog.com/en/a2a-a-new-era-of-agent-interoperability/

• Google Agent Payments Protocol announcement: https://cloud.google.com/blog/products/ai-machine-learning/announcing-agent-payments-protocol-ap2

Disclaimer: The stories and scenarios in this article are hypothetical, inspired by patterns observed across similar real-world experiences. They are used to convey key concepts more effectively and do not represent any specific individual or organization.