Why I stopped treating AI agents like smarter chatbots. A field guide to the chatbot-to-agent evolution
A guest essay by Ramana Reddy Gunda, Senior AI/ML Technical Architect
Welcome back to Diary of an AI Architect. Today, we are stepping outside our usual solo deep dives to bring you a guest essay by Ramana Reddy Gunda, AI Architecture Leader from Cognizant, with 18+ years in enterprise technology across healthcare, retail, and telecom.
Ramana makes the argument most teams learn the hard way: an LLM-based agent is not a bigger, smarter version of the traditional chatbot you already built, and treating the shift as an upgrade, instead of a paradigm change, is exactly how the engineering and governance model goes wrong. He has shipped production systems on every side of that line, from fraud-hunting clustering models to intent bots to agents that turn plain English into working data pipelines, and he recently formalized the transition in a research paper. This is the practitioner version. Below are his words. They are worth reading twice.
Table of Contents
The traditional ML era: when the question is “what pattern is hiding in this data?”
The bridge: intent-based chatbots, and why they had a ceiling
Three discontinuities, not one upgrade
RAG isn’t an enhancement - it’s a prerequisite
The part that actually requires new discipline
So which one do you actually need? Traditional ML vs LLM-Based Agent
The uncomfortable part nobody likes to admit
I’ve spent 18+ years building enterprise systems, and the last seven specifically on what feels like both sides of an industry line that a lot of people are only just starting to talk about: traditional machine learning versus LLM-based agents. I didn’t plan it that way. I moved from building unsupervised clustering models that hunt for fraud in claims data to building intent-based chatbots for enterprise support to architecting retrieval-augmented systems and generative AI agents that turn natural language requests into working data pipelines.
I got interested enough in the “why” behind that shift that I recently wrote a research paper on it - specifically on the transition from intent-based chatbots to LLM-powered agents. The conclusion I landed on, after digging through the architecture literature and lining it up against what I’d actually shipped in production: this isn’t an incremental upgrade. It’s a paradigm shift. Treating it like a bigger, smarter version of the old thing is exactly how teams get the engineering and governance model wrong.
Here’s what seven years of building on both sides of that line - and then formalizing it on paper - taught me.
The traditional ML era: when the question is “what pattern is hiding in this data?”
My earlier work was almost entirely about pattern discovery in structured, historical data. The clearest example: building an unsupervised fraud detection system for a healthcare payer, hunting for anomalous claim behavior across a huge volume of pharmacy transactions.
There was no labeled “this is fraud” dataset to train against; that’s exactly why it was a clustering problem, not a classification problem. I used K-Means and HDBSCAN together, layered with PCA for dimensionality reduction, to group pharmacy claims by behavioral similarity and then profile which clusters looked statistically abnormal. The output wasn’t a yes/no prediction; it was a risk score that told investigators where to look first.
A parallel project used the same mindset for a different question: predicting which website visitors during an open enrollment period were first-time shoppers versus returning members, based on navigation patterns and time-on-page. No LLM in sight, just feature engineering, a classification model trained on historical visit data, and a deployment pipeline that scored behavior in near-real time.
That’s still, in my experience, the right toolkit for a huge chunk of real business problems: anomaly detection, propensity scoring, churn prediction. If your problem is “find the needle in a structured haystack,” an LLM is usually the wrong needle-finder.
The bridge: intent-based chatbots, and why they had a ceiling
Before generative AI, I spent years building conversational systems the old way: intent classification, entity extraction, and deterministic dialogue state machines, the architecture that platforms like Dialogflow, LUIS, and Lex industrialized through the 2010s. I built bots for sales support, supply chain order tracking, and IT service desk requests, and within their design envelope, they worked. Deterministic flows are auditable. A state machine can be certified for compliance review in a way a language model can’t. For high-volume, bounded queries, that architecture reliably automated 60-70% of tier-1 support volume; there’s real published evidence behind that number.
But the ceiling was structural, not a matter of more engineering effort. I frame it in the paper as three inherent constraints:
Intent coverage decay. Every time a user phrased something outside the enumerated intent taxonomy, the fix was weeks of retraining, not a quick patch.
Contextual fragility. The dialogue layer tracked intent, entities, and phase, not the actual conversation. Multi-turn exchanges that needed real memory produced what felt to users like conversational amnesia.
Fallback brittleness. Out-of-scope queries had exactly one destination: an apology and a menu. There was no mechanism for handling what the system wasn’t explicitly built to handle.
Those three limits trace directly back to the architecture’s core assumptions - a closed intent ontology, a deterministic state graph, and template-bound output. You couldn’t engineer your way out of them. You had to replace the assumptions.
Three discontinuities, not one upgrade
This is the part of my research I think is most useful for anyone evaluating this shift right now: it isn’t one change, it’s three separate structural breaks, each replacing a different layer of the old architecture.
I built the diagram below the same way I built it for the paper, because seeing the three discontinuities side by side is what made the “kind, not degree” argument click for me:
First: ontology-bound classification gives way to open-vocabulary semantic inference. An LLM doesn’t sort an utterance into a predefined bucket, it infers meaning from context, history, and retrieved evidence, with no ceiling from an enumerated taxonomy. I felt this directly building a retrieval-augmented search system that let finance, HR, and sales teams ask plain-language questions and get answers pulled straight from internal documents, instead of filing a ticket or digging through folders.
Second: handcrafted state machines give way to prompt-governed generation. Behavioral specification stops being a transition graph that takes weeks to modify and becomes natural-language instructions in a system prompt, a change measured in hours.
Third: reactive single-turn response becomes autonomous multi-step execution. This is the one I think gets underrated. I recently led the architecture for a generative AI assistant that sits on top of a legacy data platform, a business user types a request in plain English, the system interprets intent, generates the right structured query, and returns usable data with no SQL knowledge required. That’s not a chatbot answering a question. That’s an agent planning, acting, and adjusting across steps.
None of these three are differences of degree. They’re differences of kind, and the research bears that out. ReAct-style agentic frameworks that interleave reasoning with tool use improved multi-hop task accuracy by 38 percentage points over non-agentic baselines on benchmark evaluation.
Retrieval grounding cut hallucination rates by roughly 21 percentage points relative to ungrounded generation. Instruction-tuned models were preferred by human evaluators over base model output 71% of the time. Those aren’t marginal gains, they’re evidence the underlying architecture changed, not just the size of the model.
RAG isn’t an enhancement - it’s a prerequisite
If there’s one thing I’d correct in how people talk about enterprise LLM deployment, it’s this: retrieval-augmented generation gets described as an optional add-on. In production, it’s not optional. Any system that needs to answer questions about policy documents, product specs, or internal knowledge bases - anything not in the model’s training data - will hallucinate with non-negligible frequency answering from parametric memory alone.
Every RAG system I’ve built follows the same shape: embed the query, retrieve the top-k relevant passages from a vector index, assemble that context, and condition generation on the retrieved evidence rather than the model’s own memory. The grounding is the architecture decision, not a prompting trick layered on afterward.
The part that actually requires new discipline
Traditional ML models are deterministic: same input, same output, every time. That’s what makes them auditable and regulator-friendly. LLM-based agents trade that determinism for flexibility, and that trade demands different engineering discipline than a classification model ever needed:
Guardrails replace feature engineering. In traditional ML, most effort goes into deciding what signal to feed the model. In LLM systems, comparable effort goes into deciding what the model is not allowed to do - validating generated queries before they execute, constraining outputs to expected formats, catching a hallucinated field before it reaches a downstream system.
Evaluation looks completely different. A classification model has a confusion matrix. An agent answering open-ended requests needs phased rollouts, starting simple before tackling complexity, and real humility about what “good enough” means before you widen scope.
The audit trail gets harder to read. An intent-based system’s trail is a clean sequence: intent classified, state transitioned, template selected. An agentic system’s trail spans multiple inference calls, tool invocations, and reasoning steps - a harder record to inspect, and one the field hasn’t fully standardized governance for yet.
Memory becomes a design choice, not a side effect. Old systems held session-scoped state and discarded it at close. LLM agents can carry short-term context, long-term vector-store memory, episodic recall, and procedural rules - turning stateless exchanges into something closer to an ongoing relationship. That’s powerful, and it’s also a governance decision every team underestimates going in.
So which one do you actually need?
Reach for traditional ML when your data is structured and historical, the question is fundamentally statistical, you need guaranteed explainability for a regulator or auditor, and cost/latency at scale matter more than flexibility.
Reach for an LLM-based agent when the bottleneck is the interface between a human and a system rather than the underlying data pattern, requests are open-ended or require synthesizing across unstructured sources, and you’re willing to invest in grounding and guardrails instead of expecting the model to be right out of the gate.
And increasingly, the best systems use both. A clustering model surfaces which cases need attention; an agent is the interface that lets a human ask follow-up questions about why. Traditional ML finds the signal. LLM agents make the signal usable by someone who isn’t a data scientist.
The uncomfortable part nobody likes to admit
The industry conversation treats LLM agents as the obvious upgrade path from “boring” traditional ML - and separately, treats today’s agents as a bigger version of yesterday’s chatbots. I don’t think either framing survives contact with the actual architecture. I’ve watched teams bolt an LLM onto a problem a simple classifier would have solved more cheaply and more reliably. And I’ve watched teams deploy an “agent” that’s really just a chatbot with a better vocabulary, without the guardrails, evaluation discipline, or audit logging that autonomous multi-step execution actually demands.
The skill that matters isn’t “know-how to build an agent.” It’s knowing which problem you’re looking at, and which of the three genuinely different architectural paradigms it calls for, before you pick a tool. That’s what seven years of building across all of them - on top of a much longer career in enterprise systems- made unavoidably clear to me.
Thank you for reading my newsletter!
Anu Karuparti
Creator, Diary of an AI Architect
How enterprises actually ship AI to production
Read by 3,000+ FDEs, AI Architects, and Engineering Leaders from Microsoft, Google, IBM, PwC and others.
Connect with me on LinkedIn.
Want to partner? Email me at anurag.karuparti@gmail.com.
P.S. Want more? 👋
1/ My visual guide to agentic AI → Gumroad
2/ Daily deep dives on agentic AI architecture → LinkedIn
3/ Real-time takes on breaking AI news → X
4/ Casual hot takes and community → Threads
5/ Visual frameworks and carousels → Instagram
6/ 60-second production lessons → TikTok
7/ The full newsletter, free → newsletter.karuparti.com
 | A guest post by
|
This gets more interesting once the agent stops being a chat surface and starts sitting inside a workflow. Context, tools and memory change the question from "did it answer well?" to "what is it now allowed to do?"
The “act” column is where the whole risk model flips. A chatbot’s worst case is a wrong answer you read and reject before it matters. An agent’s worst case is a wrong action that already happened before you saw it - the refund sent, the message out, the record changed.
So the real discontinuity isn’t just capability, it’s blast radius: you stop grading “did it answer well?” and start asking “what could it have done, and can I reach it in time to stop or undo it?” Most stacks still instrument agents like chatbots. Does your field guide treat the act step as its own layer, or as a smarter output?