Rethinking Microservices with the Rise of AI Agents

AI agents will change how we build systems.

Just a year ago, AI agents like AutoGPTs were more of an experiment, held together by fragile prompts and quick fixes. Today, they show up in start-up architectures, internal tools, and daily developer workflows.

This shift mirrors past evolutions in System Design: when monoliths gave way to microservices, containers, serverless, and platform engineering, each wave introduced tools that fundamentally changed how we design.

Now, agentic systems are becoming part of that stack.

AI agents like AutoGPT, BabyAGI, and OpenAgents don’t just call APIs. They plan, reason, use tools, remember past actions, and coordinate with each other; they’re autonomous actors, working toward outcomes through dialogue and iteration.

So, are we moving from modular services to modular intelligence?

In this newsletter, we’ll explain:

• How AI agents can change System Design

• How startups and tech giants are using agent-based architectures

• Challenges and pitfalls of agent-oriented systems

• How to prepare for the future of System Design

Let's go.

What are AI agents?

An AI agent Orchestrating workflows refers to the planning, coordination, and execution of a sequence of tasks or operations to achieve a specific goal.is a software-based system designed to autonomously perform tasks on a user’s or another system’s behalf by orchestratingOrchestrating workflows refers to the planning, coordination, and execution of a sequence of tasks or operations to achieve a specific goal. workflows and utilizing available (external) tools.

AI agents exhibit advanced capabilities such as:

• Natural language understanding

• Decision-making

• Problem-solving

• Action execution without continuous human intervention

AI agents can handle complex tasks across diverse domains. By leveraging advanced natural language processing (NLP) techniques from large language models (LLMs), these agents interpret user input, plan actions, and invoke external tools or APIs as needed in a step-by-step and context-aware manner.

For example, a customer service chatbot is a software AI agent. It interacts with users by understanding their questions (through NLP), decides the most appropriate response, and delivers answers instantly. Over time, it can learn from user interactions to improve the quality and relevance of its responses.

The following illustration shows how an AI agent works:

The rise of autonomous AI agents

https://github.com/yoheinakajima/babyagiIn early 2023, open-source projects like AutoGPThttps://github.com/Significant-Gravitas/AutoGPT and BabyAGIhttps://github.com/yoheinakajima/babyagi burst onto the scene, showcasing how AI (powered by GPT-4 or similar models) could be chained to perform multi-step tasks with minimal human input.

AutoGPT was one of the fastest-growing GitHub projects at its release. Users would give AutoGPT a high-level goal, and the agent would then generate sub-tasks, reasoning through each step autonomously. This “agent” could browse the web, write files, execute code, and more to pursue the goal.

Let’s explore how microservice architecture compares with AI agent-driven architecture and the new possibilities autonomous agents unlock.

Microservices in a nutshell (Why the comparison?)

To explore whether autonomous agents could become the next microservices, it’s useful to recall that microservices decompose monolithic applications into small, independent services, each focused on a specific business function and communicating via APIs. This modular approach enables independent development, deployment, and scaling, forming the backbone of cloud-native architecture.

The comparison to AI agents comes from a shared idea: solving big problems by dividing them into smaller, focused parts.

But instead of following fixed rules like regular services, AI agents make decisions and adapt based on what’s happening. For instance, a future e-commerce system might use an AI pricing agent to adjust prices based on trends, a customer service agent to handle routine queries, and an inventory agent to predict stock needs. These agents operate independently yet collaborate, like microservices, toward a unified application goal.

AI agents are evolving into intelligent, task-driven components capable of planning, prioritizing, and acting autonomously to achieve defined outcomes.

Another key parallel is the focus on loose coupling and well-defined responsibilities:

• Each service has a specific role in microservices and communicates through well-defined interfaces.

• Similarly, agent-based architectures assign distinct roles to each agent (like scheduling, coordination, or resource management), enabling them to interact via messages or shared data.

This decoupling allows agents to evolve independently, echoing the principle that makes microservices appealing.

While microservices revolutionized system architecture, AI agents might take this a step further. With the added elements of autonomy and reasoning, AI agents are reshaping how we approach complex, dynamic tasks.

AI agent-oriented architecture: A new paradigm

Despite similarities, AI agents introduce a new paradigm beyond traditional microservices.

Microservices will do exactly what they are coded to do (and nothing more). AI agents, on the other hand, can dynamically decide how to fulfill a request. They use AI (often large) language models to determine the best steps. An AI agent might devise a solution or workflow that a human programmer didn’t explicitly design.

For example, consider scheduling meetings. You might have a scheduling service with a fixed algorithm in a microservice world. In an agent world, you could deploy a meeting scheduler agent that knows how to use calendars, email, and video conferencing tools. If a conflict arises, the agent might negotiate a new time by contacting participants’ calendar agents, much like a human assistant would.

It’s an agent-centric design view:

• One agent handles calendar availability

• Another manages communication between attendees

• Another handles room bookings

• And so on.​.

The system runs by agents talking to other agents behind the scenes instead of rigid API calls.

The AI agent-oriented paradigm is visionary. It suggests a future where much of our software isn’t a collection of screens and forms, but a society of AI workers behind the interface.

Some technologists foresee AI agent fleetshttps://blog.mlq.ai/ai-agent-startups-to-watch, which are swarms of specialized agents interacting to solve complex tasks, potentially managing large parts of a business autonomously. It’s a futuristic idea, but one that has caught the imagination of developers and business leaders alike.

While AI agents and microservices share some architectural DNA, important differences define their behavior and scale. Let’s capture these contrasts clearly:

These differences introduce new challenges in reliability, debugging, and safety. With the conceptual groundwork laid, let’s now see these agent systems in action through real-world case studies.

Agent systems in the wild

Is any of this agent stuff being implemented?

Yes. Both startups and tech giants are actively exploring agent-based architectures. Let’s look at a few examples:

• AutoGPT and its successors: Once a hobby project, AutoGPT has raised $12M to become a full platform with a user-friendly GUI. Early users have experimented with delegating tasks like debugging, writing emails, and drafting strategies. While results vary, these use cases showcase the potential of a single AI agent in automating complex work.

• Multi-agent frameworks (CrewAI): Startups like CrewAIhttps://www.crewai.com are simplifying multi-agent collaboration. Their open-source framework lets agents share tasks and data (e.g., one agent researches competitors while another analyzes results). With $18M in funding and 150+ enterprises onboardedhttps://siliconangle.com/2024/10/22/agentic-ai-startup-crewai-closes-18m-funding-round/, CrewAI is gaining traction. As its founder notes, agents are poised to transform how companies build and deliver products.

• No-code agent builders (Relevance AI): Relevance AIhttps://relevanceai.com/ offers a no-code platform for creating custom agents using natural language, allowing non-developers to deploy them as digital co-workers. Backed by $18M in funding and used by Fortune 500 companies, it enables agent orchestration across customer support and internal operations.

• Enterprise agent orchestration (IBM and friends): Established tech giants embrace agent-based systems. IBM’s Watson X orchestratehttps://www.ibm.com/products/watsonx-orchestrate is a supervisor for AI agents—tasks can be assigned in plain English, routed to the right agent, configured with company data, and executed autonomously. With plans for more, IBM has already deployed agents for domains like HR.
  Similarly, Salesforce has launched Agentforcehttps://www.salesforce.com/ap/agentforce/, featuring 100+ prebuilt CRM agents and a no-code builder for custom needs.

• Other notable examples:

  • HuggingGPThttps://arxiv.org/abs/2303.17580 (Microsoft research) show how a “manager” agent can coordinate multiple AI models to solve multi-modal tasks, similar to microservice orchestration.

  • Stanford’s generative agentshttps://hai.stanford.edu/news/computational-agents-exhibit-believable-humanlike-behavior simulate AI-driven characters living autonomously in a virtual town, illustrating emergent behavior. Though not commercial, these experiments highlight the rapid evolution and growing interest in agent-based systems.

Now that we’ve seen how AI agents operate in various scenarios, let’s explore a real-world example of a customer support platform powered by AI agents and what its System Design might look like.

Real-world example: AI agents powering a customer support platform

Imagine you are building a customer support system for an online retail company. Instead of traditional ticketing queues and rigid workflows, you use specialized AI agents to handle different aspects of the customer experience.

Whenever a customer submits a query, such as a refund request or a shipping update, the platform dynamically assigns tasks to intelligent agents based on the query type, context, and business rules.

Let’s look at a high-level design for a customer support system powered by AI agents:

Let’s discuss its workflow.

When a customer submits a query, the query router agent understands it and classifies it as an FAQ, a technical issue, or a billing-related concern.

• If the query is identified as an FAQ, the FAQ AI agent retrieves and provides a predefined answer from the FAQ memory in the knowledge base.

• For technical issues, the technical AI agent analyzes the problem and suggests potential solutions using a relevant troubleshooting guide in the knowledge base.

• In the case of billing inquiries, the AI agent verifies the customer’s account and payment details before responding, using billing information from the knowledge base..

If the query is resolved, the response is returned to the user. If not, the human escalation manager forwards it to a human support agent for manual intervention. While all interactions are logged via a monitoring and logging service to enable performance tracking and continuous improvement

Now, let’s zoom out and explore why this agent-based approach is gaining momentum across various systems.

Benefits of agent-oriented systems

As software systems evolve, agent-oriented systems are gaining traction for their ability to handle complex, dynamic tasks. Here, we’ll explore the key advantages of using specialized AI agents over traditional monolithic or microservice-based approaches:

• Modularity: Agents are independent, self-contained entities designed to handle specific tasks. This modularity allows for easier maintenance, updates, and replacements without disrupting the system.

• Scalability: Agent-oriented systems can scale easily by adding new agents or distributing tasks across multiple agents. This makes them highly suitable for large systems or environments where demands fluctuate.

• Improved decision-making: AI agents can process large amounts of data and make decisions faster and more consistently than humans, helping streamline processes such as customer support, logisticsLogistics refers to the planning, coordination, and optimization of activities involved in the movement, storage, and delivery of goods across a network., and monitoring.

• Cost efficiency: By automating tasks traditionally performed by humans, agent-oriented systems can significantly reduce operational costs while increasing efficiency. For instance, AI agents can handle customer queries 24/7, reducing the need for large human support teams.

• Autonomy: Agents can operate autonomously to perform tasks, reducing the need for constant human intervention. This leads to more efficient and responsive systems, capable of handling repetitive or complex tasks.

Challenges and pitfalls of agent-oriented systems

Agent-oriented systems hold promise, but deploying them at scale brings several limitations:

• Overreliance on automation: Agents may appear capable of handling end-to-end processes, but fully delegating decision-making can result in poor outcomes or misaligning strategic objectives. This risk can be mitigated by maintaining human-in-the-loop oversight and regularly auditing agent decisions to ensure consistency with business goals.

• Difficulty in understanding context: AI agents often face limitations when navigating nuanced or ambiguous situations that require empathy, human judgment, or deep domain expertise. Leveraging domain-specific training data and pairing AI output with human review for complex or sensitive cases can improve reliability.

• Scalability challenges: As the number of agents grows, coordinating their interactions, managing shared resources, and maintaining consistent performance become increasingly complex. Designing systems with modular components and scalable infrastructure, supported by orchestration frameworks and real-time monitoring, helps ensure operational stability.

• Concerns regarding data privacy and security: Agents that process sensitive information, such as customer data or financial records, introduce risks related to compliance and unauthorized access. These concerns can be addressed by enforcing strong encryption for data at rest and in transit, applying role-based access controls, and adhering to established data protection standards such as GDPRGeneral data protection regulation or HIPAAHealth insurance portability and accountability act.

• Latency and cost overhead: Reasoning tasks and inter-agent communication can introduce significant delays and operational expenses, especially when agents invoke external tools or services. Setting boundaries on inter-agent interactions, using caching strategies, applying timeouts, and prioritizing tasks can help control latency and manage costs effectively.

With those traps in mind, we can now rethink core System Design principles for an agent-driven world.

System Design principles reimagined

When AI agents replace traditional microservices, the architecture will become more flexible and skill-based, with agents performing tasks based on learned goals and context.

Unlike microservices, which rely on rigid APIs and clear boundaries, AI agents evolve, adapting their behavior based on interactions with the system and users. This opens up possibilities for real-time decision-making and self-adjusting workflows, but also requires balancing adaptability and predictability.

Reimagining System Design principles requires reconsidering concepts like resilience, observability, and scalability. Instead of using traditional tools like circuit breakers or container restarts, you’ll rely on confidence-based systems to escalate decision-making when uncertainty is high.

Observability shifts from log-based tracking to analyzing feedback loops and decision histories at the agent level, offering deeper insights into how decisions evolve.

However, with these innovations come trade-offs: maintaining clear contracts between agents and managing emergent behaviors without increasing complexity is key to building a robust system.

Future of System Design

AI agents could reshape System Design, potentially becoming the microservices of the AI era—if supported by mature tooling, well-defined architectural patterns, and robust guardrails. Emerging concepts like Agent OS, agent as a service (AaaS), and vertical-specific agents hint at a future where scalable, domain-specialized agent ecosystems drive real-world applications.

However, not everything will change.

Despite increased autonomy, the need for operations engineers, debugging pipelines, observability tooling, and quality assurance layers will remain essential.

Agents may shift how systems are built and operated, but they will not eliminate the need for human oversight, structured testing, and reliable fallback mechanisms.

The evolution is promising, but the fundamentals of building resilient, maintainable systems still apply.

AI agents may not entirely replace microservices, but they could augment or surpass them in areas demanding flexibility, reasoning, and rapid iteration.

Like microservices, agents offer modularity and scalability, but they also introduce challenges around emergent behavior, predictability, and cost, especially given their reliance on expensive LLMs. Until open-source or local alternatives mature, cost remains a significant barrier.

However, the shift from coding individual functions to guiding autonomous reasoning loops marks a profound change in how we design software systems.

As we navigate this transition, understanding both classic System Design and its Generative AI evolution is no longer optional.

Whether you're architecting scalable APIs or orchestrating agent workflows with LLMs, you can hone your System Design skills with our popular courses: