Deploying AI Agents in Production Using Open Source Architecture

By -

AI agents are becoming core parts of modern applications, but deploying them in production at scale is challenging. The diversity of agent frameworks, unpredictable latency, and the need for streaming responses make it hard to use simple REST-only patterns. Most proof-of-concept deployments break down when real workloads, traffic, and observability requirements hit.

This blog post explains how to design a production-ready, open-source architecture for AI agents using FastAPI, Celery, Redis, Kubernetes, KEDA, Prometheus, Grafana, LangFuse, and LangGraph.

Why Deploying AI Agents Is Hard

  • Agent frameworks differ widely in required infrastructure.

  • Latency varies from milliseconds to minutes depending on workflow complexity.

  • Real-time streaming is needed for modern AI UX.

  • REST-only patterns can’t handle long execution, retries, or async scheduling.

  • Scaling compute-heavy agents is fundamentally different from scaling API servers.

Why the Traditional REST Architecture Fails

While Flask or FastAPI can handle simple POCs, they break under production requirements:

  • API workers block on long tasks.

  • No job scheduling, retries, or queueing.

  • No ability to scale compute separately from API layer.

  • Limited visibility into agent internals.

  • No support for streaming partial output.

To scale AI agents to thousands or millions of users, you need a dual-layer architecture: an API layer for coordination and a worker layer for execution.

An Enterprise-Ready Architecture for AI Agents



Below is a reliable, cloud-native pattern observed in real production deployments.

1. Remote Execution with FastAPI, Celery, and Redis

Pattern: FastAPI receives a request, creates a job, and immediately returns a task_id. Celery workers handle execution asynchronously.

Key points:

  • FastAPI is the control plane.

  • Celery handles asynchronous execution.

  • Redis acts as both message broker and result store.

  • Clients retrieve progress via API or WebSocket.

This keeps the API responsive while the heavy work happens elsewhere.


2. Handling Long-Running Agents with Celery

Enterprise agents often:

  • run multi-step reasoning

  • call multiple LLMs

  • access several external systems

Celery can handle long and complex workflows with features like:

  • delayed acknowledgments for safe processing

  • automatic retries

  • workflow patterns (chains, groups, chords)

Infrastructure guidelines:

  • Deploy Celery workers as a dedicated Kubernetes Deployment.

  • Use Redis StatefulSet for persistence.

  • Configure PodDisruptionBudgets for uptime.

  • Spread Redis replicas across nodes using anti-affinity rules.

3. Framework-Agnostic Agent Runtime

The architecture should run any agent framework without modification.

Supported frameworks include:

  • LangGraph

  • CrewAI

  • Agno

  • ADK

  • Custom agent runtimes

Each agent is simply wrapped in an asynchronous task executed by Celery.

4. Traceability and Observability

Production systems require deep visibility into agent execution.

Recommended tools:

  • LangFuse: trace events, token usage, cost, and errors.

  • Prometheus: track queue depth, worker load, API latency.

  • Grafana: dashboards and alerting.

This allows you to detect bottlenecks, optimize costs, and debug agent behavior.

5. Real-Time Streaming Using Redis Pub/Sub and WebSockets

AI agents often generate output incrementally. To support this:

  • Celery workers publish intermediate updates to Redis Pub/Sub.

  • A WebSocket endpoint subscribes to these channels.

  • The UI receives partial updates in real time.

This pattern supports:

  • token streaming

  • intermediate reasoning results

  • state and progress updates

It also avoids blocking API threads during long-running tasks.

Celery worker publishing events:

FastAPI WebSocket subscriber:

6. Horizontal Scalability

AI agent workloads scale in two separate dimensions.

API Scaling

  • Scale FastAPI using Kubernetes HPA.

  • Metrics: CPU, memory, or requests-per-second.

  • APIs remain stateless and lightweight.

Worker Scaling

  • Scale Celery workers with KEDA using Redis queue depth.

  • Automatically increase workers during high load.

  • Isolate agent execution from user-facing traffic.

This dual-scaling model is essential for enterprise workloads.

Keda Snippet

7. Monitoring, Metrics, and Alerting

A production deployment should expose:

  • agent execution time

  • failure rates

  • queue depth over time

  • worker concurrency

  • LLM cost per request

Prometheus and Grafana provide a reliable end-to-end observability stack.

Putting Everything Together

End-to-end architecture:

  1. Client sends request → FastAPI

  2. FastAPI → Redis queue (Celery)

  3. Celery worker runs agent workflow

  4. Worker pushes intermediate output to Redis Pub/Sub

  5. WebSocket streams updates to UI

  6. Traces → LangFuse / LangGraph

  7. Metrics → Prometheus → Grafana

  8. Scaling: HPA for API, KEDA for workers

This architecture supports:

  • long-running tasks

  • real-time streaming

  • millions of users

  • full observability

  • agent framework independence

  • cloud-native scalability

Conclusion

AI agents are powerful, but deploying them at scale requires more than a simple REST endpoint. With the right open-source components - FastAPI, Celery, Redis, Kubernetes, KEDA, Prometheus, Grafana, and LangFuse, you can build a scalable, reliable, and observable production system without relying on proprietary platforms.

If you have questions or want help applying this architecture to your use case, feel free to comment!

Location: Kochi, Kerala, India

Comments

Post a Comment

Popular posts from this blog

Welcome to Pods and Prompts

By -
Image
Hi everyone ðŸ‘‹, Welcome to my blog,  Pods and Prompts ! This is the space where I’ll be sharing my thoughts and experiences on Generative AI, Large Language Models (LLMs), and the infrastructure that powers AI applications . Everything I write here reflects my personal views  and is in no way related to  my past or present employers. A little about me: I’m Nithin Anil , an AI/ML engineer with over 13 years in the software industry . Over the years, I’ve worked with AWS, Azure, and GCP , built and managed large-scale distributed big data  systems , and designed microservice-based architectures . For the past couple of years, my focus has been on Generative AI applications,  and one of my strengths is managing infrastructure for highly available, low-latency AI systems . I’m also deeply experienced in Kubernetes and Infrastructure as Code . System design for millisecond or even microsecond latency is one of my favourite challenges. Beyond work, I’m pass...
Read more

Semantic caching for LLM Applications and AI Agents

By -
Image
Caching is one of the easiest ways to speed up applications and control cost. But LLM based systems don’t work well with traditional caching because users phrase the same idea in many different ways. This means most queries turn into cache misses. Still, caching is important. LLM agents can take time to run, and inference is expensive. A smarter caching method is needed. Why Traditional Caching Fails Works on exact matching. Natural language rarely matches exactly. Same intent written differently becomes a cache miss. Result: almost no benefit for LLM workloads. What Semantic caching Does Focuses on meaning instead of exact text. Convert each query into an embedding. Store embeddings in a vector database like Redis, Qdrant, or Milvus. Add TTL to control freshness. For each new query: Convert to embedding. Run similarity search. If similar enough, return cached output. Otherwise, run the agent. Redis Code Sample for Semantic caching Belo...
Read more