Project Overview
MetaDesign Solutions was approached by a leading AI platform to re-architect their cloud infrastructure. The client needed a robust environment to host custom AI/ML models with auto-scaling capabilities to handle unpredictable traffic spikes.
Furthermore, to successfully monetize their services, they required a custom Paywall API Gateway capable of securely managing authentication, rate limiting, and precise usage metering without adding latency to the model inference.
Cloud Infrastructure Architecture
We engineered a resilient, fault-tolerant infrastructure on AWS utilizing Amazon EKS (Kubernetes). Infrastructure as Code (IaC) was implemented using Terraform, ensuring all environments were reproducible and version-controlled.
To optimize costs, we configured Kubernetes Horizontal Pod Autoscalers (HPA) alongside Cluster Autoscaler, specifically targeting GPU node groups. This allowed the infrastructure to scale up instantly during traffic surges and scale down during quiet periods.
Custom Paywall API Gateway
We developed a high-throughput API Gateway using Node.js. To ensure sub-millisecond latency for routing and rate limiting, we integrated a Redis caching layer utilizing token bucket algorithms.
The gateway securely handles API key authentication and asynchronously flushes usage metrics (such as compute time and tokens generated) to a time-series database, integrating seamlessly with their payment processor for usage-based billing.
Have a similar challenge?
Our experts can help you build custom integrations and plugins tailored to your business workflows.
Key Challenges
Challenge 1
Scaling GPU-intensive workloads dynamically based on unpredictable API traffic.
Challenge 2
Developing a low-latency API gateway capable of complex token-based metering and billing.
Challenge 3
Ensuring strict security and isolation for proprietary customer models and data.
Challenge 4
Implementing seamless CI/CD pipelines for rapid deployment of new models.
Results & Outcomes
The new architecture transformed the client's operational capabilities. The highly scalable EKS clusters now efficiently handle over 50 million API requests daily with zero downtime. The custom Paywall API Gateway successfully monetizes the platform's usage with a processing latency of less than 50ms, enabling the client to scale their business securely and profitably.