Sign in
patterngrowingmedium complexity

Router / Gateway Pattern

Router-Gateway is an AI architecture pattern where a single entrypoint (gateway) receives all requests and a router decides which downstream model, tool, or service should handle each one. The router can use rules, heuristics, or another model to classify intent, risk, latency/cost needs, and required capabilities. This enables multi-model orchestration, cost optimization, and safer handling of diverse workloads behind a unified API. It is especially useful when different tasks require different models, modalities, or infrastructure tiers.

5implementations
2industries
Parent CategoryRetrieval Systems
01

When to Use

  • You need a single unified API for multiple models, tools, or modalities and want to hide this complexity from client applications.
  • Different tasks, tenants, or regions require different models (e.g., code vs. chat vs. vision; EU vs. US data residency).
  • You want to optimize for a mix of cost, latency, and quality by routing to different models based on request characteristics.
  • You plan to frequently experiment with or swap out models and providers without changing client code.
  • You must enforce safety, compliance, or policy-based routing (e.g., content categories, PII handling, jurisdictional rules).
02

When NOT to Use

  • You have a single primary model and a narrow, well-defined use case where routing adds unnecessary complexity and latency.
  • Your traffic volume is low and you do not expect to change models or providers frequently; a direct integration is simpler and sufficient.
  • You lack the observability, metrics, or data needed to evaluate routing decisions; you would be guessing rather than optimizing.
  • Strict regulatory or security constraints require fully isolated, simple pipelines where additional routing logic increases audit complexity.
  • Your team does not have the operational maturity to manage a central gateway (SLAs, incident response, configuration management).
03

Key Components

  • API Gateway / Entrypoint Service
  • Request Normalizer (schema validation, auth, rate limiting)
  • Router Engine (rules-based, ML-based, or hybrid)
  • Policy & Safety Layer (guardrails, PII detection, compliance rules)
  • Model Registry / Capability Catalog
  • Routing Strategies (cost, latency, quality, risk, specialization)
  • Downstream Model Connectors (LLMs, embeddings, vision, speech, etc.)
  • Tool / Service Connectors (search, RAG, transactional APIs, databases)
  • Observability & Telemetry (logging, tracing, metrics, A/B routing)
  • Feedback & Learning Loop (human feedback, auto-labeling, retraining)
Common Tools
NVIDIA LLM Router (NVIDIA-AI-Blueprints/llm-router)LangDB AI Gateway (langdb/ai-gateway)LangChain (LCEL, routers, multi-model chains)LlamaIndex (router query engines, composable graphs)Semantic Router (for intent-based routing in RAG/chatbots)Kong / NGINX / Envoy / Istio (as HTTP/API gateways and service mesh)FastAPI / Express.js / Spring Boot (custom gateway implementations)OpenAI API (multiple models, function calling, moderation)Anthropic, Google Gemini, Azure OpenAI (multi-model backends)Ray Serve / KServe / Sagemaker Endpoints (model serving & routing)Prometheus / Grafana / OpenTelemetry (metrics and tracing)Feature flag systems (LaunchDarkly, ConfigCat, custom config services)
Top Industries
mining3transportation2
04

Best Practices

  • Start with a simple rules-based router (by endpoint, task type, or tenant) before introducing ML-based routing; only add complexity when you have clear metrics and data.
  • Define a clear capability catalog for each model or tool (e.g., languages, max context, modalities, safety level, latency, cost) and base routing decisions on this metadata.
  • Separate gateway concerns (auth, rate limiting, request validation) from routing logic so you can evolve routing without breaking client integrations.
  • Implement explicit safety and compliance checks in the router path (e.g., PII detection, content filters, jurisdiction-based routing) before calling powerful models.
  • Always define fallback strategies: backup models, reduced-capability flows, or safe error messages when the preferred route fails or is overloaded.
05

Common Pitfalls

  • Overcomplicating routing logic too early (e.g., training a router model without enough data) leading to brittle behavior and hard-to-debug failures.
  • Hard-coding model-specific assumptions into client applications instead of keeping them inside the router-gateway, making migrations and experiments painful.
  • Ignoring safety and compliance in routing decisions (e.g., sending high-risk content to models without proper guardrails or regional restrictions).
  • Failing to implement robust fallbacks, causing user-visible outages when a single model or provider has issues.
  • Lack of observability into routing decisions, making it impossible to understand why a request was sent to a particular model or why performance regressed.
06

Learning Resources

tutorialai-gateway Routing Concepts (langdb/ai-gateway)tutorialNVIDIA LLM Router – Route LLM requests to the best modeltutorialLLM Traffic Control: Gateway or Router or ProxytutorialMastering RAG Chatbots: Semantic Router — RAG gatewaytutorialThe Inference Router: A Critical Component in the LLM Ecosystem
07

Example Use Cases

01Customer support platform where the gateway routes simple FAQs to a cheap small model, complex multi-step issues to a larger reasoning model, and account-specific questions to a RAG tool connected to internal systems.
02Enterprise AI assistant that routes legal queries to a high-safety, jurisdiction-specific model, engineering questions to a code-specialized model, and general chit-chat to a low-cost conversational model.
03Multilingual chatbot that routes by detected language and region to models hosted in compliant regions (e.g., EU-only models for EU users) while enforcing data residency policies.
04Content moderation pipeline where the router sends low-risk content to a fast heuristic filter and escalates borderline or high-risk content to a more accurate but slower LLM-based classifier.
05AI writing assistant that routes summarization tasks to a summarization-optimized model, code generation to a code LLM, and image requests to a text-to-image model, all behind a single /generate endpoint.
08

Solutions Using Router / Gateway Pattern

0 FOUND

Router / Gateway Pattern is a technique within Retrieval Systems. Showing solutions from the parent pattern.

No solutions found for this pattern.

Browse all patterns