How to route requests across LLM providers with a gateway

By the end of this guide you will have an LLM gateway sitting between your application and its model providers, so one OpenAI-compatible endpoint reaches several providers, requests fail over to a backup when a provider errors, and each key has a spend limit. The job is done by an AI gateway or router: you change your SDK's base URL and key, then configure models, fallbacks and budgets in one place.

If you want to set up an LLM gateway, the main decision is whether to use a hosted service, self-host an open-source one, or pick a regional option for data residency. This guide covers each route and compares seven tools from our Developer platforms category.

Key takeaways

• A gateway gives you one OpenAI-compatible endpoint for many providers. Most setups need only a base URL and API key change in your existing SDK.
• The main decision is hosted versus self-hosted: OpenRouter, Eden AI, EUrouter, FastRouter.ai and Helicone run as services, while LiteLLM and Bifrost are open source and run on your own infrastructure.
• Check the fee model. Several hosted gateways charge a platform fee on credit purchases or a markup on provider cost, while others pass through list prices.
• Test failover on purpose before you rely on it. A fallback chain you never triggered is a guess.

What you need before you start

• A working LLM integration. Ideally one that already uses the OpenAI SDK or a similar client, so the change is small.
• Provider accounts or keys. Either your own keys for each provider, or a gateway that sells credits across providers.
• A list of requirements. Which models you need, whether data must stay in a region, your error budget and your spend limits.
• A staging environment. Test failover and limits there before production.
• Time. A hosted gateway can be switched on in under an hour. Self-hosting and tuning routing rules take longer.
• Difficulty. Moderate. You change client configuration, and for self-hosted tools you also run a service and a database.

Which kind of AI tool fits this job

Pick by where the gateway runs and what matters most to you.

A hosted marketplace-style gateway. OpenRouter offers one API key for 500+ models across 80+ providers on paid plans, with automatic fallback to another provider on errors and a free plan. Credits are bought up front, and inference is passed through at the provider's list price, with a fee on credit purchases on paid plans. Eden AI puts about 1,000 models behind one OpenAI-compatible API, routes by cost, speed or latency, and has an EU endpoint. FastRouter.ai offers an auto-router, fallbacks, caching, budget caps and a weekly Insights report.

A regional gateway for data residency. EUrouter is an OpenAI-compatible gateway positioned for teams that need EU data residency and GDPR by default, with per-key spend caps and smart routing.

An open-source gateway you run yourself. LiteLLM is an open-source gateway and Python SDK with virtual keys, budgets, rate limits, fallbacks and load balancing across deployments. Bifrost is an open-source gateway from Maxim with automatic fallbacks, semantic caching, virtual keys with budgets and an MCP gateway.

A gateway combined with observability. Helicone is an open-source monitoring platform with an AI Gateway, logging cost, latency and errors for every request, and offering fallbacks and load balancing across more than 100 providers.

Step 1: List your models, providers and requirements

A gateway is easier to configure when you know what you need.

1. List every model your app uses today, and which feature calls each.
2. Write down constraints: data residency, compliance needs, latency limits and monthly budget.
3. Decide which providers should be primary and which are backups for each model.
4. Note any features you rely on, such as streaming, function calling, vision or embeddings, and check that the gateway passes them through.

Check it worked by having a one-page table of feature, primary model, backup model and constraint.

Step 2: Choose a gateway

You end this step with one gateway and a decision on hosting.

1. Decide between hosted and self-hosted. Self-hosted LiteLLM means you run and secure the gateway yourself and need Postgres for budgets and virtual keys. Bifrost's OSS edition is self-managed with community support.
2. Compare fee models. OpenRouter charges a 5.5% fee on credit purchases on Standard and 8% on Business. Eden AI charges a 5.5% platform fee when you buy credits and passes through provider prices. EUrouter adds a markup on provider cost that falls as the plan rises.
3. Check data location. EUrouter and Eden AI both describe EU routing. Hosted gateways pass your requests through their service, so read each vendor's data terms.
4. Use the comparison table below for pricing and stated limits.

Check it worked by confirming the gateway supports your required models and your hosting constraint.

Step 3: Point your SDK at the gateway

The aim is one successful request through the gateway.

1. Create an account or deploy the service, and generate a gateway API key.
2. In your existing OpenAI-compatible client, change the base URL and API key. Helicone's listing describes this as a one-line change, and EUrouter and Eden AI describe the same pattern.
3. Send a simple request to a model you already use and compare the answer with your direct call.
4. Check the response includes what you need, such as usage or cost. Eden AI lists per-request cost in every response.

Check it worked by seeing the request in the gateway's log or dashboard and getting an equivalent answer.

Step 4: Add providers and your own keys

Now connect the providers you actually plan to use.

1. Add provider credentials where the gateway supports bring-your-own-key, or buy credits for a managed account. Check whether bring-your-own-key traffic carries a platform fee. OpenRouter lists a monthly fee-free allowance and a fee above it.
2. Create virtual or scoped keys per environment, team or feature. LiteLLM and Bifrost list virtual keys with budgets and rate limits.
3. Restrict which models each key may call, if the gateway allows it. Eden AI lists model allow-lists.
4. Keep provider keys out of your application code and inside the gateway.

Check it worked by calling each configured provider once through the gateway.

Step 5: Configure fallbacks and routing rules

This is the reason most teams add a gateway.

1. For each primary model, set an ordered fallback chain of backups. OpenRouter, Eden AI, FastRouter.ai, LiteLLM, Bifrost and Helicone all list fallbacks.
2. Decide what triggers a fallback: provider errors, rate limits, timeouts.
3. Add routing rules if you need them: cheapest first, fastest first, or regional only. Eden AI routes by cost, speed or latency, and OpenRouter has routing variants for throughput and lowest price. FastRouter.ai has an auto-router that picks by cost, latency or quality.
4. Make sure backups produce acceptable quality. A backup that returns worse answers is a trade-off you should choose on purpose. See how to evaluate LLM outputs before you ship.

Check it worked by reviewing the fallback order for each feature and confirming every backup passed your test prompts.

Step 6: Set budgets, rate limits and caching

Control spend and load centrally.

1. Set a spend cap per key. EUrouter lists per-key caps that reset daily, weekly or monthly, and Eden AI lists budgets per API key. EUrouter's keys that hit a cap return a 402 Payment Required response, so make sure your app handles it.
2. Set rate limits so one feature cannot starve the others.
3. Turn on caching where identical requests repeat. Bifrost lists simple and semantic caching, and Helicone lists response caching, as does FastRouter.ai on plans above Starter. For more savings, see how to reduce your LLM API costs.
4. Add alerts for spend and error rate where the tool provides them.

Check it worked by hitting a test cap in staging and seeing the request blocked as expected.

Step 7: Test failover, then monitor

Prove the safety net works before you need it.

1. In staging, simulate a provider failure by using an invalid key or a blocked model, and confirm the request moves to the backup.
2. Send a burst of requests to test rate-limit behaviour.
3. Watch logs for the first week in production: which fallbacks fire, latency per provider and cost per request. Helicone and LiteLLM record cost for each request.
4. Review fallback chains whenever providers retire or change models.

Check it worked by seeing a deliberate failure in staging produce a successful response from the backup, with the event visible in the logs.

Tools for this job compared

Tool	Good fit for	Pricing	Stated limits
OpenRouter	One key for many hosted models, with fallback	Freemium	Free plan: 50 requests a day; fee on credit purchases is 5.5% on Standard and 8% on Business
LiteLLM	A self-hosted gateway with virtual keys and budgets	Open source	You run and secure it; budgets and virtual keys need Postgres; Enterprise pricing not published
Bifrost	An open-source gateway with caching and an MCP gateway	Open source	OSS is self-managed; guardrails, clustering, SSO and audit logs are Enterprise; benchmarks are vendor-reported
Helicone	Request logging plus a gateway with fallbacks	Freemium, paid from $79/mo (checked 24 Oct 2026)	Free tier: 10,000 requests, 1 GB storage, 1 seat; Pro and Team add usage-based charges
Eden AI	Routing by cost, speed or latency, with an EU endpoint	Paid	Prepaid credits on self-serve; higher limits and SLA support need the custom plan
EUrouter	EU data residency and per-key spend caps	Freemium, paid from €39/mo (checked 24 Oct 2026)	Markup on provider cost by plan; Free plan: 10K requests a month and 60 RPM
FastRouter.ai	Auto-routing, caching and budget caps across 200+ models	Freemium, paid from $199/mo (checked 24 Oct 2026)	Starter: 1M requests a month on one key, 7-day logs; response caching and tracing not on Starter

Prices and plan limits change, so confirm the current terms on each vendor's site before you commit.

Common mistakes to avoid

• Never testing the fallback. A chain that has not fired in staging may fail in production. Fix it by breaking a provider on purpose and confirming the backup answers.
• Backups that behave differently. A fallback model may format output differently or answer less well. Fix it by running your test prompts against each backup and choosing them deliberately.
• Ignoring fees. A fee on credits or a markup on provider cost changes your real price. Fix it by modelling your monthly volume against each gateway's fee model before you commit.
• Making the gateway a single point of failure. Everything now depends on one service. Fix it by checking the vendor's uptime commitments, and consider a self-hosted option or a direct-call escape hatch for critical paths.
• Sending sensitive data through without checking terms. Hosted gateways see your prompts and responses. Fix it by reading the data and retention terms, and by choosing a regional or self-hosted option where required.

How we put this together

This is an evidence-based guide built from each tool's listing in our directory, which is compiled from the vendor's own site and pricing page. It is not a review based on using the tools. We don't rank tools, and nobody paid to be included. Some outbound links may be affiliate links; see our disclosure.

Frequently asked questions

Is there a free LLM gateway?

Yes. LiteLLM and Bifrost are open source and free to self-host, though you pay for the servers and the model calls. OpenRouter and Helicone have free tiers with limits, and EUrouter and FastRouter.ai have free plans.

Will a gateway work with my existing OpenAI code?

Usually. OpenRouter, Eden AI, EUrouter, FastRouter.ai, LiteLLM, Bifrost and Helicone all describe OpenAI-compatible APIs, so you change the base URL and key. Check that the features you use, such as streaming or function calling, are supported.

Can I keep my data in the EU?

Some gateways offer it. EUrouter is positioned for EU data residency, and Eden AI lists an EU endpoint that refuses non-EU models. Check the vendor's data terms and the provider behind each model, since residency depends on the full path.

Do I need a gateway or just a fallback in my code?

If you use one or two providers, a few lines of retry and fallback code may be enough. A gateway pays off when you use many models, want central budgets and logs, or have several teams sharing keys.

How long does setup take?

Pointing a client at a hosted gateway is usually the quick part, since it is a base URL and key change. Self-hosting, configuring fallbacks and testing failover take longer.

Compare more options in our Developer platforms category.