Routing best practices#

Practical patterns for Routing Rules after providers are connected. Use these examples with Add a routing rule and verify in Traces.

• AI Access — Allowed providers — Target and fallback providers must be allowed for the caller’s virtual key, or routing and failover still fail after a rule matches.
• Traces — Confirm the applied routing rule and the provider/model used for each request.
• Architecture Overview — Providers and routers — When to add providers and when a routing rule is required.

Request types for Chat and API#

The gateway distinguishes two chat request types in Rule Builder and CEL Expression Preview:

Choose the condition that matches your clients:

• Chat only — request_type == "chat_completion_stream". Step-by-step: Add a single rule.

• Chat and non-streaming API clients — request_type == "chat_completion" || request_type == "chat_completion_stream". Step-by-step: Combine conditions with OR.

A condition of only Request type = Chat Completion matches non-streaming API requests only. It does not match OptScale Chat.

Configure a target provider with a fallback#

Use this pattern when one provider should handle matching requests and another provider should take over if the target provider cannot process the request. The gateway first routes requests to the target provider and then tries the configured fallback provider if necessary.

Before you start

• Two Active providers with the required models enabled.
• Both providers (and their models) assigned to the test user in AI Access — Allowed providers. Otherwise, requests cannot be routed to the fallback provider during failover.

Navigate to Providers → Routing Rules, then click + ADD. Configure the rule using the values shown in the following example.

Create the rule by following Add a routing rule, then verify it in Traces. To apply the rule only to Chat requests, set Request type to Chat Completion Stream in Rule Builder.

Weighted split across providers#

Use this pattern to distribute matching traffic across two (or more) provider/model pairs by weight. The gateway picks one target per request using a weighted random choice—not a sticky session. Target weights in the create form must sum to 1.

Before you start

• Two Active providers with the shared model enabled (for example gpt-4o on both).
• Both providers (and their models) assigned to the test user in AI Access — Allowed providers. Otherwise, requests cannot be routed to the fallback provider during failover.
• Optional fallback: configure a separate fallback destination for when the selected target fails. A fallback is not a third weight bucket.

Notes

• In Rule Builder, Model = gpt-4o matches the model name part of the request.
• Include both chat request types if you want Chat and non-streaming API clients—see Request types for Chat and API.

Create the rule by following Add a routing rule. Use + ADD TARGET to configure the second target. Then verify the rule in Traces. Over many requests, you should see traffic distributed across both targets; each individual request is routed to only one target.

For Chat-only traffic with a model filter, start with Add a single rule, then add + Add rule → Model = gpt-4o in the same AND group. The resulting expression is request_type == "chat_completion_stream" && model == "gpt-4o".

Prefer cheaper model, same provider#

Use when users select an expensive model and you want a cheaper default on the same provider connection.

Fall back to the original premium model if the cheaper path fails. Create with Add a routing rule, then verify.

Employee exception#

Use when one person must always hit a specific provider (VIP access or debugging).

Keep employee rules narrow. Disable when finished instead of deleting if you may reuse them.

Multi-provider high availability#

Use when you want ordered recovery across vendors. Fallbacks are failure recovery, not a traffic split—use weights on Targets only when you need a 70/30-style distribution (see Weighted split).

Same pattern as Configure a target provider with a fallback, with additional fallback rows via + ADD FALLBACK.

Migrate traffic by incoming provider#

Use this pattern when clients send requests to Provider A, but you want the gateway to route matching traffic to Provider B without requiring clients to change their configuration.

Create the rule by following Add a routing rule, then verify it in Traces. Requests that match the conditions are routed to Provider B. If Provider B cannot process a request, the gateway retries it using Provider A.

Match a model family with regex#

Use this pattern when multiple model IDs share a common prefix and an exact model match is too restrictive. In Rule Builder, use the regex operator for the Model field.

Create the rule by following Add a routing rule, then verify it in Traces.

Use an exact model match whenever possible. Use a regular expression only when you need to match multiple related model IDs.

Canary rollout on a team#

Use this pattern to test a new provider with a limited blast radius. Requests are distributed by weight; the selected target is determined independently for each request (not a sticky session).

Create the rule by following Add a routing rule. Use + ADD TARGET to configure the second target, then verify the rule in Traces.

Start with a pilot team. Expand the rule to the Organization scope only after validating the rollout in Traces and confirming that the required providers, models, AI Access settings, and budgets are configured correctly. See also Weighted split across providers.