Routing Rules#

Open Providers → Routing Rules to control how matching Chat and API requests reach provider and model destinations. Define scope, priority, conditions, weighted targets, and fallbacks after at least one provider is Active.

Use this page when you need to:

• Prefer a primary provider/model with automatic failover.
• Split traffic across providers by weight.
• Scope routing to an organization, team, or employee.
• Verify applied rules in Traces.

Next: start with When you need a routing rule and How routing rules work, then Add a routing rule. For worked examples, see Routing best practices.

Routing Rules tab#

Use the Routing Rules tab to review and manage rules. The table lists rules with filters and search by enabled status, scope, provider, and model. Use + ADD to create a rule. Click a rule name to open its detail page.

Next: When you need a routing rule → Add a routing rule → Verify routing → Routing checklist. Inspect results on Traces.

Routing overview#

When you need routing rules#

Create a routing rule when you need to:

• Route matching requests to a specific provider and model.
• Configure automatic fallbacks if the primary provider or model is unavailable.
• Distribute traffic across multiple provider/model pairs using weighted load balancing.
• Apply different routing behavior based on scope (organization, team, or employee) or request attributes, such as model or request type.
• Change how requests are routed without modifying applications or API clients.

You do not need a routing rule when requests should always use the provider and model selected by the user or application. In this case, users choose a provider and model in Chat, while API clients specify the model in their requests. The request is then processed using the selected provider, subject to AI Access — Allowed providers and the provider's enabled models.

Next: How routing rules work, then Add a routing rule.

How routing rules work#

Routing rules are configured by administrators on the Providers → Routing Rules tab. After a rule is created, the AI Gateway evaluates it for each matching Chat or API request before sending the request to an AI provider. Field meanings match the Add routing rule form.

The routing workflow is:

Request reaches the AI Gateway
    ↓
Identify the requesting user, team, and organization
    ↓
Evaluate applicable routing rules by scope
    ↓
Evaluate rules by priority
    ↓
Select the first rule whose conditions match
    ↓
Select a target according to its configured weight
    ↓
Send the request to the selected provider and model
    ↓
Use a fallback target if the request fails

Rule evaluation order#

The AI Gateway evaluates rules in the following order:

1. Scope — More specific rules are evaluated before broader rules. For example, an employee-level rule is evaluated before a team-level or organization-level rule (employee → team → organization).

2. Priority — Within the same scope, enabled rules are tried in ascending priority order (lower number first; 0 before 10).

3. Conditions — The request must satisfy the conditions defined in Rule Builder (CEL Expression Preview shows the generated expression). Conditions can match request attributes such as Request type, Provider, Model, or Team.

4. Targets — When a rule matches, the gateway selects one of its targets. If several targets are configured, their weights determine how traffic is distributed (weights must sum to 1).

5. Fallbacks — If the selected target cannot process the request, the gateway tries the configured fallback targets in order.

The first matching rule is applied. After a rule matches, rules with lower precedence are not evaluated.

If no routing rule matches, the gateway keeps the provider and model selected by the user or specified by the API client.

Access checks#

Routing rules do not override provider access settings. After a destination is selected, the gateway verifies that:

• the provider is Active;
• the selected model is enabled;
• the requesting user, team, or agent is allowed to use the provider via AI Access — Allowed providers;
• applicable budgets and rate limits have not been exceeded.

If these requirements are not met, the request is rejected or a configured fallback is attempted.

Routing lifecycle#

Before you begin#

Complete these steps before creating a routing rule:

1. At least one Active provider exists and passes health checks.
2. Required models are enabled on each provider detail page.
3. Users or applications that send traffic have target providers assigned in AI Access — Allowed providers.
4. For fallback or load balancing, register and enable two or more providers that expose the models you plan to route to.

Next: Add a routing rule, then verify routing and complete the routing checklist.

Add routing rule#

Complete the prerequisites first. For a worked example, see Routing Best Practices.

1. Open Providers in the main menu, go to the Routing Rules tab, and click + ADD.

2. Configure the rule settings:

• Rule name — Unique label (for example chat-openai-primary).
• Description — Optional note for operators.
• Enable Rule — Turn on to evaluate the rule for matching requests; turn off to keep the rule without applying it.
• Scope — Select Organization to apply the rule to all organization traffic, or Team or Employee to limit the rule to a specific scope. For Team or Employee, specify the team or employee in Scope ID. See Team override beats organization default.
• Priority — Numeric value; lower numbers are evaluated first within the same scope (0 before 10). Use distinct values so overrides are unambiguous.

Team override beats organization default#

Use when the organization has a broad rule and one team needs different targets. More specific Scope is evaluated first (employee → team → organization). Within a scope, lower Priority numbers run earlier. See Rule evaluation order.

Keep team and employee overrides few and intentional. Prefer Disable over Delete when pausing an override. For a single-user exception, see Employee exception.

3. Define the rule conditions in Rule Builder.

Use Rule Builder to define which AI Gateway requests the routing rule matches. Add conditions to filter requests by request type, model, team, and other supported attributes.

• Understand the logic

  • AND — every rule in the group must match.
  • OR — at least one rule in the group must match.
  • Use nested groups to combine filters—for example, (provider == "Provider A" && model == "gpt-4o") || team == "Engineering". See Use a nested rule group.

• Build the condition

  1. Choose AND or OR for the current group.
  2. (Optional) Click Add group to create a nested condition group when you need to combine multiple conditions using different AND/OR operators.
  3. Click Add rule, then select the field, operator, and value.
  4. Repeat until the rule matches only the traffic you intend.

• Verify the result

  • Check CEL Expression Preview at the bottom of Rule Builder. It shows the Common Expression Language (CEL) expression generated from your rules.
  • If no rules are added, the preview shows No expression. See Avoid empty conditions.
  • See Request types for Chat and API.
  • For step-by-step Rule Builder walkthroughs, see:
    • Add a single rule
    • Combine conditions with OR
    • Use a nested rule group

  For full routing patterns (targets, fallbacks, scope, and weights), see Routing best practices.

Avoid empty conditions#

An empty Rule Builder typically matches all requests in that scope—including embeddings, images, speech, and other types, not only chat.

• Always set at least a Request type to Chat Completion Stream when OptScale Chat must match. See Request types for Chat and API.
• Avoid organization-wide “always match” rules unless you intentionally want every request type rewritten.

4. Configure the preferred destinations in Targets.

Targets define where matching requests are sent.

• Select a Provider and Model (only enabled models appear).
• Specify a Weight value. For a single target, use 1. For multiple targets, weights must sum to 1 (for example 0.7 and 0.3). The gateway picks one target per request by weight (not a sticky session). See Weighted split across providers.
• Use + ADD TARGET to add additional destinations for load balancing.

5. Configure fallback destinations in Fallbacks (optional but recommended).

Fallbacks define where requests are retried if a target cannot process them.

• Select a fallback Provider and Model.
• Use + ADD FALLBACK to add additional fallback entries. The gateway tries them in order until a request succeeds or no fallbacks remain.

6. Click SAVE to create the routing rule.

Optional actions:

• Click CANCEL to return to the Routing Rules tab without saving changes.

Example: Add a single rule#

This walkthrough shows how to add one condition in Rule Builder. It does not configure Targets or Fallbacks—see Routing best practices for complete rule examples.

1. In Rule Builder, confirm AND is selected for the top-level group.
2. Click + Add rule.

3. Fill in the fields:

   • Field = Request type
   • Operator = =
   • Value = chat_completion_stream

4. Check CEL Expression Preview. It should look like:

request_type == "chat_completion_stream"

To match a specific model as well, click + Add rule again in the same AND group and set Model to the model name you need.

Example: Combine conditions with OR#

This walkthrough shows how to combine two conditions with OR. For when to match Chat only vs Chat and API clients, see Request types for Chat and API.

1. In Rule Builder, click OR for the top-level group.
2. Click + Add rule.

3. Fill in the fields:

   • Field = Request type
   • Operator = =
   • Value = chat_completion_stream

4. Click + Add rule again.

5. Fill in the fields:

   • Field = Request type
   • Operator = =
   • Value = chat_completion

6. Check CEL Expression Preview. It should look like:

request_type == "chat_completion_stream" || request_type == "chat_completion"

Example: Use a nested rule group#

This walkthrough shows how to nest an AND group inside an OR group. The pattern below is for learning Rule Builder structure only. To route different targets for one team, prefer Scope = Team—see Team override beats organization default.

Target condition:

(provider is Provider A AND model is gpt-4o) OR team is Engineering

1. In Rule Builder, click OR for the top-level group.
2. Click + Add rule group.
3. In the nested group, confirm AND is selected.
4. Click + Add rule.

5. Fill in the fields:

   • Field = Provider
   • Operator = =
   • Value = Provider A

6. Click + Add rule again inside the same nested group.

7. Fill in the fields:

   • Field = Model
   • Operator = =
   • Value = gpt-4o

8. Go back to the top-level group and click + Add rule.

9. Fill in the fields:

   • Field = Team
   • Operator = =
   • Value = Engineering

10. Check CEL Expression Preview. It should look like:

(provider == "Provider A" && model == "gpt-4o") || team == "Engineering"

For nested request type conditions used in production patterns, see Weighted split across providers and Migrate traffic by incoming provider.

Next: Verify routing.

Verify routing#

Verify that the routing rule is configured correctly and matching traffic follows the intended path end to end:

1. Navigate to Providers menu → Routing Rules tab. In the routing rules table, verify that the rule is Enabled and that Scope, Priority, and target Provider / Model match your design.

2. Click the rule name to open the routing rule detail page. Verify that the Condition expression matches the traffic you intend to route, and that Targets (and Fallbacks, if configured) list the expected provider and model destinations.

3. Navigate to AI Access menu. Find the test user and review the Allowed providers column. Verify that the user can access every target and fallback Provider and Model configured in the rule. See AI Access — Allowed providers.

4. Ask the test user to sign in to Chat using their credentials. Open the Chat URL for your deployment. Use a provider and model that match the rule Conditions, then complete at least a few test requests.

5. Navigate to Analytics menu → Traces. Open a matching test request and verify in Request details that the applied routing rule name, Provider, and Model match the rule Targets (or a configured Fallback if the primary target failed during testing).

6. Optional: temporarily disable the primary target provider or model, then send another matching request. Verify in Traces that subsequent requests use the expected fallback Provider and Model.

For worked examples with sample values, see Routing best practices.

Next: Routing checklist.

Routing checklist#

Before you rely on a rule in production:

1. In Rule Builder, set Request type to Chat Completion Stream to match OptScale Chat requests. See Request types for Chat and API.
2. Set Priority so lower numbers run earlier within the same scope—see Rule evaluation order.
3. Ensure target weights sum to 1 when multiple targets are configured.
4. Confirm that the user has access to every target and fallback provider/model. See AI Access — Allowed providers.
5. Prefer Disable over Delete when pausing a rule you may reuse.
6. After save, send a Chat or API request and confirm provider/model in Traces.

Next: Troubleshooting.

Troubleshooting#

If routing does not apply as expected

• Confirm the rule is Enabled and SAVE completed without validation errors.
• Confirm Scope and Scope ID match the user or team sending traffic.
• Confirm Conditions match real requests—compare CEL Expression Preview with the request type, model, and attributes shown in Traces.
• Confirm no rule with a lower priority number (higher precedence) matches the same traffic first within the same scope. Remember scope is evaluated before priority (employee → team → organization).
• Confirm target models are enabled on the provider detail page and providers are assigned in AI Access.
• Confirm target weights sum to 1 when multiple targets are configured.
• Allow a short interval after saving, then send new requests—existing in-flight requests are not re-routed retroactively.

Routing rule details#

Routing rule detail page#

Click a rule name in the Routing Rules table to open the routing rule detail page. Use EDIT to modify the rule configuration and REFRESH to reload the page.

Summary cards at the top of the page display the enabled status, number of targets, priority, and creation date.

The Summary section contains rule metadata, including the rule name, identifier, description, enabled status, scope, scope identifier, priority, creation date, and last update date. A copy control is available for the rule identifier.

The Condition section displays the matching expression that determines when the rule applies. A copy control is available for the condition expression.

The Targets table lists preferred provider and model destinations for matching traffic, including the routing weight assigned to each target. A footer below the table displays the total number of targets.

The Fallbacks section lists alternative provider and model destinations used when preferred targets are unavailable. When no fallbacks are configured, the section displays No fallbacks.

Edit routing rule#

1. Open the routing rule for editing using one of the following options:

• On the Routing Rules tab, click Edit in the rule row.
• On the routing rule detail page, click EDIT.

2. Review the pre-populated configuration and update the required settings. See the detailed field descriptions at add a routing rule.

3. Click SAVE to apply the changes.

Optional actions:

• Click CANCEL to return without saving changes.
• Click DELETE to start the routing rule removal process. See Delete routing rule for step-by-step instructions.

Delete routing rule#

1. Open Providers and select the Routing Rules tab.

2. Remove the rule using one of the following options:

• Use the Delete row action on the rule in the table, or

• Click the rule name to open the detail page, click EDIT, then click DELETE on the edit form.

3. Confirm the deletion when prompted.

The rule is removed from the organization and is no longer applied to matching requests.