Token Consumption Based Rate Limiting Policies

Stop guessing at your AI bill — cap it precisely, by the token

API-call rate limits measure how often clients access AI Gateway, not the cost. A single request to a large-context model can consume thousands of tokens while another consumes few; counting requests alone won’t prevent runaway LLM spend.
AI Gateway 1.7 lets you enforce rate-limit policies by token consumption, tracked per tenant, AI provider, model, and token group levels, so the limit you enforce reflects actual cost.

Token-based limits: Set limit.type: total_token on a rate limit rule to cap usage by tokens consumed per hour or per day instead of request count.
Example: Creating a token-based rate limit rule

POST /api/v2/aig/ratelimits
{
  "name": "rate limit rule for total token",
  "criteria": {"apply_on": "ai"},
  "limit": {
    "type": "total_token",
    "requests": 100,
    "unit": "hour"
  }
}

AI Gateway now deploys on Microsoft Azure (Beta)

Deploy AI Gateway in Microsoft Azure to place it closer to your Azure workloads

AI Gateway 1.7 adds an Azure deployment path, aligning appliance provisioning, image hardening, and enrollment with the existing AWS and GCP offerings.

UX & Configuration Boosts

Automated AWS bootstrap for AI Gateway appliances

Deploy an AI Gateway appliance on AWS without interacting with the enrollment screen

AI Gateway 1.7 introduces AWS Bootstrap, which points the appliance to a secret in AWS Secrets Manager that contains the enrollment token and, optionally, DLP and AI Guardrails configuration. The appliance enrolls and configures itself on first boot, and the TUI detects an active bootstrap and skips the manual token prompt.

Old way	New way	What this means to you
You had to boot each instance, open the TUI, paste the enrollment token, and configure DLP and AI Guardrails certificates by hand.	You store the enrollment token and optional DLP or AI Guardrails configuration once in AWS Secrets Manager. The appliance enrolls and configures itself on first boot.	You can deploy AI Gateway appliances on AWS faster, with fewer manual steps and more repeatable automation.

New “Allow” terminal action for AI Gateway policies

Permit known-good traffic explicitly rather than only blocking or alerting on malicious traffic

AI Gateway 1.7 adds Allow as a terminal action for policies. When an Allow rule matches, AI Gateway lets trusted traffic through and stops evaluating lower rules.

Old way	New way	What this means to you
You could monitor, block, or replace a transaction, but you could not definitively allow it and stop evaluating lower rules.	You can use Allow as a terminal action. When a rule matches, AI Gateway stops policy evaluation, just like RTP policies.	You can place trusted exceptions at the top of your policy and avoid rechecking them against every rule below.

Single endpoint for multi-LLM routing

One AI Gateway address for all providers. Route to any configured LLM provider from one endpoint.

AI Gateway 1.7 introduces single-endpoint multi-LLM routing. Applications and agents send AI traffic to a single AI Gateway IP or FQDN and identify the target provider with a request-header slug. AI Gateway uses that slug to route requests, reducing hardcoded endpoint mappings, simplifying provider changes, and easing integration maintenance.

Old way	New way	What this means to you
You configured a separate AI Gateway endpoint for each AI provider in every app or agent.	You point every app or agent to one AI Gateway endpoint and send the provider slug in the request header.	You can add or switch AI providers faster without updating endpoint settings across every app.

AI Gateway Release Notes Version - 1.7