Rate Limits

OpenModel enforces rate limits to ensure fair usage and platform stability. Understanding how these limits work helps you build applications that handle throttling gracefully.

Rate Limit Dimensions

Rate limits are measured across two dimensions:

A value of 0 means no limit is enforced for that dimension.

Rate Limit Levels

Limits are applied at two independent levels. A request must pass both to proceed:

User-Level Limits

Your rate limits are determined by the Group your account belongs to. Administrators configure RPM and TPM limits per group, allowing different tiers of access for different user segments.

You can check your current group and limits in the Console dashboard.

Channel-Level Limits

Each upstream channel (provider endpoint) also has its own RPM and TPM limits, configured by the platform administrator. These protect the upstream provider from being overwhelmed by too many concurrent requests.

When a channel's rate limit is reached, the gateway may automatically route your request to another available channel for the same model.

How Limits Are Enforced

OpenModel uses a Fixed Window algorithm for rate limiting:

RPM Enforcement

1. Before processing your request, the gateway increments your RPM counter
2. If the counter exceeds your limit, the request is rejected with 429 Too Many Requests
3. The counter resets at the start of each minute window

TPM Enforcement

1. Before processing, the gateway estimates your input token count (using a tokenizer service or a length / 4 heuristic as fallback)
2. This estimate is checked against your TPM limit
3. After the response completes, the counter is corrected with the actual token count from the provider

This means TPM enforcement is approximate for the pre-check but accurate over time.

Rate Limit Response

When you exceed a rate limit, the API returns a 429 Too Many Requests response in the provider-specific format matching the API you are using:

OpenAI format (/v1/responses):

{
  "error": {
    "message": "rate limit exceeded: user RPM limit",
    "type": "rate_limit_error"
  }
}

Anthropic format (/v1/messages):

{
  "type": "error",
  "error": {
    "type": "rate_limit_error",
    "message": "rate limit exceeded: user RPM limit"
  }
}

Gemini format (/v1beta/models/*):

{
  "error": {
    "code": 429,
    "message": "rate limit exceeded: user RPM limit",
    "status": "RESOURCE_EXHAUSTED"
  }
}

Handling Rate Limits

Retry with Exponential Backoff

When you receive a 429 response, wait before retrying. Use exponential backoff with jitter:

import time
import random
from openai import OpenAI, APIStatusError

client = OpenAI(
    base_url="https://api.openmodel.ai/v1",
    api_key="your-api-key",
)

def call_with_backoff(max_retries=5):
    for attempt in range(max_retries):
        try:
            return client.responses.create(
                model="gpt-5.4",
                input="Hello!",
            )
        except APIStatusError as e:
            if e.status_code == 429 and attempt < max_retries - 1:
                delay = (2 ** attempt) + random.random()
                print(f"Rate limited. Retrying in {delay:.1f}s...")
                time.sleep(delay)
            else:
                raise

Spread Requests Over Time

Rather than sending bursts of requests, distribute them evenly across the minute window. For example, if your RPM limit is 60, aim for roughly 1 request per second rather than 60 requests in the first few seconds.

import asyncio
from openai import AsyncOpenAI

client = AsyncOpenAI(
    base_url="https://api.openmodel.ai/v1",
    api_key="your-api-key",
)

async def process_batch(prompts: list[str], rps: float = 1.0):
    """Process prompts at a controlled rate (requests per second)."""
    results = []
    interval = 1.0 / rps
    for prompt in prompts:
        result = await client.responses.create(
            model="gpt-5.4",
            input=prompt,
        )
        results.append(result)
        await asyncio.sleep(interval)
    return results

Use Streaming for Better Throughput

Streaming responses begin returning data as soon as the first token is generated. While the total request duration is similar, the perceived latency is significantly lower, and you can start processing output immediately.

Implement a Request Queue

For high-throughput applications, use a request queue with concurrency control to stay within your rate limits:

import asyncio
from openai import AsyncOpenAI

client = AsyncOpenAI(
    base_url="https://api.openmodel.ai/v1",
    api_key="your-api-key",
)

async def process_with_concurrency(
    prompts: list[str],
    max_concurrent: int = 5
):
    """Process prompts with bounded concurrency."""
    semaphore = asyncio.Semaphore(max_concurrent)

    async def limited_request(prompt: str):
        async with semaphore:
            return await client.responses.create(
                model="gpt-5.4",
                input=prompt,
            )

    return await asyncio.gather(
        *[limited_request(p) for p in prompts]
    )

Monitoring Your Usage

The Console dashboard provides real-time visibility into your API usage:

• Dashboard — View request counts, token usage, and spend trends
• Usage — Per-API-key usage breakdown with date range filters
• Logs — Detailed consumption logs for every request

Monitor these metrics to understand your usage patterns and plan your rate limit needs accordingly.

Best Practices

• Design for rate limits from the start — Don't assume unlimited throughput. Build queuing and backoff into your application architecture.
• Use separate API keys for different services or environments to isolate usage.
• Cache responses when possible to reduce unnecessary API calls.
• Batch operations where the API supports it, rather than making many small requests.
• Monitor and alert — Set up monitoring on your 429 error rate to detect when you are approaching limits.