Inference Grid

Appearance

Model Selector ​

The Inference Grid supports a variety of models. When making a request, you can use either specify a list of models to allow or specify a set of capabilities that you need. In both cases, our routing engine will find you the best model for the best price.

Classic ​

The classic way to select your model is to specify one or more pipe-delimited model names as shown below:

python
model_name = "gpt-4o|claude-3.5"

You can find the list of models available through the Inference Grid on the website, along with a "pin" icon that indicates whether they are verified. If a model isn't pinned, you can still use it, it just means we haven't gotten around to verifying their performance yet.

Optimized ​

Alternatively, you can use the magic router to automatically select the best model. Instead of specifying a specific model ID, you can simply declare your requirements and let us make sure that you always get the best model for the best price.

For example, you can pass the following JSON string to the model parameter in the OpenAI-compatible API:

python
model_name = json.dumps({
   "flag_constraints": {
      "required": ["vision", "uncensored"]
    },
})

This will match any model that supports vision and is uncensored.

Configuration ​

Here are the configuration options you can use to specify your requirements.

  • flags: You can specify the following flags to indicate capabilities that you require.
    • vision: Whether the model has vision capabilities.
    • tool-use: Whether the model supports tool use.
    • uncensored: Whether the model is uncensored.

When providers join the Inference Grid, we run validators to ensure that they support the capabilities that they claim to provide and run benchmarks to classify them into tiers.