Model Validation

To ensure that providers are providing quality responses, the Inference Grid implements a validation system for workers and uses it when routing requests. Each of an provider's workers will be validated independently on an ongoing basis.

For each worker, the Inference Grid may submit up to 48 test requests per day. The reputation score attached to these requests will be set arbitrarily high but the invoices will not be paid. For reconciliation purposes, your worker will be notified that an earlier request was a test query after a period of time.

Depending on whether you declare a pinned model ID, the validation process differs slightly:

• Standard. If you do not declare a reserved model ID, then your model is simply validated by checking it's capabilities against what you declared.

• Pinned. If you declare a model ID that's reserved (i.e. a well-known model), then the validation process will also compare your model against other providers who are serving the same model to ensure that you're serving the model that users expect.

This is an ongoing process whose outputs - i.e. model quality, latency, etc. - are used by the relay to decide which provider to route each request to.