AI / ML
LLM Provider Fallbacks Need a Product Policy
By Vivek Jalondhara · 5 min read
Published
Design OpenRouter fallback behavior around compatible schemas, approved providers, bounded attempts, and draft integrity.
A fallback can turn an unavailable model endpoint into a completed request. It can also change which system receives a document, which capabilities are available, and which extraction behavior a reviewer sees. Those changes belong in the application's design rather than an undocumented default.
This illustrative Sheet Parser workflow considers a vendor document. Sheet Parser's use of OpenRouter is confirmed; the routing policy, attempt records, and failure decisions below are proposed design examples rather than production configuration.
Separate a provider change from a model change
A model can be served through different providers. Changing provider seeks a response for the selected model; changing model introduces a different extraction system. OpenRouter exposes provider preferences and a separate models parameter for model fallbacks. Configure these decisions independently.
For a sheet with merged cells, an alternate provider of an evaluated model may be an acceptable recovery path. A different model that has never seen the evaluation set should not inherit the same acceptance assumptions. Completing the HTTP request does not establish that either path produced correct product rows.
Make eligibility a hard boundary
Choose an approved set of providers for the document category before defining preferences within that set. In OpenRouter, order expresses provider preference, while only restricts the eligible providers. Setting require_parameters to true restricts routing to providers that support the supplied parameters. These fields address different constraints and should be reviewed together.
Structured-output support can differ by endpoint. Supply the required response schema and verify compatibility for every eligible route. A policy that requires structured output should fail visibly when no compatible route remains. Quietly dropping the schema requirement to increase availability changes the contract that downstream parsing depends on.
Classify failures before repeating work
Imagine the first request encounters a temporary provider outage. A permitted fallback may be useful. An invalid API credential needs an operational correction; repeated requests will not fix it. A document exceeding the application's size limit needs a different import decision. A schema-valid response with contradictory currencies needs validation or review, rather than an outage label.
Routing services may trigger fallback for error classes broader than the product wants. OpenRouter's model-fallback documentation includes several such triggers. If exact error classification matters, use an application-controlled sequence and configure routing accordingly. Keep provider recovery distinct from asking another model to reinterpret the same ambiguous content.
Budget the whole import attempt
Set an overall deadline and a bounded attempt count for the import job. Each network request consumes the remaining budget. Layering an SDK retry loop, routing fallback, and worker retry can multiply requests unexpectedly, so document which layer owns each kind of recovery and observe the resulting attempts.
A local timeout means the caller stopped waiting; it does not prove remote computation stopped. A delayed response must not overwrite a newer accepted result. Give each extraction attempt an identity, store drafts separately, and condition any job transition on the attempt still being current. Cancelling a job must also prevent later responses from promoting it.
Keep one coherent draft per attempt
Suppose one model returns the first part of a sheet before the connection fails. Appending the remainder from another attempt may create duplicated or mismatched rows. Unless the import has explicit, stable chunk boundaries, discard the incomplete candidate and preserve it only as restricted diagnostic evidence when needed.
Record enough information to explain a completed draft: import ID, attempt ID, schema version, requested routing policy, returned model identity, and validation result. Capture provider identity when the service exposes it. Avoid logging the entire vendor document by default. Review should operate on one complete candidate with identifiable provenance.
Test policy failures as well as successful recovery
Exercise an unavailable preferred provider, no eligible compatible endpoint, malformed output, an expired deadline, and a late first response arriving after a second attempt. Verify that an unapproved route is never introduced merely to obtain a successful response. Check that accepted catalog records remain unchanged during these extraction failures.
A narrower provider set can reduce recovery options. That is an explicit availability tradeoff, not automatically a configuration mistake. Widen the set only after evaluating compatibility and document-handling requirements. The useful outcome is a recoverable, explainable import whose draft still passes the same checks regardless of the route taken.
Build provider preferences from reviewed server configuration
This helper produces the provider fragment of an OpenRouter request. Pass real, reviewed provider slugs from trusted deployment configuration. The complete request must also include the selected model, messages, and required response_format schema.
Illustrative example · JavaScript · Simplified for explanation
function providerPolicy(approvedSlugs) {
if (!Array.isArray(approvedSlugs) || approvedSlugs.length === 0 ||
approvedSlugs.some(slug => typeof slug !== "string" || !slug.trim())) {
throw new Error("Configure at least one approved provider");
}
const approved = [...new Set(approvedSlugs.map(slug => slug.trim()))];
return {
only: approved,
order: [...approved],
allow_fallbacks: true,
require_parameters: true,
};
}- Check approved route capabilities
- Start bounded extraction attempt
- Validate one complete draft
- Review before catalog acceptance
Conceptual flow for the example above.
Tradeoff to keep in mind
This fragment constrains provider eligibility and preference; it does not validate provider slugs against live metadata, impose an overall timeout, or verify extraction accuracy. Keep the policy on the server and handle an empty set of available routes explicitly.