Model Compare

​

How it works

1. Fan-out: All requested models are called concurrently. The total wall-clock time is roughly that of the slowest model, not the sum of all models.
2. Error isolation: If a single model fails or times out (hard timeout of 120s), the others continue unaffected. Partial results are returned with a partial: true flag.
3. Synthesis (default): After all models respond, a separate comparison LLM analyzes the responses and produces a structured evaluation covering accuracy, completeness, clarity, and a recommendation.
4. Skip synthesis (optional): By setting skip_comparison: true, you can skip the synthesis step and receive only the raw model outputs. This is useful for parallel streaming UIs that perform their own comparison.
5. Rate limiting and Billing: The entire comparison counts as a single request against your rate limits (RPM/RPD). However, billing tracks each model call plus the comparison call as separate usage events (N+1 events).
6. Streaming: Two streaming modes are available by setting stream: true. With synthesis enabled, fan-out is non-streaming, but the final comparison text is streamed token-by-token. If skip_comparison: true is set, each fan-out model streams its tokens in real-time concurrently, tagged by model name.

​

Basic request

curl https://api.meshapi.ai/v1/chat/compare \
  -H "Authorization: Bearer <YOUR_RSK_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "models": [
      "openai/gpt-4o-mini",
      "anthropic/claude-haiku-4.5"
    ],
    "messages": [
      { "role": "user", "content": "Explain vector search in two sentences." }
    ]
  }'

const result = await client.compare.create({
  models: ["openai/gpt-4o-mini", "anthropic/claude-haiku-4.5"],
  messages: [{ role: "user", content: "Explain vector search in two sentences." }],
});

console.log(result.results);

from meshapi import ChatMessage, CompareParams, MeshAPI

client = MeshAPI(base_url="https://api.meshapi.ai", token="rsk_...")

result = client.compare.create(
    CompareParams(
        models=["openai/gpt-4o-mini", "anthropic/claude-haiku-4.5"],
        messages=[ChatMessage(role="user", content="Explain vector search in two sentences.")],
    )
)

print(result.results)

result, err := client.Compare.Create(ctx, meshapi.CompareParams{
    Models: []string{"openai/gpt-4o-mini", "anthropic/claude-haiku-4.5"},
    Messages: []meshapi.ChatMessage{
        {Role: "user", Content: "Explain vector search in two sentences."},
    },
})
if err != nil {
    log.Fatal(err)
}

fmt.Println(result.Results)

var request = new com.meshapi.sdk.types.compare.CompareRequest();
request.models = java.util.List.of("openai/gpt-4o-mini", "anthropic/claude-haiku-4.5");
request.messages = java.util.List.of(ChatMessage.user("Explain vector search in two sentences."));

var result = client.compare().create(request);
System.out.println(result.results);

​

Request fields

​

Response shape

• the compared model list
• one result per model
• optional synthesized comparison text
• latency and request metadata

​

Streaming (SSE)

​

Mode 1: With comparison (skip_comparison: false, default)

​

Mode 2: Skip comparison (skip_comparison: true)

​

SDK coverage

• Node: client.compare.create(...)
• Python: client.compare.create(...)
• Go: client.Compare.Create(...)
• Java: client.compare().create(...)

​

When to use compare

• Model selection — evaluate candidate models and find the best cost/quality trade-off for your task
• Regression testing — catch output quality regressions when a provider updates a model
• Prompt engineering — see how different phrasings affect output across several models at once
• Provider comparison — evaluate the same underlying model served by different providers
• Internal evaluations — build prompt evals on a stable request shape