Model Compare

​

How it works

1. Fan-out: All requested models are called concurrently. The total wall-clock time is roughly that of the slowest model, not the sum of all models.
2. Error isolation: If a single model fails or times out (hard timeout of 120s), the others continue unaffected. Partial results are returned with a partial: true flag.
3. Synthesis (default): After all models respond, a separate comparison LLM analyzes the responses and produces a structured evaluation covering accuracy, completeness, clarity, and a recommendation. Synthesis runs only when at least two models succeed. If every model fails, the request returns 502 all_models_failed with the error nested under detail and no request_id (see Compare troubleshooting). When you don’t name a comparison_model, the default judge is openai/gpt-4o-mini, falling back to other judges your key permits. If your key permits none, or every judge fails, comparison is null.
4. Skip synthesis (optional): By setting skip_comparison: true, you can skip the synthesis step and receive only the raw model outputs. This is useful for parallel streaming UIs that perform their own comparison.
5. Rate limiting and Billing: The entire comparison counts as a single request against your rate limits (RPM/RPD). However, billing tracks each model call plus the comparison call as separate usage events (N+1 events). If the comparison model fails and a fallback judge takes over, each judge attempt is recorded as its own usage event, but only the judge that produced the synthesis is charged.
6. Streaming: Two streaming modes are available by setting stream: true. With synthesis enabled, fan-out is non-streaming, but the final comparison text is streamed token-by-token. If skip_comparison: true is set, each fan-out model streams its tokens in real-time concurrently, tagged by model name.

​

Basic request

curl https://api.meshapi.ai/v1/chat/compare \
  -H "Authorization: Bearer <YOUR_RSK_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "models": [
      "openai/gpt-4o-mini",
      "anthropic/claude-haiku-4.5"
    ],
    "messages": [
      { "role": "user", "content": "Explain vector search in two sentences." }
    ]
  }'

const result = await client.compare.create({
  models: ["openai/gpt-4o-mini", "anthropic/claude-haiku-4.5"],
  messages: [{ role: "user", content: "Explain vector search in two sentences." }],
});

console.log(result.results);

from meshapi import ChatMessage, CompareParams, MeshAPI

client = MeshAPI(base_url="https://api.meshapi.ai", token="rsk_...")

result = client.compare.create(
    CompareParams(
        models=["openai/gpt-4o-mini", "anthropic/claude-haiku-4.5"],
        messages=[ChatMessage(role="user", content="Explain vector search in two sentences.")],
    )
)

print(result.results)

result, err := client.Compare.Create(ctx, meshapi.CompareParams{
    Models: []string{"openai/gpt-4o-mini", "anthropic/claude-haiku-4.5"},
    Messages: []meshapi.ChatMessage{
        {Role: "user", Content: "Explain vector search in two sentences."},
    },
})
if err != nil {
    log.Fatal(err)
}

fmt.Println(result.Results)

var request = new com.meshapi.sdk.types.compare.CompareRequest();
request.models = java.util.List.of("openai/gpt-4o-mini", "anthropic/claude-haiku-4.5");
request.messages = java.util.List.of(ChatMessage.user("Explain vector search in two sentences."));

var result = client.compare().create(request);
System.out.println(result.results);

​

Request fields

​

Response shape

{
  "comparison_id": "cmp_019ed1d24227774dba14",
  "object": "compare.completion",
  "created": 1781636687,
  "models": ["openai/gpt-4o-mini", "anthropic/claude-haiku-4.5"],
  "results": [
    {
      "model": "openai/gpt-4o-mini",
      "response_body": { "choices": [...], "usage": {...} },
      "content": "TCP is a connection-oriented protocol...",
      "latency_ms": 812,
      "error": null,
      "error_code": null,
      "usage": { "prompt_tokens": 24, "completion_tokens": 47, "total_tokens": 71 },
      "request_id": "req_01ARZ3NDEKTSV4RRFFQ69G5FAV::openai/gpt-4o-mini"
    }
  ],
  "comparison": "Both answers are correct; Response 1 is more precise on ordering.",
  "comparison_model": "openai/gpt-4o-mini",
  "comparison_usage": { "prompt_tokens": 245, "completion_tokens": 405, "total_tokens": 650 },
  "comparison_fallback_used": false,
  "total_latency_ms": 9016,
  "partial": false,
  "skip_comparison": false
}

​

Streaming (SSE)

​

Mode 1: With comparison (skip_comparison: false, default)

​

Mode 2: Skip comparison (skip_comparison: true)

​

SDK coverage

• Node: client.compare.create(...)
• Python: client.compare.create(...)
• Go: client.Compare.Create(...)
• Java: client.compare().create(...)

​

When to use compare

• Model selection — evaluate candidate models and find the best cost/quality trade-off for your task
• Regression testing — catch output quality regressions when a provider updates a model
• Prompt engineering — see how different phrasings affect output across several models at once
• Provider comparison — evaluate the same underlying model served by different providers
• Internal evaluations — build prompt evals on a stable request shape