Predicted outputs

Provide prediction for text revisions with mostly known output, with channel and conversion limits.

When to use it

For small revisions to existing code or documents, send the expected retained text as prediction: {type: content, content: ...} to a supporting Chat model. This is an output hint, not an enforced template, output validator, prompt cache, or answer cache. Each call still needs an actual completed answer.

The project’s GeneralOpenAIRequest.Prediction is raw JSON that the ordinary OpenAI Chat adapter can forward. The current Chat-to-Responses converter has no corresponding mapping. Confirm model support, a native Chat path, and effective parameters; arbitrary reasoning models and compatible services need not support this feature.

The example sends only text and prediction, without tools, audio, logprobs, or generation limits. Upstreams impose additional parameter-combination restrictions; see predicted outputs. No fixed speedup or savings is promised; OpenRouter performance or routing claims do not establish behavior on this deployment.

Run and validate

The script embeds harmless sample source, requests Hello → Welcome, and supplies the original as both input and prediction. It checks only Chat completion and nonempty text, returning proposed_source JSON for review. It does not overwrite files, execute returned code, or prove the revision is semantically correct.

Responses may include optional accepted/rejected prediction-token statistics. The gateway can also rebuild usage, so absent statistics do not mean zero or prove prediction was unused. This example neither requires that metadata nor computes a bill from it. Assess real benefits separately on the same model and route; local fixture timings cannot establish them.

If the parameter fails, first confirm upstream support and the effective protocol, then separately remove prediction to compare the base request. Do not automatically resubmit in a loop to hide configuration problems. See generation parameters and prompt caching for related boundaries.

Implementation sources: relaykit/dto/openai_request.go, relay/channel/openai/adaptor.go, and relaykit/relayconvert/internal/oai_chat/to_oai_responses_req.go. See Chat API.

Runnable example

Run from the documentation directory. See the REST runtime; keep http-client.mjs beside the script when copying.

export API_ROOT_URL='https://gateway.example.com'
export API_KEY='YOUR_API_KEY'
export API_MODEL='YOUR_MODEL_ID'
node examples/predicted-output.mjs

import { requestJSON } from './http-client.mjs';

try {
  const model = process.env.API_MODEL;
  if (!model?.trim()) throw new Error('Set model.');
  const original = 'export function greet(name) {\n  return `Hello, ${name}!`;\n}\n';
  const result = await requestJSON('/v1/chat/completions', {
    model,
    messages: [
      { role: 'system', content: 'Return the complete revised source only, without Markdown fences or explanations.' },
      { role: 'user', content: `Replace Hello with Welcome in this source. Keep everything else unchanged.\n${original}` },
    ],
    prediction: { type: 'content', content: original },
  });
  const choice = result.choices?.[0];
  const message = choice?.message;
  if (!Array.isArray(result.choices) || result.choices.length !== 1 || choice?.index !== 0
    || choice.finish_reason !== 'stop' || message?.role !== 'assistant'
    || message.refusal != null || message.tool_calls != null || message.function_call != null
    || typeof message.content !== 'string' || !message.content.trim()) throw new Error('Incomplete text.');
  // Generated code is returned for review, never written over source files or executed.
  console.log(JSON.stringify({ proposed_source: message.content }));
} catch {
  console.error('Prediction example failed. Check native Chat support and completion status. No automatic retry.');
  process.exitCode = 1;
}

阅读中文