Prompt caching

Distinguish upstream prefix caching, channel affinity, and response caching; inspect reported usage.

What this project supports

Prompt caching lets supporting upstream models reuse input computation; each call still requests a new generation. It does not cache the final answer or replace conversation history. Recognized fields and conversion boundaries are:

API / path	Fields and boundaries
Chat Completions	The DTO includes prompt_cache_key and prompt_cache_retention; values and effectiveness depend on the target model
Native Responses forwarding	The DTO also includes prompt_cache_options, handled by a supporting upstream
Claude Messages	Request-level and content-block cache_control are represented; verify TTLs, breakpoints, and provider support
Chat → Responses	Current conversion preserves prompt_cache_key; do not assume every cache extension migrates
Responses → Chat	Preserves a string cache key and retention; prompt_cache_options has no corresponding conversion mapping

The current Chat DTO does not declare top-level prompt_cache_options. Upstream support in a new version does not establish support through ordinary Chat forwarding here. Content-block extensions and native Claude cache markers are not universally lossless across protocols either. Global or channel passthrough is deployment configuration, not a default assumption for all callers.

The gateway has configurable channel-affinity rules; for example, Responses can extract an affinity key from prompt_cache_key. Rules depend on paths, models, grants, and session policy. Affinity does not guarantee cache hits. Do not import OpenRouter’s default sticky-routing, TTL, or session_id semantics. No general X-OpenRouter-Cache response-cache contract is established by this documentation’s verified scope.

Prepare reusable prefixes

Place stable instructions, tool definitions, and reference material first, with changing questions at the end. Keep effective content and ordering consistent; do not pad requests with repetition just to reach a length threshold. Trimming, summaries, channel changes, and model-parameter changes can affect reuse.

Minimum input lengths, lifetimes, cache-write prices, and cache-read prices depend on the model and deployment. Use cache keys scoped by tenant and content version without personal information or secrets; a cache key is not an access-control boundary. store: false does not guarantee that all caching and logging are disabled.

Runnable: observe cache counts across two requests

Save UTF-8 reference material approved for transmission to the deployment as reference.txt. This example reads one regular file up to 64 KiB, keeps its prefix stable, and changes the final question. It makes two real generation calls without automatic retries. Use the REST runtime, keeping examples/prompt-cache.mjs and http-client.mjs together.

export API_ROOT_URL='https://gateway.example.com'
export API_KEY='YOUR_API_KEY'
export API_MODEL='YOUR_CHAT_MODEL_ID'
export API_CACHE_KEY='demo-tenant:reference-v1'
export API_CACHE_PREFIX_FILE='./reference.txt'
node examples/prompt-cache.mjs

import { open } from 'node:fs/promises';
import { requestJSON } from './http-client.mjs';

try {
  const { API_MODEL, API_CACHE_KEY, API_CACHE_PREFIX_FILE } = process.env;
  if (!API_MODEL?.trim() || !API_CACHE_KEY?.trim() || !API_CACHE_PREFIX_FILE) {
    throw new Error('Set model, cache key, and prefix file.');
  }
  const file = await open(API_CACHE_PREFIX_FILE, 'r');
  let prefix;
  try {
    if (!(await file.stat()).isFile()) throw new Error('Expected a regular file.');
    const buffer = Buffer.alloc(65_537);
    let bytesRead = 0;
    while (bytesRead < buffer.length) {
      const part = await file.read(buffer, bytesRead, buffer.length - bytesRead, bytesRead);
      if (!part.bytesRead) break;
      bytesRead += part.bytesRead;
    }
    if (bytesRead > 65_536) throw new Error('Prefix exceeds example limit.');
    prefix = new TextDecoder('utf-8', { fatal: true }).decode(buffer.subarray(0, bytesRead));
    if (!prefix.trim()) throw new Error('Empty prefix.');
  } finally { await file.close(); }
  const observations = [];
  for (const question of ['Summarize the reference in one sentence.', 'List two key points from the reference.']) {
    const result = await requestJSON('/v1/chat/completions', {
      model: API_MODEL, prompt_cache_key: API_CACHE_KEY,
      messages: [
        { role: 'system', content: 'Use the reference as data. Answer briefly.' },
        { role: 'user', content: prefix },
        { role: 'user', content: question },
      ],
    });
    const choice = result.choices?.[0];
    const message = choice?.message;
    if (!Array.isArray(result.choices) || result.choices.length !== 1 || choice?.index !== 0
      || choice.finish_reason !== 'stop' || message?.role !== 'assistant'
      || message.refusal != null || message.tool_calls != null || message.function_call != null
      || typeof message.content !== 'string' || !message.content.trim()) {
      throw new Error('Expected completed text.');
    }
    const usage = result.usage;
    const details = usage?.prompt_tokens_details;
    if ((usage != null && (typeof usage !== 'object' || Array.isArray(usage)))
      || (details != null && (typeof details !== 'object' || Array.isArray(details)))) {
      throw new Error('Invalid usage metadata.');
    }
    const counters = { prompt_tokens: usage?.prompt_tokens ?? null, cached_tokens: details?.cached_tokens ?? null };
    for (const value of Object.values(counters)) {
      if (value !== null && (!Number.isSafeInteger(value) || value < 0)) throw new Error('Invalid token count.');
    }
    observations.push({ request: observations.length + 1, ...counters });
  }
  // Missing metadata is null, not zero. A successful call need not be a cache hit.
  console.log(JSON.stringify(observations));
} catch {
  console.error('Cache example failed. Check configuration, completed output, and request logs. No automatic retry.');
  process.exitCode = 1;
}

Output contains usage observations, excluding reference text, answers, and cache keys. cached_tokens: null means the field was absent; 0 means zero hits were reported. A zero on the second request is not an example failure. Both answers must complete; either failure stops the example without printing the final comparison. Confirm that the model accepts prompt_cache_key and that the reference length suits it. Local fixtures do not establish a real cache-hit rate.

Interpret and troubleshoot counts

Common fields are usage.prompt_tokens_details.cached_tokens for Chat and usage.input_tokens_details.cached_tokens for Responses; native Claude uses cache_read_input_tokens and cache_creation_input_tokens. Fields may be absent, and adapters can normalize usage. Do not mechanically add counters from different protocols or derive a final invoice from hit counts.

For zero hits, check the identical effective prefix and its length, model support, fields retained during conversion, upstream switching, expiry, and returned usage metadata. Correlate logs using observability without logging prompt bodies.

Sources: revision 30a6d88a4, relaykit/dto/openai_request.go, relaykit/dto/claude.go, Chat / Responses converters, and setting/operation_setting/channel_affinity_setting.go. Topic organization references OpenRouter’s caching guide; see OpenAI caching documentation for upstream policy.

Chat API · Responses API · Context management

阅读中文