> ## Documentation Index
> Fetch the complete documentation index at: https://docs.inputapi.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Prompt caching

> Distinguish upstream prefix caching, channel affinity, and response caching; inspect reported usage.

## What this project supports

Prompt caching lets supporting upstream models reuse input computation; each call still requests a new generation. It does not cache the final answer or replace conversation history. Recognized fields and conversion boundaries are:

| API / path | Fields and boundaries |
| - | - |
| Chat Completions | The DTO includes `prompt_cache_key` and `prompt_cache_retention`; values and effectiveness depend on the target model |
| Native Responses forwarding | The DTO also includes `prompt_cache_options`, handled by a supporting upstream |
| Claude Messages | Request-level and content-block `cache_control` are represented; verify TTLs, breakpoints, and provider support |
| Chat → Responses | Current conversion preserves `prompt_cache_key`; do not assume every cache extension migrates |
| Responses → Chat | Preserves a string cache key and retention; `prompt_cache_options` has no corresponding conversion mapping |

The current Chat DTO does not declare top-level `prompt_cache_options`. Upstream support in a new version does not establish support through ordinary Chat forwarding here. Content-block extensions and native Claude cache markers are not universally lossless across protocols either. Global or channel passthrough is deployment configuration, not a default assumption for all callers.

The gateway has configurable channel-affinity rules; for example, Responses can extract an affinity key from `prompt_cache_key`. Rules depend on paths, models, grants, and session policy. Affinity does not guarantee cache hits. Do not import OpenRouter's default sticky-routing, TTL, or `session_id` semantics. No general `X-OpenRouter-Cache` response-cache contract is established by this documentation's verified scope.

## Prepare reusable prefixes

Place stable instructions, tool definitions, and reference material first, with changing questions at the end. Keep effective content and ordering consistent; do not pad requests with repetition just to reach a length threshold. Trimming, summaries, channel changes, and model-parameter changes can affect reuse.

Minimum input lengths, lifetimes, cache-write prices, and cache-read prices depend on the model and deployment. Use cache keys scoped by tenant and content version without personal information or secrets; a cache key is not an access-control boundary. `store: false` does not guarantee that all caching and logging are disabled.

## Runnable: observe cache counts across two requests

Save UTF-8 reference material approved for transmission to the deployment as `reference.txt`. This example reads one regular file up to 64 KiB, keeps its prefix stable, and changes the final question. It makes two real generation calls without automatic retries. Use the [REST runtime](/en/guides/http-runtime.md), keeping `examples/prompt-cache.mjs` and `http-client.mjs` together.

```sh theme={null}
export API_ROOT_URL='https://gateway.example.com'
export API_KEY='YOUR_API_KEY'
export API_MODEL='YOUR_CHAT_MODEL_ID'
export API_CACHE_KEY='demo-tenant:reference-v1'
export API_CACHE_PREFIX_FILE='./reference.txt'
node examples/prompt-cache.mjs
```

```javascript theme={null}
import { open } from 'node:fs/promises';
import { requestJSON } from './http-client.mjs';

try {
  const { API_MODEL, API_CACHE_KEY, API_CACHE_PREFIX_FILE } = process.env;
  if (!API_MODEL?.trim() || !API_CACHE_KEY?.trim() || !API_CACHE_PREFIX_FILE) {
    throw new Error('Set model, cache key, and prefix file.');
  }
  const file = await open(API_CACHE_PREFIX_FILE, 'r');
  let prefix;
  try {
    if (!(await file.stat()).isFile()) throw new Error('Expected a regular file.');
    const buffer = Buffer.alloc(65_537);
    let bytesRead = 0;
    while (bytesRead < buffer.length) {
      const part = await file.read(buffer, bytesRead, buffer.length - bytesRead, bytesRead);
      if (!part.bytesRead) break;
      bytesRead += part.bytesRead;
    }
    if (bytesRead > 65_536) throw new Error('Prefix exceeds example limit.');
    prefix = new TextDecoder('utf-8', { fatal: true }).decode(buffer.subarray(0, bytesRead));
    if (!prefix.trim()) throw new Error('Empty prefix.');
  } finally { await file.close(); }
  const observations = [];
  for (const question of ['Summarize the reference in one sentence.', 'List two key points from the reference.']) {
    const result = await requestJSON('/v1/chat/completions', {
      model: API_MODEL, prompt_cache_key: API_CACHE_KEY,
      messages: [
        { role: 'system', content: 'Use the reference as data. Answer briefly.' },
        { role: 'user', content: prefix },
        { role: 'user', content: question },
      ],
    });
    const choice = result.choices?.[0];
    const message = choice?.message;
    if (!Array.isArray(result.choices) || result.choices.length !== 1 || choice?.index !== 0
      || choice.finish_reason !== 'stop' || message?.role !== 'assistant'
      || message.refusal != null || message.tool_calls != null || message.function_call != null
      || typeof message.content !== 'string' || !message.content.trim()) {
      throw new Error('Expected completed text.');
    }
    const usage = result.usage;
    const details = usage?.prompt_tokens_details;
    if ((usage != null && (typeof usage !== 'object' || Array.isArray(usage)))
      || (details != null && (typeof details !== 'object' || Array.isArray(details)))) {
      throw new Error('Invalid usage metadata.');
    }
    const counters = { prompt_tokens: usage?.prompt_tokens ?? null, cached_tokens: details?.cached_tokens ?? null };
    for (const value of Object.values(counters)) {
      if (value !== null && (!Number.isSafeInteger(value) || value < 0)) throw new Error('Invalid token count.');
    }
    observations.push({ request: observations.length + 1, ...counters });
  }
  // Missing metadata is null, not zero. A successful call need not be a cache hit.
  console.log(JSON.stringify(observations));
} catch {
  console.error('Cache example failed. Check configuration, completed output, and request logs. No automatic retry.');
  process.exitCode = 1;
}
```

Output contains usage observations, excluding reference text, answers, and cache keys. `cached_tokens: null` means the field was absent; `0` means zero hits were reported. A zero on the second request is not an example failure. Both answers must complete; either failure stops the example without printing the final comparison. Confirm that the model accepts `prompt_cache_key` and that the reference length suits it. Local fixtures do not establish a real cache-hit rate.

## Interpret and troubleshoot counts

Common fields are `usage.prompt_tokens_details.cached_tokens` for Chat and `usage.input_tokens_details.cached_tokens` for Responses; native Claude uses `cache_read_input_tokens` and `cache_creation_input_tokens`. Fields may be absent, and adapters can normalize usage. Do not mechanically add counters from different protocols or derive a final invoice from hit counts.

For zero hits, check the identical effective prefix and its length, model support, fields retained during conversion, upstream switching, expiry, and returned usage metadata. Correlate logs using [observability](/en/guides/observability.md) without logging prompt bodies.

Sources: revision `30a6d88a4`, `relaykit/dto/openai_request.go`, `relaykit/dto/claude.go`, Chat / Responses converters, and `setting/operation_setting/channel_affinity_setting.go`. Topic organization references [OpenRouter's caching guide](https://openrouter.ai/docs/guides/best-practices/prompt-caching); see [OpenAI caching documentation](https://developers.openai.com/api/docs/guides/prompt-caching) for upstream policy.

[Chat API](/en/api-reference/chat-completions.md) · [Responses API](/en/api-reference/responses.md) · [Context management](/en/guides/context-management.md)

[阅读中文](/zh/guides/prompt-caching.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.