Prompt caching

Changing a request does not necessarily discard an existing cache entry. What matters is whether a subsequent request has the same prefix and can find an eligible matching breakpoint. The main settings to check are:

For GPT-5.6 and later, cache writes cost 1.25× the standard, uncached input-token rate. It is worth incurring this charge when you know a prefix will be reused, because subsequent reads cost only 0.1× that rate. Writing a prefix once and fully reusing it once costs 1.35× its ordinary input cost, compared with 2× for processing it twice without caching. The savings grow with each additional cache read: across ten requests, one write and nine full reads cost 2.15×, compared with 10× without caching.

Both implicit and explicit caching are supported, where explicit caching gives you more control over which context is written to cache.

Explicit mode: You choose where to place cache breakpoints based on your context management.

• Set prompt_cache_options.mode to explicit to use only developer-selected breakpoints and mark each desired breakpoint by adding prompt_cache_breakpoint: { "mode": "explicit" } to a supported content block inside an input message.
• When no explicit breakpoints are placed, the request does not use prompt caching or create cache writes.
• Explicit-only mode lets you choose where cache writes end. Content after the last selected breakpoint is processed at the uncached input-token rate without a cache-write charge, so you can avoid writing changing content that is unlikely to be reused.
• Multiple explicit breakpoints can preserve prefixes that change at different rates. Each request can create up to four cache writes.
• additional_tools input items do not currently accept prompt_cache_breakpoint.

Implicit mode: OpenAI chooses breakpoint locations out of the box that work well for most use cases.

• When prompt_cache_options.mode is implicit, OpenAI places a breakpoint at the end of the latest eligible message. Eligible messages are:
  • user messages
  • the last tool response in a consecutive group of tool responses
  • the last developer message in the initial consecutive group of developer messages.
• You can add explicit breakpoints without turning off the implicit breakpoint; an implicit breakpoint uses one of the four cache write slots to leave three usable explicit cache write slots.

Only implicit caching is supported. OpenAI places implicit breakpoints at model-dependent intervals, counted from the beginning of the hidden OpenAI system message. Only breakpoints at or beyond the minimum cacheable length (counted from the end of the hidden context) are eligible.

Reported cached_tokens is calculated by subtracting the hidden system tokens from the last matched breakpoint, then rounding down to the nearest multiple of 128.

Use prompt_cache_options.ttl to control the minimum cache lifetime. The only supported value, 30m, is also the default. A cached prefix remains eligible for reuse for 30 minutes after its most recent write or reuse, though OpenAI may retain it longer.

Use prompt_cache_retention, with supported values that depend on the model:

• in_memory: Entries typically remain active for around 5 to 10 minutes of inactivity, up to one hour.
• 24h: Extended retention typically keeps entries available for around 30 minutes and can retain them for up to 24 hours.

Retention defaults and Zero Data Retention

Prompt caching may store encrypted key/value tensors in GPU-local storage as application state. For models that support both in_memory and 24h, the default depends on your organization’s data retention policy:

• Organizations without Zero Data Retention enabled default to 24h.
• Organizations with Zero Data Retention enabled default to in_memory.

Verify the available retention policies for your model and organization before selecting a value.

prompt_cache_key is an optional control for maintaining separate cache accounting for customers or users within your application. OpenAI handles cache routing automatically; you can omit the key for normal caching.

Using separate keys can make cached token usage and billing easier to explain for each customer or user. For example, separate keys help prevent cache-hit probing across users: submitting candidate prompts and observing cache hits to learn whether matching content was previously cached. See Separate cache accounting with keys.

In multi-turn applications, reusing the growing conversation history can save more input tokens than caching only the initial instructions. Preserve earlier messages and tool results so later turns can reuse the full shared prefix.

• Keep the prefix stable. Put stable developer instructions and shared reference material first. If developer instructions or shared material contain timestamps, user-specific content, or other dynamic content, place those at the end rather than the beginning, or move them into later conversation messages.
• Preserve conversation history. Append new messages rather than rewriting earlier turns. Summarization, compaction, or context truncation can change the prefix and reset cache reuse.
• Change reasoning effort without rewriting the prefix. On GPT-6 Astra, append a configuration_update input item to change reasoning effort between responses while keeping request-level reasoning.effort unchanged. This preserves the original prefix for cache reuse. See Change reasoning mid-conversation for examples and compatibility limits.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26{
  "model": "gpt-5.6",
  "reasoning": { "effort": "low", "context": "all_turns" },
  "text": { "verbosity": "medium" },
  "prompt_cache_options": { "mode": "explicit" },
  "input": [
    {
      "role": "developer",
      "content": [
        {
          "type": "input_text",
          "text": "Stable instructions and shared reference material...",
          "prompt_cache_breakpoint": { "mode": "explicit" }
        }
      ]
    },
    {
      "role": "developer",
      "content": "Dynamic developer instructions, such as user-specific content and timestamps..."
    },
    {
      "role": "user",
      "content": "The user's current question..."
    }
  ]
}

On supported GPT-6 and later models, append a configuration_update input item to change reasoning effort during a conversation while preserving the earlier cached prefix. Keep the top-level reasoning.effort at its original value as changing that setting can rewrite instructions in the hidden system instructions.

The latest configuration update controls the reasoning effort for subsequent responses. For example, append this item to the existing input array to switch to high reasoning for the subsequent requests:

1
2
3
4{
  "type": "configuration_update",
  "reasoning": { "effort": "high" }
}

When the tools your application needs vary between requests, change which tools are callable while keeping their definitions stable to preserve reusable prefixes.

• Keep tools consistent. Preserve tool definitions, ordering, and schemas.
• Disable tool use for a request. Set tool_choice to "none" instead of removing the tool definitions.
• Enable only selected tools. Use allowed_tools to restrict which tools are callable while keeping the supplied tools list stable.
• Load tools when needed. Use tool search with defer_loading: true to reduce input tokens spent on tool definitions in early requests of multi-turn threads. Discovered tools are appended at the end of context, preserving earlier reusable content.
• Preserve tool-loading history. Use a developer-role additional_tools input item to add tools during a thread according to your application’s logic.

On GPT-5.6 and later, two controls determine where cache breakpoints are placed: prompt_cache_options.mode selects implicit or explicit-only caching, and prompt_cache_breakpoint marks a boundary you choose.

• Place breakpoints automatically. Use implicit caching to place a breakpoint at the end of the latest eligible message. This is convenient for multi-turn threads that append to existing context.
• Choose breakpoints deliberately. Place explicit markers at the end of stable content. Use explicit-only mode to avoid unnecessary cache writes for changing suffixes.

Use prompt_cache_key when you want to maintain separate cache accounting for customers, users, or workspaces within your application. This can make cached token usage and billing easier to explain within each group. The key is optional and is not needed to optimize caching.

• Choose how to separate cache accounting. Assign a distinct key to each customer or user whose cache accounting should remain separate. For example, support:customer_123 and support:customer_456 maintain separate cache accounting for two customers, even when their requests contain the same prefix.
• Keep keys stable within each group. Reuse the same key for a customer’s related requests. Generate a separate key for a session or thread only when it needs its own cache accounting.
• Apply keys consistently. Use the customer’s key across their requests to maintain separate cache accounting. This also helps prevent cache-hit probing across customers.

For earlier models, prefer setting prompt_cache_retention to "24h" for extended retention when the model and your data-retention requirements allow it. See Cache lifetime for supported settings and defaults.

If many requests reuse the same developer instructions and tool definitions, but that shared prefix falls below the model’s minimum cacheable length, consider shortening it or expanding it with useful, stable instructions, examples, or reference material. Measure whether cache reuse offsets the additional input tokens and any cache-write charges, and ensure evaluations and behaviour remain stable.

The chart highlights the minimum cacheable length cost trap where short prefix lengths can cost more uncached than expanding to the minimum cacheable token length.

• Measure actual cache performance. Track usage.input_tokens_details.cached_tokens, usage.input_tokens_details.cache_write_tokens, input-token counts, latency, and realized cost. Track the token cache-hit rate by dividing total cached tokens by total input tokens, aggregating both counts by user, workspace, day, or another useful grouping.
• Calculate input cost. Use the token counts in response.usage and the model’s prices per million tokens.
• Use the prompt caching dashboard. Monitor cache hit rates in the Prompt Caching Dashboard.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18function calculateInputCost(
  usage,
  inputPricePerMillion,
  cacheInputMultiplier = 0.1,
  cacheWriteMultiplier = 1.25
) {
  const inputTokens = usage.input_tokens;
  const cachedTokens = usage.input_tokens_details.cached_tokens;
  const cacheWriteTokens = usage.input_tokens_details.cache_write_tokens;
  const ordinaryInputTokens = inputTokens - cachedTokens - cacheWriteTokens;

  const weightedInputTokens =
    ordinaryInputTokens +
    cachedTokens * cacheInputMultiplier +
    cacheWriteTokens * cacheWriteMultiplier;
  const inputCost = (weightedInputTokens * inputPricePerMillion) / 1_000_000;
  return inputCost;
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21from openai.types.responses import ResponseUsage


def calculate_input_cost(
    usage: ResponseUsage,
    input_price_per_million: float,
    cache_input_multiplier: float = 0.1,
    cache_write_multiplier: float = 1.25,
) -> float:
    input_tokens = usage.input_tokens
    cached_tokens = usage.input_tokens_details.cached_tokens
    cache_write_tokens = usage.input_tokens_details.cache_write_tokens
    ordinary_input_tokens = input_tokens - cached_tokens - cache_write_tokens

    weighted_input_tokens = (
        ordinary_input_tokens
        + cached_tokens * cache_input_multiplier
        + cache_write_tokens * cache_write_multiplier
    )
    input_cost = weighted_input_tokens * input_price_per_million / 1_000_000
    return input_cost

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18def calculate_input_cost(
  usage,
  input_price_per_million,
  cache_input_multiplier = 0.1,
  cache_write_multiplier = 1.25
)
  input_tokens = usage.input_tokens
  details = usage.input_tokens_details
  cached_tokens = details.cached_tokens
  cache_write_tokens = details.cache_write_tokens
  ordinary_input_tokens = input_tokens - cached_tokens - cache_write_tokens

  weighted_input_tokens =
    ordinary_input_tokens +
    (cached_tokens * cache_input_multiplier) +
    (cache_write_tokens * cache_write_multiplier)
  (weighted_input_tokens * input_price_per_million) / 1_000_000
end

• Keep existing stable prefixes.
• If you use prompt_cache_key, keep existing values to preserve separate cache accounting for customers or users.
• Replace prompt_cache_retention with prompt_cache_options.ttl.
• Confirm that reusable prefixes meet the model’s minimum cacheable length.
• If the default breakpoint includes content that changes between requests, add an explicit breakpoint after the stable prefix.
• Use prompt_cache_options.mode: "explicit" when later content is not worth writing.
• Compare cached_tokens, cache_write_tokens, latency, and total cost before and after migration.

Consider a single-turn LLM judge that determines whether a completed interaction shows evidence that the user is satisfied after an interaction with a chatbot. Each request uses the same grading rubric and labeled few-shot examples to evaluate a different interaction.

• Preserving the prefix: The fixed rubric and examples come first. Their combined length is deliberately kept just above the model’s minimum cacheable length, using material that helps calibrate the judge. The interaction being evaluated comes last.
• Caching mode and breakpoint: Explicit-only caching is enabled, with a breakpoint after the fixed rubric and examples. The user–chatbot conversation being evaluated comes after that breakpoint and is not written to the cache, avoiding a cache-write charge for content that is unlikely to be reused.

An example deployment using these principles reported a token cache-hit rate of ~70%. This figure illustrates a possible outcome. Actual cache-hit rate ceilings will depend upon your context and application usage.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22{
  "model": "gpt-5.6-sol",
  "reasoning": { "effort": "medium", "context": "all_turns" },
  "text": { "verbosity": "low" },
  "prompt_cache_options": { "mode": "explicit" },
  "input": [
    {
      "role": "developer",
      "content": [
        {
          "type": "input_text",
          "text": "Judge whether the completed interaction provides evidence that the user is satisfied. Return true or false. Full grading rubric and labeled few-shot examples...",
          "prompt_cache_breakpoint": { "mode": "explicit" }
        }
      ]
    },
    {
      "role": "user",
      "content": "Completed interaction to evaluate..."
    }
  ]
}

Consider a multi-turn agent with long, shared developer instructions and frequent tool calls. Typical usage sees users running multiple sessions with the agent at once, and often forking the threads.

• Preserving the prefix: Each turn appends new messages, tool calls, and results without rewriting earlier context, so the reusable prefix grows over time.
• Optional prompt cache key: This example uses agent_123_v1:user_456 to maintain separate cache accounting for user 456, making their cached token usage and billing easier to explain. This also helps prevent cache-hit probing across users. The key stays the same across that user’s sessions and forks with the agent. Omit it if your application does not need this separation.
• Implicit caching mode: Implicit caching is enabled so the latest eligible user or tool message provides a breakpoint.
• Explicit breakpoints: A breakpoint is added after each tool result to preserve earlier reusable prefixes and improve cache efficiency of forking.

An example deployment using these principles reported a token cache-hit rate >90%. This figure illustrates a possible outcome. Actual cache-hit rate ceilings will depend upon your context and application usage.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41{
  "model": "gpt-5.6-sol",
  "reasoning": { "effort": "medium", "context": "all_turns" },
  "text": { "verbosity": "medium" },
  "prompt_cache_key": "agent_123_v1:user_456",
  "prompt_cache_options": { "mode": "implicit" },
  "tools": [
    {
      "type": "function",
      "name": "function_name",
      "description": "Function description",
      "parameters": { "...": "..." }
    }
  ],
  "input": [
    {
      "role": "developer",
      "content": "Stable developer instructions and reference material..."
    },
    { "role": "user", "content": "Can you do...?" },
    {
      "type": "function_call",
      "call_id": "call_123",
      "name": "function_name",
      "arguments": "..."
    },
    {
      "type": "function_call_output",
      "call_id": "call_123",
      "output": [
        {
          "type": "input_text",
          "text": "Tool result...",
          "prompt_cache_breakpoint": { "mode": "explicit" }
        }
      ]
    },
    { "role": "assistant", "content": "Assistant response..." },
    { "role": "user", "content": "Can you also do...?" }
  ]
}

This is particularly prevalent when migrating from earlier models to GPT-5.6 or later due to the change in implicit caching behaviour. If requests share a long prefix but have different suffixes, caching the first complete request implicitly-only does not make the shorter shared prefix reusable.

Consider a static developer message followed by a dynamic user message in each request. This request writes through the dynamic content. Changing that content in the next request does not match the longer cached prefix, and there is no separate breakpoint after the static content.

1
2
3
4
5
6
7
8
9
10{
  "model": "gpt-5.6-sol",
  "reasoning": { "effort": "medium", "context": "all_turns" },
  "text": { "verbosity": "low" },
  "prompt_cache_options": { "mode": "implicit" },
  "input": [
    { "role": "developer", "content": "Static content..." },
    { "role": "user", "content": "Dynamic content..." }
  ]
}

To remediate, place an explicit breakpoint after the static content in both requests. The first request writes the reusable prefix; the next can reuse it even when the dynamic content changes. This example uses explicit-only mode to avoid writing the dynamic content to cache.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17{
  "model": "gpt-5.6-sol",
  "reasoning": { "effort": "medium", "context": "all_turns" },
  "text": { "verbosity": "low" },
  "prompt_cache_options": { "mode": "explicit" },
  "input": [
    {
      "role": "developer",
      "content": [{
        "type": "input_text",
        "text": "Static content...",
        "prompt_cache_breakpoint": { "mode": "explicit" }
      }]
    },
    { "role": "user", "content": "Dynamic content..." }
  ]
}

Suppose request 1 uses implicit mode and caches a prefix through the end of a user message, then follow-up request 2 preserves that prefix but switches to prompt_cache_options.mode: "explicit". As explained in How prefix matching works, request 2 checks only the explicit breakpoints in its own input, so it will not reuse that saved implicit prefix from request 1 (unless one of the explicit breakpoints in request 2 matches the cached endpoint from request 1).

▼ = breakpoint

- Request 1: implicit mode
  [Developer message][User message] ▼

- Request 2: explicit-only mode. Does not hit cache.
  [Developer message][User message][Follow-up] ▼

To reuse the implicit prefix from request 1, place an explicit breakpoint at the matching content-block boundary in request 2, or keep implicit mode enabled so the earlier eligible message ending remains a lookup candidate.

Even when both requests use implicit mode, preserving the same initial tokens is not always enough. Suppose request 1 ends with a user message containing Content A, then follow-up request 2 extends that same message to Content A + Content B. The old endpoint after Content A is now inside a message, rather than at its end. As explained in How prefix matching works, without an explicit breakpoint at that boundary, request 2 does not reuse the prefix saved there.

▼ = breakpoint

- Request 1: implicit mode
  [Developer message][User message: Content A] ▼

- Request 2: implicit mode. Cannot reuse the prefix through Content A.
  [Developer message][User message: Content A + Content B] ▼

When the conversation structure permits, preserve the original message and append a new message instead. Otherwise, keep the reusable text in a separate content block and place an explicit breakpoint after it in both requests.

In implicit mode, developer messages after the initial consecutive block of developer messages are not automatic cache lookup boundaries. Add an explicit breakpoint at the end of the reusable developer message to preserve that breakpoint in subsequent requests so OpenAI can check for a matching cached prefix.

A prefix that qualifies for caching on one model may be too short on another. Check the model comparison and measure the reusable prefix with the model and settings you actually use. When changing models, repeat that check rather than assuming the previous model’s threshold still applies.

Compaction replaces earlier conversation context with a shorter representation. That can change the prefix, so the first request after compaction may reuse less of the previous cache even when the conversation is logically the same.

Keep reusable instructions and reference material stable where possible, then let subsequent turns build on the compacted context. Compare total input cost before and after compaction: fewer input tokens can still save money even when the cache-hit rate falls.

No. Prompt caching does not change how the model generates output tokens. The model generates a new response using the cached prefix, so identical requests are not guaranteed to produce identical outputs.

No. Manual cache clearing is not currently available. Cache entries expire according to the model’s cache lifetime and retention settings.

Yes. Cached input tokens still count toward tokens-per-minute limits. Prompt caching does not change how rate limits are calculated.