Stateful Executors

Relevant source files

• LLama.Examples/Assets/chat-with-kunkun-chinese.json
• LLama.Examples/Examples/ChatChineseGB2312.cs
• LLama.Examples/Examples/ChatSessionStripRoleName.cs
• LLama.Examples/Examples/ChatSessionWithHistory.cs
• LLama.Examples/Examples/ChatSessionWithRestart.cs
• LLama.Examples/Examples/ChatSessionWithRoleName.cs
• LLama.Examples/Examples/GrammarJsonResponse.cs
• LLama.Examples/Examples/InstructModeExecute.cs
• LLama.Examples/Examples/InteractiveModeExecute.cs
• LLama.Examples/Examples/LoadAndSaveSession.cs
• LLama.Examples/Examples/LoadAndSaveState.cs
• LLama.Examples/Examples/StatelessModeExecute.cs
• LLama.Examples/Examples/TalkToYourself.cs
• LLama/LLamaExecutorBase.cs
• LLama/LLamaInstructExecutor.cs
• LLama/LLamaInteractExecutor.cs
• LLama/LLamaStatelessExecutor.cs

Purpose and Scope

Stateful executors maintain conversation state across multiple inference calls, allowing for multi-turn interactions where the model retains memory of previous exchanges. This is achieved by preserving the KV cache and tracking token positions within the context window. This page documents the two primary stateful executor implementations: InteractiveExecutor and InstructExecutor.

For one-shot inference without conversation history, see StatelessExecutor LLama/LLamaStatelessExecutor.cs19-20 For higher-level conversation management with templating and history transforms, see ChatSession LLama/LLama.Examples/Examples/ChatSessionWithHistory.cs20-27

---

Class Hierarchy

The ILLamaExecutor interface provides the base abstraction for all execution modes LLama/LLamaStatelessExecutor.cs19-20 Stateful logic is centralized in the StatefulExecutorBase abstract class, which implements common logic for token tracking, session file management, and multimodal integration LLama/LLamaExecutorBase.cs20-21

Code Entity Space Diagram

This diagram maps the high-level executor abstractions to their specific implementation classes and primary dependencies.

Sources: LLama/LLamaExecutorBase.cs20-21 LLama/LLamaInteractExecutor.cs23-24 LLama/LLamaInstructExecutor.cs22-23 LLama/LLamaStatelessExecutor.cs19-20

---

State Management Architecture

Stateful executors maintain several categories of state to enable efficient multi-turn conversations and KV cache reuse.

State Components

State Field	Purpose	Type
_pastTokensCount	Tokens already evaluated in KV cache (n_past)	int LLama/LLamaExecutorBase.cs29
_consumedTokensCount	Tokens consumed from input queue (n_consume)	int LLama/LLamaExecutorBase.cs33
_embed_inps	Input token queue awaiting evaluation	List<LLamaToken> LLama/LLamaExecutorBase.cs53
_embeds	Tokens currently being processed in the batch	List<LLamaToken> LLama/LLamaExecutorBase.cs49
_last_n_tokens	Recent token history for repetition penalties	FixedSizeQueue<LLamaToken> LLama/LLamaExecutorBase.cs61
_session_tokens	Tokens recovered from session file	List<LLamaToken> LLama/LLamaExecutorBase.cs57
_n_session_consumed	Session tokens already reused in current run	int LLama/LLamaExecutorBase.cs37
_n_matching_session_tokens	Tokens in prompt matching session cache prefix	int LLama/LLamaExecutorBase.cs41

Inference Data Flow

The following diagram illustrates how data flows through the stateful executor during an inference cycle, transitioning from managed strings to native evaluations.

Sources: LLama/LLamaExecutorBase.cs28-61 LLama/LLamaInteractExecutor.cs143-144 LLama/LLamaExecutorBase.cs108-114

---

InteractiveExecutor

The InteractiveExecutor provides a conversational interface with minimal input formatting. It is designed for scenarios where the user and the assistant alternate turns in a continuous stream.

Key Characteristics

• Minimal Preprocessing: It adds newlines to user continuations but avoids heavy templating, allowing the model to naturally follow the chat flow LLama/LLamaInteractExecutor.cs155-158
• Prompt Run Detection: It tracks whether it is executing the initial prompt or a follow-up via _is_prompt_run LLama/LLamaInteractExecutor.cs26
• Continuation Support: The executor accepts null text in InferAsync to trigger a continuation of the model's previous thought LLama/LLamaInteractExecutor.cs152-153

Initialization and Multimodal Support

The executor can be initialized for standard text-only inference or multimodal inference using MtmdWeights LLama/LLamaInteractExecutor.cs49-52 Multimodal input is handled via PreprocessMtmd which processes images and markers LLama/LLamaInteractExecutor.cs147

Sources: LLama/LLamaInteractExecutor.cs38-52 LLama/LLamaExecutorBase.cs122-126 LLama/LLamaInteractExecutor.cs145-148

---

InstructExecutor

The InstructExecutor is specialized for instruction-following models (e.g., Alpaca, Vicuna). It automatically wraps user inputs with specific instruction and response markers.

Key Characteristics

• Automatic Formatting: Wraps every user input with an _instructionPrefix and _instructionSuffix LLama/LLamaInstructExecutor.cs26-29
• Default Templates: Uses \n\n### Instruction:\n\n and \n\n### Response:\n\n as default markers LLama/LLamaInstructExecutor.cs39-40
• Antiprompt Injection: Automatically ensures the instruction prefix is treated as an antiprompt to stop the model from hallucinating the next user turn LLama/LLamaInstructExecutor.cs141-142

Preprocessing Logic

When _is_prompt_run is false, the InstructExecutor modifies the input queue as follows:

1. Append _inp_pfx (Instruction Prefix).
2. Append tokenized user text.
3. Append _inp_sfx (Response Prefix).

Sources: LLama/LLamaInstructExecutor.cs162-176 LLama/LLamaInstructExecutor.cs38-48

---

Session File Optimization

Stateful executors support session file caching via WithSessionFile. This mechanism allows the executor to reuse the KV cache state from a previous run, significantly reducing "time to first token" for long prompts.

Prefix Reuse Logic

When a session file is loaded, the executor compares the tokens in the file with the current prompt:

1. It loads tokens from the file using NativeApi.llama_state_load_file LLama/LLamaExecutorBase.cs147
2. It calculates _n_matching_session_tokens by iterating through _session_tokens and _embed_inps LLama/LLamaExecutorBase.cs161-170
3. If a match is found, the executor can skip re-evaluating those tokens by aligning the _pastTokensCount with the matching count during the evaluation loop LLama/LLamaExecutorBase.cs161-175

Sources: LLama/LLamaExecutorBase.cs135-154 LLama/LLamaExecutorBase.cs161-170 LLama/LLamaExecutorBase.cs147

---

Context Shifting and Overflow Strategies

When the conversation exceeds the context window (ContextSize), stateful executors perform context shifting to make room for new tokens while attempting to preserve relevant history.

Overflow Strategies

Strategy	Behavior
ThrowException	Generation stops and a ContextOverflowException is thrown LLama/LLamaStatelessExecutor.cs151
TruncateAndReprefill	Discards a percentage of the context and re-evaluates the remaining history LLama/LLamaStatelessExecutor.cs176-184

Shifting Mechanism (TruncateAndReprefill)

1. Detection: Triggered when n_past plus new tokens exceeds Context.ContextSize LLama/LLamaStatelessExecutor.cs147-148
2. Discarding: A portion of the context is discarded. The amount is determined by ContextTruncationPercentage LLama/LLamaStatelessExecutor.cs175-176
3. Preservation: A fixed number of tokens (TokensKeep) are kept at the beginning of the context (usually the system prompt) LLama/LLamaStatelessExecutor.cs155-165

Sources: LLama/LLamaStatelessExecutor.cs147-184

---

State Serialization and Persistence

Executors can capture their entire internal state into a serializable object, allowing conversations to be saved to disk and resumed later.

State Classes and Persistence Methods

Executor	State Class	Capture Method	Restore Method
InteractiveExecutor	InteractiveExecutorState	GetStateData() LLama/LLamaInteractExecutor.cs55	LoadState(data) LLama/LLamaInteractExecutor.cs75
InstructExecutor	InstructExecutorState	GetStateData() LLama/LLamaInstructExecutor.cs64	LoadState(data) LLama/LLamaInstructExecutor.cs85

These states include the full _embed_inps queue, _last_n_tokens history, and current _pastTokensCount LLama/LLamaInteractExecutor.cs57-70 Serialization is typically performed using JsonSerializer to a file stream LLama/LLamaInteractExecutor.cs101-104

Sources: LLama/LLamaInteractExecutor.cs55-114 LLama/LLamaInstructExecutor.cs64-129