StatelessExecutor provides a stateless inference pattern for LLamaSharp where each inference request operates independently with a fresh context. Unlike stateful executors (InteractiveExecutor and InstructExecutor), StatelessExecutor does not maintain conversation history or KV cache state across multiple calls. Each invocation of InferAsync creates a new LLamaContext, processes the prompt, generates tokens, and disposes the context.
This executor is suitable for one-shot inference tasks where requests are independent, such as single-prompt completions, classification tasks, or stateless API endpoints.
Sources: LLama/LLamaStatelessExecutor.cs15-20 LLama/LLamaStatelessExecutor.cs72-80
Sources: LLama/LLamaExecutorBase.cs20-21 LLama/LLamaStatelessExecutor.cs19-21 LLama/LLamaInteractExecutor.cs23 LLama/LLamaInstructExecutor.cs22-23
| Aspect | StatelessExecutor | StatefulExecutorBase |
|---|---|---|
| Context Lifecycle | Created per InferAsync call, disposed after completion | Single context persists across all calls |
| State Retention | No state between calls | Maintains _pastTokensCount, _consumedTokensCount, _embed_inps |
| Session Files | Not supported | Supports KV cache save/load via WithSessionFile |
| Memory Usage | Lower (context released after each call) | Higher (persistent KV cache) |
| Use Case | Independent requests | Conversational multi-turn inference |
| Multimodal Support | Not supported (IsMultiModal = false) | Supported via MtmdWeights |
Sources: LLama/LLamaStatelessExecutor.cs28-39 LLama/LLamaExecutorBase.cs29-65 LLama/LLamaExecutorBase.cs135-158
Sources: LLama/LLamaStatelessExecutor.cs22-49
The constructor initializes the executor with model weights and context parameters, then creates and immediately disposes a placeholder context to validate parameters:
The initial context creation at LLama/LLamaStatelessExecutor.cs66-67 validates that the weights and parameters are compatible, but the context is disposed immediately because actual contexts are created fresh for each inference call inside InferAsync.
Sources: LLama/LLamaStatelessExecutor.cs58-68
Sources: LLama/LLamaStatelessExecutor.cs72
Sources: LLama/LLamaStatelessExecutor.cs72-191
Unlike stateful executors which maintain a single LLamaContext for the lifetime of the executor, StatelessExecutor creates and destroys contexts per inference call:
Sources: LLama/LLamaStatelessExecutor.cs58-68 LLama/LLamaStatelessExecutor.cs74-80
At LLama/LLamaStatelessExecutor.cs75-76 the executor checks if the previous context handle is still open and disposes it if necessary. This ensures that even if an exception occurred during a previous inference, resources are properly cleaned up before creating a new context.
The using statement at LLama/LLamaStatelessExecutor.cs79 ensures automatic disposal when InferAsync completes or throws an exception.
Sources: LLama/LLamaStatelessExecutor.cs74-80
When ApplyTemplate is set to true, the executor transforms the user prompt using the model's chat template before tokenization:
The chat template follows llama.cpp's llama_chat_apply_template rules, formatting the prompt according to the model's defined template. This is handled by LLamaTemplate and PromptTemplateTransformer.
Sources: LLama/LLamaStatelessExecutor.cs42-49 LLama/LLamaStatelessExecutor.cs97-104
When the number of past tokens (n_past) plus new tokens exceeds Context.ContextSize, the executor implements truncation strategies to make room for new tokens:
Algorithm:
tokensKeep from inferenceParams.TokensKeep. If not specified, it defaults to the initial prompt length LLama/LLamaStatelessExecutor.cs155-166n_discard based on the configured ContextTruncationPercentage (default 0.5f) LLama/LLamaStatelessExecutor.cs175-176MemorySequenceRemove on the native context handle to clear a middle section of the KV cache LLama/LLamaStatelessExecutor.cs179MemorySequenceAdd to shift the remaining tokens back, ensuring sequence continuity LLama/LLamaStatelessExecutor.cs180n_past to reflect the new reduced token count LLama/LLamaStatelessExecutor.cs182If OverflowStrategy is set to ThrowException, a ContextOverflowException is thrown instead LLama/LLamaStatelessExecutor.cs149-152
Sources: LLama/LLamaStatelessExecutor.cs142-183
Each iteration of the generation loop:
inferenceParams.SamplingPipeline.Sample(Context.NativeHandle, _batch.TokenCount - 1) at LLama/LLamaStatelessExecutor.cs128IsEndOfGeneration(_weights.Vocab) at LLama/LLamaStatelessExecutor.cs131-132StreamingTokenDecoder and read partial UTF-8 decoded text at LLama/LLamaStatelessExecutor.cs135-136yield return at LLama/LLamaStatelessExecutor.cs137antiprocessor.Add(decoded) at LLama/LLamaStatelessExecutor.cs140-141Context.DecodeAsync at LLama/LLamaStatelessExecutor.cs186-190Sources: LLama/LLamaStatelessExecutor.cs118-191
| Parameter | Type | Purpose |
|---|---|---|
weights | LLamaWeights | Loaded model weights shared across contexts LLama/LLamaStatelessExecutor.cs22 |
params | IContextParams | Context configuration (size, batch size, threading, etc.) LLama/LLamaStatelessExecutor.cs23 |
logger | ILogger? | Optional logger for diagnostic output LLama/LLamaStatelessExecutor.cs24 |
Sources: LLama/LLamaStatelessExecutor.cs58-68
| Property | Type | Default | Description |
|---|---|---|---|
ApplyTemplate | bool | false | Apply model's chat template to prompts LLama/LLamaStatelessExecutor.cs44 |
SystemMessage | string? | null | System message for template application LLama/LLamaStatelessExecutor.cs49 |
Sources: LLama/LLamaStatelessExecutor.cs42-50
The inferenceParams argument to InferAsync controls sampling behavior:
MaxTokens: Maximum tokens to generate LLama/LLamaStatelessExecutor.cs124TokensKeep: Tokens to retain when context overflows LLama/LLamaStatelessExecutor.cs155AntiPrompts: Sequences that terminate generation LLama/LLamaStatelessExecutor.cs95SamplingPipeline: Token selection strategy (temperature, top-k, etc.) LLama/LLamaStatelessExecutor.cs83DecodeSpecialTokens: Whether to decode special tokens to text LLama/LLamaStatelessExecutor.cs93OverflowStrategy: How to handle context exhaustion LLama/LLamaStatelessExecutor.cs149Refresh this wiki