OpenTelemetry for AI Apps in .NET: Trace Semantic Kernel + Track Token Costs

1. Opening

Three weeks ago, one of our endpoints went from a 800ms p95 to 6 seconds. No deploy that day. No infra change. Just a support ticket: “the assistant feels slow now.”

We had logs. Logs told us the request came in and the response went out. They didn’t tell us where the 5.2 seconds went. Was it the LLM call taking longer because of a model-side slowdown? Was it our pricing-lookup plugin hitting a downstream API that started throttling us? Was it our own code a bad await chain, a lock we forgot about? I spent an evening adding Stopwatch calls around every method, redeploying, and squinting at Application Insights traces that only showed "HTTP POST /chat 6021ms" as a single flat block. Nothing inside it.

That’s the moment you realize you don’t have observability, you have a heartbeat monitor. It tells you the patient is alive. It doesn’t tell you what’s wrong.

Definition: Open Telemetry (OTel) is a vendor-neutral standard for collecting three kinds of signals from your app traces (the path a request takes through your system, broken into timed spans), metrics (numeric measurements over time, like counters and histograms), and logs. For AI apps specifically, the interesting part is that Semantic Kernel already emits Activity…