How supplyAsync() without an Executor Froze Our Internal Common Pool

On a Thursday afternoon, our real-time notification engine (which delivers email notifications, push notifications, and SMS alerts to millions of active users) silently stopped working.

There were no crashes. No JVM OutOfMemory errors. No database outages.

But at 3:15 PM, our customer onboarding metrics dipped to zero. New users registering on our platform were not receiving their verification OTPs. Transaction confirmations were delayed by hours. The queue depth on our notification processor grew to 120,000 pending jobs.

Our APM logs showed:

[2026-07-03 15:15:02.991] WARN  [ForkJoinPool.commonPool-worker-3] c.c.n.s.EmailDeliveryService: Slow email SMTP dispatch took 30005ms
[2026-07-03 15:15:33.001] WARN  [ForkJoinPool.commonPool-worker-1] c.c.n.s.SmsDeliveryService: SMS gateway handshake delayed by 15000ms

A quick check revealed that the application was completely responsive to health checks, but any process requiring asynchronous background execution was indefinitely hanging.

The Day the Notification Engine Went Dead Silent

When our systems team looked at the threads, they noticed a strange pattern. The CPU utilization was sitting at 1% on…