Caveats in lifting the thread context

In this blog I will try to discuss why the way we pass a thread context from caller thread to executor service threads, needs a sharp look on how our thread pool is configured.

First, I will start with a brief information on why do we even need to carry-forward the context to the executor service threads. Then a detail arrangement of the thread pool + context lifting mechanism which introduced a bug into our system and lastly the root cause analysis and fix for it.

Why do we even need passing thread context to the executor service threads:

For the app servers a thread in execution usually configured to hold several information like the transaction id, trace id, invoking user information etc. Now for various reason when we submit tasks to threadpool the same information do not pass automatically, so in a sense the executor service thread looses these thread local informations which the caller thread holds. The easiest way to visualize this: is by seeing logs that’s generated by the executor service threads. These logs won’t print the transaction id or any such thread locals configured to be printed on each of the logs.

Next I am putting the arrangement(faulty) we had to pass the thread context:

It’s gonna be little lengthy but is really required to understand the cavecat.

Below is the thread pool configuration snapshot. The main thing here is to note how we are creating means of having a backpressure when there’s too many tasks submitted to this executor.

1. By having a blocking queue of fixed capacity.
2. By setting the policy to “CallerRunsPolicy” (new ThreadPoolExecutor.CallerRunsPolicy()).

This policy suggests that if there are too many tasks submitted than the number of available threads to execute in this pool, the parent thread(caller thread) which submits the task to this executor will execute the task itself, so in a way the rate of submitting tasks decreases automatically, as the thread which submits task itself is busy in processing the task.

I really love this clever way of engineering the reduced rate of task submissions.

// Bounded queue to avoid unbounded growth
final var queue = new LinkedBlockingQueue<Runnable>(queueCapacity);
this.customExecutorService = new ThreadPoolExecutor(
     corePoolSize,        // core pool size - minimum number of threads always presnet on the thread pool
     maxPoolSize,        // max pool size - on pressure thread count can increase till max pool size
     keepAliveDuration, // Threads above corePoolSize can be removed after staying idle for this duration.
     queue, //blocking queue to have a backpressure
     threadFactory,
     new ThreadPoolExecutor.CallerRunsPolicy()  // handler for the rejected(queue is filled and all the threads are busy) tasks
    //this particular policy makes sure in case of rejection the caller thread itself will execute the task
);

Next, as we have our executor configured it’s time to look into how we can pass the context from caller thread to the executor service threads. So we had this decorator which does this job.

Get Rituraj Ghosh’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Copy context from caller thread — push the same context to the executor service thread — clear the context of the executor service thread once the execution is finished.

public static Runnable decorate(final Runnable runnable) {
        final Map<String, String> contextMap = MDC.getCopyOfContextMap();
        return () -> {
            if (contextMap != null) {
                MDC.setContextMap(contextMap);
            }
            try {
                runnable.run();
            } finally {
                MDC.clear();
            }
        };
 }

and wherever we submit tasks to that executor service we wrapped the decorator to the runable:

customExecutorService.submit(MDCRunnableDecoratorUtil.decorate(() -> {//runnable code}));

Lastly, The Caveat — RCA — fix:

caveat: We noticed in case of high volume processing, the executor service threads are failing to maintain the context up from it’s caller thread, somewhere the thread context is lost as we lift it up from caller to execute service threads.

RCA: If the reader noticed we are clearing the context of the executor service thread once after it finishes the execution, and that is required to demise the chances of having a corrupted context when it runs next time.

But with the policy set at the thread pool executor, is it the executor service thread every time that executes the passed runnable?

No, With the policy set as CallerRunsPolicy, in case of too much pressure and absence of available threads in the pool, the caller itself executes the submitted task. In this case when we clear the context after execution the caller threads context gets removed, and so any subsequest child threads created from this parent would also not have the context.

fix: Once we found out the RCA, the fix was simpler. We had to alter the decorator we had. We tried to hold the callee thread’s context(say prevContext) before it starts executing the runnable and putting a check after it finishes the execution: that if the previous context(prevContext) was not null then do not clear the context.

Below is the code example for the fix:

public static Runnable decorate(final Runnable runnable) {
        final Map<String, String> contextMap = MDC.getCopyOfContextMap();
        return () -> {
            final Map<String, String> originalContext = MDC.getCopyOfContextMap();
            try {
                if (contextMap != null) {
                    MDC.setContextMap(contextMap);
                }else{
                    MDC.clear();
                }
                runnable.run();
            } finally {
                // Restore original context instead of clearing completely
                // Adding this check is all about the fix.
                if (originalContext != null) {
                    MDC.setContextMap(originalContext);
                } else {
                    MDC.clear();
                }
            }
        };
   }

So conclusion wise we need to look more on how the thread pool executor is configured to properly lift the states up from caller thread to executor service threads.

Thanks.