Java 21 Virtual Threads: How They Work and When to Use Them

On the day before Thanksgiving 2020, the Amazon Kinesis data streaming service in AWS' main region US-East-1 went down for several hours. The company explained the outage in its subsequent failure report.

We were able to confirm a root cause, and it turned out this wasn't driven by memory pressure. Rather, the new capacity had caused all of the servers in the fleet to exceed the maximum number of threads allowed by an operating system configuration. As this limit was being exceeded, cache construction was failing to complete and front-end servers were ending up with useless shard-maps that left them unable to route requests to back-end clusters.

Simply put, that Amazon Kinesis system ran out of operating system threads. A machine can support only so many threads, and it must accommodate that limit. The system failed to make the accommodation and, as a result, stopped working. Because of service dependencies, the Kinesis outage affected other AWS services too, including internal ones. The result was major disruption among many popular third-party websites and businesses as well as Amazon itself just hours before Black Friday, one of the busiest shopping days of the entire year.

Accommodating thread limit has been part of a developer's work since the Linux kernel added threading in 1996. It's a laborious, detailed undertaking. Fortunately, new technologies have evolved to make working with threads easier and safer. One of the new technologies is virtual threads.

This article explains what virtual threads are and how they work, the important problem they solve, and an example of using virtual threads under Java.

Understanding the expense of threads

A thread is the smallest unit of a program's execution that the operating system's kernel manages. As a unit of execution within a process, a thread shares the same memory space and resources as other threads in the same process.

Multiple threads within a single process can run concurrently. Support for concurrency makes threads a critical component in systems that support many consumers simultaneously. A consumer in this context can be a system call in a variety of formats, such as an HTTP request on a web server or a remote procedure call under gRPC. It also can be tasks that must execute within a process, such as activities within a workflow process.

Programming using threads has become commonplace, particularly with the proliferation of large-scale systems that support millions of users. You'd be hard pressed to make a viable software system without them.

However, threads come with an expense that requires particular attention. Systems cannot make an unlimited number of threads.

The number of threads a system can support at any moment in time is constrained by two factors: CPU capacity and memory. If too many threads take up too much CPU capacity, processes can grind to a halt. Exhausting CPU capacity is possible in situations that are computationally intensive.

The more common constraint on thread capacity is memory. A Linux kernel typically requires around 2MB of memory to create a thread, according to José Paumard, a member of the Java Developer Relations team at Oracle. Large-scale websites that dedicate each HTTP request to a distinct thread can easily create a million threads, which would require 2TB of memory. This is a significant burden on system resources. Sometimes that amount of memory is available. Sometimes it's not, particularly when an application runs in a Linux container that has limited memory allocation.

There are frameworks for certain programming languages that aim to manage operating system threads. Examples include the .NET Task Parallel Library and the Erlang actor model, which has a special type of lightweight process that can be invoked on the order of millions of calls without incurring excessive memory consumption.

Java's java.util.concurrent APIs provide executors, synchronization utilities and other concurrency building blocks. Java 21 also finalized virtual threads, which make the familiar thread-per-task programming style practical for much larger numbers of concurrent tasks, particularly when those tasks spend significant time blocked on I/O.

How virtual threads work

Java 21 virtual threads are lightweight Thread instances scheduled by the Java runtime rather than being permanently tied one-to-one to operating system threads. The JDK schedules many virtual threads onto a smaller set of platform threads called carrier threads.

Application developers should generally think in terms of virtual threads and carrier threads rather than depend on internal implementation classes such as continuations. Those implementation details are not the public programming model and can evolve independently of application code.

When a virtual thread performs a supported blocking operation, the runtime can unmount it from its carrier thread. The carrier is then free to run other virtual threads. When the blocking operation can continue, the same virtual thread can later be mounted on an available carrier and resume execution.

The virtual thread is not destroyed and recreated each time it blocks. From the application's perspective it remains the same Thread, which is one reason virtual threads let developers retain straightforward blocking code instead of rewriting an application around callbacks.

To create virtual threads, use the Thread.ofVirtual() or the Executors.newVirtualThreadPerTaskExecutor() factory methods.

Virtual threads use the familiar Thread API: they can be started, joined and interrupted, and they work with many existing synchronization APIs. However, they are intended to be cheap, short-lived, task-oriented threads. Applications generally should not pool virtual threads simply to limit their number; when access to a scarce external resource must be limited, use an appropriate mechanism such as a semaphore or connection pool.

Creating virtual threads in Java 21

For an individual virtual thread, Java 21 provides the Thread.ofVirtual() builder:

Thread virtualThread = Thread.ofVirtual().start(() -> {
    System.out.println("Running in a virtual thread");
});

virtualThread.join();

For task-oriented application code, a virtual-thread-per-task executor is often even more convenient:

try (var executor = Executors.newVirtualThreadPerTaskExecutor()) {
    Future<String> future = executor.submit(() -> {
        // A blocking database, HTTP or file operation could run here.
        return "Task complete";
    });

    System.out.println(future.get());
}

The executor creates a new virtual thread for each submitted task. Unlike a traditional fixed thread pool, its purpose is not to reuse a small number of expensive threads.

Using virtual threads

To illustrate virtual threads in Java 21, we created a set of demonstration applications in a GitHub repository named SimpleVirtualThreads. The purpose is to compare the result of creating and running a million threads under Java 11 and Java 21.

The Java 11 code attempts to create a million threads constructing new threads.

Thread thread = new Thread(aTask);

The Java 21 code tries to create a million virtual threads using the static Thread.ofVirtual() method.

The following code shows an excerpt of Java 11 code that creates one million threads of a task that is represented by a class named BlockedThread, which encapsulates blocking code.

final int numberOfThreads = 1_000_000;

for (int i = 0; i < numberOfThreads; i++) {

 Thread thread = new Thread(new BlockedThread(i));

 thread.start();

 String str = String.format("Java 11 thread number %s is running.", i);

 System.out.println(str);

}

When the Java 11 code runs in a Docker container using default memory settings, the code generates errors due to memory issues as shown in Figure 2. There simply isn't enough memory to support the number of intended threads.

The following excerpt of Java 21 code runs the same thread creation logic from within a Docker container that uses the default settings as the Java 11 example. However, instead of using Thread thread = new Thread(aTask) to create a thread, this time the code uses the Thread.ofVirtual() static factory method to create a virtual thread.

final int numberOfThreads = 1_000_000;

for (int i = 0; i < numberOfThreads; i++) {

 Thread virtualThread = Thread.ofVirtual().unstarted(new BlockedThread(i));

 virtualThread.start();

 String str = String.format("Java 21 virtual thread number %s is running.", i);

 System.out.println(str);

}

As you can see in Figure 3, the code executes a million virtual threads under Java 21 without incident.

The Java 11 and Java 21 codes are available on the SimpleVirtualThreads repository with instructions to run the comparison in both Java 11 and Java 21 containers.

When should you use virtual threads?

Putting it all together: Virtual threads in Java 21

Virtual threads do not mean every application should create a million threads, nor does a commercially useful application need to support millions of users. Their advantage is that applications can use a simple thread-per-task style for far more concurrent blocking tasks than is practical with one platform thread per task.

They are particularly attractive for server applications that spend substantial time waiting for databases, HTTP services, files or other I/O. They do not make CPU-bound work inherently faster: CPU-intensive tasks are still constrained by the available processor cores.

Java 21 virtual threads therefore remove an important scalability constraint while preserving Java's familiar imperative programming model. Use them when concurrency is high and tasks frequently block, but continue to control scarce downstream resources and benchmark the complete application rather than treating thread count alone as a measure of scalability.

Bob Reselman is a software developer, system architect and writer. His expertise ranges from software development technologies to techniques and culture.