“The way that organizations and organisms anticipate the future is by taking signals from the past, most of the time. Knowing what has just happened in the past, an organism can bring that data forward to modify its future actions.” - Kevin Kelly
Anticipation is extremely difficult in a distributed context because to do so effectively we need to shorten and shrink both time (observational delay) and space (observational dimension). This is a problem for many load balancers that attempt to employ sophisticated scheduling algorithms (other than random or round robin), directing traffic based on past, present and predicted workload patterns, request performance and resource capacity availability.
There are many technical challenges to overcome in achieving optimal load balancing, such as being able to predict not just future workload patterns but the actual completion times of work already distributed, as well as the the availability of additional capacity subsequent to work completion. Ideally a load balancer would like to estimate when capacity is likely to be sufficiently freed up on one or more application backends to handle current inbound traffic. To do so accurately can be a challenge when application backends also employ queues and scheduling fronting service entry points. So whilst work might have been dispatched, it does not necessarily mean it has actually started execution, though we could probably include this wait time in the service time of a queueing model.
This is a very interesting optimization problem to solve but before we can tackle this we must be able to detect an imbalance in the distribution of workload and its execution within application server/service backends.
The rest of this article will demonstrate how effective an approach based on discrete event simulation and real-time replay is at detecting even the smallest of changes across multiple JVM instances under constant load conditions as well as the fair scheduling in the replaying of homogenous execution behavior by the service. In doing so we achieve extreme levels of application activity monitoring.