In looking at ways to improve the resilience of software based services, engineers are now turning to monkeys and the chaos they introduce....

Lets start with the case in which service A running in a JVM makes remote procedure calls into service B running in another JVM where we would like to introduce delay to see how it impacts service A. Whilst we may know the interfaces used in the service interaction and data communication between A and B we might not be sure of the mapping to actual method implementations. Ideally we want to designate a particular part of the codebase in service B as the implementation of the service and then inject latency into entry point calls (all determined dynamically at runtime) representing the service call boundary. Initially we don’t want to inject latency into every single internal method invocation in processing a request because that would make it harder to reason about how much delay was introduced and possibly call into question the impact of other system dynamics especially if B makes out-bound calls to other services which call re-entrantly back into A or B.

http://www.jinspired.com/site/monkeying-around-with-latency-injection-at-service-points-using-qos-ace