We have multiple threads wrting to a LinkedBlockingQueue. We time the queue writes and normally they are less than 5ms. When we get up to about 300 threads / second and a full GC occurs, the write times increase dramatically, to as much as 10 seconds or more. After between 10 and 40 seconds, the write times return to their normal level of less than 5ms. We log GC activity and the 'Total time for which application threads were stopped: ' is usually less than 1 second, always less than 1.5 seconds.
I can understand that the write times would increase during GC, but can anyone explain why it takes so long to return to the pre-GC performance after the GC has finished.
I should point out that for some of our tests we are using Terracotta and the queue is a Distributed Shared Object.
When we remove Terracotta from the equation, the behaviour is the same, but not quite as dramatic, i.e. the write times after GC may only rise to 5 or 6 seconds.
Thanks for any insight,