Limit the number of messages in q3 for lazy queueslimit-lazy-queues-q3

This commit introduces min and max options used when converting
messages from deltas (full on-disk) to q3 (index or full in memory).

Before this commit there could be up to 16k messages read at once
and sitting in memory (the full index segment). This has been
causing issues depending on the number of queues and the rate
of consumption.

With this commit the new default depends on the consumption rate.
The formula applied is 1.5*RateOut so we read from disk less than
once per second. If there is capacity then we may still read up
to 16k messages. If the rate is very low, then we have another
"minimum" default value of 100. So we will always read at least
100 (less if there are less than 100 messages in the queue) and
at most 16k depending on the consumption rate.

The values can be configured so that if there is a large number
of queues we can limit the amount of messages in memory to a
fixed value (for example, set min to 50 and max to 500). This
allows greater control over the memory usage of lazy queues.

0 files changed, 0 insertions, 0 deletions