| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
deliver_from_queue case, we now reduce n calls to mixed_queue:is_empty to 1 call and pass around the remaining count as the acc. l33t
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
useful. The code is thus now a good bit simpler.
|
| |
|
|
|
|
| |
to the disk_queue when operating in disk only mode and seems to have substantially improved performance (in addition to avoiding a sync call, repeated lasting for the length of a queue (erlang stdlib) with a million+ items in it can't have been cheap). It now seems to be very much the case that when coming out of disk only mode, huge back logs are recovered reliably.
Also, added reduce_memory_footprint and increase_memory_footprint to control. Both can be run twice and alter whether the disk_queue changes mode or the individual queues.
|
| |\ |
|
| | |\ |
|
| | | | |
|
| | |/
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This involved some substantial changes to the queue internal data
structures - mostly by choice; the new design is cleaner:
- We no longer keep a list of consumers in the channel
records. Now the channel records just contain a consumer count
instead, and that's only there for efficiency so we can more
easily tell when we need to register/unregister with the limiter.
- We now keep *two* consumer queues - one of active consumers
(that's the one we've always had) and one of blocked consumers.
We round-robin on the first one as before, and move things between the
two queues when blocking/unblocking channels. When doing so the
relative order of a channel's consumers is preserved, so the effects
of any round-robining the active consumers get carried
through to the blocked consumers when they get blocked and then back
to the active consumers when they get unblocked.
|
| | |\
| | |
| | |
| | |
| | |
| | | |
We point to the macports files of the default branch from our web site
and they got broken with the merge of bug20333. This hopefully fixes
that, but further qa is required.
|
| | | | |
|
| | | |
| | |
| | |
| | | |
os x su command
|
| | | |
| | |
| | |
| | | |
UNSENT_MESSAGE_LIMIT made performance better. This then made me wonder if the unblock and notify_sent messages weren't getting through fast enough, and sure enough, using pcast is much better there. Also, turning on dbg:tpl showed that the common path in mixed_queue was to call publish_delivered (i.e. the message has been delivered to a consumer, we just need to record this fact). Making sure everything in there for the non-persistent, non-durable but disk-only mode is asynchronous also helped performance massively.
|
| | | |
| | |
| | |
| | | |
persistent/durable async, and it has improved some issues. But, if you switch to disk only mode, then allow, say 10k messages to build up (use MulticastMain) then switch back to ram mode, then it won't recover - the receive rate will stay very low, and rabbitmqctl list_queues will continue to grow insanely. This is very very odd, because querying the disk_queue directly for the queue length shows it drops to 0, but at least one CPU is maxed out at 100% use, messages continue to arrive, but the delivery rate never goes back up. Mysterious.
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
Reversed order - i.e. now when swapping out, the first thing is to alter the disk_queue, and the 2nd thing is to alter the queues.
And vice versa.
The reasoning is as follows:
Changing the disk_queue is a BIG operation because it affects every message in there, from all queues. In order to minimise the impa
ct of this operation, we must do it first, not second, because if we do it first, only persistent messages from durable queues will
be in there, whereas if we do it second, then all messages from all queues will be in there.
Similarly, when swapping in, altering the individual queues is the first thing to do because it prevents the disk queue from growing
further (i.e. only persistent messages to durable queues then make it to the disk queue), and each queue pulls out from the disk qu
eue all the messages in there and so subsequent delivery from the mixed queue becomes very fast (actually, this is a total lie because of the call to rabbit_disk_queue:phantom_deliver in rabbit_mixed_queue:deliver - if I could get rid of this or at least make it async then that would greatly improve matters).
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
rabbit_queue_mode_manager:change_memory_usage(undef, true).
this will first ask all queues to switch from mixed to disk mode, and will on a 2nd call, ask the disk queue to switch to disk only mode.
rabbit_queue_mode_manager:change_memory_usage(undef, false).
moves the other way.
This all works, eg set MulticastMain pushing in messages and switch modes, and it's fine.
One immediate problem is that as soon as everything becomes disk only, the performance suffers, so as a result messages build up. This is as expected. Then, going back to the middle mode (i.e. disk queue in ram_disk mode and queues in disk mode), the switch in the disk queue eats up a lot of memory. I suspect this is the effect of converting the mnesia table from disc_only_copies to disc_copies when there are 40k+ messages in there (one row per message). As a result, this conversion on its own is very dangerous to make. It might be more sensible to use the "weird" mode, where the queues are in mixed mode and the disk queue is in disk_only mode so as to try and get the queues to drain as fast as possible, reducing the size of the mnesia table so that when it is finally converted back, it's small.
More experimentation is needed.
I'll hook the above commands into rabbitmqctl soon.
|
| |\ \ \
| |/ / |
|
| | |\ \ |
|
| | |/ / |
|
| | |\ \ |
|
| | | |/ |
|
| | |\ \
| | |/ |
|
| | | |\ |
|
| | | | | |
|
| | | | | |
|
| | | | |
| | | |
| | | |
| | | | |
not that it really matters much
|
| | |/ / |
|
| | |\ \ |
|
| | | | | |
|
| | |/ / |
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
correctly when being run in disk_only mode. This is _much_ more complicated than I had thought because of the fact that the presence of a message on disk has nothing to do with whether it is persistent or not. As a result early acking is required and requeuing operations are horrendous to say the least.
When going from disk-only mode to mixed mode, we don't ack anything at all. It's arguable that we should ack non-persistent messages at this point, but the problem there is that if the conversion fails then we lose messages. Therefore, we then arrive at the sitation where we're in mixed mode, and we have messages held in ram that are not persistent, but are still on disk, and require early acking when being delivered (again, requeue is hell).
The conversion to and from disk-only and mixed mode now seems to work well.
When starting up, non-persistent messages on disk are deleted.
Finally, in disk_queue, publish now takes an IsDelivered flag. This allows you to publish messages and mark them delivered in one go. However, the message is still available for delivery (i.e. it's not waiting for an ack).
Also in disk_queue, requeue_with_seqs is now [{AckTag, {NewSeqId, NewIsDelivered}}], which allows you to requeue and unset the delivered flag. Note however, that it is still not safe to requeue a message which isn't waiting for an ack.
(Please note, it's now very important to distinguish between messages which "AreDelivered" _and_ are waiting for an ack _and_ are not going to appear if you call deliver(Q), VERSUS messages which "AreDelivered" but are not waiting for an ack and will appear (eventually) if you call deliver(Q).
|
| | | |
| | |
| | |
| | | |
state of flux. disk mode to mixed mode in the mixed_queue is annoyingly hard.
|
| | | |
| | |
| | |
| | | |
OS calls. This is a good thing and makes writing to disk much faster. However, we can have the situation where we are trying to read a message off disk before that message has been fully written out to disk. Therefore, we need to fsync at choice times. Because fsync is quite expensive, we want to call fsync no more than absolutely necessary. Thus we now have a 'dirty' flag which tracks whether the current file has been written to sinc the last fsync, and we call fsync whenever is dirty and the file to read from is the current file. This has also had some similar changes elsewhere in the disk queue. In short however, it seems this does work as I'm no longer able to reproduce reads of messages which return all blanks.
|
| | | |
| | |
| | |
| | | |
documented in bug 20470
|
| | | | |
|
| | | | |
|
| | | | |
|
| | | | |
|
| | | | |
|
| |\ \ \
| |/ /
| | |
| | | |
test in comment #4 of bug20782 also has correct behaviour.
|
| | |\ \ |
|
