| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
| |
An Erlang/Elixir/BEAM profiler tool:
https://github.com/rabbitmq/looking_glass
Signed-off-by: Loïc Hoguin <loic@rabbitmq.com>
|
| |\
| |
| | |
Do not limit paging to disk if hibernated or resumed after credit
|
| | |\
| |/
|/| |
|
| |\ \
| | |
| | | |
Remove consumer bias & allow queues under max load to drain quickly
|
| | |\ \
| |/ /
|/| | |
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
This reverts commit 547be437426163fcd2655857e34f4d24b4994c3a.
The change in 1518b601036351e3f8686bab9c8ae87a8d5af0a3 is in line with the expected
stop behavior in 3.6.x.
For 3.7.0 we are still contemplating what it should be.
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
Given a queue process under max load, with both publishers & consumers,
if consumers are not **always** prioritised over publishers, a queue
can take 1 day (or more) to fully drain.
Even without consumer bias, queues can drain fast (i.e. 10
minutes in our case), or slow (i.e. 1 hour or more). To put it
differently, this is what a slow drain looks like:
```
___ <- 2,000,000 messages
/ \__
/ \___ _ _
/ \___/ \_____/ \___
/ \
|-------------- 1h --------------|
```
And this is what a fast drain looks like:
```
_ <- 1,500,000 messages
/ \_
/ \___
/ \
|- 10 min -|
```
We are still trying to understand the reason behind different drain
rates, but without removing consumer bias, this would **always** happen:
```
______________ <- 2,000,000 messages
/ \_______________
/ \______________ ________
/ \__/ \______
/ \
|----------------------------- 1 day ---------------------------------|
```
Other observations worth capturing:
```
| PUBLISHERS | CONSUMERS | READY MESSAGES | PUBLISH MSG/S | CONSUME ACK MSG/S |
| ---------- | --------- | -------------- | --------------- | ----------------- |
| 3 | 3 | 0 | 22,000 - 23,000 | 22,000 - 23,000 |
| 3 | 3 | 1 - 2,000,000 | 5,000 - 8,000 | 7,000 - 11,000 |
| 3 | 0 | 1 - 2,000,000 | 21,000 - 25,000 | 0 |
| 3 | 0 | 2,000,000 | 5,000 - 15,000 | 0 |
```
* Empty queues are the fastest since messages are delivered straight to
consuming channels
* With 3 publishing channels, a single queue process gets saturated at
22,000 msg/s. The client that we used for this benchmark would max at
10,000 msg/s, meaning that we needed 3 clients, each with 1 connection
& 1 channel to max the queue process. It is possible that a single
fast client using 1 connection & 1 channel would achieve a slightly
higher throughput, but we didn't measure on this occasion. It's
highly unrealistic for a production, high-throughput RabbitMQ
deployment to use 1 publishers running 1 connection & 1 channel. If
anything, there would be many more publishers with many connections &
channels.
* When a queue process gets saturated, publishing channels & their
connections will enter flow state, meaning that the publishing rates
will be throttled. This allows the consuming channels to keep up with
the publishing ones. This is a good thing! A message backlog slows
both publishers & consumers, as the above table captures.
* Adding more publishers or consumers slow down publishinig & consuming.
The queue process, and ultimately the Erlang VMs (typically 1 per
CPU), have more work to do, so it's expected for message throughput to
decrease.
Most relevant properties that we used for this benchmark:
```
| ERLANG | 19.3.6.2 |
| RABBITMQ | 3.6.12 |
| GCP INSTANCE TYPE | n1-standard-4 |
| -------------------- | ------------ |
| QUEUE | non-durable |
| MAX-LENGTH | 2,000,000 |
| -------------------- | ------------ |
| PUBLISHERS | 3 |
| PUBLISHER RATE MSG/S | 10,000 |
| MSG SIZE | 1KB |
| -------------------- | ------------ |
| CONSUMERS | 3 |
| PREFETCH | 100 |
| MULTI-ACK | every 10 msg |
```
Worth mentioning, `vm_memory_high_watermark_paging_ratio` was set to a
really high value so that messages would not be paged to disc. When
messages are paged out, all other queue operations are blocked,
including all publishes and consumes.
Artefacts attached to rabbitmq/rabbitmq-server#1378 :
- [ ] RabbitMQ management screenshots
- [ ] Observer Load Chars
- [ ] OS metrics
- [ ] RabbitMQ definitions
- [ ] BOSH manifest with all RabbitMQ deployment properties
- [ ] benchmark app CloudFoundry manifests.yml
[#151499632]
|
| | |/
|/|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Paging of messages to disk is limited by lazy_queue_explicit_gc_run_operation_threshold
This is required to avoid GC on every publish. But for big messages the
process can get hibernated or blocked by the message store credit-flow,
leaving the process with a lot of memory taken.
When hibernating or unblocking we should try to page messages to disk
regardless of the threshold, because we know that memory reduction is
required at this moment.
Fixes #1379
[#150916864]
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This reverts commit 1518b601036351e3f8686bab9c8ae87a8d5af0a3.
There is no consensus on how it should work. In general it is impossible
to tell a stopped node from a node we could not contact (with some
specific exceptions reported by modern net_kernel version such as
the Erlang cookie mismatch), so some believe stop should be
idempotent and others believe it should not.
|
| |\ \
| |/
|/| |
Ensure that exit code 0 is used for "stop" command
|
| |/
|
|
| |
Part of the fix to #1362
|
| |\
| |
| | |
Take ha-mode into account in choosing queue master
|
| | |
| |
| |
| | |
ha-mode:nodes when picking initial queue master node
|
| | |
| |
| |
| | |
pids are started
|
| | | |
|
| | | |
|
| | | |
|
| | |
| |
| |
| | |
They produce more informative match failure reports.
|
| | | |
|
| |/
|
|
| |
Fixes #1371
|
| |\
| |
| |
| |
| | |
rabbitmq/sync-rabbitmq-config-example-with-configure-page
Sync rabbitmq.config.example with PROJECT_ENV
|
| | | |
|
| | | |
|
| | | |
|
| | | |
|
| | | |
|
| | |
| |
| |
| | |
This was the last pass, there are no more left in PROJECT_ENV, promise.
|
| | | |
|
| | | |
|
| | |
| |
| |
| | |
Add links instead, as per discussion with @michaelklishin & @hairyhum
|
| | | |
|
| | |
| |
| |
| |
| | |
Having discussed with @michaelklishin & @hairyhum, it was agreed that
these details belong in http://www.rabbitmq.com/persistence-conf.html
|
| | | |
|
| | | |
|
| | | |
|
| | | |
|
| | | |
|
| | | |
|
| | | |
|
| |\ \
| | |
| | | |
Add additional ExecStop line from rabbitmq-server-release
|
| |/ /
| |
| |
| |
| |
| | |
This line is part of the release but not included here
https://github.com/rabbitmq/rabbitmq-server-release/blob/stable/packaging/debs/Debian/debian/rabbitmq-server.service#L16
|
| |\ \
| |/
|/| |
Add optional Restart and RestartSec configuration
|
| | | |
|
| |/
|
|
|
|
| |
Fixes #1359
This gives guidances for those users who wish to automatically restart RabbitMQ in the event of a failure. Tested by using the `Restart=on-failure` setting, then running `rabbitmqctl eval "erlang:halt(abort)."`
|
| |\
| |
| | |
Do not log anything when list of apps to stop is empty
|
| | | |
|
| | | |
|
| | | |
|
| | | |
|
| | | |
|
