summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* Ignore indirect dependencies of rabbit in rabbit_pluginsGerhard Lazu2017-10-172-6/+21
| | | | | | | | | | | | | | | | | The goal of maybe_keep_required_deps() is still the same: we don't want to list applications from $RABBITMQ_PLUGINS_DIR the rabbit application depends on. Now, the function handles indirect dependencies. Signed-off-by: Jean-Sébastien Pedron <jean-sebastien@rabbitmq.com> [#136346167] This commit is cherry-picked from `master` because rabbit_common now depends on recon: having rabbit_common depend on 3rd-party applications didn't happen so far in stable and wasn't supported. This broke the creation of the standalone package. (cherry picked from commit cbbcc3e32f97b6047b4f82f0776717dd3e85430c)
* TypoMichael Klishin2017-10-171-1/+1
|
* WordingMichael Klishin2017-10-171-23/+23
|
* CorrectionsMichael Klishin2017-10-171-1/+1
|
* Edits, linksMichael Klishin2017-10-171-46/+55
|
* WordingMichael Klishin2017-10-161-1/+1
|
* Updates to management plugin sectionMichael Klishin2017-10-161-4/+4
|
* Update rabbitmq-components.mkJean-Sébastien Pédron2017-10-131-0/+4
|
* Update rabbitmq-components.mkJean-Sébastien Pédron2017-10-131-4/+4
|
* Merge pull request #1389 from rabbitmq/rabbitmq-common-224Michael Klishin2017-10-122-5/+5
|\ | | | | Update default node memory calculation strategy to match rabbitmq/rabbitmq-common#224
| * Update default allocator strategy to match rabbitmq/rabbitmq-common#224Michael Klishin2017-10-112-5/+5
|/
* Merge pull request #1384 from rabbitmq/integrate-looking_glassJean-Sébastien Pédron2017-10-042-0/+61
|\ | | | | Integrate `looking_glass`
| * Ignore maps:from_list in xrefGerhard Lazu2017-10-041-0/+1
| | | | | | | | Signed-off-by: Loïc Hoguin <loic@rabbitmq.com>
| * Use maps:from_list so that it compiles on R16B03Gerhard Lazu2017-10-041-5/+14
| | | | | | | | | | | | | | Since looking_glass requires Erlang 19, maps:from_list will not be called if it's not defined in earlier Erlang versions. Signed-off-by: Loïc Hoguin <loic@rabbitmq.com>
| * Integrate looking_glassGerhard Lazu2017-10-042-0/+51
|/ | | | | | | An Erlang/Elixir/BEAM profiler tool: https://github.com/rabbitmq/looking_glass Signed-off-by: Loïc Hoguin <loic@rabbitmq.com>
* Merge pull request #1382 from rabbitmq/rabbitmq-server-1379rabbitmq_v3_6_13_milestone1Michael Klishin2017-10-031-2/+2
|\ | | | | Do not limit paging to disk if hibernated or resumed after credit
| * Merge branch 'stable' into rabbitmq-server-1379Michael Klishin2017-10-032-18/+12
| |\ | |/ |/|
* | Merge pull request #1378 from rabbitmq/always-prioritise-consumersMichael Klishin2017-10-021-13/+5
|\ \ | | | | | | Remove consumer bias & allow queues under max load to drain quickly
| * \ Merge branch 'stable' into always-prioritise-consumersMichael Klishin2017-09-301-5/+7
| |\ \ | |/ / |/| |
* | | Revert "Revert "Ensure that exit code 0 is used for "stop" command""Michael Klishin2017-09-281-5/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | This reverts commit 547be437426163fcd2655857e34f4d24b4994c3a. The change in 1518b601036351e3f8686bab9c8ae87a8d5af0a3 is in line with the expected stop behavior in 3.6.x. For 3.7.0 we are still contemplating what it should be.
| * | Remove consumer bias & allow queues under max load to drain quicklyGerhard Lazu2017-09-281-13/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Given a queue process under max load, with both publishers & consumers, if consumers are not **always** prioritised over publishers, a queue can take 1 day (or more) to fully drain. Even without consumer bias, queues can drain fast (i.e. 10 minutes in our case), or slow (i.e. 1 hour or more). To put it differently, this is what a slow drain looks like: ``` ___ <- 2,000,000 messages / \__ / \___ _ _ / \___/ \_____/ \___ / \ |-------------- 1h --------------| ``` And this is what a fast drain looks like: ``` _ <- 1,500,000 messages / \_ / \___ / \ |- 10 min -| ``` We are still trying to understand the reason behind different drain rates, but without removing consumer bias, this would **always** happen: ``` ______________ <- 2,000,000 messages / \_______________ / \______________ ________ / \__/ \______ / \ |----------------------------- 1 day ---------------------------------| ``` Other observations worth capturing: ``` | PUBLISHERS | CONSUMERS | READY MESSAGES | PUBLISH MSG/S | CONSUME ACK MSG/S | | ---------- | --------- | -------------- | --------------- | ----------------- | | 3 | 3 | 0 | 22,000 - 23,000 | 22,000 - 23,000 | | 3 | 3 | 1 - 2,000,000 | 5,000 - 8,000 | 7,000 - 11,000 | | 3 | 0 | 1 - 2,000,000 | 21,000 - 25,000 | 0 | | 3 | 0 | 2,000,000 | 5,000 - 15,000 | 0 | ``` * Empty queues are the fastest since messages are delivered straight to consuming channels * With 3 publishing channels, a single queue process gets saturated at 22,000 msg/s. The client that we used for this benchmark would max at 10,000 msg/s, meaning that we needed 3 clients, each with 1 connection & 1 channel to max the queue process. It is possible that a single fast client using 1 connection & 1 channel would achieve a slightly higher throughput, but we didn't measure on this occasion. It's highly unrealistic for a production, high-throughput RabbitMQ deployment to use 1 publishers running 1 connection & 1 channel. If anything, there would be many more publishers with many connections & channels. * When a queue process gets saturated, publishing channels & their connections will enter flow state, meaning that the publishing rates will be throttled. This allows the consuming channels to keep up with the publishing ones. This is a good thing! A message backlog slows both publishers & consumers, as the above table captures. * Adding more publishers or consumers slow down publishinig & consuming. The queue process, and ultimately the Erlang VMs (typically 1 per CPU), have more work to do, so it's expected for message throughput to decrease. Most relevant properties that we used for this benchmark: ``` | ERLANG | 19.3.6.2 | | RABBITMQ | 3.6.12 | | GCP INSTANCE TYPE | n1-standard-4 | | -------------------- | ------------ | | QUEUE | non-durable | | MAX-LENGTH | 2,000,000 | | -------------------- | ------------ | | PUBLISHERS | 3 | | PUBLISHER RATE MSG/S | 10,000 | | MSG SIZE | 1KB | | -------------------- | ------------ | | CONSUMERS | 3 | | PREFETCH | 100 | | MULTI-ACK | every 10 msg | ``` Worth mentioning, `vm_memory_high_watermark_paging_ratio` was set to a really high value so that messages would not be paged to disc. When messages are paged out, all other queue operations are blocked, including all publishes and consumes. Artefacts attached to rabbitmq/rabbitmq-server#1378 : - [ ] RabbitMQ management screenshots - [ ] Observer Load Chars - [ ] OS metrics - [ ] RabbitMQ definitions - [ ] BOSH manifest with all RabbitMQ deployment properties - [ ] benchmark app CloudFoundry manifests.yml [#151499632]
| | * Do not limit paging to disk if hibernated or resumed after creditDaniil Fedotov2017-09-281-2/+2
| |/ |/| | | | | | | | | | | | | | | | | | | | | | | Paging of messages to disk is limited by lazy_queue_explicit_gc_run_operation_threshold This is required to avoid GC on every publish. But for big messages the process can get hibernated or blocked by the message store credit-flow, leaving the process with a lot of memory taken. When hibernating or unblocking we should try to page messages to disk regardless of the threshold, because we know that memory reduction is required at this moment. Fixes #1379 [#150916864]
* | Revert "Ensure that exit code 0 is used for "stop" command"Michael Klishin2017-09-281-7/+5
| | | | | | | | | | | | | | | | | | | | This reverts commit 1518b601036351e3f8686bab9c8ae87a8d5af0a3. There is no consensus on how it should work. In general it is impossible to tell a stopped node from a node we could not contact (with some specific exceptions reported by modern net_kernel version such as the Erlang cookie mismatch), so some believe stop should be idempotent and others believe it should not.
* | Merge pull request #1376 from rabbitmq/rabbitmq-server-1362Michael Klishin2017-09-281-5/+7
|\ \ | |/ |/| Ensure that exit code 0 is used for "stop" command
| * Ensure that exit code 0 is used for "stop" commandLuke Bakken2017-09-261-5/+7
|/ | | | Part of the fix to #1362
* Merge pull request #1372 from rabbitmq/rabbitmq-server-1371Michael Klishin2017-09-236-33/+149
|\ | | | | Take ha-mode into account in choosing queue master
| * Be smarter about when to use all cluster nodes vs just those specified by ↵Luke Bakken2017-09-223-8/+22
| | | | | | | | ha-mode:nodes when picking initial queue master node
| * Fix test for ha-mode exactly. Requires sync-ing the queues to ensure slave ↵Luke Bakken2017-09-222-15/+22
| | | | | | | | pids are started
| * Squash a warning => compiles from scratchMichael Klishin2017-09-223-26/+57
| |
| * Add three tests for different ha-mode scenarios with min-mastersLuke Bakken2017-09-221-3/+60
| |
| * Handle suggested_queue_nodes output when slave nodes are emptyLuke Bakken2017-09-223-7/+12
| |
| * Use EUnit assertions hereMichael Klishin2017-09-221-6/+6
| | | | | | | | They produce more informative match failure reports.
| * CosmeticsMichael Klishin2017-09-223-4/+4
| |
| * Take ha-mode into account in choosing queue masterLuke Bakken2017-09-223-6/+8
|/ | | | Fixes #1371
* Merge pull request #1366 from ↵Michael Klishin2017-09-222-36/+160
|\ | | | | | | | | rabbitmq/sync-rabbitmq-config-example-with-configure-page Sync rabbitmq.config.example with PROJECT_ENV
| * Edits, more linksMichael Klishin2017-09-221-34/+48
| |
| * Additions & changes to example config as per rabbitmq-server#1366Gerhard Lazu2017-09-221-8/+16
| |
| * Replace commented line with empty lineGerhard Lazu2017-09-201-1/+1
| |
| * Remove redundant forGerhard Lazu2017-09-201-1/+1
| |
| * Fix typo: SSH -> SSFix typo: SSH -> SSLGerhard Lazu2017-09-201-1/+1
| |
| * Add last missing config properties to example configGerhard Lazu2017-09-201-0/+14
| | | | | | | | This was the last pass, there are no more left in PROJECT_ENV, promise.
| * Add a few extra missing properties & correct defaults in example configGerhard Lazu2017-09-201-6/+19
| |
| * Add other missing config properties from PROJECT_ENV to example configGerhard Lazu2017-09-201-4/+41
| |
| * Remove extra details from new config properties added to example configGerhard Lazu2017-09-201-20/+8
| | | | | | | | Add links instead, as per discussion with @michaelklishin & @hairyhum
| * Add msg_store_io_batch_size to example configGerhard Lazu2017-09-201-0/+7
| |
| * Remove extra details from msg_store_credit_disc_bound from example configGerhard Lazu2017-09-201-14/+0
| | | | | | | | | | Having discussed with @michaelklishin & @hairyhum, it was agreed that these details belong in http://www.rabbitmq.com/persistence-conf.html
| * Explain the IoBatchSize role in push_betas_to_deltasGerhard Lazu2017-09-201-0/+3
| |
| * Clarify IO_BATCH_SIZE in betas -> deltas (not gammas!) commentGerhard Lazu2017-09-201-6/+6
| |
| * Add msg_store_credit_disc_bound to example configGerhard Lazu2017-09-201-0/+20
| |
| * Add mirroring_sync_batch_size to example configGerhard Lazu2017-09-201-0/+18
| |
View Lorry Controller Status