summaryrefslogtreecommitdiff
path: root/apps
Commit message (Collapse)AuthorAgeFilesLines
* Stream Queuekjnilsson2020-09-301-1/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is an aggregated commit of all changes related to the initial implementation of queue types and on top of that the stream queue type. The varios commit messages have simply been included mostly un-edited below. Make rabbit_amqqueue:not_found_or_absent_dirty/1 visible For use in the stream plugin. Use bigger retention policy on max-age test Set coordinator timeout to 30s Handle coordinator unavailable error Handle operator policies as maps when checking if is applicable Add is_policy_applicable/2 to classic queues Ignore restart commands if the stream has been deleted It could happen that after termination some of the monitors are still up and trigger writer/replica restarts Policy support on stream queues Remove subscription events on stream coordinator Ensure old leaders are removed from monitors Introduce delay when retrying a failed phase Note that this ensures monitor is setup, there was a bug where no monitor was really started when re-trying the same phase Restart replicas after leader election instead of relying on old monitors Use timer for stream coordinator retries Fix stream stats for members/online Multiple fixes for replica monitoring and restart Ensure pending commands are appended at the end and re-run Ensure phase is reset with the state Remove duplicates from replica list Restart current phase on state_enter Remove unused import Ensure rabbit is running when checking for stream quorum Restart replicas Add a close/1 function to queue types So that we can get a chance of cleaning up resources if needed. Stream queues close their osiris logs at this point. fix compiler errors stream-queue: take retention into account When calculating ready messages metrics. Add osiris to the list of rabbit deps Retry restart of replicas Do not restart replicas or leaders after receiving a delete cluster command Add more logging to the stream coordinator Monitor subscribed processes on the stream coordinator Memory breakdown for stream queues Update quorum queue event formatter rabbit_msg_record fixes Refactor channel confirms Remove old unconfirmed_messages module that was designed to handle multiple queue fan in logic including all ha mirrors etc. Replaced with simpler rabbit_confirms module that handles the fan out and leaves any queue specific logic (such as confirms from mirrors) to the queue type implemention. Also this module has a dedicated test module. Which is nice. Backward compatibility with 3.8.x events Supports mixed version cluster upgrades Match specification when stream queue already exists Max age retention for stream queues Stop all replicas before starting leader election stream: disallow global qos remove IS_CLASSIC|QUORUM macros Ensure only classic queues are notified on channel down This also removes the delivering_queues map in the channel state as it should not be needed for this and just cause additional unecessary accounting. Polish AMQP 1.0/0.9.1 properties conversion Support byte in application properties, handle 1-bit representation for booleans. Use binary in header for long AMQP 1.0 ID Fix AMQP 1.0 to 0.9.1 conversion Fix test due to incorrect type Convert timestamp application properties to/from seconds AMQP 1.0 uses milliseconds for timestamp and AMQP 0.9.1 uses seconds, so conversion needed. Dialyzer fixes Handle all message-id types AMQP 1.0 is more liberal in it's allowed types of message-id and correlation-id - this adds headers to describe the type of the data in the message_id / correlation_id properties and also handles the case where the data cannot fit by again using headers. Resize stream coordinator cluster when broker configuration changes convert timestamp to and fro seconds user_id should be a binary message annotations keys need to be symbols stream-queue: default exchange and routing key As these won't be present for data written using the rabbitmq-stream plugin. Add exchange, routing key as message annotations To the AMQP 1.0 formatted data to enable roundtrip. Add osiris logging module config And update logging config test suite. Restart election when start of new leader fails The node might have just gone down so we need to try another one Only aux keeps track of phase now, as it might change if the leader election fails Stream coordinator refactor - all state is kept on the ra machine Ensure any ra cluster not a qq is not cleaned up Fixes to recovery and monitoring Add AMQP 1.0 common to dependencies Add rabbit_msg_record module To handle conversions into internal stream storage format. Use rabbitmq-common stream-queue branch Use SSH for osiris dependency Stream coordinator: delete replica Stream coordinator: add replica Stream coordinator: leader failover Stream coordinator: declare and delete Test consuming from a random offset Previous offsets should not be delivered to consumers Consume from stream replicas and multiple test fixes Use max-length-bytes and add new max-segment-size Use SSH for osiris dependency Basic cancel for stream queues Publish stream queues and settle/reject/requeue refactor Consume from stream queues Fix recovery Publish stream messages Add/delete stream replicas Use safe queue names Set retention policy for stream queues Required by the ctl command [#171207092] Stream queue delete queue fix missing callback impl Stream queue declare Queue type abstraction And use the implementing module as the value of the amqqueue record `type` field. This will allow for easy dispatch to the queue type implementation. Queue type abstraction Move queue declare into rabbit_queue_type Move queue delete into queue type implementation Queue type: dequeue/basic_get Move info inside queue type abstraction Move policy change into queue type interface Add purge to queue type Add recovery to the queue type interface Rename amqqueue quorum_nodes field To a more generic an extensible opaque queue type specific map. Fix tests and handle classic API response Fix HA queue confirm bug All mirrors need to be present as queue names. This introduces context linking allowing additional queue refs to be linked to a single "master" queue ref contining the actual queue context. Fix issue with events of deleted queues Also update queue type smoke test to use a cluster by default. correct default value of amqqueue getter Move classic queues further inside queue type interface why [TrackerId] Dialyzer fixes
* Increase aten poll intervalaten-poll-intervalkjnilsson2020-09-181-0/+9
| | | | | | | | | A longer interval will reduce the likely hood that unnecessary leader elections are triggered in quorum queues due to transient network issues. This increases it from the default of 1s to 5s which increases the time it takes to detect genuine network partitions somewhat. Ra still uses monitors so connection failures will not have to wait this additional time.
* rabbit_prelaunch_errors, rabbit_core_ff: Fix style consistencyJean-Sébastien Pédron2020-07-231-3/+5
|
* rabbit_prelaunch_conf: Integrate application load error to existing messageJean-Sébastien Pédron2020-07-221-5/+4
| | | | | This fixes the layout of the logged list and the style of the source file.
* rabbit_prelaunch_conf: log application loading failuresMichael Klishin2020-07-211-1/+3
|
* A typoMichael Klishin2020-07-171-1/+1
|
* Improve a couple of error messagesMichael Klishin2020-07-171-3/+3
|
* Hide maintenance mode behind a feature flagMichael Klishin2020-07-141-3/+3
| | | | | | | | | | | | | | Otherwise we'd lose mixed version cluster compatibility. We cannot pre-create a table since it will be checked for existence on cluster peers. We cannot use an upgrade function since it assumes a cluster-wide restart. A feature flag avoids as many headaches as possible and when flags are detected to be compatible on node start, they are all enabled automatically. References #2321.
* Switch to Mozilla Public License 2.0 (MPL 2.0)dcorbacho2020-07-103-34/+9
|
* rabbit_prelaunch_early_logging: Configure `rabbit_log` as a Logger handlerJean-Sébastien Pédron2020-07-061-0/+6
| | | | | | | | | | | | | | | Currently, when the new Logger API exposed by Erlang is being used by an application, messages are lost once Lager is configured. That is because Lager does not know about Logger in this context. To fix this, `rabbit_log` is configured as a Logger handler which calls Lager API. This means that Logger messages are passed to Lager. Logger's log level is to set to pass all messages (including debug). Message routing/filtering is done by Lager instead. This change depends on new log/2 function being added to the `rabbit_log` module in rabbitmq-common.
* rabbit_prelaunch: Load configuration before Erlang dist. is startedJean-Sébastien Pédron2020-06-101-4/+4
| | | | | | | | The reason is that kernel's `net_ticktime` can be configured in the Cuttlefish or Erlang configuration files. If we start the distribution first, then the configuration parameter is ignored. Fixes #2377.
* rabbit_prelaunch_conf: Style fixesJean-Sébastien Pédron2020-06-091-15/+20
| | | | Things like staying in the 80-column limit and spaces around '='.
* Pass expected {encrypted, _} form to credentials_obfuscationLuke Bakken2020-05-201-6/+4
|
* CosmeticsMichael Klishin2020-05-211-1/+1
|
* Refresh credentials_obfuscation configurationLuke Bakken2020-05-191-0/+3
|
* Set credentials_obfuscation secret to cookieLuke Bakken2020-05-191-0/+7
| | | | | | Requires this PR and version 2.0.0 of the library: https://github.com/rabbitmq/credentials-obfuscation/pull/5
* Changed or overridden default configuration values related to quorum queuesVanlightly2020-05-011-1/+2
| | | | | | Specifically: - changed default quorum_commands_soft_limit from 256 to 32 - override Ra wal_max_batch_size to 4096
* rabbit_prelaunch_conf: Always handle config. files with CuttlefishJean-Sébastien Pédron2020-04-231-51/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This has several benefits: 1. It simplifies the code, all configuration being handled by the same code path (no more condition on Erlang-term-based vs. Cuttlefish). `rabbit_config` shrinks quite a lot in the process. 2. We can use additional configuration files AND an Erlang-term-based configuration file. In other words, it is possible to use the same existing Erlang-term-based file and introduce Cuttlefish files when needed. It allows a user to run RabbitMQ with: RABBITMQ_CONFIG_FILE=/path/to/rabbitmq.config \ RABBITMQ_CONFIG_FILES=/path/to/conf.d/*.conf \ ./sbin/rabbitmq-server A developer can do the same with `make run-broker`: make run-broker \ RABBITMQ_CONFIG_FILES=/path/to/conf.d/*.conf In the example above, the main configuration file generated by rabbitmq-run.mk is an Erlang-term-based one. This is implemented by calling Cuttlefish with a (possibly empty) list of additional files and the Erlang-term-based file as the advanced configuration file. References #2180.
* Remove Ra segment_max_entries overridekjnilsson2020-04-031-2/+0
| | | | | So that it uses Ra's internal default of 4096 instead which is safer for larger message sizes.
* Update copyright (year 2020)Jean-Sébastien Pédron2020-03-103-3/+3
|
* rabbit_prelaunch_conf: Load additional Cuttlefish configuration filesJean-Sébastien Pédron2020-02-281-7/+46
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | By default, in addition to `$RABBITMQ_CONFIG_FILE`, we also load all matching `$RABBITMQ_CONFIG_FILES`. It allows to split the configuration into many files for easier management. Here is the behavior in more details: * If `$RABBITMQ_CONFIG_FILES` is a directory, all files directly inside it are considered. * If `$RABBITMQ_CONFIG_FILES` is a glob pattern, all files matching the pattern are considered. * In both cases, subdirectories (and whatever they contain) are ignored. * `$RABBITMQ_CONFIG_FILES` is only relevant when the main configuration is either missing (the file does not exist or is empty) or uses the Cuttlefish format. * Additional configuration files must use the Cuttlefish format. The default value of `$RABBITMQ_CONFIG_FILES` is: * `/etc/rabbitmq/conf.d/*.conf` on Unix * `%APPDATA%\RabbitMQ\conf.d\*.conf` on Windows Error messages related to Cuttlefish parsing were improved in the process. [#171491267]
* rabbit_prelaunch_conf: Remove out-of-date commentJean-Sébastien Pédron2020-02-281-1/+0
| | | | We already try both formats when we parse the configuration.
* rabbit_prelaunch_sighandler: Comment out code which is unusedJean-Sébastien Pédron2020-02-261-5/+8
| | | | | | | | | | As it is ready to handle the stop of RabbitMQ and has been tested, we can keep the code around just in case. However, this causes a Dialyzer error which must be fixed, so let's comment it out. [#171234776]
* rabbitmq_prelaunch: Skip signal handler on WindowsJean-Sébastien Pédron2020-02-241-7/+12
| | | | | | | This is specific to Unix, and `os:set_signal()` reminded it to me with a `badarg` exception. [#171234776]
* Only handle SIGHUP and SIGTSTPJean-Sébastien Pédron2020-02-202-0/+88
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Here is a summary of RabbitMQ signal handling: == SIGTERM == After #2180, `rabbit` is a regular Erlang application and `application:stop(rabbit)` terminates RabbitMQ gracefully. This means that `init:stop()` shuts the service down properly. Therefore, the default handling of SIGTERM, which calls `init:stop()`, is correct. rabbitmq-server(8) already relies on this mechanism. This commit restores the default signal handler which already does the right thing. No need to do it ourselves. == SIGHUP and SIGTSTP == SIHGUP is usually used to reload the configuration without restarting the service and/or reopen log files after log file rotation. SIGTSTP is sent when a user types Ctrl+Z to pause a program and get back to the shell. Both signals have common behavior we can't satisfy currently. Note that we don't handle SIGCONT which is the one used to resume a program after SIGTSTP. The system default behavior is already good (the signal is discarded). To be consistent with rabbitmq-server(8) signal handling, the signals are ignored until we can do something about them. == SIGQUIT == This signal is meant to terminate the process immediately and create a core dump. If possible, temporary files should even be kept around. The default behavior in Erlang is to call `erlang:halt()` which is a sane default: we should not stop RabbitMQ gracefully. This commit restores this behavior. == SIGUSR1 and SIGUSR2 == Erlang uses SIGUSR1 to crash the VM and create an `erl_crash.dump` file. We already used this in the past to debug RabbitMQ. Again, a sane default. This commit restores this behavior. == Other signals == We keep the default behavior of all other signals. None of them are meant to stop the program gracefully anyway. If a user wants to stop RabbitMQ, he will already use the common accepted signal for this purpose (i.e. SIGTERM). Another change in this commit is the way we setup the signal handler: * We don't replace the default Erlang signal handler, just add ours. * We do it very early in rabbitmq_prelaunch. Like other things configured by this application, we do not uninstall the handler when the application is stopped. This reverts: * commit 6a4d2721d06b8c70a36e29e6c51bbef6608def55 * commit fa607e4a25d6142bb17a90b44ef757572a923c09
* rabbitmq_prelaunch: Fix all warnings reported by DialyzerJean-Sébastien Pédron2020-02-115-25/+42
| | | | | | They are all return values being unmatched. Many were related to list comprehensions being used as a loop mechanism but the result was unused. These list comprehensions were replaced by lists:foreach/2.
* Convert systemd notification to prelaunch stepsPhilip Kuryloski2020-02-105-68/+323
| | | | | | | | | | | | | | | | | | Introduce the rabbit_boot_state module, which extracts boot state management out of rabbit_prelaunch. External boot state listeners, such as systemd, now live under the rabbit_boot_state_sup supervisor, which dispatches boot state updates to all of its children as a gen_server cast. Additionally: - the systemd listener now reads the NOTIFY_SOCKET env var directly, rather than through rabbit_env, to avoid the need to wait for the rabbit_env context to initialize - the sytemd listener now only checks for the sd_notify module and NOTIFY_SOCKET env var once upon startup, exiting gracefully when not needed - systemd related log messages are now routed through lager
* Move all RabbitMQ-specific environment variables to `rabbit_env`Jean-Sébastien Pédron2020-02-031-0/+10
| | | | | | | | | | The reading of `$NOTIFY_SOCKET` is also moved at the same time. This is in preparation of the work around start/stop status. There is an associated commit in rabbitmq-common to update `rabbit_env` and record the origin of each variable. [#170149339]
* rabbit_prelaunch_early_logging: Setup all categories earlyJean-Sébastien Pédron2020-01-131-15/+60
| | | | | | | ... instead of `prelaunch` category only. This allows to see the messages logged to the new `feature_flags` category as they are logged before the final configuration of logging (because logging requires configuration which requires feature flags).
* rabbitmq_prelaunch: Run rabbit_prelaunch_conf:setup/1 earlierLuke Bakken2019-12-192-1/+493
| | | | | | | | | | | | | | | | | This fixes the issue where applications' configuration is applied after they are started (and they do not read their environment again after startup). This is the case of applications such as `ra` or `sysmon_handler`: they are dependencies of `rabbit` and the Erlang application controller will start them before. Now, the configuration is loaded during the first prelaunch phase by rabbitmq_prelaunch, hopefully before those applications are started. To permit this change, the code updating the `enabled_plugins_file` was moved to its own module. This one can't be moved to the rabbitmq_prelaunch application because it depends on `rabbit_plugins`. While here, add a couple assertions by checking return values.
* rabbitmq_prelaunch: Fix xref errorsLuke Bakken2019-12-192-4/+4
|
* Move most of shell scripts to Erlang codeJean-Sébastien Pédron2019-12-119-0/+644
A large part of the rabbitmq-server(8) and CLI scripts, both Bourne-shell and Windows Batch versions, was moved to Erlang code and the RabbitMQ startup procedure was reorganized to be closer to a regular Erlang application. A new application called `rabbitmq_prelaunch` is responsible for: 1. Querying the environment variables to initialize important variables (using the new `rabbit_env` module in rabbitmq-common). 2. Checking the compatibility with the Erlang/OTP runtime. 3. Configuring Erlang distribution. 5. Writing the PID file. The application is started early (i.e. it is started before `rabbit`). The `rabbit` application runs the second half of the prelaunch sequence at the beginning of the application `start()` function. This second phase is responsible for the following steps: 1. Preparing the feature flags registry. 2. Reading and validating the configuration. 3. Configuring logging. 4. Running the various cluster checks. In addition to this prelaunch sequence, the `rabbit` application start procedure ends with a "postlaunch" sequence which takes care of starting enabled plugins. Thanks to this, RabbitMQ can be started with `application:start(rabbit)` as any other Erlang application. The only caveats are: * Mnesia must be stopped at the time `rabbit_prelaunch` is started, and must remain stopped when `rabbit` is started, to allow the Erlang distribution setup and cluster checks. `rabbit` takes care of starting Mnesia. * Likewise for Ra, because it relies on the `ra` application environment to be configured. Transitioning from scripts to Erlang code has the following benefits: * RabbitMQ start behavior should be identical between Unix and Windows. Also, features should be on par now. For instance, RabbitMQ now writes a PID file on Windows, like it always did on Unix-based systems. * The difference between published packages and a development environment are greatly reduced. In fact, we removed all the "if this is a dev working copy, then ..." blocks. As part of that, the `rabbit` application is now treated like its plugins: it is packaged as an `.ez` archive and written to the `plugins` directory (even though it is not technically a plugin). Also in a development copy, the CLI is copied to the top-level project. So when testing a plugin for instance, the CLI to use is `sbin/rabbitmqctl` in the current directory, not the master copy in `rabbit/scripts`. * As a consequence of the previous two points, maintaining and testing on Windows is now made easy. It should even be possible to setup CI on Windows. * There are less issues with paths containing non-US-ASCII characters, which can happen on Windows because RabbitMQ stores its data in user directories by default. This process brings at least one more benefit: we now have early logging during this prelaunch phase, which eases diagnostics and debugging. There are also behavior changes: * The new format configuration files used to be converted to an Erlang-term-based file by the Cuttlefish CLI. To do that, configuration schemas were copied to a temporary directory and the generated configuration file was written to RabbitMQ data directory. Now, Cuttlefish is used as a library: everything happens in memory. No schemas are copied, no generated configuration is written to disk. * The PID file is removed when the Erlang VM exits. * The `rabbit_config` module was trimmed significantly because most of the configuration handling is done in `rabbit_prelaunch_conf` now. * The RabbitMQ nodename does not appear on the command line, therefore it is missing from ps(1) and top(1) output. * The `rabbit:start()` function will probably behave differently in some ways because it defers everything to the Erlang application controller (instead of reimplementing it).
View Lorry Controller Status