Fixes #2394
A large part of the rabbitmq-server(8) and CLI scripts, both
Bourne-shell and Windows Batch versions, was moved to Erlang code and
the RabbitMQ startup procedure was reorganized to be closer to a regular
Erlang application.
A new application called `rabbitmq_prelaunch` is responsible for:
1. Querying the environment variables to initialize important
variables (using the new `rabbit_env` module in rabbitmq-common).
2. Checking the compatibility with the Erlang/OTP runtime.
3. Configuring Erlang distribution.
5. Writing the PID file.
The application is started early (i.e. it is started before `rabbit`).
The `rabbit` application runs the second half of the prelaunch sequence
at the beginning of the application `start()` function. This second
phase is responsible for the following steps:
1. Preparing the feature flags registry.
2. Reading and validating the configuration.
3. Configuring logging.
4. Running the various cluster checks.
In addition to this prelaunch sequence, the `rabbit` application start
procedure ends with a "postlaunch" sequence which takes care of
starting enabled plugins.
Thanks to this, RabbitMQ can be started with `application:start(rabbit)`
as any other Erlang application. The only caveats are:
* Mnesia must be stopped at the time `rabbit_prelaunch` is started,
and must remain stopped when `rabbit` is started, to allow the
Erlang distribution setup and cluster checks. `rabbit` takes care of
starting Mnesia.
* Likewise for Ra, because it relies on the `ra` application
environment to be configured.
Transitioning from scripts to Erlang code has the following benefits:
* RabbitMQ start behavior should be identical between Unix and
Windows. Also, features should be on par now. For instance, RabbitMQ
now writes a PID file on Windows, like it always did on Unix-based
systems.
* The difference between published packages and a development
environment are greatly reduced. In fact, we removed all the "if
this is a dev working copy, then ..." blocks.
As part of that, the `rabbit` application is now treated like its
plugins: it is packaged as an `.ez` archive and written to the
`plugins` directory (even though it is not technically a plugin).
Also in a development copy, the CLI is copied to the top-level
project. So when testing a plugin for instance, the CLI to use is
`sbin/rabbitmqctl` in the current directory, not the master copy in
`rabbit/scripts`.
* As a consequence of the previous two points, maintaining and testing
on Windows is now made easy. It should even be possible to setup CI
on Windows.
* There are less issues with paths containing non-US-ASCII characters,
which can happen on Windows because RabbitMQ stores its data in user
directories by default.
This process brings at least one more benefit: we now have early logging
during this prelaunch phase, which eases diagnostics and debugging.
There are also behavior changes:
* The new format configuration files used to be converted to an
Erlang-term-based file by the Cuttlefish CLI. To do that,
configuration schemas were copied to a temporary directory and the
generated configuration file was written to RabbitMQ data directory.
Now, Cuttlefish is used as a library: everything happens in memory.
No schemas are copied, no generated configuration is written to
disk.
* The PID file is removed when the Erlang VM exits.
* The `rabbit_config` module was trimmed significantly because most of
the configuration handling is done in `rabbit_prelaunch_conf` now.
* The RabbitMQ nodename does not appear on the command line, therefore
it is missing from ps(1) and top(1) output.
* The `rabbit:start()` function will probably behave differently in
some ways because it defers everything to the Erlang application
controller (instead of reimplementing it).
This commit updates URLs to prefer the https protocol. Redirects are not followed to avoid accidentally expanding intentionally shortened URLs (i.e. if using a URL shortener). # HTTP URLs that Could Not Be Fixed These URLs were unable to be fixed. Please review them to see if they can be manually resolved. * http://alvaro-videla.com/2013/09/rabbitmq-internals-credit-flow-for-erlang-processes.html (200) with 1 occurrences could not be migrated: ([https](https://alvaro-videla.com/2013/09/rabbitmq-internals-credit-flow-for-erlang-processes.html) result SSLHandshakeException). * http://blog.listincomprehension.com/search/label/procket (200) with 2 occurrences could not be migrated: ([https](https://blog.listincomprehension.com/search/label/procket) result ClosedChannelException). * http://dozzie.jarowit.net/trac/wiki/TOML (200) with 2 occurrences could not be migrated: ([https](https://dozzie.jarowit.net/trac/wiki/TOML) result SSLHandshakeException). * http://dozzie.jarowit.net/trac/wiki/subproc (200) with 2 occurrences could not be migrated: ([https](https://dozzie.jarowit.net/trac/wiki/subproc) result SSLHandshakeException). * http://e2project.org (200) with 2 occurrences could not be migrated: ([https](https://e2project.org) result AnnotatedConnectException). * http://erlang.org/doc/man/kernel_app.html (200) with 2 occurrences could not be migrated: ([https](https://erlang.org/doc/man/kernel_app.html) result ConnectTimeoutException). * http://erlang.org/pipermail/erlang-questions/2012-September/069320.html (200) with 1 occurrences could not be migrated: ([https](https://erlang.org/pipermail/erlang-questions/2012-September/069320.html) result ConnectTimeoutException). * http://nitrogenproject.com/ (200) with 4 occurrences could not be migrated: ([https](https://nitrogenproject.com/) result ConnectTimeoutException). * http://proper.softlab.ntua.gr (200) with 2 occurrences could not be migrated: ([https](https://proper.softlab.ntua.gr) result SSLHandshakeException). * http://proper.softlab.ntua.gr/ (200) with 1 occurrences could not be migrated: ([https](https://proper.softlab.ntua.gr/) result SSLHandshakeException). * http://rubybunny.info (200) with 1 occurrences could not be migrated: ([https](https://rubybunny.info) result AnnotatedConnectException). * http://yaws.hyber.org (200) with 2 occurrences could not be migrated: ([https](https://yaws.hyber.org) result AnnotatedConnectException). * http://choven.ca (503) with 2 occurrences could not be migrated: ([https](https://choven.ca) result ConnectTimeoutException). # Fixed URLs ## Fixed But Review Recommended These URLs were fixed, but the https status was not OK. However, the https status was the same as the http request or http redirected to an https URL, so they were migrated. Your review is recommended. * http://fixprotocol.org/ (301) with 1 occurrences migrated to: https://fixtrading.org ([https](https://fixprotocol.org/) result SSLHandshakeException). * http://erldb.org (UnknownHostException) with 1 occurrences migrated to: https://erldb.org ([https](https://erldb.org) result UnknownHostException). * http://www.erlang.org/cgi-bin/ezmlm-cgi?2:mss:1569 (404) with 1 occurrences migrated to: https://www.erlang.org/cgi-bin/ezmlm-cgi?2:mss:1569 ([https](https://www.erlang.org/cgi-bin/ezmlm-cgi?2:mss:1569) result 404). * http://www.rabbitmq.com/quorum-queues.html (404) with 1 occurrences migrated to: https://www.rabbitmq.com/quorum-queues.html ([https](https://www.rabbitmq.com/quorum-queues.html) result 404). ## Fixed Success These URLs were switched to an https URL with a 2xx status. While the status was successful, your review is still recommended. * http://cloudi.org/ with 27 occurrences migrated to: https://cloudi.org/ ([https](https://cloudi.org/) result 200). * http://erlware.org/ with 1 occurrences migrated to: https://erlware.org/ ([https](https://erlware.org/) result 200). * http://inaka.github.io/cowboy-trails/ with 1 occurrences migrated to: https://inaka.github.io/cowboy-trails/ ([https](https://inaka.github.io/cowboy-trails/) result 200). * http://ninenines.eu with 6 occurrences migrated to: https://ninenines.eu ([https](https://ninenines.eu) result 200). * http://pivotal.io with 1 occurrences migrated to: https://pivotal.io ([https](https://pivotal.io) result 200). * http://pubs.opengroup.org/onlinepubs/009695399/utilities/kill.html with 1 occurrences migrated to: https://pubs.opengroup.org/onlinepubs/009695399/utilities/kill.html ([https](https://pubs.opengroup.org/onlinepubs/009695399/utilities/kill.html) result 200). * http://www.actordb.com/ with 2 occurrences migrated to: https://www.actordb.com/ ([https](https://www.actordb.com/) result 200). * http://www.cs.kent.ac.uk/projects/wrangler/Home.html with 1 occurrences migrated to: https://www.cs.kent.ac.uk/projects/wrangler/Home.html ([https](https://www.cs.kent.ac.uk/projects/wrangler/Home.html) result 200). * http://www.erlang.org/ with 1 occurrences migrated to: https://www.erlang.org/ ([https](https://www.erlang.org/) result 200). * http://www.rabbitmq.com/access-control.html with 3 occurrences migrated to: https://www.rabbitmq.com/access-control.html ([https](https://www.rabbitmq.com/access-control.html) result 200). * http://www.rabbitmq.com/authentication.html with 2 occurrences migrated to: https://www.rabbitmq.com/authentication.html ([https](https://www.rabbitmq.com/authentication.html) result 200). * http://www.rabbitmq.com/clustering.html with 5 occurrences migrated to: https://www.rabbitmq.com/clustering.html ([https](https://www.rabbitmq.com/clustering.html) result 200). * http://www.rabbitmq.com/configure.html with 2 occurrences migrated to: https://www.rabbitmq.com/configure.html ([https](https://www.rabbitmq.com/configure.html) result 200). * http://www.rabbitmq.com/confirms.html with 1 occurrences migrated to: https://www.rabbitmq.com/confirms.html ([https](https://www.rabbitmq.com/confirms.html) result 200). * http://www.rabbitmq.com/dlx.html with 1 occurrences migrated to: https://www.rabbitmq.com/dlx.html ([https](https://www.rabbitmq.com/dlx.html) result 200). * http://www.rabbitmq.com/documentation.html with 1 occurrences migrated to: https://www.rabbitmq.com/documentation.html ([https](https://www.rabbitmq.com/documentation.html) result 200). * http://www.rabbitmq.com/download.html with 2 occurrences migrated to: https://www.rabbitmq.com/download.html ([https](https://www.rabbitmq.com/download.html) result 200). * http://www.rabbitmq.com/heartbeats.html with 1 occurrences migrated to: https://www.rabbitmq.com/heartbeats.html ([https](https://www.rabbitmq.com/heartbeats.html) result 200). * http://www.rabbitmq.com/lazy-queues.html with 1 occurrences migrated to: https://www.rabbitmq.com/lazy-queues.html ([https](https://www.rabbitmq.com/lazy-queues.html) result 200). * http://www.rabbitmq.com/ldap.html with 4 occurrences migrated to: https://www.rabbitmq.com/ldap.html ([https](https://www.rabbitmq.com/ldap.html) result 200). * http://www.rabbitmq.com/management.html with 6 occurrences migrated to: https://www.rabbitmq.com/management.html ([https](https://www.rabbitmq.com/management.html) result 200). * http://www.rabbitmq.com/memory-use.html with 3 occurrences migrated to: https://www.rabbitmq.com/memory-use.html ([https](https://www.rabbitmq.com/memory-use.html) result 200). * http://www.rabbitmq.com/memory.html with 2 occurrences migrated to: https://www.rabbitmq.com/memory.html ([https](https://www.rabbitmq.com/memory.html) result 200). * http://www.rabbitmq.com/monitoring.html with 1 occurrences migrated to: https://www.rabbitmq.com/monitoring.html ([https](https://www.rabbitmq.com/monitoring.html) result 200). * http://www.rabbitmq.com/nettick.html with 2 occurrences migrated to: https://www.rabbitmq.com/nettick.html ([https](https://www.rabbitmq.com/nettick.html) result 200). * http://www.rabbitmq.com/networking.html with 7 occurrences migrated to: https://www.rabbitmq.com/networking.html ([https](https://www.rabbitmq.com/networking.html) result 200). * http://www.rabbitmq.com/partitions.html with 2 occurrences migrated to: https://www.rabbitmq.com/partitions.html ([https](https://www.rabbitmq.com/partitions.html) result 200). * http://www.rabbitmq.com/persistence-conf.html with 3 occurrences migrated to: https://www.rabbitmq.com/persistence-conf.html ([https](https://www.rabbitmq.com/persistence-conf.html) result 200). * http://www.rabbitmq.com/plugins.html with 1 occurrences migrated to: https://www.rabbitmq.com/plugins.html ([https](https://www.rabbitmq.com/plugins.html) result 200). * http://www.rabbitmq.com/previous.html with 1 occurrences migrated to: https://www.rabbitmq.com/previous.html ([https](https://www.rabbitmq.com/previous.html) result 200). * http://www.rabbitmq.com/shovel.html with 2 occurrences migrated to: https://www.rabbitmq.com/shovel.html ([https](https://www.rabbitmq.com/shovel.html) result 200). * http://www.rabbitmq.com/ssl.html with 2 occurrences migrated to: https://www.rabbitmq.com/ssl.html ([https](https://www.rabbitmq.com/ssl.html) result 200). * http://www.rabbitmq.com/stomp.html with 1 occurrences migrated to: https://www.rabbitmq.com/stomp.html ([https](https://www.rabbitmq.com/stomp.html) result 200). * http://www.rebar3.org with 1 occurrences migrated to: https://www.rebar3.org ([https](https://www.rebar3.org) result 200). * http://contributor-covenant.org with 1 occurrences migrated to: https://contributor-covenant.org ([https](https://contributor-covenant.org) result 301). * http://contributor-covenant.org/version/1/3/0/ with 1 occurrences migrated to: https://contributor-covenant.org/version/1/3/0/ ([https](https://contributor-covenant.org/version/1/3/0/) result 301). * http://inaka.github.com/apns4erl with 1 occurrences migrated to: https://inaka.github.com/apns4erl ([https](https://inaka.github.com/apns4erl) result 301). * http://inaka.github.com/edis/ with 1 occurrences migrated to: https://inaka.github.com/edis/ ([https](https://inaka.github.com/edis/) result 301). * http://lasp-lang.org/ with 1 occurrences migrated to: https://lasp-lang.org/ ([https](https://lasp-lang.org/) result 301). * http://rabbitmq.com//cluster-formation.html with 1 occurrences migrated to: https://rabbitmq.com//cluster-formation.html ([https](https://rabbitmq.com//cluster-formation.html) result 301). * http://rabbitmq.com/access-control.html with 4 occurrences migrated to: https://rabbitmq.com/access-control.html ([https](https://rabbitmq.com/access-control.html) result 301). * http://rabbitmq.com/authentication.html with 2 occurrences migrated to: https://rabbitmq.com/authentication.html ([https](https://rabbitmq.com/authentication.html) result 301). * http://rabbitmq.com/clustering.html with 1 occurrences migrated to: https://rabbitmq.com/clustering.html ([https](https://rabbitmq.com/clustering.html) result 301). * http://rabbitmq.com/configure.html with 1 occurrences migrated to: https://rabbitmq.com/configure.html ([https](https://rabbitmq.com/configure.html) result 301). * http://rabbitmq.com/documentation.html with 2 occurrences migrated to: https://rabbitmq.com/documentation.html ([https](https://rabbitmq.com/documentation.html) result 301). * http://rabbitmq.com/heartbeats.html with 1 occurrences migrated to: https://rabbitmq.com/heartbeats.html ([https](https://rabbitmq.com/heartbeats.html) result 301). * http://rabbitmq.com/lazy-queues.html with 1 occurrences migrated to: https://rabbitmq.com/lazy-queues.html ([https](https://rabbitmq.com/lazy-queues.html) result 301). * http://rabbitmq.com/ldap.html with 3 occurrences migrated to: https://rabbitmq.com/ldap.html ([https](https://rabbitmq.com/ldap.html) result 301). * http://rabbitmq.com/logging.html with 1 occurrences migrated to: https://rabbitmq.com/logging.html ([https](https://rabbitmq.com/logging.html) result 301). * http://rabbitmq.com/management.html with 5 occurrences migrated to: https://rabbitmq.com/management.html ([https](https://rabbitmq.com/management.html) result 301). * http://rabbitmq.com/memory.html with 1 occurrences migrated to: https://rabbitmq.com/memory.html ([https](https://rabbitmq.com/memory.html) result 301). * http://rabbitmq.com/networking.html with 3 occurrences migrated to: https://rabbitmq.com/networking.html ([https](https://rabbitmq.com/networking.html) result 301). * http://rabbitmq.com/persistence-conf.html with 1 occurrences migrated to: https://rabbitmq.com/persistence-conf.html ([https](https://rabbitmq.com/persistence-conf.html) result 301). * http://rabbitmq.com/plugins.html with 1 occurrences migrated to: https://rabbitmq.com/plugins.html ([https](https://rabbitmq.com/plugins.html) result 301). * http://rabbitmq.com/shovel.html with 1 occurrences migrated to: https://rabbitmq.com/shovel.html ([https](https://rabbitmq.com/shovel.html) result 301). * http://rabbitmq.com/ssl.html with 1 occurrences migrated to: https://rabbitmq.com/ssl.html ([https](https://rabbitmq.com/ssl.html) result 301). * http://rabbitmq.com/stomp.html with 1 occurrences migrated to: https://rabbitmq.com/stomp.html ([https](https://rabbitmq.com/stomp.html) result 301). * http://saleyn.github.com/erlexec with 1 occurrences migrated to: https://saleyn.github.com/erlexec ([https](https://saleyn.github.com/erlexec) result 301). * http://www.erlang.org/doc/man/inet.html with 2 occurrences migrated to: https://www.erlang.org/doc/man/inet.html ([https](https://www.erlang.org/doc/man/inet.html) result 301). * http://www.erlang.org/doc/man/sys.html with 1 occurrences migrated to: https://www.erlang.org/doc/man/sys.html ([https](https://www.erlang.org/doc/man/sys.html) result 301). * http://www.mozilla.org/MPL/ with 228 occurrences migrated to: https://www.mozilla.org/MPL/ ([https](https://www.mozilla.org/MPL/) result 301). * http://zhongwencool.github.io/observer_cli with 1 occurrences migrated to: https://zhongwencool.github.io/observer_cli ([https](https://zhongwencool.github.io/observer_cli) result 301).
We use rabbitmqctl(8) inside `rabbitmq-env` to query the settings of the remote nodes (path to plugins, feature flags file, the database, etc). However, before this patch, we didn't pass the name of the remote node as specified by the user with `-n`. Therefore, the default node name was used (`rabbit@$hostname`) and that node may not exist at all. This caused the executed script to run with possibly incorrect settings. In particular, this prevented rabbitmq-plugins(8) from working properly on a node started with `gmake run-broker`. This should now be fixed because we extract the remote node name from the command line arguments and pass it to the child rabbitmqctl(8).
The goal is that each breaking change is implemented in way that a RabbitMQ node having that change can work in the same cluster as other nodes which do not know about it. The feature flags are here to indicate which breaking changes are "available" (i.e. the node knows about the breaking changes, but it still runs in a backward-compatible mode) and which ones are currently activated (i.e. the node uses the new code and older node can not be part of the cluster anymore). Enabling a feature flag (i.e. activating a breaking change) is a manual operation where the user validates that the cluster is ready to switch to the new world after the breaking change. Therefore the subsystem ensures RabbitMQ nodes can talk to each other even if they don't have the same code as long as they agree on using the common subset, and prevents RabbitMQ nodes from talking to each other once the new code is being used and one node does not understand it. The consequence is that if a breaking change is implemented using this new subsystem, a cluster can be upgraded one node at a time instead of shutting down the entire cluster to upgrade. Of course, the ability to implement a breaking change in such a way entirely depends on the nature of that change. This new subsystem does not guarantee that a cluster shutdown will never be required again. [#159298729]