Move most of shell scripts to Erlang code

A large part of the rabbitmq-server(8) and CLI scripts, both
Bourne-shell and Windows Batch versions, was moved to Erlang code and
the RabbitMQ startup procedure was reorganized to be closer to a regular
Erlang application.

A new application called `rabbitmq_prelaunch` is responsible for:

  1. Querying the environment variables to initialize important
     variables (using the new `rabbit_env` module in rabbitmq-common).
  2. Checking the compatibility with the Erlang/OTP runtime.
  3. Configuring Erlang distribution.
  5. Writing the PID file.

The application is started early (i.e. it is started before `rabbit`).

The `rabbit` application runs the second half of the prelaunch sequence
at the beginning of the application `start()` function. This second
phase is responsible for the following steps:

  1. Preparing the feature flags registry.
  2. Reading and validating the configuration.
  3. Configuring logging.
  4. Running the various cluster checks.

In addition to this prelaunch sequence, the `rabbit` application start
procedure ends with a "postlaunch" sequence which takes care of
starting enabled plugins.

Thanks to this, RabbitMQ can be started with `application:start(rabbit)`
as any other Erlang application. The only caveats are:

  * Mnesia must be stopped at the time `rabbit_prelaunch` is started,
    and must remain stopped when `rabbit` is started, to allow the
    Erlang distribution setup and cluster checks. `rabbit` takes care of
    starting Mnesia.

  * Likewise for Ra, because it relies on the `ra` application
    environment to be configured.

Transitioning from scripts to Erlang code has the following benefits:

  * RabbitMQ start behavior should be identical between Unix and
    Windows. Also, features should be on par now. For instance, RabbitMQ
    now writes a PID file on Windows, like it always did on Unix-based
    systems.

  * The difference between published packages and a development
    environment are greatly reduced. In fact, we removed all the "if
    this is a dev working copy, then ..." blocks.

    As part of that, the `rabbit` application is now treated like its
    plugins: it is packaged as an `.ez` archive and written to the
    `plugins` directory (even though it is not technically a plugin).

    Also in a development copy, the CLI is copied to the top-level
    project. So when testing a plugin for instance, the CLI to use is
    `sbin/rabbitmqctl` in the current directory, not the master copy in
    `rabbit/scripts`.

  * As a consequence of the previous two points, maintaining and testing
    on Windows is now made easy. It should even be possible to setup CI
    on Windows.

  * There are less issues with paths containing non-US-ASCII characters,
    which can happen on Windows because RabbitMQ stores its data in user
    directories by default.

This process brings at least one more benefit: we now have early logging
during this prelaunch phase, which eases diagnostics and debugging.

There are also behavior changes:

  * The new format configuration files used to be converted to an
    Erlang-term-based file by the Cuttlefish CLI. To do that,
    configuration schemas were copied to a temporary directory and the
    generated configuration file was written to RabbitMQ data directory.

    Now, Cuttlefish is used as a library: everything happens in memory.
    No schemas are copied, no generated configuration is written to
    disk.

  * The PID file is removed when the Erlang VM exits.

  * The `rabbit_config` module was trimmed significantly because most of
    the configuration handling is done in `rabbit_prelaunch_conf` now.

  * The RabbitMQ nodename does not appear on the command line, therefore
    it is missing from ps(1) and top(1) output.

  * The `rabbit:start()` function will probably behave differently in
    some ways because it defers everything to the Erlang application
    controller (instead of reimplementing it).

5 files changed, 66 insertions, 30 deletions

diff --git a/test/clustering_management_SUITE.erl b/test/clustering_management_SUITE.erl
index 90bb75636e..7335767398 100644
--- a/test/clustering_management_SUITE.erl
+++ b/test/clustering_management_SUITE.erl
@@ -224,8 +224,7 @@ join_cluster_bad_operations(Config) ->
     ok = stop_app(Hare),
     assert_failure(fun () -> start_app(Hare) end),
     ok = start_app(Rabbit),
-    %% The Erlang VM has stopped after previous rabbit app failure
-    ok = rabbit_ct_broker_helpers:start_node(Config, Hare),
+    ok = start_app(Hare),
     ok.
 
 %% This tests that the nodes in the cluster are notified immediately of a node
diff --git a/test/feature_flags_SUITE.erl b/test/feature_flags_SUITE.erl
index 16f11733d5..8f672314f0 100644
--- a/test/feature_flags_SUITE.erl
+++ b/test/feature_flags_SUITE.erl
@@ -136,14 +136,18 @@ init_per_group(clustering, Config) ->
                 [{rmq_nodes_count, 2},
                  {rmq_nodes_clustered, false},
                  {start_rmq_with_plugins_disabled, true}]),
-    build_my_plugin(Config1);
+    rabbit_ct_helpers:run_setup_steps(Config1, [
+        fun build_my_plugin/1
+      ]);
 init_per_group(activating_plugin, Config) ->
     Config1 = rabbit_ct_helpers:set_config(
                 Config,
                 [{rmq_nodes_count, 2},
                  {rmq_nodes_clustered, true},
                  {start_rmq_with_plugins_disabled, true}]),
-    build_my_plugin(Config1);
+    rabbit_ct_helpers:run_setup_steps(Config1, [
+        fun build_my_plugin/1
+      ]);
 init_per_group(_, Config) ->
     Config.
 
@@ -910,13 +914,31 @@ build_my_plugin(Config) ->
                 {ok, _} ->
                     {_, OtherPlugins1} = list_my_plugin_plugins(PluginSrcDir),
                     remove_other_plugins(PluginSrcDir, OtherPlugins1),
-                    Config1;
+                    update_cli_path(Config1, PluginSrcDir);
                 {error, _} ->
                     {skip, "Failed to compile the `my_plugin` test plugin"}
             end;
         _ ->
             remove_other_plugins(PluginSrcDir, OtherPlugins),
-            Config1
+            update_cli_path(Config1, PluginSrcDir)
+    end.
+
+update_cli_path(Config, PluginSrcDir) ->
+    SbinDir = filename:join(PluginSrcDir, "sbin"),
+    Rabbitmqctl = filename:join(SbinDir, "rabbitmqctl"),
+    RabbitmqPlugins = filename:join(SbinDir, "rabbitmq-plugins"),
+    RabbitmqQueues = filename:join(SbinDir, "rabbitmq-queues"),
+    case filelib:is_regular(Rabbitmqctl) of
+        true ->
+            ct:pal(?LOW_IMPORTANCE,
+                   "Switching to CLI in e.g. ~s", [Rabbitmqctl]),
+            rabbit_ct_helpers:set_config(
+              Config,
+              [{rabbitmqctl_cmd, Rabbitmqctl},
+               {rabbitmq_plugins_cmd, RabbitmqPlugins},
+               {rabbitmq_queues_cmd, RabbitmqQueues}]);
+        false ->
+            Config
     end.
 
 list_my_plugin_plugins(PluginSrcDir) ->
diff --git a/test/unit_SUITE.erl b/test/unit_SUITE.erl
index 2bea590342..e8abd2ae90 100644
--- a/test/unit_SUITE.erl
+++ b/test/unit_SUITE.erl
@@ -44,8 +44,6 @@ groups() ->
             ]},
           content_framing,
           content_transcoding,
-          decrypt_config,
-          listing_plugins_from_multiple_directories,
           rabbitmqctl_encode,
           pmerge,
           plmerge,
@@ -72,7 +70,9 @@ groups() ->
           decrypt_start_app,
           decrypt_start_app_file,
           decrypt_start_app_undefined,
-          decrypt_start_app_wrong_passphrase
+          decrypt_start_app_wrong_passphrase,
+          decrypt_config,
+          listing_plugins_from_multiple_directories
         ]}
     ].
 
@@ -81,20 +81,13 @@ end_per_group(_, Config) -> Config.
 
 init_per_testcase(TC, Config) when TC =:= decrypt_start_app;
                                    TC =:= decrypt_start_app_file;
-                                   TC =:= decrypt_start_app_undefined ->
-    application:load(rabbit),
-    application:set_env(rabbit, feature_flags_file, ""),
+                                   TC =:= decrypt_start_app_undefined;
+                                   TC =:= decrypt_start_app_wrong_passphrase ->
+    application:set_env(rabbit, feature_flags_file, "", [{persistent, true}]),
     Config;
 init_per_testcase(_Testcase, Config) ->
     Config.
 
-end_per_testcase(TC, _Config) when TC =:= decrypt_start_app;
-                                   TC =:= decrypt_start_app_file;
-                                   TC =:= decrypt_start_app_undefined ->
-    application:unload(rabbit),
-    application:unload(rabbit_shovel_test);
-end_per_testcase(decrypt_config, _Config) ->
-    application:unload(rabbit);
 end_per_testcase(_TC, _Config) ->
     ok.
 
@@ -177,7 +170,7 @@ decrypt_config(_Config) ->
     ok.
 
 do_decrypt_config(Algo = {C, H, I, P}) ->
-    application:load(rabbit),
+    ok = application:load(rabbit),
     RabbitConfig = application:get_all_env(rabbit),
     %% Encrypt a few values in configuration.
     %% Common cases.
@@ -205,10 +198,10 @@ do_decrypt_config(Algo = {C, H, I, P}) ->
     TCPOpts2 = lists:keyreplace(linger, 1, TCPOpts1, {linger, {encrypted, EncLinger}}),
     application:set_env(rabbit, tcp_listen_options, TCPOpts2),
     %% Decrypt configuration.
-    rabbit:decrypt_config([rabbit], Algo),
+    rabbit_prelaunch_conf:decrypt_config([rabbit], Algo),
     %% Check that configuration was decrypted properly.
     RabbitConfig = application:get_all_env(rabbit),
-    application:unload(rabbit),
+    ok = application:unload(rabbit),
     ok.
 
 encrypt_value(Key, {C, H, I, P}) ->
@@ -229,7 +222,7 @@ do_decrypt_start_app(Config, Passphrase) ->
         {hash, sha512},
         {iterations, 1000},
         {passphrase, Passphrase}
-    ]),
+    ], [{persistent, true}]),
     %% Add the path to our test application.
     code:add_path(?config(data_dir, Config) ++ "/lib/rabbit_shovel_test/ebin"),
     %% Attempt to start our test application.
@@ -256,7 +249,7 @@ decrypt_start_app_undefined(Config) ->
         {hash, sha512},
         {iterations, 1000}
         %% No passphrase option!
-    ]),
+    ], [{persistent, true}]),
     %% Add the path to our test application.
     code:add_path(?config(data_dir, Config) ++ "/lib/rabbit_shovel_test/ebin"),
     %% Attempt to start our test application.
@@ -265,7 +258,7 @@ decrypt_start_app_undefined(Config) ->
     try
         rabbit:start_apps([rabbit_shovel_test], #{rabbit => temporary})
     catch
-        exit:{bad_configuration, config_entry_decoder} -> ok;
+        throw:{bad_config_entry_decoder, missing_passphrase} -> ok;
         _:Exception -> exit({unexpected_exception, Exception})
     end.
 
@@ -276,7 +269,7 @@ decrypt_start_app_wrong_passphrase(Config) ->
         {hash, sha512},
         {iterations, 1000},
         {passphrase, "wrong passphrase"}
-    ]),
+    ], [{persistent, true}]),
     %% Add the path to our test application.
     code:add_path(?config(data_dir, Config) ++ "/lib/rabbit_shovel_test/ebin"),
     %% Attempt to start our test application.
@@ -285,7 +278,7 @@ decrypt_start_app_wrong_passphrase(Config) ->
     try
         rabbit:start_apps([rabbit_shovel_test], #{rabbit => temporary})
     catch
-        exit:{decryption_error,_,_} -> ok;
+        throw:{config_decryption_error, _, _} -> ok;
         _:Exception -> exit({unexpected_exception, Exception})
     end.
 
@@ -961,6 +954,10 @@ listing_plugins_from_multiple_directories(Config) ->
               end,
     Path = FirstDir ++ PathSep ++ SecondDir,
     Got = lists:sort([{Name, Vsn} || #plugin{name = Name, version = Vsn} <- rabbit_plugins:list(Path)]),
+    %% `rabbit` was loaded automatically by `rabbit_plugins:list/1`.
+    %% We want to unload it now so it does not interfere with other
+    %% testcases.
+    application:unload(rabbit),
     Expected = [{plugin_both, "2"}, {plugin_first_dir, "3"}, {plugin_second_dir, "4"}],
     case Got of
         Expected ->
diff --git a/test/unit_inbroker_non_parallel_SUITE.erl b/test/unit_inbroker_non_parallel_SUITE.erl
index 120a7fdbd9..2b4a216977 100644
--- a/test/unit_inbroker_non_parallel_SUITE.erl
+++ b/test/unit_inbroker_non_parallel_SUITE.erl
@@ -503,7 +503,7 @@ log_file_fails_to_initialise_during_startup1(_Config, NonWritableDir) ->
         ok -> exit({got_success_but_expected_failure,
                     log_rotation_no_write_permission_dir_test})
     catch
-        _:could_not_initialise_logger -> ok
+        throw:{error, {rabbit, {{cannot_log_to_file, _, _}, _}}} -> ok
     end,
 
     %% start application with logging to a subdirectory which
@@ -530,7 +530,7 @@ log_file_fails_to_initialise_during_startup1(_Config, NonWritableDir) ->
         ok -> exit({got_success_but_expected_failure,
                     log_rotation_parent_dirs_test})
     catch
-        _:could_not_initialise_logger -> ok
+        throw:{error, {rabbit, {{cannot_log_to_file, _, _}, _}}} -> ok
     end,
 
     %% clean up
diff --git a/test/unit_log_config_SUITE.erl b/test/unit_log_config_SUITE.erl
index 93c050edda..a13c90a362 100644
--- a/test/unit_log_config_SUITE.erl
+++ b/test/unit_log_config_SUITE.erl
@@ -132,6 +132,9 @@ sink_rewrite_sinks() ->
      {rabbit_log_mirroring_lager_event,
         [{handlers,[{lager_forwarder_backend,[lager_event,inherit]}]},
          {rabbit_handlers,[{lager_forwarder_backend,[lager_event,inherit]}]}]},
+     {rabbit_log_prelaunch_lager_event,
+        [{handlers,[{lager_forwarder_backend,[lager_event,inherit]}]},
+         {rabbit_handlers,[{lager_forwarder_backend,[lager_event,inherit]}]}]},
      {rabbit_log_queue_lager_event,
         [{handlers,[{lager_forwarder_backend,[lager_event,inherit]}]},
          {rabbit_handlers,[{lager_forwarder_backend,[lager_event,inherit]}]}]},
@@ -225,6 +228,9 @@ sink_handlers_merged_with_lager_extra_sinks_handlers(_) ->
          {rabbit_log_mirroring_lager_event,
             [{handlers,[{lager_forwarder_backend,[lager_event,inherit]}]},
              {rabbit_handlers,[{lager_forwarder_backend,[lager_event,inherit]}]}]},
+         {rabbit_log_prelaunch_lager_event,
+            [{handlers,[{lager_forwarder_backend,[lager_event,inherit]}]},
+             {rabbit_handlers,[{lager_forwarder_backend,[lager_event,inherit]}]}]},
          {rabbit_log_queue_lager_event,
             [{handlers,[{lager_forwarder_backend,[lager_event,inherit]}]},
              {rabbit_handlers,[{lager_forwarder_backend,[lager_event,inherit]}]}]},
@@ -285,7 +291,7 @@ config_sinks_level(_) ->
     rabbit_lager:configure_lager(),
 
     ExpectedSinks = sort_sinks(level_sinks()),
-    ExpectedSinks = sort_sinks(application:get_env(lager, extra_sinks, undefined)).
+    ?assertEqual(ExpectedSinks, sort_sinks(application:get_env(lager, extra_sinks, undefined))).
 
 level_sinks() ->
     [{error_logger_lager_event,
@@ -310,6 +316,9 @@ level_sinks() ->
      {rabbit_log_mirroring_lager_event,
         [{handlers,[{lager_forwarder_backend,[lager_event,error]}]},
          {rabbit_handlers,[{lager_forwarder_backend,[lager_event,error]}]}]},
+     {rabbit_log_prelaunch_lager_event,
+        [{handlers,[{lager_forwarder_backend,[lager_event,inherit]}]},
+         {rabbit_handlers,[{lager_forwarder_backend,[lager_event,inherit]}]}]},
      {rabbit_log_queue_lager_event,
         [{handlers,[{lager_forwarder_backend,[lager_event,inherit]}]},
          {rabbit_handlers,[{lager_forwarder_backend,[lager_event,inherit]}]}]},
@@ -409,6 +418,9 @@ file_sinks() ->
      {rabbit_log_mirroring_lager_event,
         [{handlers,[{lager_forwarder_backend,[lager_event,inherit]}]},
          {rabbit_handlers,[{lager_forwarder_backend,[lager_event,inherit]}]}]},
+     {rabbit_log_prelaunch_lager_event,
+        [{handlers,[{lager_forwarder_backend,[lager_event,inherit]}]},
+         {rabbit_handlers,[{lager_forwarder_backend,[lager_event,inherit]}]}]},
      {rabbit_log_queue_lager_event,
         [{handlers,[{lager_forwarder_backend,[lager_event,inherit]}]},
          {rabbit_handlers,[{lager_forwarder_backend,[lager_event,inherit]}]}]},
@@ -650,6 +662,9 @@ default_expected_sinks(UpgradeFile) ->
      {rabbit_log_mirroring_lager_event,
         [{handlers,[{lager_forwarder_backend,[lager_event,inherit]}]},
          {rabbit_handlers,[{lager_forwarder_backend,[lager_event,inherit]}]}]},
+     {rabbit_log_prelaunch_lager_event,
+        [{handlers,[{lager_forwarder_backend,[lager_event,inherit]}]},
+         {rabbit_handlers,[{lager_forwarder_backend,[lager_event,inherit]}]}]},
      {rabbit_log_queue_lager_event,
         [{handlers,[{lager_forwarder_backend,[lager_event,inherit]}]},
          {rabbit_handlers,[{lager_forwarder_backend,[lager_event,inherit]}]}]},
@@ -722,6 +737,9 @@ tty_expected_sinks() ->
      {rabbit_log_mirroring_lager_event,
         [{handlers,[{lager_forwarder_backend,[lager_event,inherit]}]},
          {rabbit_handlers,[{lager_forwarder_backend,[lager_event,inherit]}]}]},
+     {rabbit_log_prelaunch_lager_event,
+        [{handlers,[{lager_forwarder_backend,[lager_event,inherit]}]},
+         {rabbit_handlers,[{lager_forwarder_backend,[lager_event,inherit]}]}]},
      {rabbit_log_queue_lager_event,
         [{handlers,[{lager_forwarder_backend,[lager_event,inherit]}]},
          {rabbit_handlers,[{lager_forwarder_backend,[lager_event,inherit]}]}]},