| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |\ |
|
| | | |
|
| | |\ |
|
| | |/
| |
| |
| |
| |
| | |
The old way did not work if erl was not in ${ERLANG_HOME}/bin. The new way
looks like it should only work if erl is on the ${PATH} - but in fact
the Erlang VM arranges things so that is always true!
|
| | | |
|
| | | |
|
| | |\ |
|
| | | | |
|
| | | |
| | |
| | |
| | |
| | | |
Otherwise, in handle_dead_rabbit(), if the specified node is already
back in the cluster, "partitions" will not be emptied.
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
... in pause conditions, especially if the remote node is back in the
cluster already.
This gives the node a chance to enter a pause mode. Without this,
partitions may never be fixed.
|
| | |/
| |
| |
| |
| |
| |
| |
| | |
If the node is already back in the cluster at the time
rabbit_networking:on_node_down() is called don't remove the specified
node's listeners. Otherwise, we would loose the record from the entire
cluster, creating an inconsistency between the running listeners and the
recorded ones.
|
| | |\ |
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
At least, this fixes the following crash, when the partial partition
detection tries to close the connection while RabbitMQ is stopping:
** Reason for termination ==
** {timeout,{gen_server,call,
[application_controller,
{set_env,kernel,dist_auto_connect,never}]}}
|
| | |\ \
| | |/
| |/| |
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
rabbit_node_monitor:run_outside_applications/2 gains a new argument to
indicate if the caller is willing is let a possible current outside
application process to finish before its own function can run. If this
argument is false, the current behaviour is kept: the function is not
executed.
This fixes a bug where autoheal's outside application function was not
executed if pause_if_all_down's outside application function was still
running (ie. rabbit:start() didn't return yet). The bug caused the
loser to not stop and thus the autoheal process to never complete.
Now, the autoheal outside application function waits for the
pause_if_all_down function to finish before stopping the loser.
|
| | |\ \ |
|
| | | | |
| | | |
| | | |
| | | | |
... and simplify the handle_msg({winner_is, _}, ...) clause.
|
| | | | |
| | | |
| | | |
| | | | |
While here, document the message flow.
|
| | | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
Before, the leader was monitoring the losers itself (exactly like the
winner). When they were all down, it was going back to the "not_healing"
state.
Therefore, there was a possibility that the leader and winner went
out-of-sync regarding the autoheal state.
Now, the leader simply waits for a confirmation from the winner that the
autoheal process is over. If the leader is a loser too, the autoheal
state is saved in the application environment to survive the restart.
When the leader is back up, it asks the winner to possibly notify it
again.
|
| | | |/
| |/| |
|
| | |\ \
| | |/
| |/| |
|
| | |/
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
In rabbit_node_monitor:disconnect/1, we change the "dist_auto_connect"
kernel parameter to force a disconnection and give some time to all
components to handle the subsequent "nodedown" event.
"global" doesn't handle this situation very well. With an unfortunate
sequence of messages and bad timings, this can trigger an inconsistency
in its internal state. When this happens, global:sync() never returns.
See bug 26556, comment #5 for a detailed description.
The workaround consists of a process who parses the "global" internal
state if global:sync/0 didn't return in 15 seconds. If the state
contains in-progress synchronisation older than 10 seconds, the
spawned process sends fake nodedown/nodeup events to "global" on both
inconsistent nodes so they restart their synchronisation.
This workaround will be removed once the real bugs are fixed and
"dist_auto_connect" is left untouched.
|
| | |\
| |/
|/| |
|
| |\ \ |
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
If the list returned by `nodes()` didn't change since last call to
`pause_minority_guard()`, return the previous state, not `ok`.
This fixes a bug where the first call to `pause_minority_guard()` could
return `pausing` but the subsequent calls would return `ok`, leading to
channels resuming the send of confirms even though the node is about to
enter pause mode.
|
| | | |\
| |_|/
|/| | |
|
| | | | |
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | | |
OK, the previous attempt was a bit misguided. Rather than catch any
exception "inside" starting an app, log it and rethrow a token saying
no further logging is needed, let's just let the exception bubble out
and catch it once at the top level.
|
| | | |\
| |_|/
|/| | |
|
| |\ \ \ |
|
| | | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
We used to have a weird situation where an error inside a boot step
would cause us to go through error logging twice. The exception would
get caught in run_step/3, then go through boot_error() etc, and
basic_boot_error/3 would exit again, leading to more stuff being
logged as an application start failure. So to fix that, let's
exit with a special atom which prevents further logging. Also
rename functions to be a bit more meaningful.
|
| | | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
We want to clean up basic_boot_error/3, but these call sites make it
harder. Really, trying to produce nicely formatted errors here is a
waste of time, these errors can only be caused when adding boot steps.
Normal users should never see them. So don't complicate things by
handling them specially.
|
| |/ / /
| | |
| | |
| | |
| | | |
...and replace {boot_step, _, _} wrapping with an extra parameter which
might make it harder to make similar mistakes in future.
|
| | | |\
| |_|/
|/| | |
|
| |/ /
| |
| |
| |
| | |
Conflicts:
.gitignore
|
| | |\
| | |
| | | |
Add systemd notification support
|
| | | | |
|
| | |\ \
| | | |
| | | | |
When ERLANG_HOME is not set, exit with code 1
|
| | |/ / |
|
| | |\ \ |
|
| | | | |
| | | |
| | | |
| | | | |
No part of RabbitMQ does this type of checking, as pointed out by Simon.
|
| | | | |
| | | |
| | | |
| | | |
| | | | |
An admin could add nodes to the pause_if_all_down list before creating
the nodes.
|
| | | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
Now that 'pause_if_all_down' accepts a list of preferred nodes, it is
possible that these nodes are spread across multiple partitions. For
example, suppose we have nodes A and B in datacenter #1 and nodes C and
D in datacenter #2, and we set {pause_if_all_down, [A, C]}, If the link
between both datacenters is lost, A/B and C/D forms two partitions.
RabbitMQ continues to run at both sites because all nodes see at least
one node from the preferred nodes list. When the link comes back, we
need to handle the recovery.
Therefore, a user can specify the strategy:
o {pause_if_all_down, [...], ignore} (default)
o {pause_if_all_down, [...], autoheal}
This third parameter is mandatory.
If the strategy is 'ignore', RabbitMQ is started again on paused nodes,
as soon as they see another node from the preferred nodes list. This is
the default behaviour.
If the strategy is 'autoheal', RabbitMQ is started again, like in
'ignore' mode, but when all nodes are up, autohealing kicks in as well.
Compared to plain 'autoheal' mode, the chance of loosing data is low
because paused nodes never drifted away from the cluster. When they
start again, they join the cluster and resume operations as any starting
node.
|
| | | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
Now, a partition is paused if all nodes from the pause_if_all_down list
are seen as down. If a fraction of the list is alive, the nodes in the
partition remain up.
Compared to the previous version, some of the listed nodes can be taken
down for maintenance without risking service interruption. However, this
raises the problem of listed nodes distributed in multiple partitions:
we need to handle recovery. This will be addressed in a followup commit.
|
| | | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
The syntax is:
{cluster_partition_management, {keep_preferred, node@domain}}
The specified node name is used to determine which partition should run
or be suspended. Nodes which can still reach the specified node continue
to run. Nodes which can't are suspended.
Compared to pause_minority, this allows the admin to determine which
nodes to prioritize in case of partitions with an equal number of nodes.
|
| | | | | |
|
| | | | | |
|
| | |\ \ \
| |/ / /
|/| | | |
|
| | | | | |
|
| | |\ \ \ |
|
