QPID-6035: HA clearly distinguish qpid-ha commands intended for cluster manager.

This commit adds a --cluster-manager flag to qpid-ha tool.

Without this flag
- the 'promote' command is not listed in the tool help.
- using the promote command raises an error saying that it is only for cluster manager use
  and mentioning the --cluster-manager flag.

With the flag: promote functions as before.

The qpid-ha help text for promote is also more clear now that it is for cluster
manager only.

Originally the idea was to split qpid-ha into two tools but I have kept one tool
with the flag and warning messages because it:

- avoids packaging changes that might trip things up.

- helps people who are already using qpid-ha promote: their scripts will
  break but the error message explains how to fix it.

I think the special role of promote is sufficiently clear now even if it is
part of the same tool.

This commit also updates the following to take account of the new flag:
- rgmanager qpidd-primary script.
- qpidd tests.
- qpid book HA chapter.

NOTE: THIS WILL BREAK TEST HARNESSES that do promotion outside of rgmanager.
You'll need to add the --cluster-manager flag in the relevant places.

git-svn-id: https://svn.apache.org/repos/asf/qpid/trunk/qpid@1619877 13f79535-47bb-0310-9956-ffa450edef68

4 files changed, 76 insertions, 35 deletions

diff --git a/cpp/etc/qpidd-primary.in b/cpp/etc/qpidd-primary.in
index e79d8cc09c..86bc76d5e1 100755
--- a/cpp/etc/qpidd-primary.in
+++ b/cpp/etc/qpidd-primary.in
@@ -76,7 +76,7 @@ start() {
     lock
     $QPID_INIT start primary || return $?
     echo -n $"Promoting to primary: "
-    err=$($QPID_HA promote 2>&1)
+    err=$($QPID_HA promote --cluster-manager 2>&1)
     RETVAL=$?
     [ $RETVAL = 0 ] && success || { echo -n "$err: "; failure; }
     echo
diff --git a/cpp/src/tests/ha_test.py b/cpp/src/tests/ha_test.py
index f09a4cd573..9adad45ed4 100755
--- a/cpp/src/tests/ha_test.py
+++ b/cpp/src/tests/ha_test.py
@@ -176,7 +176,7 @@ acl allow all all
         except Exception, e:
             raise Exception("Error in qpid_ha -b %s %s: %s"%(url, args,e))
 
-    def promote(self): self.ready(); self.qpid_ha(["promote"])
+    def promote(self): self.ready(); self.qpid_ha(["promote", "--cluster-manager"])
     def replicate(self, from_broker, queue): self.qpid_ha(["replicate", from_broker, queue])
     @property
     def agent(self):
diff --git a/doc/book/src/cpp-broker/Active-Passive-Cluster.xml b/doc/book/src/cpp-broker/Active-Passive-Cluster.xml
index fbdf321c56..26d701df64 100644
--- a/doc/book/src/cpp-broker/Active-Passive-Cluster.xml
+++ b/doc/book/src/cpp-broker/Active-Passive-Cluster.xml
@@ -927,29 +927,35 @@ acl allow <replaceable>USER</replaceable>@QPID all all
       </itemizedlist>
     </para>
     <para>
-      The <command>qpid-ha</command> command allows you to check if a broker is primary,
-      and to promote a backup to primary.
+      The <command>qpid-ha</command> command allows you to check if a broker is
+      primary, and to promote a backup to primary.
     </para>
     <para>
       To test if a broker is the primary:
-      <programlisting>
-qpid-ha -b <replaceable>broker-address</replaceable> status --expect=primary
-      </programlisting>
-      This command will return 0 if the broker at <replaceable>broker-address</replaceable>
-      is the primary, non-0 otherwise.
+    </para>
+    <programlisting>qpid-ha -b <replaceable>broker-address</replaceable> status --expect=primary</programlisting>
+    <para>
+      This will return 0 if the broker at <replaceable>broker-address</replaceable> is the primary,
+      non-0 otherwise.
     </para>
     <para>
       To promote a broker to primary:
-      <programlisting>
-qpid-ha -b <replaceable>broker-address</replaceable> promote
-      </programlisting>
+    <programlisting>qpid-ha --cluster-manager -b <replaceable>broker-address</replaceable> promote</programlisting>
+    </para>
+    <para>
+      Note that <literal>promote</literal> is considered a "cluster manager
+      only" command. Incorrect use of <literal>promote</literal> outside of the
+      cluster manager could create a cluster with multiple primaries. Such a
+      cluster will malfunction and lose data.  "Cluster manager only" commands
+      are not accessible in <command>qpid-ha</command> without the
+      <literal>--cluster-manager</literal> option.
     </para>
     <para>
-      <command>qpid-ha --help</command> gives information on other commands and options available.
-      You can also use <command>qpid-ha</command> to manually examine and promote brokers. This
-      can be useful for testing failover scenarios without having to set up a full resource manager,
-      or to simulate a cluster on a single node. For deployment, a resource manager is required.
+      To list the full set of commands use:
     </para>
+    <programlisting>
+qpid-ha --cluster-manager --help
+    </programlisting>
   </section>
 
   <section id ="ha-store">
@@ -999,8 +1005,9 @@ warning Client closed connection with 320: ACL denied anonymous@QPID creating a
       <para>
 	Set the HA security configuration and ACL file as described in <xref
 	linkend="ha-security"/>.  Once the cluster is running and the primary is
-	promoted , run <literal>qpid-ha</literal> to make sure that the brokers
-	are running as one cluster.
+	promoted , run:
+	<programlisting>qpid-ha status --all</programlisting>
+	to make sure that the brokers are running as one cluster.
       </para>
     </section>
     <section id="ha-troubleshoot-slow-recovery">
diff --git a/tools/src/py/qpid-ha b/tools/src/py/qpid-ha
index 640463c09a..d7b2cb48f6 100755
--- a/tools/src/py/qpid-ha
+++ b/tools/src/py/qpid-ha
@@ -50,12 +50,17 @@ def find_qpidd_conf():
         if p.isfile(conf): return conf
     return None
 
-class Command:
-    commands = {}
+class Command(object):
+    """
+    Common options and logic for all commands. Subclasses provide additional
+    options and execution logic.
+    """
+
+    commands = []
 
     def __init__(self, name, help, arg_names=[], connect_agent=True):
         """@param connect_agent true if we should establish a QMF agent connection"""
-        Command.commands[name] = self
+        Command.commands.append(self)
         self.name = name
         self.connect_agent = connect_agent
         self.arg_names = arg_names
@@ -141,6 +146,21 @@ class Command:
     def do_execute(self, qmf_broker, opts, args):
         raise Exception("Command '%s' is not yet implemented"%self.name)
 
+class ManagerCommand(Command):
+    """
+    Base for commands that should only be used by a cluster manager tool that ensures
+    cluster consistency.
+    """
+
+    manager_commands = []           # Cluster manager commands
+
+    def __init__(self, name, help, arg_names=[], connect_agent=True):
+        """@param connect_agent true if we should establish a QMF agent connection"""
+        super(ManagerCommand, self).__init__(name, "[Cluster manager only] "+help, arg_names, connect_agent)
+        self.commands.remove(self)   # Not a user command
+        self.manager_commands.append(self)
+
+
 class PingCmd(Command):
     def __init__(self):
         Command.__init__(self, "ping","Check if the broker is alive and responding", connect_agent=False)
@@ -148,11 +168,13 @@ class PingCmd(Command):
         self.connection.session() # Make sure we can establish a session.
 PingCmd()
 
-class PromoteCmd(Command):
+class PromoteCmd(ManagerCommand):
     def __init__(self):
-        Command.__init__(self, "promote","Promote a backup broker to primary. Note this command will not detect if another broker is already primary, and creating a second primary will make the cluster inconsistent. It is up to the caller (normally the cluster resource manager) to ensure there is only one primary.")
+        super(PromoteCmd, self).__init__("promote", "Promote a backup broker to primary. This command should *only* be used by a cluster manager (such as rgmanager) that ensures only one broker is primary at a time. Promoting more than one broker to primary at the same time will make the cluster inconsistent and will cause data loss and unexpected behavior.")
+
     def do_execute(self, qmf_broker, ha_broker, opts, args):
         qmf_broker._method("promote", {}, HA_BROKER, timeout=opts.timeout)
+
 PromoteCmd()
 
 
@@ -226,25 +248,37 @@ def print_usage(prog):
         print "  %-12s %s."%(name, help.split(".")[0])
     print "\nFor help with a command type: %s <command> --help\n"%prog
 
-def find_command(args):
+def find_command(args, commands):
     """Find a command among the arguments and options"""
     for arg in args:
-        if arg in Command.commands:
-            return Command.commands[arg]
+        cmds = [cmd for cmd in commands if cmd.name == arg]
+        if cmds: return cmds[0]
     return None
 
 def main_except(argv):
     """This version of main raises exceptions"""
-    args=argv[1:]
-    if args and args[0] == "--help-all":
-        for c in Command.commands.itervalues():
-            c.op.print_help(); print
+    args = argv[1:]
+    commands = Command.commands
+    if "--cluster-manager" in args:
+        commands += ManagerCommand.manager_commands
+        args.remove("--cluster-manager")
+    if "--help-all" in args:
+        for c in commands: c.op.print_help(); print
     else:
-        command = find_command(args)
-        if not command:
-            print_usage(os.path.basename(argv[0]));
-            raise Exception("Command not found")
-        command.execute(args)
+        command = find_command(args, commands)
+        if command:
+            command.execute(args)
+        else:
+            # Check for attempt to use a manager command without --cluster-manager
+            command = find_command(args, ManagerCommand.manager_commands)
+            if command:
+                message="""'%s' should only be called by the cluster manager.
+Incorrect use of '%s' will cause cluster malfunction.
+To call from a cluster manager use '%s --cluster-manager'. """
+                raise Exception(message%((command.name,)*3))
+            else:
+                print_usage(os.path.basename(argv[0]));
+                raise Exception("No valid command")
 
 def main(argv):
     try: