QPID-4006: [Java Broker] Docbook documentation for BDB HA (first version)

Applied patch from Oleksandr Rudyy <orudyy@gmail.com> and myself.

git-svn-id: https://svn.apache.org/repos/asf/qpid/trunk@1345487 13f79535-47bb-0310-9956-ffa450edef68

16 files changed, 957 insertions, 1 deletions

diff --git a/qpid/doc/book/src/java-broker/AMQP-Messaging-Broker-Java-Book.xml b/qpid/doc/book/src/java-broker/AMQP-Messaging-Broker-Java-Book.xml
index 0bb40052cd..73240a81cb 100644
--- a/qpid/doc/book/src/java-broker/AMQP-Messaging-Broker-Java-Book.xml
+++ b/qpid/doc/book/src/java-broker/AMQP-Messaging-Broker-Java-Book.xml
@@ -31,7 +31,7 @@
       <listitem><para>Implemented in Java - Fully JMS compliant, runs on any Java platform.</para></listitem>
     </itemizedlist>
 
-    <para>Both AMQP messaging brokers support clients in multiple languages, as long as the messaging client and the messaging broker use the same version of AMQP. See <link linkend="AMQP-Compatibility"/> to see which messaging clients work with each broker.</para>
+    <para>Both AMQP messaging brokers support clients in multiple languages, as long as the messaging client and the messaging broker use the same version of AMQP.</para>
 
     <para>This manual contains information specific to the broker that is implemented in Java.</para>
   </preface>
@@ -44,6 +44,7 @@
         <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="Java-Environment-Variables.xml"/>
         <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="Qpid-Troubleshooting-Guide.xml"/>
         <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="Broker-Configuration-Guide.xml"/>
+        <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="HA-Guide.xml"/>
 </chapter>
 
 <chapter  id="Qpid-Java-Broker-HowTos">
diff --git a/qpid/doc/book/src/java-broker/HA-Guide.xml b/qpid/doc/book/src/java-broker/HA-Guide.xml
new file mode 100644
index 0000000000..1c9ea078e1
--- /dev/null
+++ b/qpid/doc/book/src/java-broker/HA-Guide.xml
@@ -0,0 +1,937 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!DOCTYPE urls [
+<!ENTITY oracleBdbProductOverviewUrl "http://www.oracle.com/technetwork/products/berkeleydb/overview/index-093405.html">
+<!ENTITY oracleBdbRepGuideUrl "http://oracle.com/cd/E17277_02/html/ReplicationGuide/">
+<!ENTITY oracleBdbJavaDocUrl "http://docs.oracle.com/cd/E17277_02/html/java/">
+<!ENTITY oracleJdkDocUrl "http://oracle.com/javase/6/docs/api/">
+]>
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements.  See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership.  The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License.  You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied.  See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+<section>
+  <title>High Availability</title>
+  <section role="h3" id="HAGeneralIntroduction">
+    <title>General Introduction</title>
+    <para>The term High Availability (HA) usually refers to having a number of instances of a service such as a Message Broker
+      available so that should a service unexpectedly fail, or requires to be shutdown for maintenance, users may quickly connect
+      to another instance and continue their work with minimal interuption. HA is one way to make a overall system more resilient
+      by eliminating a single point failure from a system.</para>
+    <para>HA offerings are usually categorised as <emphasis role="bold">Active/Active</emphasis> or <emphasis role="bold">Active/Passive</emphasis>.
+      An Active/Active system is one where all nodes within the cluster are usuaully available for use by clients all of the time.  In an
+      Active/Passive system, one only node within the clsuter is available for use by clients at any one time, whilst the others are in
+      some kind of standby state, awaiting to quickly step-in in the event the active node becomes unavailable.
+    </para>
+  </section>
+  <section role="h3" id="HAOfferingsOfJavaBroker">
+    <title>HA offerings of the Java Broker</title>
+    <para>The Java Broker's HA offering became available at release <emphasis role="bold">0.18</emphasis>.  HA is provided by way of the HA
+      features built into the <ulink url="&oracleBdbProductOverviewUrl;">Java Edition of the Berkley Database (BDB JE)</ulink> and as such
+      is currently available to to Java Broker users who use the BDB Store.</para>
+    <para>HA in the Java Broker provides an <emphasis role="bold">Active/Passive</emphasis> mode of operation with Virtual hosts being
+      the unit of replication.  The Active node (referred to as the <emphasis role="bold">Master</emphasis>) accepts all work from all the clients.
+       The Passive nodes (referred to as <emphasis role="bold">Replicas</emphasis>) are unavailable for work: the only task they must perform is
+       to remain in synch with the Master node by consuming a replication stream containing all data and state.</para>
+    <para>If the Master node fails, a Replica node is elected to become the new Master node.  All clients automatically failover
+      <footnote><para>The automatic failover feature is available only for AMQP connections from the Java client.  Management connections (JMX)
+        do not current offer this feature.</para></footnote> to the new Master and continue their work.</para>
+    <para>The Java Broker HA solution is incompatible with the HA solution offered by the CPP Broker.  It is not possible to co-locate Java and CPP
+       Brokers within the same cluster.</para>
+    <para>HA is not currently available for those using the the <emphasis role="bold">Derby Store</emphasis> or <emphasis role="bold">Memory
+      Message Store</emphasis>.</para>
+  </section>
+  <section role="h3" id="HATwoNodeCluster">
+    <title>Two Node Cluster</title>
+    <section role="h4">
+      <title>Overview</title>
+      <para>In this HA solution, a cluster is formed with two nodes. one node serves as
+        <emphasis role="bold">master</emphasis> and the other is a <emphasis role="bold">replica</emphasis>.
+      </para>
+      <para>All data and state required for the operation of the virtual host is automatically sent from the
+        master to the replica. This is called the replication stream. The master virtual host confirms each
+        message is on the replica before the client transaction completes. The exact way the client awaits
+        for the master and replica is gorverned by the <link linkend="HADurabilityGuarantee">durability</link>
+        configuration, which is discussed later. In this way, the replica remains ready to take over the
+        role of the master if the master becomes unavailable.
+      </para>
+      <para>It is important to note that there is an inherent limitation of two node clusters is that
+        the replica node cannot make itself master automatically in the event of master failure.  This
+        is because the replica has no way to distinguish between a network partition (with potentially
+        the master still alive on the other side of the partition) and the case of genuine master failure.
+        (If the replica were to elect itself as master, the cluster would run the risk of a
+        <ulink url="http://en.wikipedia.org/wiki/Split-brain_(computing)">split-brain</ulink> scenario).
+        In the event of a master failure, a third party must designate the replica as primary.  This process
+        is described in more detail later.
+      </para>
+      <para>Clients connect to the cluster using a <link linkend="HAClientFailover">failover url</link>.
+        This allows the client to maintain a connection to the master in a way that is transparent
+        to the client application.</para>
+    </section>
+    <section role="h4">
+      <title>Depictions of cluster operation</title>
+      <para>In this section, the operation of the cluster is depicted through a series of figures
+        supported by explanatory text.</para>
+      <figure>
+        <title>Key for figures</title>
+        <mediaobject>
+          <imageobject>
+            <imagedata fileref="images/HA-2N-Key.png" format="PNG" scalefit="1"/>
+          </imageobject>
+          <textobject>
+            <phrase>Key to figures</phrase>
+          </textobject>
+        </mediaobject>
+      </figure>
+      <section role="h5" id="HATwoNodeNormalOperation">
+        <title>Normal Operation</title>
+        <para>The figure below illustrates normal operation.  Clients connecting to the cluster by way
+	  of the failover over URL achieve a connection to the master.   A clients perform work (message
+	  production, consumption, queue creation etc), the master additionally sends this data to the
+	  replica over the network.</para>
+        <figure>
+          <title>Normal operation of a two-node cluster</title>
+          <mediaobject>
+            <imageobject>
+              <imagedata fileref="images/HA-2N-Normal.png" format="PNG" scalefit="1"/>
+            </imageobject>
+            <textobject>
+              <phrase>Normal operation</phrase>
+            </textobject>
+          </mediaobject>
+        </figure>
+      </section>
+      <section role="h5" id="HATwoNodeMasterFailure">
+        <title>Master Failure and Recovery</title>
+        <para>The figure below illustrates a sequence of events whereby the master suffers a failure
+	  and the replica is made the master to allow the clients to continue to work. Later the
+	  old master is repaired and comes back on-line in replica role.</para>
+        <para>The item numbers in this list apply to the numbered boxes in the figure below.</para>
+        <orderedlist>
+          <listitem>
+            <para>System operating normally</para>
+          </listitem>
+          <listitem>
+            <para>Master suffers a failure and disconnects all clients. Replica realises that it is no
+	      longer in contact with master.  Clients begin to try to reconnect to the cluster although these
+	      connection attempts will fail at this point.</para>
+          </listitem>
+          <listitem>
+            <para>A third-party (an operator, a script or a combination of the two) verifies that the master has truely
+	       failed. If it has truely failed, the decision is made to  designate the replica as primary.
+               This primary designation is performed using <link linkend="HAJMXAPI">JMX</link>.</para>
+          </listitem>
+          <listitem>
+            <para>Client connections to the new master succeed and the <emphasis role="bold">service is restored
+	      </emphasis>, albeit without a replica.</para>
+          </listitem>
+          <listitem>
+            <para>The old master is repaired and brought back on-line.  It automatically rejoins the cluster
+	       in the <emphasis role="bold">replica</emphasis> role.</para>
+          </listitem>
+        </orderedlist>
+        <figure>
+          <title>Failure of master and recovery sequence</title>
+          <mediaobject>
+            <imageobject>
+              <imagedata fileref="images/HA-2N-MasterFail.png" format="PNG" scalefit="1"/>
+            </imageobject>
+            <textobject>
+              <phrase>Failure of master and subsequent recovery sequence</phrase>
+            </textobject>
+          </mediaobject>
+        </figure>
+      </section>
+      <section role="h5" id="HATwoNodeReplicaFailure">
+        <title>Replica Failure and Recovery</title>
+        <para>The figure that follows illustrates a sequence of events whereby the replica suffers a failure
+	   leaving the master to continue processing alone.  Later the replica is repaired and is restarted.
+	   It rejoins the cluster so that it is once again ready to take over in the event of master failure.</para>
+        <para>The behavior of the replica failure case is governed by the <varname>designatedPrimary</varname>
+	   configuration item.   If true on the master, the master will continue to operate solo without outside
+           intervention. If false, a third-party must designate as primary in order for the master to continue solo.</para>
+        <para>The item numbers in this list apply to the numbered boxes in the figure below. This example assumes
+	   that <varname>designatedPrimary</varname> is true.</para>
+        <orderedlist>
+          <listitem>
+            <para>System operating normally</para>
+          </listitem>
+          <listitem>
+            <para>Replica suffers a failure. Master realises that replica longer in contact but as
+	      <varname>designatedPrimary</varname> is true, master continues processing solo thus client
+	      connections are uninterrupted by the loss of the replica. System operating normally albeit with
+	      single node.</para>
+          </listitem>
+          <listitem>
+            <para>Replica is repaired.</para>
+          </listitem>
+          <listitem>
+            <para>After catching up with missed work, replica is once again ready to take over in the event of master failure.</para>
+          </listitem>
+        </orderedlist>
+        <figure>
+          <title>Failure of replica and subsequent recovery sequence</title>
+          <mediaobject>
+            <imageobject>
+              <imagedata fileref="images/HA-2N-ReplicaFail.png" format="PNG" scalefit="1"/>
+            </imageobject>
+            <textobject>
+              <phrase>Failure of replica and subsequent recovery sequence</phrase>
+            </textobject>
+          </mediaobject>
+        </figure>
+      </section>
+      <section role="h5" id="HATwoNodeNetworkPartition">
+        <title>Network Partition and Recovery</title>
+        <para>The figure below illustrates the sequence of events that would occur if the network between
+	  master and replica were to suffer a partition, and the nodes were out of contact with one and other.</para>
+        <para>As with <link linkend="HATwoNodeReplicaFailure">Replica Failure and Recovery</link>, the
+	  behaviour is governed by the <varname>designatedPrimary</varname>.
+	  Only if <varname>designatedPrimary</varname> is true on the master, wil the master continue solo.</para>
+        <para>The item numbers in this list apply to the numbered boxes in the figure below. This example assumes
+	   that <varname>designatedPrimary</varname> is true.</para>
+        <orderedlist>
+          <listitem>
+            <para>System operating normally</para>
+          </listitem>
+          <listitem>
+            <para>Network suffers a failure. Master realises that replica longer in contact but as
+	      <varname>designatedPrimary</varname> is true, master continues processing solo thus client
+	      connections are uninterrupted by the network partition.</para>
+          </listitem>
+          <listitem>
+            <para>Network is repaired.</para>
+          </listitem>
+          <listitem>
+            <para>After catching up with missed work, replica is once again ready to take over in the event of master failure.
+	    System operating normally again.</para>
+          </listitem>
+        </orderedlist>
+        <figure>
+          <title>Partition of the network separating master and replica</title>
+          <mediaobject>
+            <imageobject>
+              <imagedata fileref="images/HA-2N-NetworkPartition.png" format="PNG" scalefit="1"/>
+            </imageobject>
+            <textobject>
+              <phrase>Network Partition and Recovery</phrase>
+            </textobject>
+          </mediaobject>
+        </figure>
+      </section>
+      <section role="h5" id="HATwoNodeSplitBrain">
+        <title>Split Brain</title>
+        <para>A <ulink url="http://en.wikipedia.org/wiki/Split-brain_(computing)">split-brain</ulink>
+	  is a situation where the two node cluster has two masters.  BDB normal strives to prevent
+	  this situation arising by preventing two nodes in a cluster being master at the same time.
+	  However, if the network suffers a partition, and the third-party intervenes incorrectly
+	  and makes the replica a second master a split-brain will be formed and both masters will
+	  proceed to perform work  <emphasis role="bold">independently</emphasis> of one and other.</para>
+        <para>There is no automatic recovery from a split-brain.</para>
+        <para>Manual intervention will be required to choose which store will be retained as master
+	  and which will be discarded.  Manual intervention will be required to identify and repeat the
+          lost business transactions.</para>
+        <para>The item numbers in this list apply to the numbered boxes in the figure below.</para>
+        <orderedlist>
+          <listitem>
+            <para>System operating normally</para>
+          </listitem>
+          <listitem>
+            <para>Network suffers a failure. Master realises that replica longer in contact but as
+	      <varname>designatedPrimary</varname> is true, master continues processing solo.  Client
+	      connections are uninterrupted by the network partition.</para>
+            <para>A third-party <emphasis role="bold">erroneously</emphasis> designates the replica as primary.</para>
+          </listitem>
+          <listitem>
+            <para>As the nodes cannot see one and other, both behave as masters.  Clients perform work against
+	      both master nodes.</para>
+          </listitem>
+        </orderedlist>
+        <figure>
+          <title>Split Brain</title>
+          <mediaobject>
+            <imageobject>
+              <imagedata fileref="images/HA-2N-SplitBrain.png" format="PNG" scalefit="1"/>
+            </imageobject>
+            <textobject>
+              <phrase>Split Brain</phrase>
+            </textobject>
+          </mediaobject>
+        </figure>
+      </section>
+    </section>
+  </section>
+  <section role="h3" id="HAMultiNodeCluster">
+    <title>Multi Node Cluster</title>
+    <para>Multi node clusters, that is clusters where the number of nodes is three or more, are not yet
+         ready for use.</para>
+  </section>
+  <section role="h3" id="HAConfiguration">
+    <title>Configuring a Virtual Host to be a node</title>
+    <para>To configure a virtualhost as a cluster node, configure the virtualhost.xml in the following manner:</para>
+    <para>
+      <programlisting><![CDATA[
+<virtualhost>
+  <name>myhost</name>
+  <myvhost>
+    <store>
+      <class>org.apache.qpid.server.store.berkeleydb.BDBHAMessageStore</class>
+      <environment-path>${work}/bdbhastore</environment-path>
+      <highAvailability>
+        <groupName>myclustername</groupName>
+        <nodeName>mynode1</nodeName>
+        <nodeHostPort>node1host:port</nodeHostPort>
+        <helperHostPort>node1host:port</helperHostPort>
+        <durability>NO_SYNC\,NO_SYNC\,SIMPLE_MAJORITY</durability>
+        <coalescingSync>true|false</coalescingSync>
+        <designatedPrimary>true|false</designatedPrimary>
+      </highAvailability>
+    </store>
+    ...
+ </myvhost>
+</virtualhost>]]></programlisting>
+    </para>
+    <para>The <varname>groupName</varname> is the name of logical name of the cluster.  All nodes within the
+      cluster must use the same <varname>groupName</varname> in order to be consider part of the cluster.</para>
+    <para>The <varname>nodeName</varname> is the logical name of the node.  All nodes within the cluster must have a
+      unique name.  It is recommended that the node name should be chosen from a different nomenclature from that of
+      the servers on which they are hosted, in case the need arises to move node to a new server in the future.</para>
+    <para>The <varname>nodeHostPort</varname> is the hostname and port number used by this node to communicate with the
+      the other nodes in the cluster. For the hostname, an IP address, hostname or fully qualified hostname may be used.
+      For the port number, any free port can be used.  It is important that this address is stable over time, as BDB
+      records and uses this address internally.</para>
+    <para>The <varname>helperHostPort</varname> is the hostname and port number that new nodes use to discover other
+      nodes within the cluster when they are newly introduced to the cluster.  When configuring the first node, set the
+      <varname>helperHostPort</varname> to its own <varname>nodeHostPort</varname>.  For the second and subsequent nodes,
+      set their <varname>helperHostPort</varname> to that of of the first node.</para>
+    <para><varname>durability</varname> controls the <link linkend="HADurabilityGuarantee">durability</link>
+      guarantees made by the cluster. It is important that all nodes use the same value. The default value is
+      NO_SYNC\,NO_SYNC\,SIMPLE_MAJORITY.  Owning to the internal use of Apache Commons Config, it is necessary
+      to escape the commas within the durability string.</para>
+    <para><varname>coalescingSync</varname> controls the <link linkend="HADurabilityGuarantee_CoalescingSync">coalescing-sync</link>
+      mode of Qpid. It is important that all nodes use the same value. If omitted, it defaults to true.</para>
+    <para>The <varname>designatedPrimary</varname> is applicable only to the <link linkend="HATwoNodeCluster">two-node
+      case.</link>  It governs the behaviour of the master node when the replica fails or becomes uncontactable.  If true,
+      the master will automatically designate itself as primary and service will continue from the single node.  If false,
+      the master will transition to an unavailable state until a third-party manually designates the node as primary
+      or the replica is restored. It is suggested that the node that normally fulfils the role of master is set true in
+      config file and the node that is normally replica is set false.  Be aware that setting both nodes to true will lead
+      to a <link linkend="HATwoNodeSplitBrain">split-brain</link> in the case of network partition.</para>
+    <section role="h4" id="HAConfiguration_BDBEnvVars">
+      <title>Passing BDB environment and replication configuration options</title>
+      <para>It is possible to pass BDB <ulink url="&oracleBdbJavaDocUrl;com/sleepycat/je/EnvironmentConfig.html">
+         environment</ulink> and <ulink url="&oracleBdbJavaDocUrl;com/sleepycat/je/rep/ReplicationConfig.html">
+         replication</ulink> configuration options from the virtualhost.xml. Environment configuration options are passed using
+         the <varname>envConfig</varname> element, and replication config using <varname>repConfig</varname>.</para>
+      <para>For example, to override the BDB environment configuration options <varname>je.cleaner.threads</varname> and
+        <varname>je.txn.timeout</varname></para>
+      <programlisting><![CDATA[
+         ...
+      </highAvailability>
+      <envConfig>
+        <name>je.cleaner.threads</name>
+        <value>2</value>
+      </envConfig>
+      <envConfig>
+        <name>je.txn.timeout</name>
+        <value>15 min</value>
+      </envConfig>
+      ...
+    </store>]]></programlisting>
+      <para>And to override the BDB replication configuration options <varname>je.rep.insufficientReplicasTimeout</varname>.</para>
+      <programlisting><![CDATA[
+         ...
+      </highAvailability>
+      ...
+      <repConfig>
+        <name>je.rep.insufficientReplicasTimeout</name>
+        <value>2</value>
+      </envConfig>
+      <envConfig>
+        <name>je.txn.timeout</name>
+        <value>10 s</value>
+      </envConfig>
+      ...
+    </store>]]></programlisting>
+    </section>
+  </section>
+  <section role="h3" id="HADurabilityGuarantee">
+    <title>Durability Guarantees</title>
+    <para>The term <ulink url="http://en.wikipedia.org/wiki/ACID#Durability">durability</ulink> is used to mean that once a
+      transaction is committed, it remains committed regardless of subsequent failures. A highly durable system is one where
+      loss of a committed transaction is extermely unlikely, whereas with a less durable system loss of a transaction is likely
+      in a greater number of scenarios.  Typically, the more highly durable a system the slower and more costly it will be.</para>
+    <para>Qpid exposes the all the
+      <ulink url="&oracleBdbRepGuideUrl;txn-management.html#durabilitycontrols">durability controls</ulink>
+      offered by by BDB JE JA and a Qpid specific optimisation called <emphasis role="bold">coalescing-sync</emphasis> which defaults
+      to enabled.</para>
+    <section role="h4" id="HADurabilityGuarantee_BDBControls">
+      <title>BDB Durability Controls</title>
+      <para>BDB expresses durability as a triplet with the following form:</para>
+      <programlisting><![CDATA[<master sync policy>,<replica sync policy>,<replica acknowledgement policy>]]></programlisting>
+      <para>The sync polices controls whether the thread performing the committing thread awaits the successful completion of the
+        write, or the write and sync before continuing. The master sync policy and replica sync policy need not be the same.</para>
+      <para>For master and replic sync policies, the available values are:
+        <ulink url="&oracleBdbJavaDocUrl;com/sleepycat/je/Durability.SyncPolicy.html#SYNC">SYNC</ulink>,
+        <ulink url="&oracleBdbJavaDocUrl;com/sleepycat/je/Durability.SyncPolicy.html#WRITE_NO_SYNC">WRITE_NO_SYNC</ulink>,
+        <ulink url="&oracleBdbJavaDocUrl;com/sleepycat/je/Durability.SyncPolicy.html#NO_SYNC">NO_SYNC</ulink>. SYNC
+        is offers the highest durability whereas NO_SYNC the lowest.</para>
+      <para>Note: the combination of a master sync policy of SYNC and <link linkend="HADurabilityGuarantee_CoalescingSync">coalescing-sync</link>
+        true would result in poor performance with no corresponding increase in durability guarantee.  It cannot not be used.</para>
+      <para>The acknowledgement policy defines whether when a master commits a transaction, it also awaits for the replica(s) to
+         commit the same transaction before continuing.  For the two-node case, ALL and SIMPLE_MAJORITY are equal.</para>
+      <para>For acknowledgement policy, the available value are:
+         <ulink url="&oracleBdbJavaDocUrl;com/sleepycat/je/Durability.ReplicaAckPolicy.html#ALL">ALL</ulink>,
+         <ulink url="&oracleBdbJavaDocUrl;com/sleepycat/je/Durability.ReplicaAckPolicy.html#SIMPLE_MAJORITY">SIMPLE_MAJORITY</ulink>
+         <ulink url="&oracleBdbJavaDocUrl;com/sleepycat/je/Durability.ReplicaAckPolicy.html#NONE">NONE</ulink>.</para>
+    </section>
+    <section role="h4" id="HADurabilityGuarantee_CoalescingSync">
+      <title>Coalescing-sync</title>
+      <para>If enabled (the default) Qpid works to reduce the number of separate
+        <ulink url="&oracleJdkDocUrl;java/io/FileDescriptor.html#sync()">file-system sync</ulink> operations
+        performed by the <emphasis role="bold">master</emphasis> on the underlying storage device thus improving performance.  It does
+        this coalescing separate sync operations arising from the different client commits operations occuring at approximately the same time.
+        It does this in such a manner not to reduce the ACID guarantees of the system.</para>
+      <para>Coalescing-sync has no effect on the behaviour of the replicas.</para>
+    </section>
+    <section role="h4" id="HADurabilityGuarantee_Default">
+      <title>Default</title>
+      <para>The default durability guarantee is <constant>NO_SYNC, NO_SYNC, SIMPLE_MAJORITY</constant> with coalescing-sync enabled. The effect
+         of this combination is described in the table below. It offers a good compromise between durability guarantee and performance
+         with writes being guaranteed on the master and the additional guarantee that a majority of replicas have received the
+         transaction.</para>
+    </section>
+    <section role="h4" id="HADurabilityGuarantee_Examples">
+      <title>Examples</title>
+      <para>Here are some examples illustrating the effects of the durability and coalescing-sync settings.</para>
+      <para>
+        <table>
+          <title>Effect of different durability guarantees</title>
+          <tgroup cols="4">
+            <thead>
+              <row>
+                <entry/>
+                <entry>Durability</entry>
+                <entry>Coalescing-sync</entry>
+                <entry>Description</entry>
+              </row>
+            </thead>
+            <tbody>
+              <row>
+                <entry>1</entry>
+                <entry>NO_SYNC, NO_SYNC, SIMPLE_MAJORITY</entry>
+                <entry>true</entry>
+                <entry>Before the commit returns to the client, the transaction will be written/sync'd to the Master's disk (effect of
+                   coalescing-sync) and a majority of the replica(s) will have acknowledged the <emphasis role="bold">receipt</emphasis>
+                   of the transaction.  The replicas will write and sync the transaction to their disk at a point in the future governed by
+                   <ulink url="&oracleBdbJavaDocUrl;com/sleepycat/je/rep/ReplicationMutableConfig.html#LOG_FLUSH_TASK_INTERVAL">ReplicationMutableConfig#LOG_FLUSH_INTERVAL</ulink>.
+                </entry>
+              </row>
+              <row>
+                <entry>2</entry>
+                <entry>NO_SYNC, WRITE_NO_SYNC, SIMPLE_MAJORITY</entry>
+                <entry>true</entry>
+                <entry>Before the commit returns to the client, the transaction will be written/sync'd to the Master's disk (effect of
+                  coalescing-sync and a majority of the replica(s) will have acknowledged the <emphasis role="bold">write</emphasis> of
+                  the transaction to their disk.  The replicas will sync the transaction to disk at a point in the future governed by
+                  ReplicationMutableConfig#LOG_FLUSH_INTERVAL.</entry>
+              </row>
+              <row>
+                <entry>3</entry>
+                <entry>NO_SYNC, NO_SYNC, NONE</entry>
+                <entry>false</entry>
+                <entry>After the commit returns to the client, the transaction is neither guaranteed to be written to the disk of the master
+                   nor received by any of the replicas. The master and replicas will write and sync the transaction to their disk at a point
+                   in the future governed by ReplicationMutableConfig#LOG_FLUSH_INTERVAL. This offers the weakest durability guarantee.</entry>
+              </row>
+            </tbody>
+          </tgroup>
+        </table>
+      </para>
+    </section>
+  </section>
+  <section id="HAClientFailover">
+    <title>Client failover configuration</title>
+    <para>The details about format of Qpid connection URLs can be found at section
+        <ulink url="../../Programming-In-Apache-Qpid/html/QpidJNDI.html">Connection URLs</ulink>
+        of book <ulink url="../../Programming-In-Apache-Qpid/html/">Programming In Apache Qpid</ulink>.</para>
+    <para>The failover policy option in connection URL for HA Cluster should be set to <emphasis>roundrobin</emphasis>.
+      The Master broker should be put into a first place in <emphasis>brokerlist</emphasis> URL option.
+      The recommended value for <emphasis>connectdelay</emphasis> option in broker URL should be set to
+      the value greater than 1000 milliseconds. If it is desired that clients re-connect automatically after a
+      master to replica failure, <varname>cyclecount</varname> should be tuned so that the retry period is longer than
+      the expected length of time to perform the failover.</para>
+    <example><title>Example of connection URL for the HA Cluster</title><![CDATA[
+amqp://guest:guest@clientid/test?brokerlist='tcp://localhost:5672?connectdelay='2000'&retries='3';tcp://localhost:5671?connectdelay='2000'&retries='3';tcp://localhost:5673?connectdelay='2000'&retries='3''&failover='roundrobin?cyclecount='30''
+        ]]></example>
+  </section>
+  <section role="h3" id="HAJMXAPI">
+    <title>Qpid JMX API for HA</title>
+    <para>Qpid exposes HA store information via JMX interface and provides JMX API to remove Node from the group, update Node IP address
+    and assign Node as a designated primary.</para>
+    <para>MBean <classname>BDBHAMessageStore</classname> is instantiated by the Qpid broker for the each instance of HA store.</para>
+    <para>The reference to this MBean can be obtained via JMX API using object name like <emphasis>org.apache.qpid:type=BDBHAMessageStore,name=&lt;host name&gt;</emphasis>
+                where &lt;host name&gt; is the name of the virtual host of the broker.</para>
+    <table border="1">
+      <title>Mbean <classname>BDBHAMessageStore</classname> attributes</title>
+      <thead>
+        <tr>
+          <td>Name</td>
+          <td>Type</td>
+          <td>Accessibility</td>
+          <td>Description</td>
+        </tr>
+      </thead>
+      <tbody>
+        <tr>
+          <td>GroupName</td>
+          <td>String</td>
+          <td>Read only</td>
+          <td>Name identifying the group</td>
+        </tr>
+        <tr>
+          <td>NodeName</td>
+          <td>String</td>
+          <td>Read only</td>
+          <td>Unique name identifying the node within the group</td>
+        </tr>
+        <tr>
+          <td>NodeHostPort</td>
+          <td>String</td>
+          <td>Read only</td>
+          <td>Host/port used to replicate data between this node and others in the group</td>
+        </tr>
+        <tr>
+          <td>HelperHostPort</td>
+          <td>String</td>
+          <td>Read only</td>
+          <td>Host/port used to allow a new node to discover other group members</td>
+        </tr>
+        <tr>
+          <td>NodeState</td>
+          <td>String</td>
+          <td>Read only</td>
+          <td>Current state of the node</td>
+        </tr>
+        <tr>
+          <td>ReplicationPolicy</td>
+          <td>String</td>
+          <td>Read only</td>
+          <td>Node replication durability</td>
+        </tr>
+        <tr id="JMXDesignatedPrimary">
+          <td>DesignatedPrimary</td>
+          <td>boolean</td>
+          <td>Read/Write</td>
+          <td>Designated primary flag. Applicable to the two node case.</td>
+        </tr>
+        <tr>
+          <td>getAllNodesInGroup</td>
+          <td>TabularData</td>
+          <td>Read only</td>
+          <td>Get all nodes within the group, regardless of whether currently attached or not</td>
+        </tr>
+      </tbody>
+    </table>
+    <table border="1">
+      <title>Mbean <classname>BDBHAMessageStore</classname> operations</title>
+      <thead>
+        <tr>
+          <td>Operation</td>
+          <td>Parameters</td>
+          <td>Returns</td>
+          <td>Description</td>
+        </tr>
+      </thead>
+      <tbody>
+        <tr>
+          <td>removeNodeFromGroup</td>
+          <td>
+            <para><emphasis>nodeName</emphasis>, name of node, string</para>
+          </td>
+          <td>void</td>
+          <td>Remove an existing node from the group</td>
+        </tr>
+        <tr>
+          <td>updateAddress</td>
+          <td>
+            <itemizedlist>
+              <itemizedlist>
+                <para><emphasis>nodeName</emphasis>, name of node, string</para>
+              </itemizedlist>
+              <itemizedlist>
+                <para><emphasis>newHostName</emphasis>, new host name, string</para>
+              </itemizedlist>
+              <itemizedlist>
+                <para><emphasis>newPort</emphasis>, new port number, int</para>
+              </itemizedlist>
+            </itemizedlist>
+          </td>
+          <td>void</td>
+          <td>Update the address of another node. The node must be in a STOPPED state.</td>
+        </tr>
+      </tbody>
+    </table>
+    <figure>
+      <title>BDBHAMessageStore view from jconsole.</title>
+      <graphic fileref="images/HA-BDBHAMessageStore-MBean-jconsole.png"/>
+    </figure>
+    <example>
+      <title>Example of java code to get the node state value</title>
+      <programlisting language="java"><![CDATA[
+Map<String, Object> environment = new HashMap<String, Object>();
+
+// credentials: user name and password
+environment.put(JMXConnector.CREDENTIALS, new String[] {"admin","admin"});
+JMXServiceURL url =  new JMXServiceURL("service:jmx:rmi:///jndi/rmi://localhost:9001/jmxrmi");
+JMXConnector jmxConnector = JMXConnectorFactory.connect(url, environment);
+MBeanServerConnection mbsc =  jmxConnector.getMBeanServerConnection();
+
+ObjectName queueObjectName = new ObjectName("org.apache.qpid:type=BDBHAMessageStore,name=test");
+String state = (String)mbsc.getAttribute(queueObjectName, "NodeState");
+
+System.out.println("Node state:" + state);
+        ]]></programlisting>
+      <para>Example system output:</para>
+      <screen><![CDATA[Node state:MASTER]]></screen>
+    </example>
+  </section>
+  <section id="BDB-HA-Monitoring-cluster">
+    <title>Monitoring cluster</title>
+    <para>In order to discover the potential issues with HA Cluster early, all nodes in the Cluster should be monitored on regular basis
+    using the following techniques:</para>
+    <itemizedlist>
+      <listitem>
+        <para>Broker log files scrapping for WARN or ERROR entries and operational log entries like:</para>
+        <itemizedlist>
+          <listitem>
+            <para><emphasis>MST-1007 :</emphasis> Store Passivated. It can indicate that Master virtual host is down.</para>
+          </listitem>
+          <listitem>
+            <para><emphasis>MST-1006 :</emphasis> Recovery Complete. It can indicate that former Replica virtual host is up and became Master.</para>
+          </listitem>
+        </itemizedlist>
+      </listitem>
+      <listitem>
+        <para>Disk space usage and system load using system tools.</para>
+      </listitem>
+      <listitem>
+        <para>Berkeley HA node status using <classname>DbPing</classname> utility.</para>
+        <example><title>Using <classname>DbPing</classname> utility for monitoring HA nodes.</title><![CDATA[
+java -jar je-5.0.48.jar DbPing -groupName TestClusterGroup -nodeName Node-5001 -nodeHost localhost:5001 -socketTimeout 10000
+]]><screen>
+Current state of node: Node-5001 from group: TestClusterGroup
+  Current state: MASTER
+  Current master: Node-5001
+  Current JE version: 5.0.48
+  Current log version: 8
+  Current transaction end (abort or commit) VLSN: 165
+  Current master transaction end (abort or commit) VLSN: 0
+  Current active feeders on node: 0
+  Current system load average: 0.35
+</screen></example>
+        <para>In the example above <classname>DbPing</classname> utility requested status of Cluster node with name
+            <emphasis>Node-5001</emphasis> from replication group <emphasis>TestClusterGroup</emphasis> running on host <emphasis>localhost:5001</emphasis>.
+            The state of the node was reported into a system output.
+            </para>
+      </listitem>
+      <listitem>
+        <para>Using Qpid broker JMX interfaces.</para>
+        <para>Mbean <classname>BDBHAMessageStore</classname> can be used to request the following node information:</para>
+        <itemizedlist>
+          <listitem>
+            <para><emphasis>NodeState</emphasis> indicates whether node is a Master or Replica.</para>
+          </listitem>
+          <listitem>
+            <para><emphasis>Durability</emphasis> replication durability.</para>
+          </listitem>
+          <listitem>
+            <para><emphasis>DesignatedPrimary</emphasis> indicates whether Master node is designated primary.</para>
+          </listitem>
+          <listitem>
+            <para><emphasis>GroupName</emphasis> replication group name.</para>
+          </listitem>
+          <listitem>
+            <para><emphasis>NodeName</emphasis> node name.</para>
+          </listitem>
+          <listitem>
+            <para><emphasis>NodeHostPort</emphasis> node host and port.</para>
+          </listitem>
+          <listitem>
+            <para><emphasis>HelperHostPort</emphasis> helper host and port.</para>
+          </listitem>
+          <listitem>
+            <para><emphasis>AllNodesInGroup</emphasis> lists of all nodes in the replication group including their names, hosts and ports.</para>
+          </listitem>
+        </itemizedlist>
+        <para>For more details about <classname>BDBHAMessageStore</classname> MBean please refer section <link linkend="HAJMXAPI">Qpid JMX API for HA</link></para>
+      </listitem>
+    </itemizedlist>
+  </section>
+  <section id="HADiskSpace">
+    <title>Disk space requirements</title>
+    <para>Disk space is a critical resource for the HA Qpid broker.</para>
+    <para>In case when Replica goes down (or falls behind the Master in 2 node cluster where Master is designated primary)
+    and Master continues running, the non-replicated logs are stored on a Master disk for the period of time
+    as specified in <emphasis>je.rep.repStreamTimeout</emphasis> JE setting in order to replicate this data later
+    when Replica is back. This setting is set to 1 hour by default on QPid broker.  The setting can be overridden as described
+    <xref linkend="HAConfiguration_BDBEnvVars"/>.</para>
+    <para>Depending from the application publishing/consuming rates and message sizes,
+    the disk space might become overfull during this period of time due to preserved logs.
+    Please, make sure to allocate enough space on your disk to avoid this from happening.
+    </para>
+  </section>
+  <section id="BDB-HA-Network-Requirements">
+    <title>Network Requirements</title>
+    <para>The HA Cluster performance depends from the network bandwidth, its load by existing traffic and quality of service.</para>
+    <para>In order to achieve a better performance it is recommended to use a separate network infrastructure for the Qpid HA Nodes
+     which might include installation of dedicated network hardware on Broker hosts, assigning a higher priority to replication ports,
+     installing a cluster in a separate network not impacted by any other traffic.</para>
+  </section>
+  <section id="BDB-HA-Security">
+    <title>Security</title>
+    <para>At the moment Berkeley replication API supports only TCP/IP protocol to transfer replication data between Master and Replicas.</para>
+    <para>As result, the replicated data is unprotected and can be intercepted by anyone having access to the replication network.</para>
+    <para>Also, anyone who can access to this network can introduce a new node and therefore receive a copy of the data.</para>
+    <para>In order to reduce the security risks the entire HA cluster is recommended to run in a separate network protected from general access.</para>
+  </section>
+  <section id="BDB-HA-Backup">
+    <title>Backups</title>
+    <para>In order to protect the entire cluster from some cataclysms which might destroy all cluster nodes,
+    the backups of a Master store should be taken on a regular basis.</para>
+    <para>Qpid Broker distribution includes the "hot" backup utility <emphasis>backup.sh</emphasis> which can be found at broker bin folder.
+         This utility can perform the backup when broker is running.</para>
+    <para><emphasis>backup.sh</emphasis> script invokes <classname>org.apache.qpid.server.store.berkeleydb.BDBBackup</classname> to do the job.</para>
+    <para>You can also run this class from command line like in an example below:</para>
+    <example><title>Performing store backup by using <classname>BDBBackup</classname> class directly</title>
+        java -cp qpid-bdbstore-0.18.jar org.apache.qpid.server.store.berkeleydb.BDBBackup -fromdir path/to/store/folder -todir path/to/backup/folder
+    </example>
+    <para>In the example above BDBBackup utility is called from qpid-bdbstore-0.18.jar to backup the store at <emphasis>path/to/store/folder</emphasis> and copy store logs into <emphasis>path/to/backup/folder</emphasis>.</para>
+    <para>Linux and Unix users can take advantage of <emphasis>backup.sh</emphasis> bash script by running this script in a similar way.</para>
+    <example><title>Performing store backup by using <classname>backup.sh</classname> bash script</title>
+        backup.sh -fromdir path/to/store/folder -todir path/to/backup/folder
+    </example>
+    <note>
+      <para>Please, do not forget to switch to do backups of a new Master store when cluster Master is changed.</para>
+    </note>
+  </section>
+  <section id="HAMigrationFromNonHA">
+    <title>Migration of a non-HA store to HA</title>
+    <para>Non HA stores starting from schema version 4 (0.14 Qpid release) can be automatically converted into HA store on broker startup if replication is previously enabled with DbEnableReplication utility from Berkeley JE jar.</para>
+    <para>DbEnableReplication converts non HA store into HA store and can be used as follows:</para>
+    <example><title>Enabling replication</title><![CDATA[
+java -jar je-5.0.48.jar DbEnableReplication -h /path/to/store -groupName MyReplicationGroup -nodeName MyNode1 -nodeHostPort  localhost:5001
+        ]]></example>
+    <para>In the examples above, je jar of version 5.0.48 is used to convert store at <emphasis>/path/to/store</emphasis> into HA store having replication group name <emphasis>MyReplicationGroup</emphasis>, node name <emphasis>MyNode1</emphasis> and running on host <emphasis>localhost</emphasis> and port <emphasis>5001</emphasis>.</para>
+    <para>After running DbEnableReplication and changing virtual host store to HA message store, like in example below, on broker start up the store schema will be upgraded to the recent version and broker can be used as normal.</para>
+    <example>
+      <title>Example of XML configuration for HA message store</title>
+      <programlisting language="xml"><![CDATA[
+<store>
+    <class>org.apache.qpid.server.store.berkeleydb.BDBHAMessageStore</class>
+    <environment-path>/path/to/store</environment-path>
+    <highAvailability>
+        <groupName>MyReplicationGroup</groupName>
+        <nodeName>MyNode1</nodeName>
+        <nodeHostPort>localhost:5001</nodeHostPort>
+        <helperHostPort>localhost:5001</helperHostPort>
+    </highAvailability>
+</store>]]></programlisting>
+    </example>
+    <para>The Replica nodes can be started with empty stores. The data will be automatically copied from Master to Replica on Replica start-up.</para>
+    <note>
+      <para>Due to existing caveats in Berkeley JE with copying of data from Master into Replica it is recommended to restart the Master node after store schema upgrade is finished before starting the Replica nodes.</para>
+    </note>
+  </section>
+  <section id="HAPerformance">
+    <title>Performance</title>
+    <para>The aim of this section is not providing Qpid broker performance metrics for HA as it depends from the test environment but rather showing an impact of HA on Qpid Broker performance in comparison with Non HA case.</para>
+    <para>For testing of impact of HA on a broker performance a special test script was written using Qpid performance test framework.
+    The script opened a number of connections to the Qpid broker, created producers and consumers on separate connections,
+    and published test messages with concurrent producers into a test queue and consumed them with concurrent consumers.
+    The table below shows the number of producers/consumers used in the tests.
+    The overall throughput was collected for each of the number.
+    </para>
+    <table border="1">
+      <title>Number of producers/consumers in performance tests</title>
+      <thead>
+        <tr>
+          <th>Test</th>
+          <th>Number of producers</th>
+          <th>Number of consumers</th>
+        </tr>
+      </thead>
+      <tbody>
+        <tr>
+          <td>1</td>
+          <td>1</td>
+          <td>1</td>
+        </tr>
+        <tr>
+          <td>2</td>
+          <td>2</td>
+          <td>2</td>
+        </tr>
+        <tr>
+          <td>3</td>
+          <td>4</td>
+          <td>4</td>
+        </tr>
+        <tr>
+          <td>4</td>
+          <td>8</td>
+          <td>8</td>
+        </tr>
+        <tr>
+          <td>5</td>
+          <td>16</td>
+          <td>16</td>
+        </tr>
+        <tr>
+          <td>6</td>
+          <td>32</td>
+          <td>32</td>
+        </tr>
+        <tr>
+          <td>7</td>
+          <td>64</td>
+          <td>64</td>
+        </tr>
+      </tbody>
+    </table>
+    <para>The test was run against the following Qpid Brokers</para>
+    <itemizedlist>
+      <listitem>
+        <para>Non HA Broker</para>
+      </listitem>
+      <listitem>
+        <para>HA 2 Nodes Cluster with durability <emphasis>SYNC,SYNC,ALL</emphasis></para>
+      </listitem>
+      <listitem>
+        <para>HA 2 Nodes Cluster with durability <emphasis>WRITE_NO_SYNC,WRITE_NO_SYNC,ALL</emphasis></para>
+      </listitem>
+      <listitem>
+        <para>HA 2 Nodes Cluster with durability <emphasis>WRITE_NO_SYNC,WRITE_NO_SYNC,ALL</emphasis> and <emphasis>coalescing-sync</emphasis> Qpid mode</para>
+      </listitem>
+      <listitem>
+        <para>HA 2 Nodes Cluster with durability <emphasis>WRITE_NO_SYNC,NO_SYNC,ALL</emphasis> and <emphasis>coalescing-sync</emphasis> Qpid mode</para>
+      </listitem>
+      <listitem>
+        <para>HA 2 Nodes Cluster with durability <emphasis>NO_SYNC,NO_SYNC,ALL</emphasis> and <emphasis>coalescing-sync</emphasis> Qpid option</para>
+      </listitem>
+    </itemizedlist>
+    <para>Test Environment we tested performance on was not really representative as we got only 2 test servers
+        having 4core processors Intel(r) Xeon(R) CPU 5150@2.66GHz, 4G of RAM
+        and running under OS Red Hat Enterprise Linux AS release 4 (Nahant Update 4).
+        Network bandwidth was 1Mbit.
+    </para>
+    <para>We ran Master node on a first server and Replica and clients(both consumers and producers) on a second server.</para>
+    <para>In non-HA case Qpid Broker was run on a first server and clients were run on a second server.</para>
+    <para>The table below contains the test results we measured on this environment for different Broker configurations.</para>
+    <para>Each result is represented by throughput value in KB/second and difference in % between HA configuration and non HA case for the same number of clients.</para>
+    <table border="1">
+      <title>Performance Comparison</title>
+      <thead>
+        <tr>
+          <td>Test/Broker</td>
+          <td>No HA</td>
+          <td>SYNC, SYNC, ALL</td>
+          <td>WRITE_NO_SYNC, WRITE_NO_SYNC, ALL</td>
+          <td>WRITE_NO_SYNC, WRITE_NO_SYNC, ALL - coalescing-sync</td>
+          <td>WRITE_NO_SYNC, NO_SYNC,ALL - coalescing-sync</td>
+          <td>NO_SYNC, NO_SYNC, ALL - coalescing-sync</td>
+        </tr>
+      </thead>
+      <tbody>
+        <tr>
+          <td>1 (1/1)</td>
+          <td>0.0%</td>
+          <td>-61.4%</td>
+          <td>117.0%</td>
+          <td>-16.02%</td>
+          <td>-9.58%</td>
+          <td>-25.47%</td>
+        </tr>
+        <tr>
+          <td>2 (2/2)</td>
+          <td>0.0%</td>
+          <td>-75.43%</td>
+          <td>67.87%</td>
+          <td>-66.6%</td>
+          <td>-69.02%</td>
+          <td>-30.43%</td>
+        </tr>
+        <tr>
+          <td>3 (4/4)</td>
+          <td>0.0%</td>
+          <td>-84.89%</td>
+          <td>24.19%</td>
+          <td>-71.02%</td>
+          <td>-69.37%</td>
+          <td>-43.67%</td>
+        </tr>
+        <tr>
+          <td>4 (8/8)</td>
+          <td>0.0%</td>
+          <td>-91.17%</td>
+          <td>-22.97%</td>
+          <td>-82.32%</td>
+          <td>-83.42%</td>
+          <td>-55.5%</td>
+        </tr>
+        <tr>
+          <td>5 (16/16)</td>
+          <td>0.0%</td>
+          <td>-91.16%</td>
+          <td>-21.42%</td>
+          <td>-86.6%</td>
+          <td>-86.37%</td>
+          <td>-46.99%</td>
+        </tr>
+        <tr>
+          <td>6 (32/32)</td>
+          <td>0.0%</td>
+          <td>-94.83%</td>
+          <td>-51.51%</td>
+          <td>-92.15%</td>
+          <td>-92.02%</td>
+          <td>-57.59%</td>
+        </tr>
+        <tr>
+          <td>7 (64/64)</td>
+          <td>0.0%</td>
+          <td>-94.2%</td>
+          <td>-41.84%</td>
+          <td>-89.55%</td>
+          <td>-89.55%</td>
+          <td>-50.54%</td>
+        </tr>
+      </tbody>
+    </table>
+    <para>The figure below depicts the graphs for the performance test results</para>
+    <figure>
+      <title>Test results</title>
+      <graphic fileref="images/HA-perftests-results.png"/>
+    </figure>
+    <para>On using durability <emphasis>SYNC,SYNC,ALL</emphasis> (without coalescing-sync) the performance drops significantly (by 62-95%) in comparison with non HA broker.</para>
+    <para>Whilst, on using durability <emphasis>WRITE_NO_SYNC,WRITE_NO_SYNC,ALL</emphasis> (without coalescing-sync) the performance drops by only half, but with loss of durability guarantee, so is not recommended.</para>
+    <para>In order to have better performance with HA, Qpid Broker comes up with the special mode called <link linkend="HADurabilityGuarantee_CoalescingSync">coalescing-sync</link>,
+    With this mode enabled, Qpid broker batches the concurrent transaction commits and syncs transaction data into Master disk in one go.
+    As result, the HA performance only drops by 25-60% for durability <emphasis>NO_SYNC,NO_SYNC,ALL</emphasis> and by 10-90% for <emphasis>WRITE_NO_SYNC,WRITE_NO_SYNC,ALL</emphasis>.</para>
+  </section>
+</section>
diff --git a/qpid/doc/book/src/java-broker/images/HA-2N-Key.png b/qpid/doc/book/src/java-broker/images/HA-2N-Key.png
new file mode 100644
index 0000000000..3c1856d496
--- /dev/null
+++ b/qpid/doc/book/src/java-broker/images/HA-2N-Key.png
diff --git a/qpid/doc/book/src/java-broker/images/HA-2N-Key.svg b/qpid/doc/book/src/java-broker/images/HA-2N-Key.svg
new file mode 100644
index 0000000000..9567f385d5
--- /dev/null
+++ b/qpid/doc/book/src/java-broker/images/HA-2N-Key.svg
@@ -0,0 +1,3 @@
+<?xml version="1.0"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xl="http://www.w3.org/1999/xlink" version="1.1" viewBox="-7 -3 435 195" width="435pt" height="195pt"><metadata xmlns:dc="http://purl.org/dc/elements/1.1/"><dc:date>2012-05-31 06:19Z</dc:date><!-- Produced by OmniGraffle Professional 5.3.6 --></metadata><defs><filter id="Shadow" filterUnits="userSpaceOnUse"><feGaussianBlur in="SourceAlpha" result="blur" stdDeviation="3.488"/><feOffset in="blur" result="offset" dx="0" dy="4"/><feFlood flood-color="black" flood-opacity=".75" result="flood"/><feComposite in="flood" in2="offset" operator="in"/></filter><linearGradient x1="0" x2="1" id="Gradient" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#2824aa"/></linearGradient><linearGradient id="Obj_Gradient" xl:href="#Gradient" gradientTransform="translate(52.803375 70.5) rotate(90) scale(24)"/><font-face font-family="Helvetica" font-size="12" units-per-em="1000" underline-position="-75.683594" underline-thickness="49.316406" slope="0" x-height="522.94922" cap-height="717.28516" ascent="770.01953" descent="-229.98047" font-weight="500"><font-face-src><font-face-name name="Helvetica"/></font-face-src></font-face><linearGradient x1="0" x2="1" id="Gradient_2" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#0caa25"/></linearGradient><linearGradient id="Obj_Gradient_2" xl:href="#Gradient_2" gradientTransform="translate(127.99037 28.999998) rotate(90) scale(25.5938)"/><linearGradient x1="0" x2="1" id="Gradient_3" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#aa172d"/></linearGradient><linearGradient id="Obj_Gradient_3" xl:href="#Gradient_3" gradientTransform="translate(160.86441 28.999998) rotate(90) scale(25.5938)"/><linearGradient x1="0" x2="1" id="Gradient_4" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#6923aa"/></linearGradient><linearGradient id="Obj_Gradient_4" xl:href="#Gradient_4" gradientTransform="translate(52.803375 107.5) rotate(90) scale(24)"/><radialGradient cx="0" cy="0" r="1" id="Gradient_5" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset=".5" stop-color="#b5d0ea"/><stop offset="1" stop-color="#aaa"/></radialGradient><radialGradient id="Obj_Gradient_5" xl:href="#Gradient_5" gradientTransform="translate(56.678375 41.796898) scale(37.736065)"/><marker orient="auto" overflow="visible" markerUnits="strokeWidth" id="Arrow_Marker" viewBox="-1 -4 10 8" markerWidth="10" markerHeight="8" color="black"><g><path d="M 8 0 L 0 -3 L 0 3 Z" fill="none" stroke="currentColor" stroke-width="1"/></g></marker><marker orient="auto" overflow="visible" markerUnits="strokeWidth" id="FilledArrow_Marker" viewBox="-1 -3 6 6" markerWidth="6" markerHeight="6" color="black"><g><path d="M 3.7333333 0 L 0 -1.4 L 0 1.4 Z" fill="currentColor" stroke="currentColor" stroke-width="1"/></g></marker><linearGradient x1="0" x2="1" id="Gradient_6" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#b5011d"/></linearGradient><linearGradient id="Obj_Gradient_6" xl:href="#Gradient_6" gradientTransform="translate(287.5 113.94637) rotate(90) scale(26.035202)"/><radialGradient cx="0" cy="0" r="1" id="Gradient_7" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset=".5" stop-color="#ea061f"/><stop offset="1" stop-color="#aaa"/></radialGradient><radialGradient id="Obj_Gradient_7" xl:href="#Gradient_7" gradientTransform="translate(280 41) scale(26.907248)"/><font-face font-family="Arial Unicode MS" font-size="36" panose-1="2 11 6 4 2 2 2 2 2 4" units-per-em="1000" underline-position="-100.097656" underline-thickness="49.804688" slope="0" x-height="529.78516" cap-height="728.02734" ascent="1068.84766" descent="-270.9961" font-weight="500"><font-face-src><font-face-name name="ArialUnicodeMS"/></font-face-src></font-face></defs><g stroke="none" stroke-opacity="1" stroke-dasharray="none" fill="none" fill-opacity="1"><title>Canvas 7</title><g><title>Layer 1</title><g><use xl:href="#id240_Graphic" filter="url(#Shadow)"/><use xl:href="#id239_Graphic" filter="url(#Shadow)"/><use xl:href="#id238_Graphic" filter="url(#Shadow)"/><use xl:href="#id237_Graphic" filter="url(#Shadow)"/><use xl:href="#id236_Graphic" filter="url(#Shadow)"/><use xl:href="#id235_Graphic" filter="url(#Shadow)"/><use xl:href="#id229_Graphic" filter="url(#Shadow)"/><use xl:href="#id227_Graphic" filter="url(#Shadow)"/></g><g id="id240_Graphic"><rect x="13.999878" y="13" width="394.00012" height="155" fill="white"/><rect x="13.999878" y="13" width="394.00012" height="155" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/></g><g id="id239_Graphic"><rect x="28.177876" y="70.5" width="49.251003" height="24" fill="url(#Obj_Gradient)"/><rect x="28.177876" y="70.5" width="49.251003" height="24" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><text transform="translate(33.177876 75.5)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="500" x="1.9536247" y="11" textLength="35.34375">Broker</tspan></text></g><g id="id238_Graphic"><path d="M 115.652176 41.796898 L 121.821274 28.999998 L 134.15947 28.999998 L 140.32857 41.796898 L 134.15947 54.593796 L 121.821274 54.593796 Z" fill="url(#Obj_Gradient_2)"/><path d="M 115.652176 41.796898 L 121.821274 28.999998 L 134.15947 28.999998 L 140.32857 41.796898 L 134.15947 54.593796 L 121.821274 54.593796 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/></g><g id="id237_Graphic"><path d="M 148.526215 41.796898 L 154.69531 28.999998 L 167.03351 28.999998 L 173.20261 41.796898 L 167.03351 54.593796 L 154.69531 54.593796 Z" fill="url(#Obj_Gradient_3)"/><path d="M 148.526215 41.796898 L 154.69531 28.999998 L 167.03351 28.999998 L 173.20261 41.796898 L 167.03351 54.593796 L 154.69531 54.593796 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/></g><g id="id236_Graphic"><path d="M 32.853378 107.5 L 72.753372 107.5 C 77.472977 107.5 81.303375 112.876 81.303375 119.5 C 81.303375 126.124 77.472977 131.5 72.753372 131.5 L 32.853378 131.5 C 28.133776 131.5 24.303375 126.124 24.303375 119.5 C 24.303375 112.876 28.133776 107.5 32.853378 107.5" fill="url(#Obj_Gradient_4)"/><path d="M 32.853378 107.5 L 72.753372 107.5 C 77.472977 107.5 81.303375 112.876 81.303375 119.5 C 81.303375 126.124 77.472977 131.5 72.753372 131.5 L 32.853378 131.5 C 28.133776 131.5 24.303375 126.124 24.303375 119.5 C 24.303375 112.876 28.133776 107.5 32.853378 107.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><text transform="translate(35.003376 112.5)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="500" x="2.4601574" y="11" textLength="30.679688">Client</tspan></text></g><g id="id235_Graphic"><path d="M 30.434645 43.17154 C 18.959625 41.796898 23.535576 30.224663 41.840797 32.199223 C 43.539116 28.35017 64.825623 28.974915 64.686462 32.199223 C 78.033752 28.075294 95.09079 36.298325 83.649857 40.422256 C 97.37842 42.421642 83.476616 53.194073 72.209625 51.394573 C 71.307922 54.39391 51.165936 55.443512 49.398033 51.394573 C 37.992596 55.718647 14.210434 49.070145 30.434645 43.17154 Z" fill="url(#Obj_Gradient_5)"/><path d="M 30.434645 43.17154 C 18.959625 41.796898 23.535576 30.224663 41.840797 32.199223 C 43.539116 28.35017 64.825623 28.974915 64.686462 32.199223 C 78.033752 28.075294 95.09079 36.298325 83.649857 40.422256 C 97.37842 42.421642 83.476616 53.194073 72.209625 51.394573 C 71.307922 54.39391 51.165936 55.443512 49.398033 51.394573 C 37.992596 55.718647 14.210434 49.070145 30.434645 43.17154 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><text transform="translate(36.828377 34.796898)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="500" x=".84511757" y="11" textLength="38.009766">Cluster</tspan></text></g><text transform="translate(194.40121 29)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="500" x="0" y="11" textLength="8.0039062">V</tspan><tspan font-family="Helvetica" font-size="12" font-weight="500" x="7.7929688" y="11" textLength="26.009766">irtual</tspan><tspan font-family="Helvetica" font-size="12" font-weight="500" x="0" y="25" textLength="22.68164">host</tspan></text><line x1="115.65217" y1="84" x2="162.7522" y2="84" marker-end="url(#Arrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><line x1="114.77717" y1="114.406006" x2="155.878815" y2="114.99225" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="3"/><text transform="translate(181.40121 70.5)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="500" x="0" y="11" textLength="59.367188">Replication</tspan><tspan font-family="Helvetica" font-size="12" font-weight="500" x="0" y="25" textLength="36.673828">stream</tspan></text><text transform="translate(181.90121 93.906006)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="500" x="0" y="25" textLength="30.679688">Client</tspan><tspan font-family="Helvetica" font-size="12" font-weight="500" x="0" y="39" textLength="58.04297">connection</tspan></text><g id="id229_Graphic"><path d="M 257.6145 123.341385 L 303.64609 123.13183 C 303.64609 123.13183 305.97821 110.84691 310.80124 114.69145 C 315.62408 118.53591 319 121.44636 319 121.44636 L 315.92612 122.44486 L 307.22827 122.069695 L 307.22821 132.69258 L 315.47678 133.06787 C 315.47678 133.06787 315.92618 136.46542 315.92618 136.78604 C 315.92618 137.10628 311.77792 139.65259 309.63474 139.972885 C 307.49097 140.29318 304.06018 131.63016 304.06018 131.63039 C 304.06018 131.63039 257.61444 130.06921 257.48056 130.06921 C 257.34662 130.06921 256.13666 132.15001 256.00693 126.86558 C 255.87709 121.581276 257.6145 123.341385 257.6145 123.341385 Z" fill="url(#Obj_Gradient_6)"/><path d="M 257.6145 123.341385 L 303.64609 123.13183 C 303.64609 123.13183 305.97821 110.84691 310.80124 114.69145 C 315.62408 118.53591 319 121.44636 319 121.44636 L 315.92612 122.44486 L 307.22827 122.069695 L 307.22821 132.69258 L 315.47678 133.06787 C 315.47678 133.06787 315.92618 136.46542 315.92618 136.78604 C 315.92618 137.10628 311.77792 139.65259 309.63474 139.972885 C 307.49097 140.29318 304.06018 131.63016 304.06018 131.63039 C 304.06018 131.63039 257.61444 130.06921 257.48056 130.06921 C 257.34662 130.06921 256.13666 132.15001 256.00693 126.86558 C 255.87709 121.581276 257.6145 123.341385 257.6145 123.341385 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/></g><text transform="translate(334.09879 119.963974)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="500" x="0" y="11" textLength="35.34961">Repair</tspan></text><g id="id227_Graphic"><path d="M 260 32.310394 L 275.19995 42.241386 L 263.13727 54.738464 L 267.99988 59 L 281.17648 45.481422 L 295.20001 54.655197 L 300 49.68975 L 284.79993 39.137974 L 297.59991 27.965553 L 290.39993 23 L 279.19989 35.413807 L 266.4 25.482777 Z" fill="url(#Obj_Gradient_7)"/><path d="M 260 32.310394 L 275.19995 42.241386 L 263.13727 54.738464 L 267.99988 59 L 281.17648 45.481422 L 295.20001 54.655197 L 300 49.68975 L 284.79993 39.137974 L 297.59991 27.965553 L 290.39993 23 L 279.19989 35.413807 L 266.4 25.482777 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/></g><text transform="translate(329.02448 34)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="500" x="0" y="11" textLength="26.677734">Fault</tspan></text><text transform="translate(264 59.5)" fill="#262626"><tspan font-family="Arial Unicode MS" font-size="36" font-weight="500" fill="#262626" x="0" y="38" textLength="36">♛</tspan></text><text transform="translate(320.52448 72)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="500" x="0" y="11" textLength="54.035156">Designate</tspan><tspan font-family="Helvetica" font-size="12" font-weight="500" x="0" y="25" textLength="41.332031">Primary</tspan></text></g></g></svg>
diff --git a/qpid/doc/book/src/java-broker/images/HA-2N-MasterFail.png b/qpid/doc/book/src/java-broker/images/HA-2N-MasterFail.png
new file mode 100644
index 0000000000..b839d291b9
--- /dev/null
+++ b/qpid/doc/book/src/java-broker/images/HA-2N-MasterFail.png
diff --git a/qpid/doc/book/src/java-broker/images/HA-2N-MasterFail.svg b/qpid/doc/book/src/java-broker/images/HA-2N-MasterFail.svg
new file mode 100644
index 0000000000..35b2c643ff
--- /dev/null
+++ b/qpid/doc/book/src/java-broker/images/HA-2N-MasterFail.svg
@@ -0,0 +1,3 @@
+<?xml version="1.0"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xl="http://www.w3.org/1999/xlink" version="1.1" viewBox="-20 -16 600 871" width="50pc" height="871pt"><metadata xmlns:dc="http://purl.org/dc/elements/1.1/"><dc:date>2012-05-31 06:19Z</dc:date><!-- Produced by OmniGraffle Professional 5.3.6 --></metadata><defs><radialGradient cx="0" cy="0" r="1" id="Gradient" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset=".5" stop-color="#b5d0ea"/><stop offset="1" stop-color="#aaa"/></radialGradient><radialGradient id="Obj_Gradient" xl:href="#Gradient" gradientTransform="translate(200.35501 136.5) scale(145.18695)"/><linearGradient x1="0" x2="1" id="Gradient_2" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#2824aa"/></linearGradient><linearGradient id="Obj_Gradient_2" xl:href="#Gradient_2" gradientTransform="translate(212.855 146) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_3" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#0caa25"/></linearGradient><linearGradient id="Obj_Gradient_3" xl:href="#Gradient_3" gradientTransform="translate(212.85451 162.5) rotate(90) scale(36)"/><linearGradient x1="0" x2="1" id="Gradient_4" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#6923aa"/></linearGradient><linearGradient id="Obj_Gradient_4" xl:href="#Gradient_4" gradientTransform="translate(73.18171 111) rotate(90) scale(24)"/><linearGradient id="Obj_Gradient_5" xl:href="#Gradient_2" gradientTransform="translate(195.355 59.999996) rotate(90) scale(69.000007)"/><linearGradient x1="0" x2="1" id="Gradient_5" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#aa172d"/></linearGradient><linearGradient id="Obj_Gradient_6" xl:href="#Gradient_5" gradientTransform="translate(195.35451 78) rotate(90) scale(36)"/><marker orient="auto" overflow="visible" markerUnits="strokeWidth" id="FilledArrow_Marker" viewBox="-1 -3 6 6" markerWidth="6" markerHeight="6" color="black"><g><path d="M 3.7333333 0 L 0 -1.4 L 0 1.4 Z" fill="currentColor" stroke="currentColor" stroke-width="1"/></g></marker><marker orient="auto" overflow="visible" markerUnits="strokeWidth" id="Arrow_Marker" viewBox="-1 -4 10 8" markerWidth="10" markerHeight="8" color="black"><g><path d="M 8 0 L 0 -3 L 0 3 Z" fill="none" stroke="currentColor" stroke-width="1"/></g></marker><radialGradient id="Obj_Gradient_7" xl:href="#Gradient" gradientTransform="translate(470.5 133.5) scale(145.18695)"/><linearGradient id="Obj_Gradient_8" xl:href="#Gradient_2" gradientTransform="translate(488 141.5) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_6" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#aaa3a7"/></linearGradient><linearGradient id="Obj_Gradient_9" xl:href="#Gradient_6" gradientTransform="translate(487.9995 158) rotate(90) scale(36)"/><linearGradient id="Obj_Gradient_10" xl:href="#Gradient_4" gradientTransform="translate(350.21 85) rotate(90) scale(24)"/><linearGradient id="Obj_Gradient_11" xl:href="#Gradient_2" gradientTransform="translate(470.5 55.5) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_7" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#aaaaa3"/></linearGradient><linearGradient id="Obj_Gradient_12" xl:href="#Gradient_7" gradientTransform="translate(470.4995 73.5) rotate(90) scale(36)"/><radialGradient cx="0" cy="0" r="1" id="Gradient_8" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset=".5" stop-color="#ea061f"/><stop offset="1" stop-color="#aaa"/></radialGradient><radialGradient id="Obj_Gradient_13" xl:href="#Gradient_8" gradientTransform="translate(488 176) scale(38.183766)"/><font-face font-family="Helvetica" font-size="12" units-per-em="1000" underline-position="-75.683594" underline-thickness="49.316406" slope="0" x-height="532.22656" cap-height="719.72656" ascent="770.01953" descent="-229.98047" font-weight="bold"><font-face-src><font-face-name name="Helvetica-Bold"/></font-face-src></font-face><radialGradient id="Obj_Gradient_14" xl:href="#Gradient" gradientTransform="translate(474.08 415.255) scale(145.18695)"/><linearGradient id="Obj_Gradient_15" xl:href="#Gradient_2" gradientTransform="translate(491.58 423.255) rotate(90) scale(69)"/><linearGradient id="Obj_Gradient_16" xl:href="#Gradient_6" gradientTransform="translate(491.5795 439.755) rotate(90) scale(36)"/><linearGradient id="Obj_Gradient_17" xl:href="#Gradient_4" gradientTransform="translate(353.79 366.755) rotate(90) scale(24.00003)"/><linearGradient id="Obj_Gradient_18" xl:href="#Gradient_2" gradientTransform="translate(474.08 337.255) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_9" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#21aa1a"/></linearGradient><linearGradient id="Obj_Gradient_19" xl:href="#Gradient_9" gradientTransform="translate(474.0795 355.255) rotate(90) scale(36)"/><radialGradient id="Obj_Gradient_20" xl:href="#Gradient" gradientTransform="translate(192.79001 696.5) scale(145.18695)"/><linearGradient id="Obj_Gradient_21" xl:href="#Gradient_2" gradientTransform="translate(210.28999 704.5) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_10" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#aa1f43"/></linearGradient><linearGradient id="Obj_Gradient_22" xl:href="#Gradient_10" gradientTransform="translate(210.2895 721) rotate(90) scale(36)"/><linearGradient id="Obj_Gradient_23" xl:href="#Gradient_4" gradientTransform="translate(72.5 648) rotate(90) scale(24)"/><linearGradient id="Obj_Gradient_24" xl:href="#Gradient_2" gradientTransform="translate(192.78999 618.5) rotate(90) scale(69)"/><linearGradient id="Obj_Gradient_25" xl:href="#Gradient_9" gradientTransform="translate(192.7895 636.5) rotate(90) scale(36)"/><radialGradient id="Obj_Gradient_26" xl:href="#Gradient" gradientTransform="translate(166.45009 415.255) scale(145.18695)"/><linearGradient id="Obj_Gradient_27" xl:href="#Gradient_2" gradientTransform="translate(183.95009 423.255) rotate(90) scale(69)"/><linearGradient id="Obj_Gradient_28" xl:href="#Gradient_6" gradientTransform="translate(183.9496 439.755) rotate(90) scale(36)"/><linearGradient id="Obj_Gradient_29" xl:href="#Gradient_4" gradientTransform="translate(72.5 372.755) rotate(90) scale(24)"/><linearGradient id="Obj_Gradient_30" xl:href="#Gradient_2" gradientTransform="translate(180.08009 343.255) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_11" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#a9a5a6"/></linearGradient><linearGradient id="Obj_Gradient_31" xl:href="#Gradient_11" gradientTransform="translate(180.0796 361.255) rotate(90) scale(36)"/><radialGradient id="Obj_Gradient_32" xl:href="#Gradient_8" gradientTransform="translate(183.95009 457.755) scale(38.183766)"/><font-face font-family="Arial Unicode MS" font-size="36" panose-1="2 11 6 4 2 2 2 2 2 4" units-per-em="1000" underline-position="-100.097656" underline-thickness="49.804688" slope="0" x-height="529.78516" cap-height="728.02734" ascent="1068.84766" descent="-270.9961" font-weight="500"><font-face-src><font-face-name name="ArialUnicodeMS"/></font-face-src></font-face></defs><g stroke="none" stroke-opacity="1" stroke-dasharray="none" fill="none" fill-opacity="1"><title>Canvas 2</title><g><title>Layer 1</title><rect x="0" y="0" width="278" height="273" fill="white"/><rect x="0" y="0" width="278" height="273" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 146.38904 149.98122 C 122.79251 136.5 132.20221 23.010353 169.84393 42.375 C 173.33626 4.6271057 217.1085 10.754013 216.82236 42.375 C 244.26889 1.931366 279.34393 82.575165 255.81749 123.01879 C 284.04803 142.62691 255.46124 248.27281 232.29251 230.625 C 230.43831 260.03967 189.01958 270.33319 185.38417 230.625 C 161.930725 273.03143 113.02657 207.82918 146.38904 149.98122 Z" fill="url(#Obj_Gradient)"/><path d="M 146.38904 149.98122 C 122.79251 136.5 132.20221 23.010353 169.84393 42.375 C 173.33626 4.6271057 217.1085 10.754013 216.82236 42.375 C 244.26889 1.931366 279.34393 82.575165 255.81749 123.01879 C 284.04803 142.62691 255.46124 248.27281 232.29251 230.625 C 230.43831 260.03967 189.01958 270.33319 185.38417 230.625 C 161.930725 273.03143 113.02657 207.82918 146.38904 149.98122 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="179.355" y="146" width="67" height="69" fill="url(#Obj_Gradient_2)"/><rect x="179.355" y="146" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 192.524 180.5 L 202.68925 162.5 L 223.01976 162.5 L 233.18501 180.5 L 223.01976 198.5 L 202.68925 198.5 Z" fill="url(#Obj_Gradient_3)"/><path d="M 192.524 180.5 L 202.68925 162.5 L 223.01976 162.5 L 233.18501 180.5 L 223.01976 198.5 L 202.68925 198.5 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 62.68171 111 L 83.68171 111 C 86.16571 111 88.18171 116.376 88.18171 123 C 88.18171 129.62399 86.16571 135 83.68171 135 L 62.68171 135 C 60.197708 135 58.18171 129.62399 58.18171 123 C 58.18171 116.376 60.197708 111 62.68171 111" fill="url(#Obj_Gradient_4)"/><path d="M 62.68171 111 L 83.68171 111 C 86.16571 111 88.18171 116.376 88.18171 123 C 88.18171 129.62399 86.16571 135 83.68171 135 L 62.68171 135 C 60.197708 135 58.18171 129.62399 58.18171 123 C 58.18171 116.376 60.197708 111 62.68171 111" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="161.855" y="60" width="67" height="69" fill="url(#Obj_Gradient_5)"/><rect x="161.855" y="60" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 175.024 96 L 185.18925 78 L 205.51976 78 L 215.68501 96 L 205.51976 114 L 185.18925 114 Z" fill="url(#Obj_Gradient_6)"/><path d="M 175.024 96 L 185.18925 78 L 205.51976 78 L 215.68501 96 L 205.51976 114 L 185.18925 114 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 74.355003 137 C 80.38701 150.765915 72.80871 171.11536 92.452835 178.3019 C 109.12348 184.40063 145.40762 181.02281 176.33522 180.23158" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="3"/><path d="M 222.38084 162.05576 C 227.03842 153.03807 238.59448 143.85422 236.355 135 C 234.66139 128.303986 225.07715 121.79347 216.5538 115.25858" marker-end="url(#Arrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="281.29001" y="0" width="278.00003" height="273" fill="white"/><rect x="281.29001" y="0" width="278.00003" height="273" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 416.53403 146.98122 C 392.9375 133.5 402.3472 20.010353 439.98892 39.375 C 443.48123 1.6271057 487.2535 7.754013 486.96735 39.375 C 514.41388 -1.06863403 549.4889 79.575165 525.96246 120.01879 C 554.19305 139.62691 525.60626 245.27281 502.4375 227.625 C 500.5833 257.03967 459.16455 267.33319 455.52917 227.625 C 432.07571 270.03143 383.17157 204.82918 416.53403 146.98122 Z" fill="url(#Obj_Gradient_7)"/><path d="M 416.53403 146.98122 C 392.9375 133.5 402.3472 20.010353 439.98892 39.375 C 443.48123 1.6271057 487.2535 7.754013 486.96735 39.375 C 514.41388 -1.06863403 549.4889 79.575165 525.96246 120.01879 C 554.19305 139.62691 525.60626 245.27281 502.4375 227.625 C 500.5833 257.03967 459.16455 267.33319 455.52917 227.625 C 432.07571 270.03143 383.17157 204.82918 416.53403 146.98122 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="454.5" y="141.5" width="67" height="69" fill="url(#Obj_Gradient_8)"/><rect x="454.5" y="141.5" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 467.669 176 L 477.83426 158 L 498.16476 158 L 508.33002 176 L 498.16476 194 L 477.83426 194 Z" fill="url(#Obj_Gradient_9)"/><path d="M 467.669 176 L 477.83426 158 L 498.16476 158 L 508.33002 176 L 498.16476 194 L 477.83426 194 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 339.71 85 L 360.71 85 C 363.194 85 365.21 90.376 365.21 97 C 365.21 103.624 363.194 109 360.71 109 L 339.71 109 C 337.22598 109 335.21 103.624 335.21 97 C 335.21 90.376 337.22598 85 339.71 85" fill="url(#Obj_Gradient_10)"/><path d="M 339.71 85 L 360.71 85 C 363.194 85 365.21 90.376 365.21 97 C 365.21 103.624 363.194 109 360.71 109 L 339.71 109 C 337.22598 109 335.21 103.624 335.21 97 C 335.21 90.376 337.22598 85 339.71 85" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="437" y="55.5" width="67" height="69" fill="url(#Obj_Gradient_11)"/><rect x="437" y="55.5" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 450.169 91.5 L 460.33426 73.5 L 480.66476 73.5 L 490.83002 91.5 L 480.66476 109.5 L 460.33426 109.5 Z" fill="url(#Obj_Gradient_12)"/><path d="M 450.169 91.5 L 460.33426 73.5 L 480.66476 73.5 L 490.83002 91.5 L 480.66476 109.5 L 460.33426 109.5 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 461 162.96559 L 481.51996 177.86208 L 465.23532 196.6077 L 471.79984 203 L 489.58826 182.72214 L 508.52 196.48279 L 515 189.03462 L 494.4799 173.20695 L 511.7599 156.44833 L 502.0399 149 L 486.91986 167.62071 L 469.64001 152.72417 Z" fill="url(#Obj_Gradient_13)"/><path d="M 461 162.96559 L 481.51996 177.86208 L 465.23532 196.6077 L 471.79984 203 L 489.58826 182.72214 L 508.52 196.48279 L 515 189.03462 L 494.4799 173.20695 L 511.7599 156.44833 L 502.0399 149 L 486.91986 167.62071 L 469.64001 152.72417 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><text transform="translate(5.8400302 4)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="bold" x=".16308594" y="11" textLength="6.673828">1</tspan></text><rect x="281.29001" y="278.755" width="278.00003" height="273" fill="white"/><rect x="281.29001" y="278.755" width="278.00003" height="273" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 420.11401 428.7362 C 396.51749 415.255 405.92719 301.76535 443.5689 321.13 C 447.06122 283.38211 490.8335 289.50903 490.54733 321.13 C 517.9939 280.68637 553.0689 361.33017 529.54248 401.7738 C 557.773 421.38193 529.18622 527.02783 506.0175 509.38 C 504.1633 538.79468 462.74454 549.0882 459.10913 509.38 C 435.6557 551.78644 386.75156 486.58417 420.11401 428.7362 Z" fill="url(#Obj_Gradient_14)"/><path d="M 420.11401 428.7362 C 396.51749 415.255 405.92719 301.76535 443.5689 321.13 C 447.06122 283.38211 490.8335 289.50903 490.54733 321.13 C 517.9939 280.68637 553.0689 361.33017 529.54248 401.7738 C 557.773 421.38193 529.18622 527.02783 506.0175 509.38 C 504.1633 538.79468 462.74454 549.0882 459.10913 509.38 C 435.6557 551.78644 386.75156 486.58417 420.11401 428.7362 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="458.08" y="423.255" width="66.99997" height="69" fill="url(#Obj_Gradient_15)"/><rect x="458.08" y="423.255" width="66.99997" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 471.249 457.755 L 481.41425 439.755 L 501.74475 439.755 L 511.91 457.755 L 501.74475 475.755 L 481.41425 475.755 Z" fill="url(#Obj_Gradient_16)"/><path d="M 471.249 457.755 L 481.41425 439.755 L 501.74475 439.755 L 511.91 457.755 L 501.74475 475.755 L 481.41425 475.755 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 343.29 366.755 L 364.29 366.755 C 366.77402 366.755 368.79 372.13101 368.79 378.755 C 368.79 385.379 366.77402 390.755 364.29 390.755 L 343.29 390.755 C 340.806 390.755 338.79 385.379 338.79 378.755 C 338.79 372.13101 340.806 366.755 343.29 366.755" fill="url(#Obj_Gradient_17)"/><path d="M 343.29 366.755 L 364.29 366.755 C 366.77402 366.755 368.79 372.13101 368.79 378.755 C 368.79 385.379 366.77402 390.755 364.29 390.755 L 343.29 390.755 C 340.806 390.755 338.79 385.379 338.79 378.755 C 338.79 372.13101 340.806 366.755 343.29 366.755" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="440.57999" y="337.255" width="67" height="69" fill="url(#Obj_Gradient_18)"/><rect x="440.57999" y="337.255" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 453.749 373.255 L 463.91425 355.255 L 484.24475 355.255 L 494.41 373.255 L 484.24475 391.255 L 463.91425 391.255 Z" fill="url(#Obj_Gradient_19)"/><path d="M 453.749 373.255 L 463.91425 355.255 L 484.24475 355.255 L 494.41 373.255 L 484.24475 391.255 L 463.91425 391.255 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 365.46664 366.5985 C 373.74026 357.98486 375.04666 340.75385 390.29 340.755 C 402.3567 340.75592 423.16241 351.55505 442.3288 360.32996" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="3"/><rect x="0" y="558" width="278" height="273" fill="white"/><rect x="0" y="558" width="278" height="273" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 138.824036 709.9812 C 115.22751 696.5 124.63721 583.01038 162.27893 602.375 C 165.77126 564.62708 209.5435 570.75403 209.25735 602.375 C 236.70389 561.9314 271.77893 642.57513 248.25249 683.0188 C 276.48303 702.6269 247.89624 808.27283 224.72751 790.625 C 222.8733 820.03967 181.45457 830.3332 177.81917 790.625 C 154.36572 833.03143 105.46157 767.82916 138.824036 709.9812 Z" fill="url(#Obj_Gradient_20)"/><path d="M 138.824036 709.9812 C 115.22751 696.5 124.63721 583.01038 162.27893 602.375 C 165.77126 564.62708 209.5435 570.75403 209.25735 602.375 C 236.70389 561.9314 271.77893 642.57513 248.25249 683.0188 C 276.48303 702.6269 247.89624 808.27283 224.72751 790.625 C 222.8733 820.03967 181.45457 830.3332 177.81917 790.625 C 154.36572 833.03143 105.46157 767.82916 138.824036 709.9812 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="176.78999" y="704.5" width="67" height="69" fill="url(#Obj_Gradient_21)"/><rect x="176.78999" y="704.5" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 189.959 739 L 200.12425 721 L 220.45476 721 L 230.62001 739 L 220.45476 757 L 200.12425 757 Z" fill="url(#Obj_Gradient_22)"/><path d="M 189.959 739 L 200.12425 721 L 220.45476 721 L 230.62001 739 L 220.45476 757 L 200.12425 757 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 62 648 L 83 648 C 85.484 648 87.5 653.37598 87.5 660 C 87.5 666.62402 85.484 672 83 672 L 62 672 C 59.516 672 57.5 666.62402 57.5 660 C 57.5 653.37598 59.516 648 62 648" fill="url(#Obj_Gradient_23)"/><path d="M 62 648 L 83 648 C 85.484 648 87.5 653.37598 87.5 660 C 87.5 666.62402 85.484 672 83 672 L 62 672 C 59.516 672 57.5 666.62402 57.5 660 C 57.5 653.37598 59.516 648 62 648" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="159.28999" y="618.5" width="67" height="69" fill="url(#Obj_Gradient_24)"/><rect x="159.28999" y="618.5" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 172.459 654.5 L 182.62425 636.5 L 202.95476 636.5 L 213.12001 654.5 L 202.95476 672.5 L 182.62425 672.5 Z" fill="url(#Obj_Gradient_25)"/><path d="M 172.459 654.5 L 182.62425 636.5 L 202.95476 636.5 L 213.12001 654.5 L 202.95476 672.5 L 182.62425 672.5 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 84.176636 647.8435 C 92.450264 639.22986 93.75663 621.99884 109 622 C 121.06667 622.0009 141.87247 632.80005 161.03886 641.575" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="3"/><path d="M 181.93205 672.16388 C 174.38545 684.44135 157.17926 699.40002 159.28999 709 C 160.93364 716.4756 174.29166 720.7052 185.68005 725.4799" marker-end="url(#Arrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><text transform="translate(286 8)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="bold" x=".16308594" y="11" textLength="6.673828">2</tspan></text><text transform="translate(286 289.62299)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="bold" x=".16308594" y="11" textLength="6.673828">4</tspan></text><text transform="translate(10.3773594 571.472)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="bold" x=".16308594" y="11" textLength="6.673828">5</tspan></text><rect x="0" y="278.755" width="278" height="273" fill="white"/><rect x="0" y="278.755" width="278" height="273" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 112.48411 428.7362 C 88.88759 415.255 98.297287 301.76535 135.93901 321.13 C 139.431335 283.38211 183.20358 289.50903 182.91743 321.13 C 210.36397 280.68637 245.439 361.33017 221.91257 401.7738 C 250.14313 421.38193 221.55634 527.02783 198.38759 509.38 C 196.53339 538.79468 155.11465 549.0882 151.47925 509.38 C 128.02582 551.78644 79.12165 486.58417 112.48411 428.7362 Z" fill="url(#Obj_Gradient_26)"/><path d="M 112.48411 428.7362 C 88.88759 415.255 98.297287 301.76535 135.93901 321.13 C 139.431335 283.38211 183.20358 289.50903 182.91743 321.13 C 210.36397 280.68637 245.439 361.33017 221.91257 401.7738 C 250.14313 421.38193 221.55634 527.02783 198.38759 509.38 C 196.53339 538.79468 155.11465 549.0882 151.47925 509.38 C 128.02582 551.78644 79.12165 486.58417 112.48411 428.7362 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="150.45009" y="423.255" width="67" height="69" fill="url(#Obj_Gradient_27)"/><rect x="150.45009" y="423.255" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 163.61909 457.755 L 173.78435 439.755 L 194.11485 439.755 L 204.2801 457.755 L 194.11485 475.755 L 173.78435 475.755 Z" fill="url(#Obj_Gradient_28)"/><path d="M 163.61909 457.755 L 173.78435 439.755 L 194.11485 439.755 L 204.2801 457.755 L 194.11485 475.755 L 173.78435 475.755 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 62 372.755 L 83 372.755 C 85.484 372.755 87.5 378.13101 87.5 384.755 C 87.5 391.379 85.484 396.755 83 396.755 L 62 396.755 C 59.516 396.755 57.5 391.379 57.5 384.755 C 57.5 378.13101 59.516 372.755 62 372.755" fill="url(#Obj_Gradient_29)"/><path d="M 62 372.755 L 83 372.755 C 85.484 372.755 87.5 378.13101 87.5 384.755 C 87.5 391.379 85.484 396.755 83 396.755 L 62 396.755 C 59.516 396.755 57.5 391.379 57.5 384.755 C 57.5 378.13101 59.516 372.755 62 372.755" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="146.58009" y="343.255" width="67" height="69" fill="url(#Obj_Gradient_30)"/><rect x="146.58009" y="343.255" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 159.7491 379.255 L 169.91435 361.255 L 190.24486 361.255 L 200.41011 379.255 L 190.24486 397.255 L 169.91435 397.255 Z" fill="url(#Obj_Gradient_31)"/><path d="M 159.7491 379.255 L 169.91435 361.255 L 190.24486 361.255 L 200.41011 379.255 L 190.24486 397.255 L 169.91435 397.255 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 156.95009 444.72058 L 177.47003 459.6171 L 161.18541 478.3627 L 167.74992 484.755 L 185.53835 464.47714 L 204.47009 478.2378 L 210.95009 470.78964 L 190.42998 454.96198 L 207.70998 438.20334 L 197.98997 430.755 L 182.86996 449.3757 L 165.59009 434.47916 Z" fill="url(#Obj_Gradient_32)"/><path d="M 156.95009 444.72058 L 177.47003 459.6171 L 161.18541 478.3627 L 167.74992 484.755 L 185.53835 464.47714 L 204.47009 478.2378 L 210.95009 470.78964 L 190.42998 454.96198 L 207.70998 438.20334 L 197.98997 430.755 L 182.86996 449.3757 L 165.59009 434.47916 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><text transform="translate(210.37738 346.23645)" fill="#262626"><tspan font-family="Arial Unicode MS" font-size="36" font-weight="500" fill="#262626" x="0" y="38" textLength="36">♛</tspan></text><text transform="translate(5.8400302 287.736)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="bold" x=".16308594" y="11" textLength="6.673828">3</tspan></text></g></g></svg>
diff --git a/qpid/doc/book/src/java-broker/images/HA-2N-NetworkPartition.png b/qpid/doc/book/src/java-broker/images/HA-2N-NetworkPartition.png
new file mode 100644
index 0000000000..cd4a196924
--- /dev/null
+++ b/qpid/doc/book/src/java-broker/images/HA-2N-NetworkPartition.png
diff --git a/qpid/doc/book/src/java-broker/images/HA-2N-NetworkPartition.svg b/qpid/doc/book/src/java-broker/images/HA-2N-NetworkPartition.svg
new file mode 100644
index 0000000000..375d88a7db
--- /dev/null
+++ b/qpid/doc/book/src/java-broker/images/HA-2N-NetworkPartition.svg
@@ -0,0 +1,3 @@
+<?xml version="1.0"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xl="http://www.w3.org/1999/xlink" version="1.1" viewBox="-20 -12 600 596" width="50pc" height="596pt"><metadata xmlns:dc="http://purl.org/dc/elements/1.1/"><dc:date>2012-05-31 06:19Z</dc:date><!-- Produced by OmniGraffle Professional 5.3.6 --></metadata><defs><radialGradient cx="0" cy="0" r="1" id="Gradient" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset=".5" stop-color="#b5d0ea"/><stop offset="1" stop-color="#aaa"/></radialGradient><radialGradient id="Obj_Gradient" xl:href="#Gradient" gradientTransform="translate(474.08 425.5) scale(145.18695)"/><linearGradient x1="0" x2="1" id="Gradient_2" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#2824aa"/></linearGradient><linearGradient id="Obj_Gradient_2" xl:href="#Gradient_2" gradientTransform="translate(466.66 343.723) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_3" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#aa1f43"/></linearGradient><linearGradient id="Obj_Gradient_3" xl:href="#Gradient_3" gradientTransform="translate(466.65952 360.223) rotate(90) scale(36)"/><linearGradient x1="0" x2="1" id="Gradient_4" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#6923aa"/></linearGradient><linearGradient id="Obj_Gradient_4" xl:href="#Gradient_4" gradientTransform="translate(353.79 377) rotate(90) scale(24.00003)"/><linearGradient id="Obj_Gradient_5" xl:href="#Gradient_2" gradientTransform="translate(474.08 438) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_5" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#21aa1a"/></linearGradient><linearGradient id="Obj_Gradient_6" xl:href="#Gradient_5" gradientTransform="translate(474.0795 456) rotate(90) scale(36)"/><marker orient="auto" overflow="visible" markerUnits="strokeWidth" id="FilledArrow_Marker" viewBox="-1 -3 6 6" markerWidth="6" markerHeight="6" color="black"><g><path d="M 3.7333333 0 L 0 -1.4 L 0 1.4 Z" fill="currentColor" stroke="currentColor" stroke-width="1"/></g></marker><marker orient="auto" overflow="visible" markerUnits="strokeWidth" id="Arrow_Marker" viewBox="-1 -4 10 8" markerWidth="10" markerHeight="8" color="black"><g><path d="M 8 0 L 0 -3 L 0 3 Z" fill="none" stroke="currentColor" stroke-width="1"/></g></marker><radialGradient id="Obj_Gradient_7" xl:href="#Gradient" gradientTransform="translate(470.5 137.5) scale(145.18695)"/><linearGradient id="Obj_Gradient_8" xl:href="#Gradient_2" gradientTransform="translate(488 145.5) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_6" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#6ead6a"/></linearGradient><linearGradient id="Obj_Gradient_9" xl:href="#Gradient_6" gradientTransform="translate(487.9995 162) rotate(90) scale(36)"/><linearGradient id="Obj_Gradient_10" xl:href="#Gradient_4" gradientTransform="translate(350.21 89) rotate(90) scale(24)"/><radialGradient cx="0" cy="0" r="1" id="Gradient_7" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset=".5" stop-color="#ea061f"/><stop offset="1" stop-color="#aaa"/></radialGradient><radialGradient id="Obj_Gradient_11" xl:href="#Gradient_7" gradientTransform="translate(487.17532 136.54291) scale(20.071926)"/><linearGradient id="Obj_Gradient_12" xl:href="#Gradient_2" gradientTransform="translate(466.66 55.445) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_8" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#b6b4b5"/></linearGradient><linearGradient id="Obj_Gradient_13" xl:href="#Gradient_8" gradientTransform="translate(466.65952 71.945) rotate(90) scale(36)"/><font-face font-family="Helvetica" font-size="12" units-per-em="1000" underline-position="-75.683594" underline-thickness="49.316406" slope="0" x-height="532.22656" cap-height="719.72656" ascent="770.01953" descent="-229.98047" font-weight="bold"><font-face-src><font-face-name name="Helvetica-Bold"/></font-face-src></font-face><radialGradient id="Obj_Gradient_14" xl:href="#Gradient" gradientTransform="translate(200.35501 140.5) scale(145.18695)"/><linearGradient id="Obj_Gradient_15" xl:href="#Gradient_2" gradientTransform="translate(212.855 150) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_9" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#0caa25"/></linearGradient><linearGradient id="Obj_Gradient_16" xl:href="#Gradient_9" gradientTransform="translate(212.85451 166.5) rotate(90) scale(36)"/><linearGradient id="Obj_Gradient_17" xl:href="#Gradient_4" gradientTransform="translate(80.355003 115) rotate(90) scale(24)"/><linearGradient id="Obj_Gradient_18" xl:href="#Gradient_2" gradientTransform="translate(195.355 63.999996) rotate(90) scale(69.000007)"/><linearGradient x1="0" x2="1" id="Gradient_10" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#aa172d"/></linearGradient><linearGradient id="Obj_Gradient_19" xl:href="#Gradient_10" gradientTransform="translate(195.35451 82) rotate(90) scale(36)"/><radialGradient id="Obj_Gradient_20" xl:href="#Gradient" gradientTransform="translate(180.29001 423.5) scale(145.18695)"/><linearGradient id="Obj_Gradient_21" xl:href="#Gradient_2" gradientTransform="translate(172.37 346) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_11" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#aaa3a7"/></linearGradient><linearGradient id="Obj_Gradient_22" xl:href="#Gradient_11" gradientTransform="translate(172.36951 360.5) rotate(90) scale(36)"/><linearGradient id="Obj_Gradient_23" xl:href="#Gradient_4" gradientTransform="translate(72.5 379) rotate(90) scale(24)"/><linearGradient id="Obj_Gradient_24" xl:href="#Gradient_2" gradientTransform="translate(180.953 435.5) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_12" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#66ab61"/></linearGradient><linearGradient id="Obj_Gradient_25" xl:href="#Gradient_12" gradientTransform="translate(180.95251 452) rotate(90) scale(36)"/><linearGradient x1="0" x2="1" id="Gradient_13" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#b5011d"/></linearGradient><linearGradient id="Obj_Gradient_26" xl:href="#Gradient_13" gradientTransform="translate(169.38802 421.22614) rotate(45) scale(21.988586)"/></defs><g stroke="none" stroke-opacity="1" stroke-dasharray="none" fill="none" fill-opacity="1"><title>Canvas 5</title><g><title>Layer 1</title><rect x="281.29001" y="287" width="278.00003" height="273" fill="white"/><rect x="281.29001" y="287" width="278.00003" height="273" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 420.11401 438.9812 C 396.51749 425.5 405.92719 312.01035 443.5689 331.375 C 447.06122 293.6271 490.8335 299.75403 490.54733 331.375 C 517.9939 290.93137 553.0689 371.57516 529.54248 412.0188 C 557.773 431.62692 529.18622 537.27283 506.0175 519.625 C 504.1633 549.03967 462.74454 559.3332 459.10913 519.625 C 435.6557 562.03143 386.75156 496.82916 420.11401 438.9812 Z" fill="url(#Obj_Gradient)"/><path d="M 420.11401 438.9812 C 396.51749 425.5 405.92719 312.01035 443.5689 331.375 C 447.06122 293.6271 490.8335 299.75403 490.54733 331.375 C 517.9939 290.93137 553.0689 371.57516 529.54248 412.0188 C 557.773 431.62692 529.18622 537.27283 506.0175 519.625 C 504.1633 549.03967 462.74454 559.3332 459.10913 519.625 C 435.6557 562.03143 386.75156 496.82916 420.11401 438.9812 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="433.16" y="343.723" width="67" height="69" fill="url(#Obj_Gradient_2)"/><rect x="433.16" y="343.723" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 446.329 378.223 L 456.49426 360.223 L 476.82477 360.223 L 486.99002 378.223 L 476.82477 396.223 L 456.49426 396.223 Z" fill="url(#Obj_Gradient_3)"/><path d="M 446.329 378.223 L 456.49426 360.223 L 476.82477 360.223 L 486.99002 378.223 L 476.82477 396.223 L 456.49426 396.223 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 343.29 377 L 364.29 377 C 366.77402 377 368.79 382.376 368.79 389 C 368.79 395.624 366.77402 401 364.29 401 L 343.29 401 C 340.806 401 338.79 395.624 338.79 389 C 338.79 382.376 340.806 377 343.29 377" fill="url(#Obj_Gradient_4)"/><path d="M 343.29 377 L 364.29 377 C 366.77402 377 368.79 382.376 368.79 389 C 368.79 395.624 366.77402 401 364.29 401 L 343.29 401 C 340.806 401 338.79 395.624 338.79 389 C 338.79 382.376 340.806 377 343.29 377" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="440.57999" y="438" width="67" height="69" fill="url(#Obj_Gradient_5)"/><rect x="440.57999" y="438" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 453.749 474 L 463.91425 456 L 484.24475 456 L 494.41 474 L 484.24475 492 L 463.91425 492 Z" fill="url(#Obj_Gradient_6)"/><path d="M 453.749 474 L 463.91425 456 L 484.24475 456 L 494.41 474 L 484.24475 492 L 463.91425 492 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 358.11069 401.47244 C 364.4065 419.64648 360.82266 444.49713 377 456 C 390.09668 465.31238 416.1513 465.88107 439.42624 468.3237" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="3"/><path d="M 480.27563 455.52597 C 482.5702 448.68466 488.31772 444.80447 487.16 435 C 486.26978 427.46088 481.2962 416.41544 476.94943 405.90878" marker-end="url(#Arrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="281.29001" y="4" width="278.00003" height="273" fill="white"/><rect x="281.29001" y="4" width="278.00003" height="273" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 416.53403 150.98122 C 392.9375 137.5 402.3472 24.010353 439.98892 43.375 C 443.48123 5.6271057 487.2535 11.754013 486.96735 43.375 C 514.41388 2.931366 549.4889 83.575165 525.96246 124.01879 C 554.19305 143.62691 525.60626 249.27281 502.4375 231.625 C 500.5833 261.03967 459.16455 271.33319 455.52917 231.625 C 432.07571 274.03143 383.17157 208.82918 416.53403 150.98122 Z" fill="url(#Obj_Gradient_7)"/><path d="M 416.53403 150.98122 C 392.9375 137.5 402.3472 24.010353 439.98892 43.375 C 443.48123 5.6271057 487.2535 11.754013 486.96735 43.375 C 514.41388 2.931366 549.4889 83.575165 525.96246 124.01879 C 554.19305 143.62691 525.60626 249.27281 502.4375 231.625 C 500.5833 261.03967 459.16455 271.33319 455.52917 231.625 C 432.07571 274.03143 383.17157 208.82918 416.53403 150.98122 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="454.5" y="145.5" width="67" height="69" fill="url(#Obj_Gradient_8)"/><rect x="454.5" y="145.5" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 467.669 180 L 477.83426 162 L 498.16476 162 L 508.33002 180 L 498.16476 198 L 477.83426 198 Z" fill="url(#Obj_Gradient_9)"/><path d="M 467.669 180 L 477.83426 162 L 498.16476 162 L 508.33002 180 L 498.16476 198 L 477.83426 198 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 339.71 89 L 360.71 89 C 363.194 89 365.21 94.376 365.21 101 C 365.21 107.624 363.194 113 360.71 113 L 339.71 113 C 337.22598 113 335.21 107.624 335.21 101 C 335.21 94.376 337.22598 89 339.71 89" fill="url(#Obj_Gradient_10)"/><path d="M 339.71 89 L 360.71 89 C 363.194 89 365.21 94.376 365.21 101 C 365.21 107.624 363.194 113 360.71 113 L 339.71 113 C 337.22598 113 335.21 107.624 335.21 101 C 335.21 94.376 337.22598 89 339.71 89" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 474.53464 129.01602 L 484.14154 137.61832 L 476.51752 148.44287 L 479.59082 152.134415 L 487.91891 140.42473 L 496.7823 148.37096 L 499.816 144.06992 L 490.20901 134.93007 L 498.2991 125.25254 L 493.74844 120.95141 L 486.66962 131.70415 L 478.57965 123.101974 Z" fill="url(#Obj_Gradient_11)"/><path d="M 474.53464 129.01602 L 484.14154 137.61832 L 476.51752 148.44287 L 479.59082 152.134415 L 487.91891 140.42473 L 496.7823 148.37096 L 499.816 144.06992 L 490.20901 134.93007 L 498.2991 125.25254 L 493.74844 120.95141 L 486.66962 131.70415 L 478.57965 123.101974 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="433.16" y="55.445" width="67" height="69" fill="url(#Obj_Gradient_12)"/><rect x="433.16" y="55.445" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 446.329 89.945 L 456.49426 71.945 L 476.82477 71.945 L 486.99002 89.945 L 476.82477 107.945 L 456.49426 107.945 Z" fill="url(#Obj_Gradient_13)"/><path d="M 446.329 89.945 L 456.49426 71.945 L 476.82477 71.945 L 486.99002 89.945 L 476.82477 107.945 L 456.49426 107.945 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 487.69427 161.50006 C 487.55954 153.3342 489.4468 145.84863 487.29 137 C 485.7971 130.87509 482.3659 124.09454 479.00916 117.33957" marker-end="url(#Arrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><text transform="translate(5.8400302 4)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="bold" x=".16308594" y="11" textLength="6.673828">1</tspan></text><text transform="translate(286 6)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="bold" x=".16308594" y="11" textLength="6.673828">2</tspan></text><rect x="0" y="4" width="278" height="273" fill="white"/><rect x="0" y="4" width="278" height="273" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 146.38904 153.98122 C 122.79251 140.5 132.20221 27.010353 169.84393 46.375 C 173.33626 8.6271057 217.1085 14.754013 216.82236 46.375 C 244.26889 5.931366 279.34393 86.575165 255.81749 127.01879 C 284.04803 146.62691 255.46124 252.27281 232.29251 234.625 C 230.43831 264.03967 189.01958 274.33319 185.38417 234.625 C 161.930725 277.03143 113.02657 211.82918 146.38904 153.98122 Z" fill="url(#Obj_Gradient_14)"/><path d="M 146.38904 153.98122 C 122.79251 140.5 132.20221 27.010353 169.84393 46.375 C 173.33626 8.6271057 217.1085 14.754013 216.82236 46.375 C 244.26889 5.931366 279.34393 86.575165 255.81749 127.01879 C 284.04803 146.62691 255.46124 252.27281 232.29251 234.625 C 230.43831 264.03967 189.01958 274.33319 185.38417 234.625 C 161.930725 277.03143 113.02657 211.82918 146.38904 153.98122 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="179.355" y="150" width="67" height="69" fill="url(#Obj_Gradient_15)"/><rect x="179.355" y="150" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 192.524 184.5 L 202.68925 166.5 L 223.01976 166.5 L 233.18501 184.5 L 223.01976 202.5 L 202.68925 202.5 Z" fill="url(#Obj_Gradient_16)"/><path d="M 192.524 184.5 L 202.68925 166.5 L 223.01976 166.5 L 233.18501 184.5 L 223.01976 202.5 L 202.68925 202.5 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 69.855003 115 L 90.855003 115 C 93.339005 115 95.355003 120.376 95.355003 127 C 95.355003 133.62399 93.339005 139 90.855003 139 L 69.855003 139 C 67.371002 139 65.355003 133.62399 65.355003 127 C 65.355003 120.376 67.371002 115 69.855003 115" fill="url(#Obj_Gradient_17)"/><path d="M 69.855003 115 L 90.855003 115 C 93.339005 115 95.355003 120.376 95.355003 127 C 95.355003 133.62399 93.339005 139 90.855003 139 L 69.855003 139 C 67.371002 139 65.355003 133.62399 65.355003 127 C 65.355003 120.376 67.371002 115 69.855003 115" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="161.855" y="64" width="67" height="69" fill="url(#Obj_Gradient_18)"/><rect x="161.855" y="64" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 175.024 100 L 185.18925 82 L 205.51976 82 L 215.68501 100 L 205.51976 118 L 185.18925 118 Z" fill="url(#Obj_Gradient_19)"/><path d="M 175.024 100 L 185.18925 82 L 205.51976 82 L 215.68501 100 L 205.51976 118 L 185.18925 118 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 88 139 C 101.11702 162.9976 109.51097 202.49907 127.355 211 C 141.45248 217.71606 161.45502 205.08597 180.36313 196.03654" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="3"/><path d="M 222.38084 166.05576 C 227.03842 157.03807 238.59448 147.85422 236.355 139 C 234.66139 132.304 225.07715 125.79347 216.5538 119.25858" marker-end="url(#Arrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><text transform="translate(286 293)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="bold" x=".16308594" y="11" textLength="6.673828">4</tspan></text><text transform="translate(5.8400302 6)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="bold" x=".16308594" y="11" textLength="6.673828">1</tspan></text><text transform="translate(5.840027 287)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="bold" x=".16308594" y="11" textLength="6.673828">3</tspan></text><rect x="0" y="287" width="278" height="273" fill="white"/><rect x="0" y="287" width="278" height="273" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 126.32403 436.9812 C 102.72751 423.5 112.13721 310.01035 149.77893 329.375 C 153.27126 291.6271 197.0435 297.75403 196.75735 329.375 C 224.20389 288.93137 259.27893 369.57516 235.75249 410.0188 C 263.98303 429.62692 235.39624 535.27283 212.22751 517.625 C 210.37331 547.03967 168.95457 557.3332 165.31917 517.625 C 141.86572 560.03143 92.96157 494.82916 126.32403 436.9812 Z" fill="url(#Obj_Gradient_20)"/><path d="M 126.32403 436.9812 C 102.72751 423.5 112.13721 310.01035 149.77893 329.375 C 153.27126 291.6271 197.0435 297.75403 196.75735 329.375 C 224.20389 288.93137 259.27893 369.57516 235.75249 410.0188 C 263.98303 429.62692 235.39624 535.27283 212.22751 517.625 C 210.37331 547.03967 168.95457 557.3332 165.31917 517.625 C 141.86572 560.03143 92.96157 494.82916 126.32403 436.9812 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="138.869995" y="346" width="67" height="69" fill="url(#Obj_Gradient_21)"/><rect x="138.869995" y="346" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 152.039 378.5 L 162.20425 360.5 L 182.53476 360.5 L 192.70001 378.5 L 182.53476 396.5 L 162.20425 396.5 Z" fill="url(#Obj_Gradient_22)"/><path d="M 152.039 378.5 L 162.20425 360.5 L 182.53476 360.5 L 192.70001 378.5 L 182.53476 396.5 L 162.20425 396.5 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 62 379 L 83 379 C 85.484 379 87.5 384.376 87.5 391 C 87.5 397.624 85.484 403 83 403 L 62 403 C 59.516 403 57.5 397.624 57.5 391 C 57.5 384.376 59.516 379 62 379" fill="url(#Obj_Gradient_23)"/><path d="M 62 379 L 83 379 C 85.484 379 87.5 384.376 87.5 391 C 87.5 397.624 85.484 403 83 403 L 62 403 C 59.516 403 57.5 397.624 57.5 391 C 57.5 384.376 59.516 379 62 379" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="147.453" y="435.5" width="67" height="69" fill="url(#Obj_Gradient_24)"/><rect x="147.453" y="435.5" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 160.62201 470 L 170.78726 452 L 191.11777 452 L 201.28302 470 L 191.11777 488 L 170.78726 488 Z" fill="url(#Obj_Gradient_25)"/><path d="M 160.62201 470 L 170.78726 452 L 191.11777 452 L 201.28302 470 L 191.11777 488 L 170.78726 488 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 181.17079 451.50006 C 181.27284 442.8504 182.34718 434.63248 181.47696 425.54852 C 180.89874 419.51263 179.46176 413.09204 178.022 406.67056" marker-end="url(#Arrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 162.48625 439.34933 L 182.67911 418.90622 C 182.67911 418.90622 176.37192 410.54022 180.79669 410.70734 C 185.22139 410.87448 187.68919 413.07565 187.68919 413.07565 L 183.62592 416.69077 L 189.96997 423.03482 L 193.83488 419.61807 C 193.83488 419.61807 196.0623 421.44876 196.25372 421.64017 C 196.44499 421.83145 196.1347 425.18307 195.37999 426.3204 C 194.62503 427.45789 187.93712 423.79865 187.93724 423.79877 C 187.93724 423.79877 166.50412 443.3672 166.44502 443.4263 C 166.38591 443.4854 165.92685 444.32736 163.88141 442.16357 C 161.83583 439.99976 162.48625 439.34933 162.48625 439.34933 Z" fill="url(#Obj_Gradient_26)"/><path d="M 162.48625 439.34933 L 182.67911 418.90622 C 182.67911 418.90622 176.37192 410.54022 180.79669 410.70734 C 185.22139 410.87448 187.68919 413.07565 187.68919 413.07565 L 183.62592 416.69077 L 189.96997 423.03482 L 193.83488 419.61807 C 193.83488 419.61807 196.0623 421.44876 196.25372 421.64017 C 196.44499 421.83145 196.1347 425.18307 195.37999 426.3204 C 194.62503 427.45789 187.93712 423.79865 187.93724 423.79877 C 187.93724 423.79877 166.50412 443.3672 166.44502 443.4263 C 166.38591 443.4854 165.92685 444.32736 163.88141 442.16357 C 161.83583 439.99976 162.48625 439.34933 162.48625 439.34933 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 75.526443 403.48596 C 80.05397 422.16476 74.80702 448.81583 89.11038 459.52802 C 100.46647 468.0329 124.151474 466.49478 145.480865 466.87387" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="3"/><text transform="translate(10 297)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="bold" x=".16308594" y="11" textLength="6.673828">3</tspan></text></g></g></svg>
diff --git a/qpid/doc/book/src/java-broker/images/HA-2N-Normal.png b/qpid/doc/book/src/java-broker/images/HA-2N-Normal.png
new file mode 100644
index 0000000000..eeaad4f230
--- /dev/null
+++ b/qpid/doc/book/src/java-broker/images/HA-2N-Normal.png
diff --git a/qpid/doc/book/src/java-broker/images/HA-2N-Normal.svg b/qpid/doc/book/src/java-broker/images/HA-2N-Normal.svg
new file mode 100644
index 0000000000..c4fac9d37a
--- /dev/null
+++ b/qpid/doc/book/src/java-broker/images/HA-2N-Normal.svg
@@ -0,0 +1,3 @@
+<?xml version="1.0"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xl="http://www.w3.org/1999/xlink" version="1.1" viewBox="-11 -10 318 313" width="318pt" height="313pt"><metadata xmlns:dc="http://purl.org/dc/elements/1.1/"><dc:date>2012-05-31 06:19Z</dc:date><!-- Produced by OmniGraffle Professional 5.3.6 --></metadata><defs><radialGradient cx="0" cy="0" r="1" id="Gradient" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset=".5" stop-color="#b5d0ea"/><stop offset="1" stop-color="#aaa"/></radialGradient><radialGradient id="Obj_Gradient" xl:href="#Gradient" gradientTransform="translate(209.35501 142.5) scale(145.18695)"/><linearGradient x1="0" x2="1" id="Gradient_2" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#2824aa"/></linearGradient><linearGradient id="Obj_Gradient_2" xl:href="#Gradient_2" gradientTransform="translate(221.85501 152) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_3" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#0caa25"/></linearGradient><linearGradient id="Obj_Gradient_3" xl:href="#Gradient_3" gradientTransform="translate(221.85452 168.5) rotate(90) scale(36)"/><linearGradient x1="0" x2="1" id="Gradient_4" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#6923aa"/></linearGradient><linearGradient id="Obj_Gradient_4" xl:href="#Gradient_4" gradientTransform="translate(71.35501 115) rotate(90) scale(24)"/><linearGradient id="Obj_Gradient_5" xl:href="#Gradient_2" gradientTransform="translate(204.35501 66) rotate(90) scale(69.000007)"/><linearGradient x1="0" x2="1" id="Gradient_5" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#aa172d"/></linearGradient><linearGradient id="Obj_Gradient_6" xl:href="#Gradient_5" gradientTransform="translate(204.35452 84) rotate(90) scale(36)"/><marker orient="auto" overflow="visible" markerUnits="strokeWidth" id="FilledArrow_Marker" viewBox="-1 -3 6 6" markerWidth="6" markerHeight="6" color="black"><g><path d="M 3.7333333 0 L 0 -1.4 L 0 1.4 Z" fill="currentColor" stroke="currentColor" stroke-width="1"/></g></marker><marker orient="auto" overflow="visible" markerUnits="strokeWidth" id="Arrow_Marker" viewBox="-1 -4 10 8" markerWidth="10" markerHeight="8" color="black"><g><path d="M 8 0 L 0 -3 L 0 3 Z" fill="none" stroke="currentColor" stroke-width="1"/></g></marker></defs><g stroke="none" stroke-opacity="1" stroke-dasharray="none" fill="none" fill-opacity="1"><title>Canvas 1</title><g><title>Layer 1</title><rect x="9" y="6" width="278" height="273" fill="white"/><rect x="9" y="6" width="278" height="273" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 155.38904 155.98122 C 131.79251 142.5 141.20221 29.010353 178.84393 48.375 C 182.33626 10.627106 226.1085 16.754013 225.82236 48.375 C 253.26889 7.931366 288.34393 88.575165 264.8175 129.01878 C 293.04803 148.62691 264.46124 254.27281 241.29251 236.625 C 239.43831 266.03967 198.01958 276.33319 194.38417 236.625 C 170.93073 279.03143 122.02657 213.82918 155.38904 155.98122 Z" fill="url(#Obj_Gradient)"/><path d="M 155.38904 155.98122 C 131.79251 142.5 141.20221 29.010353 178.84393 48.375 C 182.33626 10.627106 226.1085 16.754013 225.82236 48.375 C 253.26889 7.931366 288.34393 88.575165 264.8175 129.01878 C 293.04803 148.62691 264.46124 254.27281 241.29251 236.625 C 239.43831 266.03967 198.01958 276.33319 194.38417 236.625 C 170.93073 279.03143 122.02657 213.82918 155.38904 155.98122 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="188.35501" y="152" width="67" height="69" fill="url(#Obj_Gradient_2)"/><rect x="188.35501" y="152" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 201.52402 186.5 L 211.68927 168.5 L 232.01978 168.5 L 242.18503 186.5 L 232.01978 204.5 L 211.68927 204.5 Z" fill="url(#Obj_Gradient_3)"/><path d="M 201.52402 186.5 L 211.68927 168.5 L 232.01978 168.5 L 242.18503 186.5 L 232.01978 204.5 L 211.68927 204.5 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 60.85501 115 L 81.85501 115 C 84.33901 115 86.35501 120.376 86.35501 127 C 86.35501 133.62399 84.33901 139 81.85501 139 L 60.85501 139 C 58.37101 139 56.35501 133.62399 56.35501 127 C 56.35501 120.376 58.37101 115 60.85501 115" fill="url(#Obj_Gradient_4)"/><path d="M 60.85501 115 L 81.85501 115 C 84.33901 115 86.35501 120.376 86.35501 127 C 86.35501 133.62399 84.33901 139 81.85501 139 L 60.85501 139 C 58.37101 139 56.35501 133.62399 56.35501 127 C 56.35501 120.376 58.37101 115 60.85501 115" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="170.85501" y="66" width="67" height="69" fill="url(#Obj_Gradient_5)"/><rect x="170.85501" y="66" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 184.02402 102 L 194.18927 84 L 214.51978 84 L 224.68503 102 L 214.51978 120 L 194.18927 120 Z" fill="url(#Obj_Gradient_6)"/><path d="M 184.02402 102 L 194.18927 84 L 214.51978 84 L 224.68503 102 L 214.51978 120 L 194.18927 120 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 83.35501 143 C 101.01991 166.33099 116.23704 204.83237 136.35501 213 C 152.19005 219.42882 171.06885 207.06767 189.39609 198.1017" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="3"/><path d="M 231.38086 168.05576 C 236.03844 159.03807 247.59447 149.85422 245.35498 141 C 243.66136 134.304 234.07715 127.79349 225.55383 121.2586" marker-end="url(#Arrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/></g></g></svg>
diff --git a/qpid/doc/book/src/java-broker/images/HA-2N-ReplicaFail.png b/qpid/doc/book/src/java-broker/images/HA-2N-ReplicaFail.png
new file mode 100644
index 0000000000..769fc959fc
--- /dev/null
+++ b/qpid/doc/book/src/java-broker/images/HA-2N-ReplicaFail.png
diff --git a/qpid/doc/book/src/java-broker/images/HA-2N-ReplicaFail.svg b/qpid/doc/book/src/java-broker/images/HA-2N-ReplicaFail.svg
new file mode 100644
index 0000000000..aa872a30e3
--- /dev/null
+++ b/qpid/doc/book/src/java-broker/images/HA-2N-ReplicaFail.svg
@@ -0,0 +1,3 @@
+<?xml version="1.0"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xl="http://www.w3.org/1999/xlink" version="1.1" viewBox="-20 -16 600 590" width="50pc" height="590pt"><metadata xmlns:dc="http://purl.org/dc/elements/1.1/"><dc:date>2012-05-31 06:19Z</dc:date><!-- Produced by OmniGraffle Professional 5.3.6 --></metadata><defs><radialGradient cx="0" cy="0" r="1" id="Gradient" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset=".5" stop-color="#b5d0ea"/><stop offset="1" stop-color="#aaa"/></radialGradient><radialGradient id="Obj_Gradient" xl:href="#Gradient" gradientTransform="translate(470.5 133.5) scale(145.18695)"/><linearGradient x1="0" x2="1" id="Gradient_2" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#2824aa"/></linearGradient><linearGradient id="Obj_Gradient_2" xl:href="#Gradient_2" gradientTransform="translate(488 141.5) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_3" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#6cad67"/></linearGradient><linearGradient id="Obj_Gradient_3" xl:href="#Gradient_3" gradientTransform="translate(487.9995 158) rotate(90) scale(36)"/><linearGradient x1="0" x2="1" id="Gradient_4" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#6923aa"/></linearGradient><linearGradient id="Obj_Gradient_4" xl:href="#Gradient_4" gradientTransform="translate(350.21 85) rotate(90) scale(24)"/><linearGradient id="Obj_Gradient_5" xl:href="#Gradient_2" gradientTransform="translate(470.5 55.5) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_5" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#aaaaa3"/></linearGradient><linearGradient id="Obj_Gradient_6" xl:href="#Gradient_5" gradientTransform="translate(470.4995 73.5) rotate(90) scale(36)"/><marker orient="auto" overflow="visible" markerUnits="strokeWidth" id="FilledArrow_Marker" viewBox="-1 -3 6 6" markerWidth="6" markerHeight="6" color="black"><g><path d="M 3.7333333 0 L 0 -1.4 L 0 1.4 Z" fill="currentColor" stroke="currentColor" stroke-width="1"/></g></marker><radialGradient cx="0" cy="0" r="1" id="Gradient_6" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset=".5" stop-color="#ea061f"/><stop offset="1" stop-color="#aaa"/></radialGradient><radialGradient id="Obj_Gradient_7" xl:href="#Gradient_6" gradientTransform="translate(472.145 91.5) scale(38.183766)"/><font-face font-family="Helvetica" font-size="12" units-per-em="1000" underline-position="-75.683594" underline-thickness="49.316406" slope="0" x-height="532.22656" cap-height="719.72656" ascent="770.01953" descent="-229.98047" font-weight="bold"><font-face-src><font-face-name name="Helvetica-Bold"/></font-face-src></font-face><radialGradient id="Obj_Gradient_8" xl:href="#Gradient" gradientTransform="translate(474.08 415.5) scale(145.18695)"/><linearGradient id="Obj_Gradient_9" xl:href="#Gradient_2" gradientTransform="translate(466.66 333.723) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_7" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#aa1f43"/></linearGradient><linearGradient id="Obj_Gradient_10" xl:href="#Gradient_7" gradientTransform="translate(466.65952 350.223) rotate(90) scale(36)"/><linearGradient id="Obj_Gradient_11" xl:href="#Gradient_4" gradientTransform="translate(353.79 367) rotate(90) scale(24.00003)"/><linearGradient id="Obj_Gradient_12" xl:href="#Gradient_2" gradientTransform="translate(474.08 428) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_8" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#21aa1a"/></linearGradient><linearGradient id="Obj_Gradient_13" xl:href="#Gradient_8" gradientTransform="translate(474.0795 446) rotate(90) scale(36)"/><marker orient="auto" overflow="visible" markerUnits="strokeWidth" id="Arrow_Marker" viewBox="-1 -4 10 8" markerWidth="10" markerHeight="8" color="black"><g><path d="M 8 0 L 0 -3 L 0 3 Z" fill="none" stroke="currentColor" stroke-width="1"/></g></marker><radialGradient id="Obj_Gradient_14" xl:href="#Gradient" gradientTransform="translate(180.29001 409.5) scale(145.18695)"/><linearGradient id="Obj_Gradient_15" xl:href="#Gradient_2" gradientTransform="translate(172.37 332) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_9" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#aaa3a7"/></linearGradient><linearGradient id="Obj_Gradient_16" xl:href="#Gradient_9" gradientTransform="translate(172.36951 346.5) rotate(90) scale(36)"/><linearGradient id="Obj_Gradient_17" xl:href="#Gradient_4" gradientTransform="translate(72.5 365) rotate(90) scale(24)"/><linearGradient id="Obj_Gradient_18" xl:href="#Gradient_2" gradientTransform="translate(180.953 421.5) rotate(90) scale(69)"/><linearGradient id="Obj_Gradient_19" xl:href="#Gradient_8" gradientTransform="translate(180.9525 438) rotate(90) scale(36)"/><linearGradient x1="0" x2="1" id="Gradient_10" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#b5011d"/></linearGradient><linearGradient id="Obj_Gradient_20" xl:href="#Gradient_10" gradientTransform="translate(134.69392 361.18848) rotate(45) scale(23.213103)"/><radialGradient id="Obj_Gradient_21" xl:href="#Gradient" gradientTransform="translate(200.35501 136.5) scale(145.18695)"/><linearGradient id="Obj_Gradient_22" xl:href="#Gradient_2" gradientTransform="translate(212.855 146) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_11" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#0caa25"/></linearGradient><linearGradient id="Obj_Gradient_23" xl:href="#Gradient_11" gradientTransform="translate(212.85451 162.5) rotate(90) scale(36)"/><linearGradient id="Obj_Gradient_24" xl:href="#Gradient_4" gradientTransform="translate(80.355003 111) rotate(90) scale(24)"/><linearGradient id="Obj_Gradient_25" xl:href="#Gradient_2" gradientTransform="translate(195.355 59.999996) rotate(90) scale(69.000007)"/><linearGradient x1="0" x2="1" id="Gradient_12" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#aa172d"/></linearGradient><linearGradient id="Obj_Gradient_26" xl:href="#Gradient_12" gradientTransform="translate(195.35451 78) rotate(90) scale(36)"/></defs><g stroke="none" stroke-opacity="1" stroke-dasharray="none" fill="none" fill-opacity="1"><title>Canvas 4</title><g><title>Layer 1</title><rect x="281.29001" y="0" width="278.00003" height="273" fill="white"/><rect x="281.29001" y="0" width="278.00003" height="273" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 416.53403 146.98122 C 392.9375 133.5 402.3472 20.010353 439.98892 39.375 C 443.48123 1.6271057 487.2535 7.754013 486.96735 39.375 C 514.41388 -1.06863403 549.4889 79.575165 525.96246 120.01879 C 554.19305 139.62691 525.60626 245.27281 502.4375 227.625 C 500.5833 257.03967 459.16455 267.33319 455.52917 227.625 C 432.07571 270.03143 383.17157 204.82918 416.53403 146.98122 Z" fill="url(#Obj_Gradient)"/><path d="M 416.53403 146.98122 C 392.9375 133.5 402.3472 20.010353 439.98892 39.375 C 443.48123 1.6271057 487.2535 7.754013 486.96735 39.375 C 514.41388 -1.06863403 549.4889 79.575165 525.96246 120.01879 C 554.19305 139.62691 525.60626 245.27281 502.4375 227.625 C 500.5833 257.03967 459.16455 267.33319 455.52917 227.625 C 432.07571 270.03143 383.17157 204.82918 416.53403 146.98122 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="454.5" y="141.5" width="67" height="69" fill="url(#Obj_Gradient_2)"/><rect x="454.5" y="141.5" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 467.669 176 L 477.83426 158 L 498.16476 158 L 508.33002 176 L 498.16476 194 L 477.83426 194 Z" fill="url(#Obj_Gradient_3)"/><path d="M 467.669 176 L 477.83426 158 L 498.16476 158 L 508.33002 176 L 498.16476 194 L 477.83426 194 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 339.71 85 L 360.71 85 C 363.194 85 365.21 90.376 365.21 97 C 365.21 103.624 363.194 109 360.71 109 L 339.71 109 C 337.22598 109 335.21 103.624 335.21 97 C 335.21 90.376 337.22598 85 339.71 85" fill="url(#Obj_Gradient_4)"/><path d="M 339.71 85 L 360.71 85 C 363.194 85 365.21 90.376 365.21 97 C 365.21 103.624 363.194 109 360.71 109 L 339.71 109 C 337.22598 109 335.21 103.624 335.21 97 C 335.21 90.376 337.22598 85 339.71 85" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="437" y="55.5" width="67" height="69" fill="url(#Obj_Gradient_5)"/><rect x="437" y="55.5" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 450.169 91.5 L 460.33426 73.5 L 480.66476 73.5 L 490.83002 91.5 L 480.66476 109.5 L 460.33426 109.5 Z" fill="url(#Obj_Gradient_6)"/><path d="M 450.169 91.5 L 460.33426 73.5 L 480.66476 73.5 L 490.83002 91.5 L 480.66476 109.5 L 460.33426 109.5 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 355.0216 109.46646 C 360.75049 124.30949 353.19086 143.50909 372.21 154 C 388.3057 162.87834 423.4454 165.52399 453.4511 169.84836" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="3"/><path d="M 445.145 78.46559 L 465.66495 93.362083 L 449.3803 112.107697 L 455.94482 118.5 L 473.73325 98.22214 L 492.66498 111.982796 L 499.145 104.53463 L 478.62488 88.706955 L 495.90488 71.948326 L 486.18488 64.5 L 471.06485 83.12071 L 453.78497 68.224167 Z" fill="url(#Obj_Gradient_7)"/><path d="M 445.145 78.46559 L 465.66495 93.362083 L 449.3803 112.107697 L 455.94482 118.5 L 473.73325 98.22214 L 492.66498 111.982796 L 499.145 104.53463 L 478.62488 88.706955 L 495.90488 71.948326 L 486.18488 64.5 L 471.06485 83.12071 L 453.78497 68.224167 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><text transform="translate(286 6)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="bold" x=".16308594" y="11" textLength="6.673828">2</tspan></text><rect x="281.29001" y="277" width="278.00003" height="273" fill="white"/><rect x="281.29001" y="277" width="278.00003" height="273" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 420.11401 428.9812 C 396.51749 415.5 405.92719 302.01035 443.5689 321.375 C 447.06122 283.6271 490.8335 289.75403 490.54733 321.375 C 517.9939 280.93137 553.0689 361.57516 529.54248 402.0188 C 557.773 421.62692 529.18622 527.27283 506.0175 509.625 C 504.1633 539.03967 462.74454 549.3332 459.10913 509.625 C 435.6557 552.03143 386.75156 486.82916 420.11401 428.9812 Z" fill="url(#Obj_Gradient_8)"/><path d="M 420.11401 428.9812 C 396.51749 415.5 405.92719 302.01035 443.5689 321.375 C 447.06122 283.6271 490.8335 289.75403 490.54733 321.375 C 517.9939 280.93137 553.0689 361.57516 529.54248 402.0188 C 557.773 421.62692 529.18622 527.27283 506.0175 509.625 C 504.1633 539.03967 462.74454 549.3332 459.10913 509.625 C 435.6557 552.03143 386.75156 486.82916 420.11401 428.9812 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="433.16" y="333.723" width="67" height="69" fill="url(#Obj_Gradient_9)"/><rect x="433.16" y="333.723" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 446.329 368.223 L 456.49426 350.223 L 476.82477 350.223 L 486.99002 368.223 L 476.82477 386.223 L 456.49426 386.223 Z" fill="url(#Obj_Gradient_10)"/><path d="M 446.329 368.223 L 456.49426 350.223 L 476.82477 350.223 L 486.99002 368.223 L 476.82477 386.223 L 456.49426 386.223 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 343.29 367 L 364.29 367 C 366.77402 367 368.79 372.376 368.79 379 C 368.79 385.624 366.77402 391 364.29 391 L 343.29 391 C 340.806 391 338.79 385.624 338.79 379 C 338.79 372.376 340.806 367 343.29 367" fill="url(#Obj_Gradient_11)"/><path d="M 343.29 367 L 364.29 367 C 366.77402 367 368.79 372.376 368.79 379 C 368.79 385.624 366.77402 391 364.29 391 L 343.29 391 C 340.806 391 338.79 385.624 338.79 379 C 338.79 372.376 340.806 367 343.29 367" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="440.57999" y="428" width="67" height="69" fill="url(#Obj_Gradient_12)"/><rect x="440.57999" y="428" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 453.749 464 L 463.91425 446 L 484.24475 446 L 494.41 464 L 484.24475 482 L 463.91425 482 Z" fill="url(#Obj_Gradient_13)"/><path d="M 453.749 464 L 463.91425 446 L 484.24475 446 L 494.41 464 L 484.24475 482 L 463.91425 482 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 358.11069 391.47244 C 364.4065 409.64648 360.82266 434.49713 377 446 C 390.09668 455.31238 416.1513 455.88107 439.42624 458.3237" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="3"/><path d="M 480.27567 445.52594 C 482.57022 438.68463 488.31775 434.80447 487.16 425 C 486.26974 417.46088 481.2962 406.41544 476.94943 395.90878" marker-end="url(#Arrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="0" y="277" width="278" height="273" fill="white"/><rect x="0" y="277" width="278" height="273" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 126.32403 422.9812 C 102.72751 409.5 112.13721 296.01035 149.77893 315.375 C 153.27126 277.6271 197.0435 283.75403 196.75735 315.375 C 224.20389 274.93137 259.27893 355.57516 235.75249 396.0188 C 263.98303 415.62692 235.39624 521.27283 212.22751 503.625 C 210.37331 533.03967 168.95457 543.3332 165.31917 503.625 C 141.86572 546.03143 92.96157 480.82916 126.32403 422.9812 Z" fill="url(#Obj_Gradient_14)"/><path d="M 126.32403 422.9812 C 102.72751 409.5 112.13721 296.01035 149.77893 315.375 C 153.27126 277.6271 197.0435 283.75403 196.75735 315.375 C 224.20389 274.93137 259.27893 355.57516 235.75249 396.0188 C 263.98303 415.62692 235.39624 521.27283 212.22751 503.625 C 210.37331 533.03967 168.95457 543.3332 165.31917 503.625 C 141.86572 546.03143 92.96157 480.82916 126.32403 422.9812 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="138.869995" y="332" width="67" height="69" fill="url(#Obj_Gradient_15)"/><rect x="138.869995" y="332" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 152.039 364.5 L 162.20425 346.5 L 182.53476 346.5 L 192.70001 364.5 L 182.53476 382.5 L 162.20425 382.5 Z" fill="url(#Obj_Gradient_16)"/><path d="M 152.039 364.5 L 162.20425 346.5 L 182.53476 346.5 L 192.70001 364.5 L 182.53476 382.5 L 162.20425 382.5 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 62 365 L 83 365 C 85.484 365 87.5 370.376 87.5 377 C 87.5 383.624 85.484 389 83 389 L 62 389 C 59.516 389 57.5 383.624 57.5 377 C 57.5 370.376 59.516 365 62 365" fill="url(#Obj_Gradient_17)"/><path d="M 62 365 L 83 365 C 85.484 365 87.5 370.376 87.5 377 C 87.5 383.624 85.484 389 83 389 L 62 389 C 59.516 389 57.5 383.624 57.5 377 C 57.5 370.376 59.516 365 62 365" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="147.453" y="421.5" width="67" height="69" fill="url(#Obj_Gradient_18)"/><rect x="147.453" y="421.5" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 160.62199 456 L 170.78725 438 L 191.11775 438 L 201.283 456 L 191.11775 474 L 170.78725 474 Z" fill="url(#Obj_Gradient_19)"/><path d="M 160.62199 456 L 170.78725 438 L 191.11775 438 L 201.283 456 L 191.11775 474 L 170.78725 474 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 75.38166 389.48718 C 79.4207 406.98972 73.112816 431.39597 87.5 442 C 99.01256 450.4853 123.781944 450.13696 145.894165 451.55768" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="3"/><path d="M 118.20667 389.5221 L 154.46504 352.99951 C 154.46504 352.99951 148.56358 343.41068 154.80022 342.02164 C 161.03673 340.63266 163.73999 342.85837 163.73999 342.85837 L 156.62726 349.49799 L 163.3246 356.19534 L 170.08206 349.91098 C 170.08206 349.91098 172.57938 351.69775 172.78146 351.89984 C 172.98338 352.10178 171.3094 356.98645 169.817 358.88278 C 168.32417 360.77942 160.15027 358.03 160.15039 358.03012 C 160.15039 358.03012 122.44828 393.7637 122.342445 393.86954 C 122.23656 393.9754 121.27359 395.3426 119.157745 393.0148 C 117.04174 390.68701 118.20667 389.5221 118.20667 389.5221 Z" fill="url(#Obj_Gradient_20)"/><path d="M 118.20667 389.5221 L 154.46504 352.99951 C 154.46504 352.99951 148.56358 343.41068 154.80022 342.02164 C 161.03673 340.63266 163.73999 342.85837 163.73999 342.85837 L 156.62726 349.49799 L 163.3246 356.19534 L 170.08206 349.91098 C 170.08206 349.91098 172.57938 351.69775 172.78146 351.89984 C 172.98338 352.10178 171.3094 356.98645 169.817 358.88278 C 168.32417 360.77942 160.15027 358.03 160.15039 358.03012 C 160.15039 358.03012 122.44828 393.7637 122.342445 393.86954 C 122.23656 393.9754 121.27359 395.3426 119.157745 393.0148 C 117.04174 390.68701 118.20667 389.5221 118.20667 389.5221 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="0" y="0" width="278" height="273" fill="white"/><rect x="0" y="0" width="278" height="273" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 146.38904 149.98122 C 122.79251 136.5 132.20221 23.010353 169.84393 42.375 C 173.33626 4.6271057 217.1085 10.754013 216.82236 42.375 C 244.26889 1.931366 279.34393 82.575165 255.81749 123.01879 C 284.04803 142.62691 255.46124 248.27281 232.29251 230.625 C 230.43831 260.03967 189.01958 270.33319 185.38417 230.625 C 161.930725 273.03143 113.02657 207.82918 146.38904 149.98122 Z" fill="url(#Obj_Gradient_21)"/><path d="M 146.38904 149.98122 C 122.79251 136.5 132.20221 23.010353 169.84393 42.375 C 173.33626 4.6271057 217.1085 10.754013 216.82236 42.375 C 244.26889 1.931366 279.34393 82.575165 255.81749 123.01879 C 284.04803 142.62691 255.46124 248.27281 232.29251 230.625 C 230.43831 260.03967 189.01958 270.33319 185.38417 230.625 C 161.930725 273.03143 113.02657 207.82918 146.38904 149.98122 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="179.355" y="146" width="67" height="69" fill="url(#Obj_Gradient_22)"/><rect x="179.355" y="146" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 192.524 180.5 L 202.68925 162.5 L 223.01976 162.5 L 233.18501 180.5 L 223.01976 198.5 L 202.68925 198.5 Z" fill="url(#Obj_Gradient_23)"/><path d="M 192.524 180.5 L 202.68925 162.5 L 223.01976 162.5 L 233.18501 180.5 L 223.01976 198.5 L 202.68925 198.5 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 69.855003 111 L 90.855003 111 C 93.339005 111 95.355003 116.376 95.355003 123 C 95.355003 129.62399 93.339005 135 90.855003 135 L 69.855003 135 C 67.371002 135 65.355003 129.62399 65.355003 123 C 65.355003 116.376 67.371002 111 69.855003 111" fill="url(#Obj_Gradient_24)"/><path d="M 69.855003 111 L 90.855003 111 C 93.339005 111 95.355003 116.376 95.355003 123 C 95.355003 129.62399 93.339005 135 90.855003 135 L 69.855003 135 C 67.371002 135 65.355003 129.62399 65.355003 123 C 65.355003 116.376 67.371002 111 69.855003 111" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="161.855" y="60" width="67" height="69" fill="url(#Obj_Gradient_25)"/><rect x="161.855" y="60" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 175.024 96 L 185.18925 78 L 205.51976 78 L 215.68501 96 L 205.51976 114 L 185.18925 114 Z" fill="url(#Obj_Gradient_26)"/><path d="M 175.024 96 L 185.18925 78 L 205.51976 78 L 215.68501 96 L 205.51976 114 L 185.18925 114 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 74.355003 137 C 92.019905 160.33099 107.23704 198.83237 127.355 207 C 143.19005 213.42882 162.06882 201.06767 180.39609 192.10172" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="3"/><path d="M 222.38084 162.05576 C 227.03842 153.03807 238.59448 143.85422 236.355 135 C 234.66139 128.303986 225.07715 121.79347 216.5538 115.25858" marker-end="url(#Arrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><text transform="translate(5.8400302 4)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="bold" x=".16308594" y="11" textLength="6.673828">1</tspan></text><text transform="translate(8 286)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="bold" x=".16308594" y="11" textLength="6.673828">3</tspan></text><text transform="translate(286 286)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="bold" x=".16308594" y="11" textLength="6.673828">4</tspan></text></g></g></svg>
diff --git a/qpid/doc/book/src/java-broker/images/HA-2N-SplitBrain.png b/qpid/doc/book/src/java-broker/images/HA-2N-SplitBrain.png
new file mode 100644
index 0000000000..74393648a5
--- /dev/null
+++ b/qpid/doc/book/src/java-broker/images/HA-2N-SplitBrain.png
diff --git a/qpid/doc/book/src/java-broker/images/HA-2N-SplitBrain.svg b/qpid/doc/book/src/java-broker/images/HA-2N-SplitBrain.svg
new file mode 100644
index 0000000000..c82cc5065a
--- /dev/null
+++ b/qpid/doc/book/src/java-broker/images/HA-2N-SplitBrain.svg
@@ -0,0 +1,3 @@
+<?xml version="1.0"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xl="http://www.w3.org/1999/xlink" version="1.1" viewBox="-20 -16 598 591" width="598pt" height="591pt"><metadata xmlns:dc="http://purl.org/dc/elements/1.1/"><dc:date>2012-05-31 06:19Z</dc:date><!-- Produced by OmniGraffle Professional 5.3.6 --></metadata><defs><radialGradient cx="0" cy="0" r="1" id="Gradient" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset=".5" stop-color="#b5d0ea"/><stop offset="1" stop-color="#aaa"/></radialGradient><radialGradient id="Obj_Gradient" xl:href="#Gradient" gradientTransform="translate(200.35501 136.5) scale(145.18695)"/><linearGradient x1="0" x2="1" id="Gradient_2" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#2824aa"/></linearGradient><linearGradient id="Obj_Gradient_2" xl:href="#Gradient_2" gradientTransform="translate(212.855 146) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_3" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#0caa25"/></linearGradient><linearGradient id="Obj_Gradient_3" xl:href="#Gradient_3" gradientTransform="translate(212.85451 162.5) rotate(90) scale(36)"/><linearGradient x1="0" x2="1" id="Gradient_4" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#6923aa"/></linearGradient><linearGradient id="Obj_Gradient_4" xl:href="#Gradient_4" gradientTransform="translate(80.355003 111) rotate(90) scale(24)"/><linearGradient id="Obj_Gradient_5" xl:href="#Gradient_2" gradientTransform="translate(195.355 59.999996) rotate(90) scale(69.000007)"/><linearGradient x1="0" x2="1" id="Gradient_5" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#aa172d"/></linearGradient><linearGradient id="Obj_Gradient_6" xl:href="#Gradient_5" gradientTransform="translate(195.35451 78) rotate(90) scale(36)"/><marker orient="auto" overflow="visible" markerUnits="strokeWidth" id="FilledArrow_Marker" viewBox="-1 -3 6 6" markerWidth="6" markerHeight="6" color="black"><g><path d="M 3.7333333 0 L 0 -1.4 L 0 1.4 Z" fill="currentColor" stroke="currentColor" stroke-width="1"/></g></marker><marker orient="auto" overflow="visible" markerUnits="strokeWidth" id="Arrow_Marker" viewBox="-1 -4 10 8" markerWidth="10" markerHeight="8" color="black"><g><path d="M 8 0 L 0 -3 L 0 3 Z" fill="none" stroke="currentColor" stroke-width="1"/></g></marker><font-face font-family="Helvetica" font-size="12" units-per-em="1000" underline-position="-75.683594" underline-thickness="49.316406" slope="0" x-height="532.22656" cap-height="719.72656" ascent="770.01953" descent="-229.98047" font-weight="bold"><font-face-src><font-face-name name="Helvetica-Bold"/></font-face-src></font-face><radialGradient id="Obj_Gradient_7" xl:href="#Gradient" gradientTransform="translate(468.855 133.5) scale(145.18695)"/><linearGradient id="Obj_Gradient_8" xl:href="#Gradient_2" gradientTransform="translate(486.355 141.5) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_6" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#6ead6a"/></linearGradient><linearGradient id="Obj_Gradient_9" xl:href="#Gradient_6" gradientTransform="translate(486.3545 158) rotate(90) scale(36)"/><linearGradient id="Obj_Gradient_10" xl:href="#Gradient_4" gradientTransform="translate(348.565 85) rotate(90) scale(24)"/><radialGradient cx="0" cy="0" r="1" id="Gradient_7" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset=".5" stop-color="#ea061f"/><stop offset="1" stop-color="#aaa"/></radialGradient><radialGradient id="Obj_Gradient_11" xl:href="#Gradient_7" gradientTransform="translate(483.56808 132.68761) scale(20.071926)"/><linearGradient id="Obj_Gradient_12" xl:href="#Gradient_2" gradientTransform="translate(465.01501 51.445) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_8" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#b6b4b5"/></linearGradient><linearGradient id="Obj_Gradient_13" xl:href="#Gradient_8" gradientTransform="translate(465.0145 67.945) rotate(90) scale(36)"/><font-face font-family="Arial Unicode MS" font-size="36" panose-1="2 11 6 4 2 2 2 2 2 4" units-per-em="1000" underline-position="-100.097656" underline-thickness="49.804688" slope="0" x-height="529.78516" cap-height="728.02734" ascent="1068.84766" descent="-270.9961" font-weight="500"><font-face-src><font-face-name name="ArialUnicodeMS"/></font-face-src></font-face><radialGradient id="Obj_Gradient_14" xl:href="#Gradient" gradientTransform="translate(180.29001 414.5) scale(145.18695)"/><linearGradient id="Obj_Gradient_15" xl:href="#Gradient_2" gradientTransform="translate(172.37 337) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_9" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#66ab61"/></linearGradient><linearGradient id="Obj_Gradient_16" xl:href="#Gradient_9" gradientTransform="translate(172.36951 351.5) rotate(90) scale(36)"/><linearGradient id="Obj_Gradient_17" xl:href="#Gradient_4" gradientTransform="translate(53.18649 396) rotate(90) scale(24)"/><linearGradient id="Obj_Gradient_18" xl:href="#Gradient_2" gradientTransform="translate(180.953 426.5) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_10" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#21aa1a"/></linearGradient><linearGradient id="Obj_Gradient_19" xl:href="#Gradient_10" gradientTransform="translate(180.9525 443) rotate(90) scale(36)"/><radialGradient id="Obj_Gradient_20" xl:href="#Gradient_7" gradientTransform="translate(185.10121 416.23209) scale(20.071926)"/><font-face font-family="Arial Unicode MS" font-size="72" panose-1="2 11 6 4 2 2 2 2 2 4" units-per-em="1000" underline-position="-100.097656" underline-thickness="49.804688" slope="0" x-height="529.78516" cap-height="728.02734" ascent="1068.84766" descent="-270.9961" font-weight="500"><font-face-src><font-face-name name="ArialUnicodeMS"/></font-face-src></font-face></defs><g stroke="none" stroke-opacity="1" stroke-dasharray="none" fill="none" fill-opacity="1"><title>Canvas 6</title><g><title>Layer 1</title><rect x="0" y="0" width="278" height="273" fill="white"/><rect x="0" y="0" width="278" height="273" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 146.38904 149.98122 C 122.79251 136.5 132.20221 23.010353 169.84393 42.375 C 173.33626 4.6271057 217.1085 10.754013 216.82236 42.375 C 244.26889 1.931366 279.34393 82.575165 255.81749 123.01879 C 284.04803 142.62691 255.46124 248.27281 232.29251 230.625 C 230.43831 260.03967 189.01958 270.33319 185.38417 230.625 C 161.930725 273.03143 113.02657 207.82918 146.38904 149.98122 Z" fill="url(#Obj_Gradient)"/><path d="M 146.38904 149.98122 C 122.79251 136.5 132.20221 23.010353 169.84393 42.375 C 173.33626 4.6271057 217.1085 10.754013 216.82236 42.375 C 244.26889 1.931366 279.34393 82.575165 255.81749 123.01879 C 284.04803 142.62691 255.46124 248.27281 232.29251 230.625 C 230.43831 260.03967 189.01958 270.33319 185.38417 230.625 C 161.930725 273.03143 113.02657 207.82918 146.38904 149.98122 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="179.355" y="146" width="67" height="69" fill="url(#Obj_Gradient_2)"/><rect x="179.355" y="146" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 192.524 180.5 L 202.68925 162.5 L 223.01976 162.5 L 233.18501 180.5 L 223.01976 198.5 L 202.68925 198.5 Z" fill="url(#Obj_Gradient_3)"/><path d="M 192.524 180.5 L 202.68925 162.5 L 223.01976 162.5 L 233.18501 180.5 L 223.01976 198.5 L 202.68925 198.5 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 69.855003 111 L 90.855003 111 C 93.339005 111 95.355003 116.376 95.355003 123 C 95.355003 129.62399 93.339005 135 90.855003 135 L 69.855003 135 C 67.371002 135 65.355003 129.62399 65.355003 123 C 65.355003 116.376 67.371002 111 69.855003 111" fill="url(#Obj_Gradient_4)"/><path d="M 69.855003 111 L 90.855003 111 C 93.339005 111 95.355003 116.376 95.355003 123 C 95.355003 129.62399 93.339005 135 90.855003 135 L 69.855003 135 C 67.371002 135 65.355003 129.62399 65.355003 123 C 65.355003 116.376 67.371002 111 69.855003 111" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="161.855" y="60" width="67" height="69" fill="url(#Obj_Gradient_5)"/><rect x="161.855" y="60" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 175.024 96 L 185.18925 78 L 205.51976 78 L 215.68501 96 L 205.51976 114 L 185.18925 114 Z" fill="url(#Obj_Gradient_6)"/><path d="M 175.024 96 L 185.18925 78 L 205.51976 78 L 215.68501 96 L 205.51976 114 L 185.18925 114 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 83 135 C 87.728355 147.33211 78.882416 164.66258 97.186493 172 C 112.61268 178.18378 147.33044 177.27293 176.94928 178.23598" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="3"/><path d="M 222.38084 162.05576 C 227.03842 153.03807 238.59448 143.85422 236.355 135 C 234.66139 128.303986 225.07715 121.79347 216.5538 115.25858" marker-end="url(#Arrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><text transform="translate(285 6)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="bold" x=".16308594" y="11" textLength="6.673828">2</tspan></text><rect x="279.64499" y="0" width="278.00003" height="273" fill="white"/><rect x="279.64499" y="0" width="278.00003" height="273" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 414.88904 146.98122 C 391.2925 133.5 400.7022 20.010353 438.34393 39.375 C 441.83624 1.6271057 485.60852 7.754013 485.32236 39.375 C 512.76892 -1.06863403 547.84393 79.575165 524.3175 120.01879 C 552.54803 139.62691 523.96124 245.27281 500.7925 227.625 C 498.93832 257.03967 457.51956 267.33319 453.88416 227.625 C 430.43073 270.03143 381.52658 204.82918 414.88904 146.98122 Z" fill="url(#Obj_Gradient_7)"/><path d="M 414.88904 146.98122 C 391.2925 133.5 400.7022 20.010353 438.34393 39.375 C 441.83624 1.6271057 485.60852 7.754013 485.32236 39.375 C 512.76892 -1.06863403 547.84393 79.575165 524.3175 120.01879 C 552.54803 139.62691 523.96124 245.27281 500.7925 227.625 C 498.93832 257.03967 457.51956 267.33319 453.88416 227.625 C 430.43073 270.03143 381.52658 204.82918 414.88904 146.98122 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="452.855" y="141.5" width="66.99997" height="69" fill="url(#Obj_Gradient_8)"/><rect x="452.855" y="141.5" width="66.99997" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 466.024 176 L 476.18924 158 L 496.51974 158 L 506.685 176 L 496.51974 194 L 476.18924 194 Z" fill="url(#Obj_Gradient_9)"/><path d="M 466.024 176 L 476.18924 158 L 496.51974 158 L 506.685 176 L 496.51974 194 L 476.18924 194 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 338.065 85 L 359.065 85 C 361.54901 85 363.565 90.376 363.565 97 C 363.565 103.624 361.54901 109 359.065 109 L 338.065 109 C 335.581 109 333.565 103.624 333.565 97 C 333.565 90.376 335.581 85 338.065 85" fill="url(#Obj_Gradient_10)"/><path d="M 338.065 85 L 359.065 85 C 361.54901 85 363.565 90.376 363.565 97 C 363.565 103.624 361.54901 109 359.065 109 L 338.065 109 C 335.581 109 333.565 103.624 333.565 97 C 333.565 90.376 335.581 85 338.065 85" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 353.37662 109.46646 C 359.1055 124.30949 351.54587 143.50909 370.565 154 C 386.66068 162.87834 421.80042 165.52399 451.8061 169.84834" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="3"/><path d="M 470.9274 125.16071 L 480.5343 133.763016 L 472.91028 144.58757 L 475.98358 148.27911 L 484.31168 136.56943 L 493.17505 144.515656 L 496.20877 140.214615 L 486.60178 131.07477 L 494.69186 121.39724 L 490.1412 117.09611 L 483.06238 127.84885 L 474.9724 119.246674 Z" fill="url(#Obj_Gradient_11)"/><path d="M 470.9274 125.16071 L 480.5343 133.763016 L 472.91028 144.58757 L 475.98358 148.27911 L 484.31168 136.56943 L 493.17505 144.515656 L 496.20877 140.214615 L 486.60178 131.07477 L 494.69186 121.39724 L 490.1412 117.09611 L 483.06238 127.84885 L 474.9724 119.246674 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="431.51501" y="51.445" width="67" height="69" fill="url(#Obj_Gradient_12)"/><rect x="431.51501" y="51.445" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 444.684 85.945 L 454.84924 67.945 L 475.17975 67.945 L 485.345 85.945 L 475.17975 103.945 L 454.84924 103.945 Z" fill="url(#Obj_Gradient_13)"/><path d="M 444.684 85.945 L 454.84924 67.945 L 475.17975 67.945 L 485.345 85.945 L 475.17975 103.945 L 454.84924 103.945 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 485.18884 157.50099 C 484.6743 149.33481 485.7887 141.847626 483.645 133 C 482.1737 126.92762 479.1671 120.211914 476.20593 113.515625" marker-end="url(#Arrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><text transform="translate(496 56.5)" fill="#262626"><tspan font-family="Arial Unicode MS" font-size="36" font-weight="500" fill="#262626" x="0" y="38" textLength="36">♛</tspan></text><text transform="translate(6 6)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="bold" x=".16308594" y="11" textLength="6.673828">1</tspan></text><rect x="0" y="278" width="278" height="273" fill="white"/><rect x="0" y="278" width="278" height="273" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 126.32403 427.9812 C 102.72751 414.5 112.13721 301.01035 149.77893 320.375 C 153.27126 282.6271 197.0435 288.75403 196.75735 320.375 C 224.20389 279.93137 259.27893 360.57516 235.75249 401.0188 C 263.98303 420.62692 235.39624 526.27283 212.22751 508.625 C 210.37331 538.03967 168.95457 548.3332 165.31917 508.625 C 141.86572 551.03143 92.96157 485.82916 126.32403 427.9812 Z" fill="url(#Obj_Gradient_14)"/><path d="M 126.32403 427.9812 C 102.72751 414.5 112.13721 301.01035 149.77893 320.375 C 153.27126 282.6271 197.0435 288.75403 196.75735 320.375 C 224.20389 279.93137 259.27893 360.57516 235.75249 401.0188 C 263.98303 420.62692 235.39624 526.27283 212.22751 508.625 C 210.37331 538.03967 168.95457 548.3332 165.31917 508.625 C 141.86572 551.03143 92.96157 485.82916 126.32403 427.9812 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="138.869995" y="337" width="67" height="69" fill="url(#Obj_Gradient_15)"/><rect x="138.869995" y="337" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 152.039 369.5 L 162.20425 351.5 L 182.53476 351.5 L 192.70001 369.5 L 182.53476 387.5 L 162.20425 387.5 Z" fill="url(#Obj_Gradient_16)"/><path d="M 152.039 369.5 L 162.20425 351.5 L 182.53476 351.5 L 192.70001 369.5 L 182.53476 387.5 L 162.20425 387.5 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 42.68649 396 L 63.68649 396 C 66.170486 396 68.186493 401.376 68.186493 408 C 68.186493 414.624 66.170486 420 63.68649 420 L 42.68649 420 C 40.202488 420 38.18649 414.624 38.18649 408 C 38.18649 401.376 40.202488 396 42.68649 396" fill="url(#Obj_Gradient_17)"/><path d="M 42.68649 396 L 63.68649 396 C 66.170486 396 68.186493 401.376 68.186493 408 C 68.186493 414.624 66.170486 420 63.68649 420 L 42.68649 420 C 40.202488 420 38.18649 414.624 38.18649 408 C 38.18649 401.376 40.202488 396 42.68649 396" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="147.453" y="426.5" width="67" height="69" fill="url(#Obj_Gradient_18)"/><rect x="147.453" y="426.5" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 160.62199 461 L 170.78725 443 L 191.11775 443 L 201.283 461 L 191.11775 479 L 170.78725 479 Z" fill="url(#Obj_Gradient_19)"/><path d="M 160.62199 461 L 170.78725 443 L 191.11775 443 L 201.283 461 L 191.11775 479 L 170.78725 479 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 64.073936 420.37445 C 71.881844 429.24875 71.228394 440.71005 87.5 447 C 100.49186 452.02213 124.27972 453.7499 145.93179 456.12848" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="3"/><path d="M 61.40507 395.58307 C 67.269463 386.72293 63.968967 373.3653 79 369 C 90.933815 365.53418 114.428574 367.73508 135.71434 368.85406" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="3"/><text transform="translate(6 286)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="bold" x=".16308594" y="11" textLength="6.673828">3</tspan></text><path d="M 172.46051 408.7052 L 182.06741 417.3075 L 174.4434 428.13205 L 177.51671 431.82358 L 185.8448 420.1139 L 194.70818 428.06015 L 197.74191 423.7591 L 188.1349 414.61926 L 196.22501 404.9417 L 191.67433 400.6406 L 184.59552 411.39334 L 176.50554 402.79114 Z" fill="url(#Obj_Gradient_20)"/><path d="M 172.46051 408.7052 L 182.06741 417.3075 L 174.4434 428.13205 L 177.51671 431.82358 L 185.8448 420.1139 L 194.70818 428.06015 L 197.74191 423.7591 L 188.1349 414.61926 L 196.22501 404.9417 L 191.67433 400.6406 L 184.59552 411.39334 L 176.50554 402.79114 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 186.722 441.046 C 186.20738 432.87949 187.32178 425.39178 185.17799 416.544 C 183.70665 410.47153 180.69995 403.75586 177.73871 397.05966" marker-end="url(#Arrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><text transform="translate(19.372978 273)" fill="black"><tspan font-family="Arial Unicode MS" font-size="72" font-weight="500" x="0" y="77" textLength="54">☠</tspan></text></g></g></svg>
diff --git a/qpid/doc/book/src/java-broker/images/HA-BDBHAMessageStore-MBean-jconsole.png b/qpid/doc/book/src/java-broker/images/HA-BDBHAMessageStore-MBean-jconsole.png
new file mode 100644
index 0000000000..b2d52575bb
--- /dev/null
+++ b/qpid/doc/book/src/java-broker/images/HA-BDBHAMessageStore-MBean-jconsole.png
diff --git a/qpid/doc/book/src/java-broker/images/HA-perftests-results.png b/qpid/doc/book/src/java-broker/images/HA-perftests-results.png
new file mode 100644
index 0000000000..e8dbb3c123
--- /dev/null
+++ b/qpid/doc/book/src/java-broker/images/HA-perftests-results.png