diff options
| author | Justin Ross <jross@apache.org> | 2016-04-19 23:11:13 +0000 |
|---|---|---|
| committer | Justin Ross <jross@apache.org> | 2016-04-19 23:11:13 +0000 |
| commit | da7718ef463775acc7d6fbecf2d64c1bbfc39fd8 (patch) | |
| tree | 6da761b56ed0433b68f755927a180d615f7fb5b3 /qpid/cpp/docs/book/src/cpp-broker | |
| parent | eb1e7851a50c6a7901c73eb42d639516c0e3ba43 (diff) | |
| download | qpid-python-da7718ef463775acc7d6fbecf2d64c1bbfc39fd8.tar.gz | |
QPID-7207: Remove files and components that are obsolete or no longer in use; move doc and packaging pieces to the cpp subtree
git-svn-id: https://svn.apache.org/repos/asf/qpid/trunk@1740032 13f79535-47bb-0310-9956-ffa450edef68
Diffstat (limited to 'qpid/cpp/docs/book/src/cpp-broker')
19 files changed, 10456 insertions, 0 deletions
diff --git a/qpid/cpp/docs/book/src/cpp-broker/.gitignore b/qpid/cpp/docs/book/src/cpp-broker/.gitignore new file mode 100644 index 0000000000..6d31bdba7f --- /dev/null +++ b/qpid/cpp/docs/book/src/cpp-broker/.gitignore @@ -0,0 +1,20 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +/output diff --git a/qpid/cpp/docs/book/src/cpp-broker/AMQP-Compatibility.xml b/qpid/cpp/docs/book/src/cpp-broker/AMQP-Compatibility.xml new file mode 100644 index 0000000000..e5aa98cf96 --- /dev/null +++ b/qpid/cpp/docs/book/src/cpp-broker/AMQP-Compatibility.xml @@ -0,0 +1,713 @@ +<?xml version="1.0" encoding="utf-8"?> +<!-- + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +--> + +<section id="AMQP-Compatibility"> + <title> + AMQP compatibility + </title> + <para> + Qpid provides the most complete and compatible implementation + of AMQP. And is the most aggressive in implementing the latest + version of the specification. + </para> + <para> + There are two brokers: + </para> + + <itemizedlist> + <listitem><para>C++ with support for AMQP 0-10</para></listitem> + <listitem><para>Java with support for AMQP 0-8 and 0-9 (0-10 planned)</para></listitem> + </itemizedlist> + <para> + There are client libraries for C++, Java (JMS), .Net (written in + C#), python and ruby. + </para> + <itemizedlist> + <listitem><para>All clients support 0-10 and interoperate with the C++ + broker. + </para></listitem> + </itemizedlist> + <itemizedlist> + <listitem><para>The JMS client supports 0-8, 0-9 and 0-10 and interoperates + with both brokers. + </para></listitem> + </itemizedlist> + <itemizedlist> + <listitem><para>The python and ruby clients will also support all versions, + but the API is dynamically driven by the specification used and + so differs between versions. To work with the Java broker you + must use 0-8 or 0-9, to work with the C++ broker you must use + 0-10. + </para></listitem> + </itemizedlist> + <itemizedlist> + <listitem><para>There are two separate C# clients, one for 0-8 that + interoperates with the Java broker, one for 0-10 that + inteoperates with the C++ broker. + </para></listitem> + </itemizedlist> + <para> + QMF Management is supported in Ruby, Python, C++, and via QMan + for Java JMX & WS-DM. + </para> + <section role="h3" id="AMQPcompatibility-AMQPCompatibilityofQpidreleases-3A"> + <title> + AMQP + Compatibility of Qpid releases: + </title> + <para> + Qpid implements the AMQP Specification, and as the specification + has progressed Qpid is keeping up with the updates. This means + that different Qpid versions support different versions of AMQP. + Here is a simple guide on what use. + </para> + <para> + Here is a matrix that describes the different versions supported + by each release. The status symbols are interpreted as follows: + </para> + + <variablelist> + <varlistentry> + <term>Y</term> + <listitem><para>supported</para></listitem> + </varlistentry> + <varlistentry> + <term>N</term> + <listitem><para>unsupported</para></listitem> + </varlistentry> + <varlistentry> + <term>IP</term> + <listitem><para>in progress</para></listitem> + </varlistentry> + <varlistentry> + <term>P</term> + <listitem><para>planned</para></listitem> + </varlistentry> + </variablelist> + + <table> + <title>AMQP Version Support by Qpid Release</title> + <tgroup cols="6"> + <tbody> + <row> + <entry> + Component + </entry> + <entry> + Spec + </entry> + <entry> + + </entry> + <entry> + + </entry> + <entry> + + </entry> + <entry> + + </entry> + </row> + <row> + <entry> + + </entry> + <entry> + + </entry> + <entry> + M2.1 + </entry> + <entry> + M3 + </entry> + <entry> + M4 + </entry> + <entry> + 0.5 + </entry> + </row> + <row> + <entry> + java client + </entry> + <entry> + 0-10 + </entry> + <entry> + + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + </row> + <row> + <entry> + + </entry> + <entry> + 0-9 + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + </row> + <row> + <entry> + + </entry> + <entry> + 0-8 + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + </row> + <row> + <entry> + java broker + </entry> + <entry> + 0-10 + </entry> + <entry> + + </entry> + <entry> + + </entry> + <entry> + + </entry> + <entry> + P + </entry> + </row> + <row> + <entry> + + </entry> + <entry> + 0-9 + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + </row> + <row> + <entry> + + </entry> + <entry> + 0-8 + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + </row> + <row> + <entry> + c++ client/broker + </entry> + <entry> + 0-10 + </entry> + <entry> + + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + </row> + <row> + <entry> + + </entry> + <entry> + 0-9 + </entry> + <entry> + Y + </entry> + <entry> + + </entry> + <entry> + + </entry> + <entry> + + </entry> + </row> + <row> + <entry> + python client + </entry> + <entry> + 0-10 + </entry> + <entry> + + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + </row> + <row> + <entry> + + </entry> + <entry> + 0-9 + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + </row> + <row> + <entry> + + </entry> + <entry> + 0-8 + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + </row> + <row> + <entry> + ruby client + </entry> + <entry> + 0-10 + </entry> + <entry> + + </entry> + <entry> + + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + </row> + <row> + <entry> + + </entry> + <entry> + 0-8 + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + </row> + <row> + <entry> + C# client + </entry> + <entry> + 0-10 + </entry> + <entry> + + </entry> + <entry> + + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + </row> + <row> + <entry> + + </entry> + <entry> + 0-8 + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + </row> + </tbody> + </tgroup> + </table> + <!--h3--> + </section> + + <section role="h3" id="AMQPcompatibility-InteroptablebyAMQPspecificationversion"> + <title> + Interop + table by AMQP specification version + </title> + <para> + Above table represented in another format. + </para> + <table> + <title>AMQP Version Support - alternate format</title> + <tgroup cols="5"> + <tbody> + <row> + <entry> + + </entry> + <entry> + release + </entry> + <entry> + 0-8 + </entry> + <entry> + 0-9 + </entry> + <entry> + 0-10 + </entry> + </row> + <row> + <entry> + java client + </entry> + <entry> + M3 M4 0.5 + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + </row> + <row> + <entry> + java client + </entry> + <entry> + M2.1 + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + <entry> + N + </entry> + </row> + <row> + <entry> + java broker + </entry> + <entry> + M3 M4 0.5 + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + <entry> + N + </entry> + </row> + <row> + <entry> + java broker + </entry> + <entry> + trunk + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + <entry> + P + </entry> + </row> + <row> + <entry> + java broker + </entry> + <entry> + M2.1 + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + <entry> + N + </entry> + </row> + <row> + <entry> + c++ client/broker + </entry> + <entry> + M3 M4 0.5 + </entry> + <entry> + N + </entry> + <entry> + N + </entry> + <entry> + Y + </entry> + </row> + <row> + <entry> + c++ client/broker + </entry> + <entry> + M2.1 + </entry> + <entry> + N + </entry> + <entry> + Y + </entry> + <entry> + N + </entry> + </row> + <row> + <entry> + python client + </entry> + <entry> + M3 M4 0.5 + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + </row> + <row> + <entry> + python client + </entry> + <entry> + M2.1 + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + <entry> + N + </entry> + </row> + <row> + <entry> + ruby client + </entry> + <entry> + M3 M4 0.5 + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + <entry> + N + </entry> + </row> + <row> + <entry> + ruby client + </entry> + <entry> + trunk + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + <entry> + P + </entry> + </row> + <row> + <entry> + C# client + </entry> + <entry> + M3 M4 0.5 + </entry> + <entry> + Y + </entry> + <entry> + N + </entry> + <entry> + N + </entry> + </row> + <row> + <entry> + C# client + </entry> + <entry> + trunk + </entry> + <entry> + Y + </entry> + <entry> + N + </entry> + <entry> + Y + </entry> + </row> + </tbody> + </tgroup> + </table> + <!--h3--> + </section> + +</section> diff --git a/qpid/cpp/docs/book/src/cpp-broker/AMQP-Messaging-Broker-CPP-Book.xml b/qpid/cpp/docs/book/src/cpp-broker/AMQP-Messaging-Broker-CPP-Book.xml new file mode 100644 index 0000000000..6122b12e18 --- /dev/null +++ b/qpid/cpp/docs/book/src/cpp-broker/AMQP-Messaging-Broker-CPP-Book.xml @@ -0,0 +1,74 @@ +<?xml version="1.0"?> +<!-- + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +--> + +<book xmlns:xi="http://www.w3.org/2001/XInclude"> + <title>AMQP Messaging Broker (Implemented in C++)</title> + <preface> + <title>Introduction</title> + + <para>Qpid provides two AMQP messaging brokers:</para> + + <itemizedlist> + <listitem><para>Implemented in C++ - high performance, low latency, and RDMA support.</para></listitem> + <listitem><para>Implemented in Java - Fully JMS compliant, runs on any Java platform.</para></listitem> + </itemizedlist> + + <para>Both AMQP messaging brokers support clients in multiple + languages, as long as the messaging client and the messaging + broker use the same version of AMQP. See <link + linkend="AMQP-Compatibility">AMQP Compatibility</link> to see + which messaging clients work with each broker.</para> + + <para>This manual contains information specific to the broker that is implemented in C++.</para> + </preface> + +<chapter> + <title> + Running the AMQP Messaging Broker + </title> + + <xi:include href="Running-CPP-Broker.xml"/> + <xi:include href="Cheat-Sheet-for-configuring-Queue-Options.xml"/> + <xi:include href="Cheat-Sheet-for-configuring-Exchange-Options.xml"/> + <xi:include href="Using-Broker-Federation.xml"/> + <xi:include href="Security.xml"/> + <xi:include href="LVQ.xml"/> + <xi:include href="queue-state-replication.xml"/> + <xi:include href="producer-flow-control.xml"/> + <xi:include href="AMQP-Compatibility.xml"/> + <xi:include href="Qpid-Interoperability-Documentation.xml"/> + <xi:include href="Using-message-groups.xml"/> + <xi:include href="Active-Passive-Cluster.xml"/> + <xi:include href="HA-Queue-Replication.xml"/> +</chapter> + + +<chapter id="chapter-Managing-CPP-Broker"> + <title> + Managing the AMQP Messaging Broker + </title> + + <xi:include href="Managing-CPP-Broker.xml"/> + <xi:include href="Qpid-Management-Framework.xml"/> + <xi:include href="QMF-Python-Console-Tutorial.xml"/> +</chapter> +</book> diff --git a/qpid/cpp/docs/book/src/cpp-broker/Active-Passive-Cluster.xml b/qpid/cpp/docs/book/src/cpp-broker/Active-Passive-Cluster.xml new file mode 100644 index 0000000000..461b75d320 --- /dev/null +++ b/qpid/cpp/docs/book/src/cpp-broker/Active-Passive-Cluster.xml @@ -0,0 +1,1229 @@ +<?xml version="1.0" encoding="utf-8"?> +<!-- + +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +h"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. + +--> + +<section id="chapter-ha"> + <title>Active-Passive Messaging Clusters</title> + + <section id="ha-overview"> + <title>Overview</title> + <para> + + The High Availability (HA) module provides + <firstterm>active-passive</firstterm>, <firstterm>hot-standby</firstterm> + messaging clusters to provide fault tolerant message delivery. + </para> + <para> + In an active-passive cluster only one broker, known as the + <firstterm>primary</firstterm>, is active and serving clients at a time. The other + brokers are standing by as <firstterm>backups</firstterm>. Changes on the primary + are replicated to all the backups so they are always up-to-date or "hot". Backup + brokers reject client connection attempts, to enforce the requirement that clients + only connect to the primary. + </para> + <para> + If the primary fails, one of the backups is promoted to take over as the new + primary. Clients fail-over to the new primary automatically. If there are multiple + backups, the other backups also fail-over to become backups of the new primary. + </para> + <para> + This approach relies on an external <firstterm>cluster resource manager</firstterm> + to detect failures, choose the new primary and handle network partitions. <ulink + url="https://fedorahosted.org/cluster/wiki/RGManager">rgmanager</ulink> is supported + initially, but others may be supported in the future. + </para> + <section id="ha-at-least-once"> + <title>Avoiding message loss</title> + <para> + In order to avoid message loss, the primary broker <emphasis>delays + acknowledgement</emphasis> of messages received from clients until the + message has been replicated and acknowledged by all of the back-up + brokers, or has been consumed from the primary queue. + </para> + <para> + This ensures that all acknowledged messages are safe: they have either + been consumed or backed up to all backup brokers. Messages that are + consumed <emphasis>before</emphasis> they are replicated do not need to + be replicated. This reduces the work load when replicating a queue with + active consumers. + </para> + <para> + Clients keep <emphasis>unacknowledged</emphasis> messages in a buffer + <footnote> + <para> + You can control the maximum number of messages in the buffer by setting the + client's <literal>capacity</literal>. For details of how to set the capacity + in client code see "Using the Qpid Messaging API" in + <citetitle>Programming in Apache Qpid</citetitle>. + </para> + </footnote> + until they are acknowledged by the primary. If the primary fails, clients will + fail-over to the new primary and <emphasis>re-send</emphasis> all their + unacknowledged messages. + <footnote> + <para> + Clients must use "at-least-once" reliability to enable re-send of unacknowledged + messages. This is the default behaviour, no options need be set to enable it. For + details of client addressing options see "Using the Qpid Messaging API" + in <citetitle>Programming in Apache Qpid</citetitle>. + </para> + </footnote> + </para> + <para> + If the primary crashes, all the <emphasis>acknowledged</emphasis> + messages will be available on the backup that takes over as the new + primary. The <emphasis>unacknowledged</emphasis> messages will be + re-sent by the clients. Thus no messages are lost. + </para> + <para> + Note that this means it is possible for messages to be + <emphasis>duplicated</emphasis>. In the event of a failure it is possible for a + message to received by the backup that becomes the new primary + <emphasis>and</emphasis> re-sent by the client. The application must take steps + to identify and eliminate duplicates. + </para> + <para> + When a new primary is promoted after a fail-over it is initially in + "recovering" mode. In this mode, it delays acknowledgement of messages + on behalf of all the backups that were connected to the previous + primary. This protects those messages against a failure of the new + primary until the backups have a chance to connect and catch up. + </para> + <para> + Not all messages need to be replicated to the back-up brokers. If a + message is consumed and acknowledged by a regular client before it has + been replicated to a backup, then it doesn't need to be replicated. + </para> + <variablelist id="ha-broker-states"> + <title>HA Broker States</title> + <varlistentry> + <term>Stand-alone</term> + <listitem> + <para> + Broker is not part of a HA cluster. + </para> + </listitem> + </varlistentry> + <varlistentry> + <term>Joining</term> + <listitem> + <para> + Newly started broker, not yet connected to any existing primary. + </para> + </listitem> + </varlistentry> + <varlistentry> + <term>Catch-up</term> + <listitem> + <para> + A backup broker that is connected to the primary and downloading + existing state (queues, messages etc.) + </para> + </listitem> + </varlistentry> + <varlistentry> + <term>Ready</term> + <listitem> + <para> + A backup broker that is fully caught-up and ready to take over as + primary. + </para> + </listitem> + </varlistentry> + <varlistentry> + <term>Recovering</term> + <listitem> + <para> + Newly-promoted primary, waiting for backups to connect and catch up. + Clients can connect but they are stalled until the primary is active. + </para> + </listitem> + </varlistentry> + <varlistentry> + <term>Active</term> + <listitem> + <para> + The active primary broker with all backups connected and caught-up. + </para> + </listitem> + </varlistentry> + </variablelist> + </section> + <section id="limitations"> + <title>Limitations</title> + <para> + There are a some known limitations in the current implementation. These + will be fixed in future versions. + </para> + <itemizedlist> + <listitem> + <para> + Transactional changes to queue state are not replicated atomically. If + the primary crashes during a transaction, it is possible that the + backup could contain only part of the changes introduced by a + transaction. + </para> + </listitem> + <listitem> + <para> + Configuration changes (creating or deleting queues, exchanges and + bindings) are replicated asynchronously. Management tools used to + make changes will consider the change complete when it is complete + on the primary, it may not yet be replicated to all the backups. + </para> + </listitem> + <listitem> + <para> + Federation links <emphasis>to</emphasis> the primary will fail over + correctly. Federated links <emphasis>from</emphasis> the primary + will be lost in fail over, they will not be re-connected to the new + primary. It is possible to work around this by replacing the + <literal>qpidd-primary</literal> start up script with a script that + re-creates federation links when the primary is promoted. + </para> + </listitem> + </itemizedlist> + </section> + </section> + + <section id="ha-virtual-ip"> + <title>Virtual IP Addresses</title> + <para> + Some resource managers (including <command>rgmanager</command>) support + <firstterm>virtual IP addresses</firstterm>. A virtual IP address is an IP + address that can be relocated to any of the nodes in a cluster. The + resource manager associates this address with the primary node in the + cluster, and relocates it to the new primary when there is a failure. This + simplifies configuration as you can publish a single IP address rather + than a list. + </para> + <para> + A virtual IP address can be used by clients to connect to the primary. The + following sections will explain how to configure virtual IP addresses for + clients or brokers. + </para> + </section> + + <section id="ha-broker-config"> + <title>Configuring the Brokers</title> + <para> + The broker must load the <filename>ha</filename> module, it is loaded by + default. The following broker options are available for the HA module. + </para> + <note> + <para> + Broker management is required for HA to operate, it is enabled by + default. The option <literal>mgmt-enable</literal> must not be set to + "no" + </para> + </note> + <note> + <para> + Incorrect security settings are a common cause of problems when + getting started, see <xref linkend="ha-security"/>. + </para> + </note> + <table frame="all" id="ha-broker-options"> + <title>Broker Options for High Availability Messaging Cluster</title> + <tgroup align="left" cols="2" colsep="1" rowsep="1"> + <colspec colname="c1"/> + <colspec colname="c2"/> + <thead> + <row> + <entry align="center" nameend="c2" namest="c1"> + Options for High Availability Messaging Cluster + </entry> + </row> + </thead> + <tbody> + <row> + <entry> + <literal>ha-cluster <replaceable>yes|no</replaceable></literal> + </entry> + <entry> + Set to "yes" to have the broker join a cluster. + </entry> + </row> + <row> + <entry> + <literal>ha-queue-replication <replaceable>yes|no</replaceable></literal> + </entry> + <entry> + Enable replication of specific queues without joining a cluster, see <xref linkend="ha-queue-replication"/>. + </entry> + </row> + <row> + <entry> + <literal>ha-brokers-url <replaceable>URL</replaceable></literal> + </entry> + <entry> + <para> + The URL + <footnote id="ha-url-grammar"> + <para> + The full format of the URL is given by this grammar: + <programlisting> +url = ["amqp:"][ user ["/" password] "@" ] addr ("," addr)* +addr = tcp_addr / rmda_addr / ssl_addr / ... +tcp_addr = ["tcp:"] host [":" port] +rdma_addr = "rdma:" host [":" port] +ssl_addr = "ssl:" host [":" port]' + </programlisting> + </para> + </footnote> + used by cluster brokers to connect to each other. The URL should + contain a comma separated list of the broker addresses, rather than a + virtual IP address. + </para> + </entry> + </row> + <row> + <entry><literal>ha-public-url <replaceable>URL</replaceable></literal> </entry> + <entry> + <para> + This option is only needed for backwards compatibility if you + have been using the <literal>amq.failover</literal> exchange. + This exchange is now obsolete, it is recommended to use a + virtual IP address instead. + </para> + <para> + If set, this URL is advertised by the + <literal>amq.failover</literal> exchange and overrides the + broker option <literal>known-hosts-url</literal> + </para> + </entry> + </row> + <row> + <entry><literal>ha-replicate </literal><replaceable>VALUE</replaceable></entry> + <entry> + <para> + Specifies whether queues and exchanges are replicated by default. + <replaceable>VALUE</replaceable> is one of: <literal>none</literal>, + <literal>configuration</literal>, <literal>all</literal>. + For details see <xref linkend="ha-replicate-values"/>. + </para> + </entry> + </row> + <row> + <entry> + <para><literal>ha-username <replaceable>USER</replaceable></literal></para> + <para><literal>ha-password <replaceable>PASS</replaceable></literal></para> + <para><literal>ha-mechanism <replaceable>MECHANISM</replaceable></literal></para> + </entry> + <entry> + Authentication settings used by HA brokers to connect to each other, + see <xref linkend="ha-security"/> + </entry> + </row> + <row> + <entry><literal>ha-backup-timeout<replaceable>SECONDS</replaceable></literal> + <footnote id="ha-seconds-spec"> + <para> + Values specified as <replaceable>SECONDS</replaceable> can be a + fraction of a second, e.g. "0.1" for a tenth of a second. + They can also have an explicit unit, + e.g. 10s (seconds), 10ms (milliseconds), 10us (microseconds), 10ns (nanoseconds) + </para> + </footnote> + </entry> + <entry> + <para> + Maximum time that a recovering primary will wait for an expected + backup to connect and become ready. + </para> + </entry> + </row> + <row> + <entry> + <literal>link-maintenance-interval <replaceable>SECONDS</replaceable></literal> + <footnoteref linkend="ha-seconds-spec"/> + </entry> + <entry> + <para> + HA uses federation links to connect from backup to primary. + Backup brokers check the link to the primary on this interval + and re-connect if need be. Default 2 seconds. Set lower for + faster failover, e.g. 0.1 seconds. Setting too low will result + in excessive link-checking on the backups. + </para> + </entry> + </row> + <row> + <entry> + <literal>link-heartbeat-interval <replaceable>SECONDS</replaceable></literal> + <footnoteref linkend="ha-seconds-spec"/> + </entry> + <entry> + <para> + HA uses federation links to connect from backup to primary. + If no heart-beat is received for twice this interval the primary will consider that + backup dead (e.g. if backup is hung or partitioned.) + This interval is also used to time-out for broker status checks, + it may take up to this interval for rgmanager to detect a hung or partitioned broker. + Clients sending messages may be held up during this time. + Default 120 seconds: you will probably want to set this to a lower value e.g. 10. + If set too low rgmanager may consider a slow broker to have failed and kill it. + </para> + </entry> + </row> + </tbody> + </tgroup> + </table> + <para> + To configure a HA cluster you must set at least <literal>ha-cluster</literal> and + <literal>ha-brokers-url</literal>. + </para> + </section> + + <section id="ha-rm"> + <title>The Cluster Resource Manager</title> + <para> + Broker fail-over is managed by a <firstterm>cluster resource + manager</firstterm>. An integration with <ulink + url="https://fedorahosted.org/cluster/wiki/RGManager">rgmanager</ulink> is + provided, but it is possible to integrate with other resource managers. + </para> + <para> + The resource manager is responsible for starting the <command>qpidd</command> broker + on each node in the cluster. The resource manager then <firstterm>promotes</firstterm> + one of the brokers to be the primary. The other brokers connect to the primary as + backups, using the URL provided in the <literal>ha-brokers-url</literal> configuration + option. + </para> + <para> + Once connected, the backup brokers synchronize their state with the + primary. When a backup is synchronized, or "hot", it is ready to take + over if the primary fails. Backup brokers continually receive updates + from the primary in order to stay synchronized. + </para> + <para> + If the primary fails, backup brokers go into fail-over mode. The resource + manager must detect the failure and promote one of the backups to be the + new primary. The other backups connect to the new primary and synchronize + their state with it. + </para> + <para> + The resource manager is also responsible for protecting the cluster from + <firstterm>split-brain</firstterm> conditions resulting from a network partition. A + network partition divide a cluster into two sub-groups which cannot see each other. + Usually a <firstterm>quorum</firstterm> voting algorithm is used that disables nodes + in the inquorate sub-group. + </para> + </section> + + <section id="ha-rm-config"> + <title>Configuring with <command>rgmanager</command> as resource manager</title> + <para> + This section assumes that you are already familiar with setting up and configuring + clustered services using <command>cman</command> and + <command>rgmanager</command>. It will show you how to configure an active-passive, + hot-standby <command>qpidd</command> HA cluster with <command>rgmanager</command>. + </para> + <note> + <para> + Once all components are installed it is important to take the following step: + <programlisting> +chkconfig rgmanager on +chkconfig cman on +chkconfig qpidd <emphasis>off</emphasis> + </programlisting> + </para> + <para> + The qpidd service must be <emphasis>off</emphasis> in + <literal>chkconfig</literal> because <literal>rgmanager</literal> will + start and stop <literal>qpidd</literal>. If the normal system init + process also attempts to start and stop qpidd it can cause rgmanager to + lose track of qpidd processes. The symptom when this happens is that + <literal>clustat</literal> shows a <literal>qpidd</literal> service to + be stopped when in fact there is a <literal>qpidd</literal> process + running. The <literal>qpidd</literal> log will show errors like this: + <programlisting> +critical Unexpected error: Daemon startup failed: Cannot lock /var/lib/qpidd/lock: Resource temporarily unavailable + </programlisting> + </para> + </note> + <para> + You must provide a <literal>cluster.conf</literal> file to configure + <command>cman</command> and <command>rgmanager</command>. Here is + an example <literal>cluster.conf</literal> file for a cluster of 3 nodes named + node1, node2 and node3. We will go through the configuration step-by-step. + </para> + <programlisting> + <![CDATA[ +<?xml version="1.0"?> +<!-- +This is an example of a cluster.conf file to run qpidd HA under rgmanager. +This example assumes a 3 node cluster, with nodes named node1, node2 and node3. + +NOTE: fencing is not shown, you must configure fencing appropriately for your cluster. +--> + +<cluster name="qpid-test" config_version="18"> + <!-- The cluster has 3 nodes. Each has a unique nodeid and one vote + for quorum. --> + <clusternodes> + <clusternode name="node1.example.com" nodeid="1"/> + <clusternode name="node2.example.com" nodeid="2"/> + <clusternode name="node3.example.com" nodeid="3"/> + </clusternodes> + + <!-- Resouce Manager configuration. --> + + status_poll_interval is the interval in seconds that the resource manager checks the status + of managed services. This affects how quickly the manager will detect failed services. + --> + <rm status_poll_interval="1"> + <!-- + There is a failoverdomain for each node containing just that node. + This lets us stipulate that the qpidd service should always run on each node. + --> + <failoverdomains> + <failoverdomain name="node1-domain" restricted="1"> + <failoverdomainnode name="node1.example.com"/> + </failoverdomain> + <failoverdomain name="node2-domain" restricted="1"> + <failoverdomainnode name="node2.example.com"/> + </failoverdomain> + <failoverdomain name="node3-domain" restricted="1"> + <failoverdomainnode name="node3.example.com"/> + </failoverdomain> + </failoverdomains> + + <resources> + <!-- This script starts a qpidd broker acting as a backup. --> + <script file="/etc/init.d/qpidd" name="qpidd"/> + + <!-- This script promotes the qpidd broker on this node to primary. --> + <script file="/etc/init.d/qpidd-primary" name="qpidd-primary"/> + + <!-- + This is a virtual IP address for client traffic. + monitor_link="yes" means monitor the health of the NIC used for the VIP. + sleeptime="0" means don't delay when failing over the VIP to a new address. + --> + <ip address="20.0.20.200" monitor_link="yes" sleeptime="0"/> + </resources> + + <!-- There is a qpidd service on each node, it should be restarted if it fails. --> + <service name="node1-qpidd-service" domain="node1-domain" recovery="restart"> + <script ref="qpidd"/> + </service> + <service name="node2-qpidd-service" domain="node2-domain" recovery="restart"> + <script ref="qpidd"/> + </service> + <service name="node3-qpidd-service" domain="node3-domain" recovery="restart"> + <script ref="qpidd"/> + </service> + + <!-- There should always be a single qpidd-primary service, it can run on any node. --> + <service name="qpidd-primary-service" autostart="1" exclusive="0" recovery="relocate"> + <script ref="qpidd-primary"/> + <!-- The primary has the IP addresses for brokers and clients to connect. --> + <ip ref="20.0.20.200"/> + </service> + </rm> +</cluster> + ]]> + </programlisting> + + <para> + There is a <literal>failoverdomain</literal> for each node containing just that + one node. This lets us stipulate that the qpidd service should always run on all + nodes. + </para> + <para> + The <literal>resources</literal> section defines the <command>qpidd</command> + script used to start the <command>qpidd</command> service. It also defines the + <command>qpid-primary</command> script which does not + actually start a new service, rather it promotes the existing + <command>qpidd</command> broker to primary status. + </para> + <para> + The <literal>resources</literal> section also defines a virtual IP + address for clients: <literal>20.0.20.200</literal>. + </para> + <para> + <filename>qpidd.conf</filename> should contain these lines: + </para> + <programlisting> +ha-cluster=yes +ha-brokers-url=20.0.20.1,20.0.20.2,20.0.20.3 + </programlisting> + <para> + The brokers connect to each other directly via the addresses + listed in <command>ha-brokers-url</command>. Note the client and broker + addresses are on separate sub-nets, this is recommended but not required. + </para> + <para> + The <literal>service</literal> section defines 3 <literal>qpidd</literal> + services, one for each node. Each service is in a restricted fail-over + domain containing just that node, and has the <literal>restart</literal> + recovery policy. The effect of this is that rgmanager will run + <command>qpidd</command> on each node, restarting if it fails. + </para> + <para> + There is a single <literal>qpidd-primary-service</literal> using the + <command>qpidd-primary</command> script which is not restricted to a + domain and has the <literal>relocate</literal> recovery policy. This means + rgmanager will start <command>qpidd-primary</command> on one of the nodes + when the cluster starts and will relocate it to another node if the + original node fails. Running the <literal>qpidd-primary</literal> script + does not start a new broker process, it promotes the existing broker to + become the primary. + </para> + + <section id="ha-rm-shutdown-node"> + <title>Shutting down qpidd on a HA node</title> + <para> + As explained above both the per-node <literal>qpidd</literal> service + and the re-locatable <literal>qpidd-primary</literal> service are + implemented by the same <literal>qpidd</literal> daemon. + </para> + <para> + As a result, stopping the <literal>qpidd</literal> service will not stop + a <literal>qpidd</literal> daemon that is acting as primary, and + stopping the <literal>qpidd-primary</literal> service will not stop a + <literal>qpidd</literal> process that is acting as backup. + </para> + <para> + To shut down a node that is acting as primary you need to shut down the + <literal>qpidd</literal> service <emphasis>and</emphasis> relocate the + primary: + </para> + <para> + <programlisting> +clusvcadm -d somenode-qpidd-service +clusvcadm -r qpidd-primary-service + </programlisting> + </para> + <para> + This will shut down the <literal>qpidd</literal> daemon on that node and + prevent the primary service service from relocating back to the node + because the qpidd service is no longer running there. + </para> + </section> + </section> + + <section id="ha-broker-admin"> + <title>Broker Administration Tools</title> + <para> + Normally, clients are not allowed to connect to a backup broker. However + management tools are allowed to connect to a backup brokers. If you use + these tools you <emphasis>must not</emphasis> add or remove messages from + replicated queues, nor create or delete replicated queues or exchanges as + this will disrupt the replication process and may cause message loss. + </para> + <para> + <command>qpid-ha</command> allows you to view and change HA configuration settings. + </para> + <para> + The tools <command>qpid-config</command>, <command>qpid-route</command> and + <command>qpid-stat</command> will connect to a backup if you pass the flag <command>ha-admin</command> on the + command line. + </para> + </section> + + <section id="ha-replicate-values"> + <title>Controlling replication of queues and exchanges</title> + <para> + By default, queues and exchanges are not replicated automatically. You can change + the default behaviour by setting the <literal>ha-replicate</literal> configuration + option. It has one of the following values: + <itemizedlist> + <listitem> + <para> + <firstterm>all</firstterm>: Replicate everything automatically: queues, + exchanges, bindings and messages. + </para> + </listitem> + <listitem> + <para> + <firstterm>configuration</firstterm>: Replicate the existence of queues, + exchange and bindings but don't replicate messages. + </para> + </listitem> + <listitem> + <para> + <firstterm>none</firstterm>: Don't replicate anything, this is the default. + </para> + </listitem> + </itemizedlist> + </para> + <para> + You can over-ride the default for a particular queue or exchange by passing the + argument <literal>qpid.replicate</literal> when creating the queue or exchange. It + takes the same values as <literal>ha-replicate</literal> + </para> + <para> + Bindings are automatically replicated if the queue and exchange being bound both + have replication <literal>all</literal> or <literal>configuration</literal>, they + are not replicated otherwise. + </para> + <para> + You can create replicated queues and exchanges with the + <command>qpid-config</command> management tool like this: + </para> + <programlisting> +qpid-config add queue myqueue --replicate all + </programlisting> + <para> + To create replicated queues and exchanges via the client API, add a + <literal>node</literal> entry to the address like this: + </para> + <programlisting> +"myqueue;{create:always,node:{x-declare:{arguments:{'qpid.replicate':all}}}}" + </programlisting> + <para> + There are some built-in exchanges created automatically by the broker, these + exchanges are never replicated. The built-in exchanges are the default (nameless) + exchange, the AMQP standard exchanges (<literal>amq.direct, amq.topic, amq.fanout</literal> and + <literal>amq.match</literal>) and the management exchanges (<literal>qpid.management, qmf.default.direct</literal> and + <literal>qmf.default.topic</literal>) + </para> + <para> + Note that if you bind a replicated queue to one of these exchanges, the + binding will <emphasis>not</emphasis> be replicated, so the queue will not + have the binding after a fail-over. + </para> + </section> + + <section id="ha-failover"> + <title>Client Connection and Fail-over</title> + <para> + Clients can only connect to the primary broker. Backup brokers reject any + connection attempt by a client. Clients rejected by a backup broker will + automatically fail-over until they connect to the primary. + </para> + <para> + Clients are configured with the URL for the cluster (details below for + each type of client). There are two possibilities + <itemizedlist> + <listitem> + <para> + The URL contains multiple addresses, one for each broker in the cluster. + </para> + </listitem> + <listitem> + <para> + The URL contains a single <firstterm>virtual IP address</firstterm> + that is assigned to the primary broker by the resource manager. + This is the recommended configuration. + </para> + </listitem> + </itemizedlist> + In the first case, clients will repeatedly re-try each address in the URL + until they successfully connect to the primary. In the second case the + resource manager will assign the virtual IP address to the primary broker, + so clients only need to re-try on a single address. + </para> + <para> + When the primary broker fails, clients re-try all known cluster addresses + until they connect to the new primary. The client re-sends any messages + that were previously sent but not acknowledged by the broker at the time + of the failure. Similarly messages that have been sent by the broker, but + not acknowledged by the client, are re-queued. + </para> + <para> + TCP can be slow to detect connection failures. A client can configure a + connection to use a <firstterm>heartbeat</firstterm> to detect connection + failure, and can specify a time interval for the heartbeat. If heartbeats + are in use, failures will be detected no later than twice the heartbeat + interval. The following sections explain how to enable heartbeat in each + client. + </para> + <para> + Note: the following sections explain how to configure clients with + multiple dresses, but if you are using a virtual IP address you only need + to configure that one address for clients, you don't need to list all the + addresses. + </para> + <para> + Suppose your cluster has 3 nodes: <literal>node1</literal>, + <literal>node2</literal> and <literal>node3</literal> all using the + default AMQP port, and you are not using a virtual IP address. To connect + a client you need to specify the address(es) and set the + <literal>reconnect</literal> property to <literal>true</literal>. The + following sub-sections show how to connect each type of client. + </para> + <section id="ha-clients"> + <title>C++ clients</title> + <para> + With the C++ client, you specify multiple cluster addresses in a single URL + <footnote> + <para> + The full grammar for the URL is: + </para> + <programlisting> +url = ["amqp:"][ user ["/" password] "@" ] addr ("," addr)* +addr = tcp_addr / rmda_addr / ssl_addr / ... +tcp_addr = ["tcp:"] host [":" port] +rdma_addr = "rdma:" host [":" port] +ssl_addr = "ssl:" host [":" port]' + </programlisting> + </footnote> + You also need to specify the connection option + <literal>reconnect</literal> to be true. For example: + </para> + <programlisting> +qpid::messaging::Connection c("node1,node2,node3","{reconnect:true}"); + </programlisting> + <para> + Heartbeats are disabled by default. You can enable them by specifying a + heartbeat interval (in seconds) for the connection via the + <literal>heartbeat</literal> option. For example: + </para> + <programlisting> +qpid::messaging::Connection c("node1,node2,node3","{reconnect:true,heartbeat:10}"); + </programlisting> + </section> + <section id="ha-python-client"> + <title>Python clients</title> + <para> + With the python client, you specify <literal>reconnect=True</literal> + and a list of <replaceable>host:port</replaceable> addresses as + <literal>reconnect_urls</literal> when calling + <literal>Connection.establish</literal> or + <literal>Connection.open</literal> + </para> + <programlisting> +connection = qpid.messaging.Connection.establish("node1", reconnect=True, reconnect_urls=["node1", "node2", "node3"]) + </programlisting> + <para> + Heartbeats are disabled by default. You can + enable them by specifying a heartbeat interval (in seconds) for the + connection via the 'heartbeat' option. For example: + </para> + <programlisting> +connection = qpid.messaging.Connection.establish("node1", reconnect=True, reconnect_urls=["node1", "node2", "node3"], heartbeat=10) + </programlisting> + </section> + <section id="ha-jms-client"> + <title>Java JMS Clients</title> + <para> + In Java JMS clients, client fail-over is handled automatically if it is + enabled in the connection. You can configure a connection to use + fail-over using the <command>failover</command> property: + </para> + + <screen> + connectionfactory.qpidConnectionfactory = amqp://guest:guest@clientid/test?brokerlist='tcp://localhost:5672'&failover='failover_exchange' + </screen> + <para> + This property can take three values: + </para> + <variablelist> + <title>Fail-over Modes</title> + <varlistentry> + <term>failover_exchange</term> + <listitem> + <para> + If the connection fails, fail over to any other broker in the cluster. + </para> + + </listitem> + + </varlistentry> + <varlistentry> + <term>roundrobin</term> + <listitem> + <para> + If the connection fails, fail over to one of the brokers specified in the <command>brokerlist</command>. + </para> + + </listitem> + + </varlistentry> + <varlistentry> + <term>singlebroker</term> + <listitem> + <para> + Fail-over is not supported; the connection is to a single broker only. + </para> + + </listitem> + + </varlistentry> + + </variablelist> + <para> + In a Connection URL, heartbeat is set using the <command>heartbeat</command> property, which is an integer corresponding to the heartbeat period in seconds. For instance, the following line from a JNDI properties file sets the heartbeat time out to 3 seconds: + </para> + + <screen> + connectionfactory.qpidConnectionfactory = amqp://guest:guest@clientid/test?brokerlist='tcp://localhost:5672'&heartbeat='3' + </screen> + </section> + </section> + + <section id="ha-security"> + <title>Security and Access Control.</title> + <para> + This section outlines the HA specific aspects of security configuration. + Please see <xref linkend="chap-Messaging_User_Guide-Security"/> for + more details on enabling authentication and setting up Access Control Lists. + </para> + <note> + <para> + Unless you disable authentication with <literal>auth=no</literal> in + your configuration, you <emphasis>must</emphasis> set the options below + and you <emphasis>must</emphasis> have an ACL file with at least the + entry described below. + </para> + <para> + Backups will be <emphasis>unable to connect to the primary</emphasis> if + the security configuration is incorrect. See also <xref + linkend="ha-troubleshoot-security"/> + </para> + </note> + <para> + When authentication is enabled you must set the credentials used by HA + brokers with following options: + </para> + <table frame="all" id="ha-security-options"> + <title>HA Security Options</title> + <tgroup align="left" cols="2" colsep="1" rowsep="1"> + <colspec colname="c1"/> + <colspec colname="c2"/> + <thead> + <row> + <entry align="center" nameend="c2" namest="c1"> + HA Security Options + </entry> + </row> + </thead> + <tbody> + <row> + <entry><para><literal>ha-username</literal> <replaceable>USER</replaceable></para></entry> + <entry><para>User name for HA brokers. Note this must <emphasis>not</emphasis> include the <literal>@QPID</literal> suffix.</para></entry> + </row> + <row> + <entry><para><literal>ha-password</literal> <replaceable>PASS</replaceable></para></entry> + <entry><para>Password for HA brokers.</para></entry> + </row> + <row> + <entry><para><literal>ha-mechanism</literal> <replaceable>MECHANISM</replaceable></para></entry> + <entry> + <para> + Mechanism for HA brokers. Any mechanism you enable for + broker-to-broker communication can also be used by a client, so + do not use ha-mechanism=ANONYMOUS in a secure environment. + </para> + </entry> + </row> + </tbody> + </tgroup> + </table> + <para> + This identity is used to authorize federation links from backup to + primary. It is also used to authorize actions on the backup to replicate + primary state, for example creating queues and exchanges. + </para> + <para> + When authorization is enabled you must have an Access Control List with the + following rule to allow HA replication to function. Suppose + <literal>ha-username</literal>=<replaceable>USER</replaceable> + </para> + <programlisting> +acl allow <replaceable>USER</replaceable>@QPID all all + </programlisting> + </section> + + <section id="ha-other-rm"> + <title>Integrating with other Cluster Resource Managers</title> + <para> + To integrate with a different resource manager you must configure it to: + <itemizedlist> + <listitem><para>Start a qpidd process on each node of the cluster.</para></listitem> + <listitem><para>Restart qpidd if it crashes.</para></listitem> + <listitem><para>Promote exactly one of the brokers to primary.</para></listitem> + <listitem><para>Detect a failure and promote a new primary.</para></listitem> + </itemizedlist> + </para> + <para> + The <command>qpid-ha</command> command allows you to check if a broker is + primary, and to promote a backup to primary. + </para> + <para> + To test if a broker is the primary: + </para> + <programlisting>qpid-ha -b <replaceable>broker-address</replaceable> status --expect=primary</programlisting> + <para> + This will return 0 if the broker at <replaceable>broker-address</replaceable> is the primary, + non-0 otherwise. + </para> + <para> + To promote a broker to primary: + <programlisting>qpid-ha --cluster-manager -b <replaceable>broker-address</replaceable> promote</programlisting> + </para> + <para> + Note that <literal>promote</literal> is considered a "cluster manager + only" command. Incorrect use of <literal>promote</literal> outside of the + cluster manager could create a cluster with multiple primaries. Such a + cluster will malfunction and lose data. "Cluster manager only" commands + are not accessible in <command>qpid-ha</command> without the + <literal>--cluster-manager</literal> option. + </para> + <para> + To list the full set of commands use: + </para> + <programlisting> +qpid-ha --cluster-manager --help + </programlisting> + </section> + + <section id ="ha-store"> + <title>Using a message store in a cluster</title> + <para> + If you use a persistent store for your messages then each broker in a + cluster will have its own store. If the entire cluster fails and is + restarted, the *first* broker that becomes primary will recover from its + store. All the other brokers will clear their stores and get an update + from the primary to ensure consistency. + </para> + </section> + + <section id="ha-troubleshoot"> + <title>Troubleshooting a cluster</title> + <para> + This section applies to clusters that are using rgmanager as the + cluster manager. + </para> + <section id="ha-troubleshoot-no-primary"> + <title>No primary broker</title> + <para> + When you initially start a HA cluster, all brokers are in + <literal>joining</literal> mode. The brokers do not automatically select + a primary, they rely on the cluster manager <literal>rgmanager</literal> + to do so. If <literal>rgmanager</literal> is not running or is not + configured correctly, brokers will remain in the + <literal>joining</literal> state. See <xref linkend="ha-rm-config"/> + </para> + </section> + <section id="ha-troubleshoot-security"> + <title>Authentication and ACL failures</title> + <para> + If a broker is unable to establish a connection to another broker in the + cluster due to authentication or ACL problems the logs may contain + errors like the following: + <programlisting> +info SASL: Authentication failed: SASL(-13): user not found: Password verification failed + </programlisting> + <programlisting> +warning Client closed connection with 320: User anonymous@QPID federation connection denied. Systems with authentication enabled must specify ACL create link rules. + </programlisting> + <programlisting> +warning Client closed connection with 320: ACL denied anonymous@QPID creating a federation link. + </programlisting> + </para> + <para> + Set the HA security configuration and ACL file as described in <xref + linkend="ha-security"/>. Once the cluster is running and the primary is + promoted , run: + <programlisting>qpid-ha status --all</programlisting> + to make sure that the brokers are running as one cluster. + </para> + </section> + <section id="ha-troubleshoot-slow-recovery"> + <title>Slow recovery times</title> + <para> + The following configuration settings affect recovery time. The + values shown are examples that give fast recovery on a lightly + loaded system. You should run tests to determine if the values are + appropriate for your system and load conditions. + </para> + <section id="ha-troubleshoot-cluster.conf"> + <title>cluster.conf:</title> + <programlisting> +<rm status_poll_interval=1> + </programlisting> + <para> + status_poll_interval is the interval in seconds that the + resource manager checks the status of managed services. This + affects how quickly the manager will detect failed services. + </para> + <programlisting> +<ip address="20.0.20.200" monitor_link="yes" sleeptime="0"/> + </programlisting> + <para> + This is a virtual IP address for client traffic. + monitor_link="yes" means monitor the health of the network interface + used for the VIP. sleeptime="0" means don't delay when + failing over the VIP to a new address. + </para> + </section> + <section id="ha-troubleshoot-qpidd.conf"> + <title>qpidd.conf</title> + <programlisting> +link-maintenance-interval=0.1 + </programlisting> + <para> + Interval for backup brokers to check the link to the primary + re-connect if need be. Default 2 seconds. Can be set lower for + faster fail-over. Setting too low will result in excessive + link-checking activity on the broker. + </para> + <programlisting> +link-heartbeat-interval=5 + </programlisting> + <para> + Heartbeat interval for federation links. The HA cluster uses + federation links between the primary and each backup. The + primary can take up to twice the heartbeat interval to detect a + failed backup. When a sender sends a message the primary waits + for all backups to acknowledge before acknowledging to the + sender. A disconnected backup may cause the primary to block + senders until it is detected via heartbeat. + </para> + <para> + This interval is also used as the timeout for broker status + checks by rgmanager. It may take up to this interval for + rgmanager to detect a hung broker. + </para> + <para> + The default of 120 seconds is very high, you will probably want + to set this to a lower value. If set too low, under network + congestion or heavy load, a slow-to-respond broker may be + re-started by rgmanager. + </para> + </section> + </section> + <section id="ha-troubleshoot-total-cluster-failure"> + <title>Total cluster failure</title> + <para> + Note: for definition of broker states <firstterm>joining</firstterm>, + <firstterm>catch-up</firstterm>, <firstterm>ready</firstterm>, + <firstterm>recovering</firstterm> and <firstterm>active</firstterm> see + <xref linkend="ha-broker-states"/> + </para> + <para> + The cluster can only guarantee availability as long as there is at + least one active primary broker or ready backup broker left alive. + If all the brokers fail simultaneously, the cluster will fail and + non-persistent data will be lost. + </para> + <para> + While there is an active primary broker, clients can get service. + If the active primary fails, one of the "ready" backup + brokers will take over, recover and become active. Note a backup + can only be promoted to primary if it is in the "ready" + state (with the exception of the first primary in a new cluster + where all brokers are in the "joining" state) + </para> + <para> + Given a stable cluster of N brokers with one active primary and + N-1 ready backups, the system can sustain up to N-1 failures in + rapid succession. The surviving broker will be promoted to active + and continue to give service. + </para> + <para> + However at this point the system <emphasis>cannot</emphasis> + sustain a failure of the surviving broker until at least one of + the other brokers recovers, catches up and becomes a ready backup. + If the surviving broker fails before that the cluster will fail in + one of two modes (depending on the exact timing of failures) + </para> + <section id="ha-troubleshoot-the-cluster-hangs"> + <title>1. The cluster hangs</title> + <para> + All brokers are in joining or catch-up mode. rgmanager tries to + promote a new primary but cannot find any candidates and so + gives up. clustat will show that the qpidd services are running + but the the qpidd-primary service has stopped, something like + this: + </para> + <programlisting> +Service Name Owner (Last) State +------- ---- ----- ------ ----- +service:mrg33-qpidd-service 20.0.10.33 started +service:mrg34-qpidd-service 20.0.10.34 started +service:mrg35-qpidd-service 20.0.10.35 started +service:qpidd-primary-service (20.0.10.33) stopped + </programlisting> + <para> + Eventually all brokers become stuck in "joining" mode, + as shown by: <literal>qpid-ha status --all</literal> + </para> + <para> + At this point you need to restart the cluster in one of the + following ways: + <orderedlist> + <listitem><para> + Restart the entire cluster: + In <literal>luci:<replaceable>your-cluster</replaceable>:Nodes</literal> + click reboot to restart the entire cluster + </para></listitem> + <listitem><para> + Stop and restart the cluster with + <literal>ccs --stopall; ccs --startall</literal> + </para></listitem> + <listitem><para> + Restart just the Qpid services:In <literal>luci:<replaceable>your-cluster</replaceable>:Service Groups</literal> + <orderedlist> + <listitem><para>Select all the qpidd (not qpidd-primary) services, click restart</para></listitem> + <listitem><para>Select the qpidd-primary service, click restart</para></listitem> + </orderedlist> + </para></listitem> + <listitem><para> + Stop the <literal>qpidd-primary</literal> and + <literal>qpidd</literal> services with <literal>clusvcadm</literal>, + then restart (qpidd-primary last) + </para></listitem> + </orderedlist> + </para> + </section> + <section id="ha-troubleshoot-the-cluster-reboots"> + <title>2. The cluster reboots</title> + <para> + A new primary is promoted and the cluster is functional but all + non-persistent data from before the failure is lost. + </para> + </section> + </section> + <section id="ha-troubleshoot-fencing-and-network-partitions"> + <title>Fencing and network partitions</title> + <para> + A network partition is a a network failure that divides the + cluster into two or more sub-clusters, where each broker can + communicate with brokers in its own sub-cluster but not with + brokers in other sub-clusters. This condition is also referred to + as a "split brain". + </para> + <para> + Nodes in one sub-cluster can't tell whether nodes in other + sub-clusters are dead or are still running but disconnected. We + cannot allow each sub-cluster to independently declare its own + qpidd primary and start serving clients, as the cluster will + become inconsistent. We must ensure only one sub-cluster continues + to provide service. + </para> + <para> + A <emphasis>quorum</emphasis> determines which sub-cluster + continues to operate, and <emphasis>power fencing</emphasis> + ensures that nodes in non-quorate sub-clusters cannot attempt to + provide service inconsistently. For more information see: + </para> + <para> + https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html-single/High_Availability_Add-On_Overview/index.html, + chapter 2. Quorum and 4. Fencing. + </para> + </section> + </section> +</section> diff --git a/qpid/cpp/docs/book/src/cpp-broker/Cheat-Sheet-for-configuring-Exchange-Options.xml b/qpid/cpp/docs/book/src/cpp-broker/Cheat-Sheet-for-configuring-Exchange-Options.xml new file mode 100644 index 0000000000..fccdae1b9a --- /dev/null +++ b/qpid/cpp/docs/book/src/cpp-broker/Cheat-Sheet-for-configuring-Exchange-Options.xml @@ -0,0 +1,144 @@ +<?xml version="1.0" encoding="utf-8"?> + +<!-- + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +--> + +<section> + <title> + Cheat Sheet for configuring Exchange Options + </title> + + <section role="h2" id="CheatSheetforconfiguringExchangeOptions-ConfiguringExchangeOptions"> + <title> + Configuring Exchange Options + </title> + <para> + The C++ Broker M4 or later supports the following additional + Exchange options in addition to the standard AMQP define options + </para><itemizedlist> + <listitem><para>Exchange Level Message sequencing + </para></listitem> + <listitem><para>Initial Value Exchange + </para></listitem> + </itemizedlist><para> + Note that these features can be used on any exchange type, that + has been declared with the options set. + </para><para> + It also supports an additional option to the bind operation on a + direct exchange + </para><itemizedlist> + <listitem><para>Exclusive binding for key + </para></listitem> + </itemizedlist> + + <section role="h3" id="CheatSheetforconfiguringExchangeOptions-ExchangeLevelMessagesequencing"><title> + Exchange Level Message sequencing + </title> + <para> + This feature can be used to place a sequence number into each + message's headers, based on the order they pass through an + exchange. The sequencing starts at 0 and then wraps in an AMQP + int64 type. + </para><para> + The field name used is "qpid.msg_sequence" + </para><para> + To use this feature an exchange needs to be declared specifying + this option in the declare + </para> + <programlisting> +.... + FieldTable args; + args.setInt("qpid.msg_sequence",1); + +... + // now declare the exchange + session.exchangeDeclare(arg::exchange="direct", arg::arguments=args); +</programlisting> + <para> + Then each message passing through that exchange will be numbers + in the application headers. + </para> + <programlisting> + unit64_t seqNo; + //after message transfer + seqNo = message.getHeaders().getAsInt64("qpid.msg_sequence"); +</programlisting> + <!--h3--></section> + <section role="h3" id="CheatSheetforconfiguringExchangeOptions-InitialValueExchange"><title> + Initial + Value Exchange + </title> + <para> + This feature caches a last message sent to an exchange. When a + new binding is created onto the exchange it will then attempt to + route this cached messaged to the queue, based on the binding. + This allows for topics or the creation of configurations where a + new consumer can receive the last message sent to the broker, + with matching routing. + </para><para> + To use this feature an exchange needs to be declared specifying + this option in the declare + </para> + <programlisting> +.... + FieldTable args; + args.setInt("qpid.ive",1); + +... + // now declare the exchange + session.exchangeDeclare(arg::exchange="direct", arg::arguments=args); +</programlisting> + <para> + now use the exchange in the same way you would use any other + exchange. + </para> + <!--h3--></section> + + <section role="h3" id="CheatSheetforconfiguringExchangeOptions-Exclusivebindingforkey"><title> + Exclusive + binding for key + </title> + <para> + Direct exchanges in qpidd support a qpid.exclusive-binding option + on the bind operation that causes the binding specified to be the + only one for the given key. I.e. if there is already a binding at + this exchange with this key it will be atomically updated to bind + the new queue. This means that the binding can be changed + concurrently with an incoming stream of messages and each message + will be routed to exactly one queue. + </para> + <programlisting> +.... + FieldTable args; + args.setInt("qpid.exclusive-binding",1); + + //the following will cause the only binding from amq.direct with 'my-key' + //to be the one to 'my-queue'; if there were any previous bindings for that + //key they will be removed. This is atomic w.r.t message routing through the + //exchange. + session.exchangeBind(arg::exchange="amq.direct", arg::queue="my-queue", + arg::bindingKey="my-key", arg::arguments=args); + +... +</programlisting> +<!--h3--></section> +<!--h2--></section> +</section> diff --git a/qpid/cpp/docs/book/src/cpp-broker/Cheat-Sheet-for-configuring-Queue-Options.xml b/qpid/cpp/docs/book/src/cpp-broker/Cheat-Sheet-for-configuring-Queue-Options.xml new file mode 100644 index 0000000000..125372e463 --- /dev/null +++ b/qpid/cpp/docs/book/src/cpp-broker/Cheat-Sheet-for-configuring-Queue-Options.xml @@ -0,0 +1,198 @@ +<?xml version="1.0" encoding="utf-8"?> + +<!-- + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +--> + +<section><title> + Cheat Sheet for configuring Queue Options + </title> + + <section role="h2" id="CheatSheetforconfiguringQueueOptions-ConfiguringQueueOptions"><title> + Configuring + Queue Options + </title> + + <para> + The C++ Broker M4 or later supports the following additional + Queue constraints. + </para> + <itemizedlist> + <listitem><para> + <xref linkend="CheatSheetforconfiguringQueueOptions-ConfiguringQueueOptions"/> + </para></listitem> + <listitem><para> + <itemizedlist> + <listitem><para> + <xref linkend="CheatSheetforconfiguringQueueOptions-ApplyingQueueSizingConstraints"/> + </para></listitem> + <listitem><para> + <xref linkend="CheatSheetforconfiguringQueueOptions-ChangingtheQueueorderingBehaviors-28FIFO-2FLVQ-29"/> + </para></listitem> + <listitem><para> + <xref linkend="CheatSheetforconfiguringQueueOptions-Settingadditionalbehaviors"/> + </para></listitem> + <listitem><para> + <itemizedlist> + <listitem><para> + <xref linkend="CheatSheetforconfiguringQueueOptions-Queueeventgeneration"/> + </para></listitem> + </itemizedlist> + </para></listitem> + <listitem><para> + <xref linkend="CheatSheetforconfiguringQueueOptions-OtherClients"/> + </para></listitem> + </itemizedlist> + </para></listitem> + </itemizedlist> + + <para> + The 0.10 C++ Broker supports the following additional Queue configuration options: + </para> + <itemizedlist> + <listitem><para> + <xref linkend="producer-flow-control"/> + </para></listitem> + </itemizedlist> + + <section role="h3" id="CheatSheetforconfiguringQueueOptions-ApplyingQueueSizingConstraints"><title> + Applying Queue Sizing Constraints + </title> + + <para> + This allows to specify how to size a queue and what to do when + the sizing constraints have been reached. The queue size can be + limited by the number messages (message depth) or byte depth on + the queue. + </para><para> + Once the Queue meets/ exceeds these constraints the follow + policies can be applied + </para><itemizedlist> + <listitem><para>REJECT - Reject the published message + </para></listitem> + <listitem><para>FLOW_TO_DISK - Flow the messages to disk, to preserve memory + </para></listitem> + <listitem><para>RING - start overwriting messages in a ring based on sizing. + If head meets tail, advance head + </para></listitem> + <listitem><para>RING_STRICT - start overwriting messages in a ring based on + sizing. If head meets tail, AND the consumer has the tail message + acquired it will reject + </para></listitem> + </itemizedlist><para> + Examples: + </para><para> + Create a queue an auto delete queue that will support 100 000 + bytes, and then REJECT + </para> + <programlisting> +#include "qpid/client/QueueOptions.h" + + QueueOptions qo; + qo.setSizePolicy(REJECT,100000,0); + + session.queueDeclare(arg::queue=queue, arg::autoDelete=true, arg::arguments=qo); +</programlisting> + <para> + Create a queue that will support 1000 messages into a RING buffer + </para> + <programlisting> +#include "qpid/client/QueueOptions.h" + + QueueOptions qo; + qo.setSizePolicy(RING,0,1000); + + session.queueDeclare(arg::queue=queue, arg::arguments=qo); +</programlisting> + <!--h3--></section> + <section role="h3" id="CheatSheetforconfiguringQueueOptions-ChangingtheQueueorderingBehaviors-28FIFO-2FLVQ-29"><title> + Changing the Queue ordering Behaviors (FIFO/LVQ) + </title> + <para> + The default ordering in a queue in Qpid is FIFO. However + additional ordering semantics can be used namely LVQ (Last Value + Queue). Last Value Queue is define as follows. + </para><para> + If I publish symbols RHT, IBM, JAVA, MSFT, and then publish RHT + before the consumer is able to consume RHT, that message will be + over written in the queue and the consumer will receive the last + published value for RHT. + </para><para> + Example: + </para> + <programlisting> +#include "qpid/client/QueueOptions.h" + + QueueOptions qo; + qo.setOrdering(LVQ); + + session.queueDeclare(arg::queue=queue, arg::arguments=qo); + + ..... + string key; + qo.getLVQKey(key); + + .... + for each message, set the into application headers before transfer + message.getHeaders().setString(key,"RHT"); + +</programlisting> + <para> + Notes: + </para><itemizedlist> + <listitem><para>Messages that are dequeued and the re-queued will have the + following exceptions. a.) if a new message has been queued with + the same key, the re-queue from the consumer, will combine these + two messages. b.) If an update happens for a message of the same + key, after the re-queue, it will not update the re-queued + message. This is done to protect a client from being able to + adversely manipulate the queue. + </para></listitem> + <listitem><para>Acquire: When a message is acquired from the queue, no matter + it's position, it will behave the same as a dequeue + </para></listitem> + <listitem><para>LVQ does not support durable messages. If the queue or + messages are declared durable on an LVQ, the durability will be + ignored. + </para></listitem> + </itemizedlist><para> + A fully worked <xref linkend="LVQ-Example"/> can be found here + </para> + <!--h3--></section> + <section role="h3" id="CheatSheetforconfiguringQueueOptions-Settingadditionalbehaviors"><title> + Setting additional behaviors + </title> + <!--h3--></section> + <section role="h3" id="CheatSheetforconfiguringQueueOptions-OtherClients"><title> + Other + Clients + </title> + <para> + Note that these options can be set from any client. QueueOptions + just correctly formats the arguments passed to the QueueDeclare() + method. + </para> + + <!--h3--></section> + + <!--h2--></section> + + +</section> diff --git a/qpid/cpp/docs/book/src/cpp-broker/HA-Queue-Replication.xml b/qpid/cpp/docs/book/src/cpp-broker/HA-Queue-Replication.xml new file mode 100644 index 0000000000..81b55a3914 --- /dev/null +++ b/qpid/cpp/docs/book/src/cpp-broker/HA-Queue-Replication.xml @@ -0,0 +1,126 @@ +<?xml version="1.0" encoding="utf-8"?> +<!-- + +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. + +--> + +<section id="ha-queue-replication"> + <title>Replicating Queues with the HA module</title> + <para> + As well as support for an active-passive cluster, the + HA module allows you to replicate individual queues, + even if the brokers are not in a cluster. The <firstterm>original</firstterm> + queue is used as normal. The <firstterm>replica</firstterm> queue is + updated automatically as messages are added to or removed from the original + queue. + </para> + <warning> + <para> + It is not safe to modify the replica queue + other than via the automatic updates from the original. Adding or removing + messages on the replica queue will make replication inconsistent and may + cause message loss. + The HA module does <emphasis>not</emphasis> enforce + restricted access to the replica queue (as it does in the case of a cluster) + so it is up to the application to ensure the replica is not used until it has + been disconnected from the original. + </para> + </warning> + <section> + <title>Replicating queues</title> + <para> + To create a replica queue, the HA module must be + loaded on both the original and replica brokers (it is loaded by default.) + You also need to set the configuration option: + <programlisting> + ha-queue-replication=yes + </programlisting> + to enable this feature on a stand-alone broker. It is automatically + enabled for brokers that are part of a cluster. + </para> + <para> + Suppose that <command>myqueue</command> is a queue on + <command>node1</command> and we want to create a replica of + <command>myqueue</command> on <command>node2</command> (where both brokers + are using the default AMQP port.) This is accomplished by the command: + <programlisting> + qpid-config --broker=node2 add queue --start-replica node1 myqueue + </programlisting> + If <command>myqueue</command> already exists on the replica + broker you can start replication from the original queue like this: + <programlisting> + qpid-ha replicate -b node2 node1 myqueue + </programlisting> + </para> + </section> + <section> + <title>Replicating queues between clusters</title> + <para> + You can replicate queues between two standalone brokers, between a + standalone broker and a cluster, or between two clusters (see <xref + linkend="chapter-ha"/>.) For failover in a cluster there are two cases to + consider. + </para> + <orderedlist> + <listitem> + <para> + When the <emphasis>original</emphasis> queue is on the active node + of a cluster, failover is automatic. If the active node + fails, the replication link will automatically reconnect and the + replica will continue to be updated from the new primary. + </para> + </listitem> + <listitem> + <para> + When the <emphasis>replica</emphasis> queue is on the active node of a + cluster, there is no automatic failover. However you can use the + following workaround. + </para> + </listitem> + </orderedlist> + <section> + <title>Work around for fail-over of replica queue in a cluster</title> + <para> + When a primary broker fails the cluster resource manager calls a script + to promote a backup broker to be the new primary. By default this script + is <filename>/etc/init.d/qpidd-primary</filename> but you can modify + that in your <filename>cluster.conf</filename> file (see <xref + linkend="ha-rm-config"/>.) + </para> + <para> + You can modify this script (on each host in your cluster) by adding + commands to create your replica queues just before the broker is + promoted, as indicated in the following exceprt from the script: + <programlisting> +start() { + service qpidd start + echo -n $"Promoting qpid daemon to cluster primary: " + ################################ + #### Add your commands here #### + ################################ + $QPID_HA -b localhost:$QPID_PORT promote + [ "$?" -eq 0 ] && success || failure +} + </programlisting> + Your commands will be run, and your replicas created, whenever + the system fails over to a new primary. + </para> + </section> + </section> +</section> diff --git a/qpid/cpp/docs/book/src/cpp-broker/LVQ.xml b/qpid/cpp/docs/book/src/cpp-broker/LVQ.xml new file mode 100644 index 0000000000..b57c6268be --- /dev/null +++ b/qpid/cpp/docs/book/src/cpp-broker/LVQ.xml @@ -0,0 +1,181 @@ +<?xml version="1.0" encoding="utf-8"?> +<!-- + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +--> + +<section> + <title>LVQ - Last Value Queue</title> + + <section role="h2" id="LVQ-UnderstandingLVQ"> + <title>Understanding LVQ</title> + <para> + A Last Value Queue is configured with the name of a message header that + is used as a key. The queue behaves as a normal FIFO queue with the + exception that when a message is enqueued, any other message in the + queue with the same value in the key header is removed and discarded. + Thus, for any given key value, the queue holds only the most recent + message. + </para> + <para> + The following example illustrates the operation of a Last Value Queue. + The example shows an empty queue with no consumers and a sequence of + produced messages. The numbers represent the key for each message. + </para> + <programlisting> + <empty queue> + 1 => + 1 + 2 => + 1 2 + 3 => + 1 2 3 + 4 => + 1 2 3 4 + 2 => + 1 3 4 2 + 1 => + 3 4 2 1 + </programlisting> + <para> + Note that the first four messages are enqueued normally in FIFO order. + The fifth message has key '2' and is also enqueued on the tail of the + queue. However the message already in the queue with the same key is + discarded. + <note> + <para> + If the set of keys used in the messages in a LVQ is constrained, the + number of messages in the queue shall not exceed the number of + distinct keys in use. + </para> + </note> + </para> + <section role="h3" id="LVQ-UnderstandingLVQ-UseCases"> + <title>Common Use-Cases</title> + <itemizedlist> + <listitem> + <para> + LVQ with zero or one consuming subscriptions - In this case, if + the consumer drops momentarily or is slower than the producer(s), + it will only receive current information relative to the message + keys. + </para> + </listitem> + <listitem> + <para> + LVQ with zero or more browsing subscriptions - A browsing consumer + can subscribe to the LVQ and get an immediate dump of all of the + "current" messages and track updates thereafter. Any number of + independent browsers can subscribe to the same LVQ with the same + effect. Since messages are never consumed, they only disappear + when replaced with a newer message with the same key or when their + TTL expires. + </para> + </listitem> + </itemizedlist> + </section> + </section> + + <section role="h2" id="LVQ-Creating"> + <title>Creating a Last Value Queue</title> + <section role="h3" id="LVQ-Creating-Address"> + <title>Using Addressing Syntax</title> + <para> + A LVQ may be created using directives in the API's address syntax. + The important argument is "qpid.last_value_queue_key". The following + Python example shows how a producer of stock price updates can create + a LVQ to hold the latest stock prices for each ticker symbol. The + message header used to hold the ticker symbol is called "ticker". + </para> + <programlisting> + conn = Connection(url) + conn.open() + sess = conn.session() + tx = sess.sender("prices;{create:always, node:{type:queue, x-declare:{arguments:{'qpid.last_value_queue_key':'ticker'}}}}") + </programlisting> + </section> + <section role="h3" id="LVQ-Creating-Tool"> + <title>Using qpid-config</title> + <para> + The same LVQ as shown in the previous example can be created using the + qpid-config utility: + </para> + <programlisting> + $ qpid-config add queue prices --lvq-key ticker + </programlisting> + </section> + </section> + + <section role="h2" id="LVQ-Example"> + <title>LVQ Example</title> + + <section role="h3" id="LVQ-Example-Sender"> + <title>LVQ Sender</title> + <programlisting> + from qpid.messaging import Connection, Message + + def send(sender, key, message): + message.properties["ticker"] = key + sender.send(message) + + conn = Connection("localhost") + conn.open() + sess = conn.session() + tx = sess.sender("prices;{create:always, node:{type:queue,x-declare:{arguments:{'qpid.last_value_queue_key':ticker}}}}") + + msg = Message("Content") + send(tx, "key1", msg); + send(tx, "key2", msg); + send(tx, "key3", msg); + send(tx, "key4", msg); + send(tx, "key2", msg); + send(tx, "key1", msg); + + conn.close() + </programlisting> + </section> + + <section role="h3" id="LVQ-Example-Receiver"> + <title>LVQ Browsing Receiver</title> + <programlisting> + from qpid.messaging import Connection, Message + + conn = Connection("localhost") + conn.open() + sess = conn.session() + rx = sess.receiver("prices;{mode:browse}") + + while True: + msg = rx.fetch() + sess.acknowledge() + print msg + </programlisting> + </section> + </section> + + <section role="h2" id="LVQ-Deprecated"> + <title>Deprecated LVQ Modes</title> + <para> + There are two legacy modes (still implemented as of Qpid 0.14) + controlled by the qpid.last_value_queue and + qpid.last_value_queue_no_browse argument values. These modes are + deprecated and should not be used. + </para> + </section> +</section> diff --git a/qpid/cpp/docs/book/src/cpp-broker/Makefile b/qpid/cpp/docs/book/src/cpp-broker/Makefile new file mode 100644 index 0000000000..0266a0f54d --- /dev/null +++ b/qpid/cpp/docs/book/src/cpp-broker/Makefile @@ -0,0 +1,20 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +include ../Makefile.inc diff --git a/qpid/cpp/docs/book/src/cpp-broker/Managing-CPP-Broker.xml b/qpid/cpp/docs/book/src/cpp-broker/Managing-CPP-Broker.xml new file mode 100644 index 0000000000..ab72bf286b --- /dev/null +++ b/qpid/cpp/docs/book/src/cpp-broker/Managing-CPP-Broker.xml @@ -0,0 +1,480 @@ +<?xml version="1.0" encoding="utf-8"?> +<!-- + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +--> + +<section id="section-Managing-CPP-Broker"> + <title> Managing the C++ Broker </title> + <para> + There are quite a few ways to interact with the C++ broker. The + command line tools + include: + </para><itemizedlist> + <listitem><para>qpid-route - used to configure federation (a set of federated + brokers) + </para></listitem> + <listitem><para>qpid-config - used to configure queues, exchanges, bindings + and list them etc + </para></listitem> + <listitem><para>qpid-tool - used to view management information/statistics + and call any management actions on the broker + </para></listitem> + <listitem><para>qpid-printevents - used to receive and print QMF events + </para></listitem> + <listitem><para>qpid-ha - used to interact with the High Availability module + </para></listitem> + </itemizedlist> + + <section role="h3" id="MgmtC-2B-2B-Usingqpidconfig"><title> + Using qpid-config + </title> + <para> + This utility can be used to create queues exchanges and bindings, + both durable and transient. Always check for latest options by + running --help command. + </para> + <programlisting> +$ qpid-config --help +Usage: qpid-config [OPTIONS] + qpid-config [OPTIONS] exchanges [filter-string] + qpid-config [OPTIONS] queues [filter-string] + qpid-config [OPTIONS] add exchange <type> <name> [AddExchangeOptions] + qpid-config [OPTIONS] del exchange <name> + qpid-config [OPTIONS] add queue <name> [AddQueueOptions] + qpid-config [OPTIONS] del queue <name> + qpid-config [OPTIONS] bind <exchange-name> <queue-name> [binding-key] + qpid-config [OPTIONS] unbind <exchange-name> <queue-name> [binding-key] + +Options: + -b [ --bindings ] Show bindings in queue or exchange list + -a [ --broker-addr ] Address (localhost) Address of qpidd broker + broker-addr is in the form: [username/password@] hostname | ip-address [:<port>] + ex: localhost, 10.1.1.7:10000, broker-host:10000, guest/guest@localhost + +Add Queue Options: + --durable Queue is durable + --file-count N (8) Number of files in queue's persistence journal + --file-size N (24) File size in pages (64Kib/page) + --max-queue-size N Maximum in-memory queue size as bytes + --max-queue-count N Maximum in-memory queue size as a number of messages + --limit-policy [none | reject | flow-to-disk | ring | ring-strict] + Action taken when queue limit is reached: + none (default) - Use broker's default policy + reject - Reject enqueued messages + flow-to-disk - Page messages to disk + ring - Replace oldest unacquired message with new + ring-strict - Replace oldest message, reject if oldest is acquired + --order [fifo | lvq | lvq-no-browse] + Set queue ordering policy: + fifo (default) - First in, first out + lvq - Last Value Queue ordering, allows queue browsing + lvq-no-browse - Last Value Queue ordering, browsing clients may lose data + +Add Exchange Options: + --durable Exchange is durable + --sequence Exchange will insert a 'qpid.msg_sequence' field in the message header + with a value that increments for each message forwarded. + --ive Exchange will behave as an 'initial-value-exchange', keeping a reference + to the last message forwarded and enqueuing that message to newly bound + queues. +</programlisting> + <para> + Get the summary page + </para> + <programlisting> +$ qpid-config +Total Exchanges: 6 + topic: 2 + headers: 1 + fanout: 1 + direct: 2 + Total Queues: 7 + durable: 0 + non-durable: 7 +</programlisting> + <para> + List the queues + </para> + <programlisting> +$ qpid-config queues +Queue Name Attributes +================================================================= +pub_start +pub_done +sub_ready +sub_done +perftest0 --durable +reply-dhcp-100-18-254.bos.redhat.com.20713 auto-del excl +topic-dhcp-100-18-254.bos.redhat.com.20713 auto-del excl + +</programlisting> + <para> + List the exchanges with bindings + </para> + <programlisting> +$ ./qpid-config -b exchanges +Exchange '' (direct) + bind pub_start => pub_start + bind pub_done => pub_done + bind sub_ready => sub_ready + bind sub_done => sub_done + bind perftest0 => perftest0 + bind mgmt-3206ff16-fb29-4a30-82ea-e76f50dd7d15 => mgmt-3206ff16-fb29-4a30-82ea-e76f50dd7d15 + bind repl-3206ff16-fb29-4a30-82ea-e76f50dd7d15 => repl-3206ff16-fb29-4a30-82ea-e76f50dd7d15 +Exchange 'amq.direct' (direct) + bind repl-3206ff16-fb29-4a30-82ea-e76f50dd7d15 => repl-3206ff16-fb29-4a30-82ea-e76f50dd7d15 + bind repl-df06c7a6-4ce7-426a-9f66-da91a2a6a837 => repl-df06c7a6-4ce7-426a-9f66-da91a2a6a837 + bind repl-c55915c2-2fda-43ee-9410-b1c1cbb3e4ae => repl-c55915c2-2fda-43ee-9410-b1c1cbb3e4ae +Exchange 'amq.topic' (topic) +Exchange 'amq.fanout' (fanout) +Exchange 'amq.match' (headers) +Exchange 'qpid.management' (topic) + bind mgmt.# => mgmt-3206ff16-fb29-4a30-82ea-e76f50dd7d15 +</programlisting> +<!--h3--></section> + + <section role="h3" id="MgmtC-2B-2B-Usingqpidroute"><title> + Using qpid-route + </title> + <para> + This utility is to create federated networks of brokers, This + allows you for forward messages between brokers in a network. + Messages can be routed statically (using "qpid-route route add") + where the bindings that control message forwarding are supplied + in the route. Message routing can also be dynamic (using + "qpid-route dynamic add") where the messages are automatically + forwarded to clients based on their bindings to the local broker. + </para> + <programlisting> +$ qpid-route +Usage: qpid-route [OPTIONS] dynamic add <dest-broker> <src-broker> <exchange> [tag] [exclude-list] + qpid-route [OPTIONS] dynamic del <dest-broker> <src-broker> <exchange> + + qpid-route [OPTIONS] route add <dest-broker> <src-broker> <exchange> <routing-key> [tag] [exclude-list] + qpid-route [OPTIONS] route del <dest-broker> <src-broker> <exchange> <routing-key> + qpid-route [OPTIONS] queue add <dest-broker> <src-broker> <exchange> <queue> + qpid-route [OPTIONS] queue del <dest-broker> <src-broker> <exchange> <queue> + qpid-route [OPTIONS] route list [<dest-broker>] + qpid-route [OPTIONS] route flush [<dest-broker>] + qpid-route [OPTIONS] route map [<broker>] + + qpid-route [OPTIONS] link add <dest-broker> <src-broker> + qpid-route [OPTIONS] link del <dest-broker> <src-broker> + qpid-route [OPTIONS] link list [<dest-broker>] + +Options: + -v [ --verbose ] Verbose output + -q [ --quiet ] Quiet output, don't print duplicate warnings + -d [ --durable ] Added configuration shall be durable + -e [ --del-empty-link ] Delete link after deleting last route on the link + -s [ --src-local ] Make connection to source broker (push route) + -t <transport> [ --transport <transport>] + Specify transport to use for links, defaults to tcp + + dest-broker and src-broker are in the form: [username/password@] hostname | ip-address [:<port>] + ex: localhost, 10.1.1.7:10000, broker-host:10000, guest/guest@localhost +</programlisting> + <para> + A few examples: + </para> + <programlisting> +qpid-route dynamic add host1 host2 fed.topic +qpid-route dynamic add host2 host1 fed.topic + +qpid-route -v route add host1 host2 hub1.topic hub2.topic.stock.buy +qpid-route -v route add host1 host2 hub1.topic hub2.topic.stock.sell +qpid-route -v route add host1 host2 hub1.topic 'hub2.topic.stock.#' +qpid-route -v route add host1 host2 hub1.topic 'hub2.#' +qpid-route -v route add host1 host2 hub1.topic 'hub2.topic.#' +qpid-route -v route add host1 host2 hub1.topic 'hub2.global.#' +</programlisting> + <para> + The link map feature can be used to display the entire federated + network configuration by supplying a single broker as an entry + point: + </para> + <programlisting> +$ qpid-route route map localhost:10001 + +Finding Linked Brokers: + localhost:10001... Ok + localhost:10002... Ok + localhost:10003... Ok + localhost:10004... Ok + localhost:10005... Ok + localhost:10006... Ok + localhost:10007... Ok + localhost:10008... Ok + +Dynamic Routes: + + Exchange fed.topic: + localhost:10002 <=> localhost:10001 + localhost:10003 <=> localhost:10002 + localhost:10004 <=> localhost:10002 + localhost:10005 <=> localhost:10002 + localhost:10006 <=> localhost:10005 + localhost:10007 <=> localhost:10006 + localhost:10008 <=> localhost:10006 + + Exchange fed.direct: + localhost:10002 => localhost:10001 + localhost:10004 => localhost:10003 + localhost:10003 => localhost:10002 + localhost:10001 => localhost:10004 + +Static Routes: + + localhost:10003(ex=amq.direct) <= localhost:10005(ex=amq.direct) key=rkey + localhost:10003(ex=amq.direct) <= localhost:10005(ex=amq.direct) key=rkey2 +</programlisting> +<!--h3--></section> + + <section role="h3" id="MgmtC-2B-2B-Usingqpidtool"><title> + Using qpid-tool + </title> + <para> + This utility provided a telnet style interface to be able to + view, list all stats and action + all the methods. Simple capture below. Best to just play with it + and mail the list if you have + questions or want features added. + </para> + <programlisting> +qpid: +qpid: help +Management Tool for QPID +Commands: + list - Print summary of existing objects by class + list <className> - Print list of objects of the specified class + list <className> all - Print contents of all objects of specified class + list <className> active - Print contents of all non-deleted objects of specified class + list <list-of-IDs> - Print contents of one or more objects (infer className) + list <className> <list-of-IDs> - Print contents of one or more objects + list is space-separated, ranges may be specified (i.e. 1004-1010) + call <ID> <methodName> <args> - Invoke a method on an object + schema - Print summary of object classes seen on the target + schema <className> - Print details of an object class + set time-format short - Select short timestamp format (default) + set time-format long - Select long timestamp format + quit or ^D - Exit the program +qpid: list +Management Object Types: + ObjectType Active Deleted + ================================ + qpid.binding 21 0 + qpid.broker 1 0 + qpid.client 1 0 + qpid.exchange 6 0 + qpid.queue 13 0 + qpid.session 4 0 + qpid.system 1 0 + qpid.vhost 1 0 +qpid: list qpid.system +Objects of type qpid.system + ID Created Destroyed Index + ================================== + 1000 21:00:02 - host +qpid: list 1000 +Object of type qpid.system: (last sample time: 21:26:02) + Type Element 1000 + ======================================================= + config sysId host + config osName Linux + config nodeName localhost.localdomain + config release 2.6.24.4-64.fc8 + config version #1 SMP Sat Mar 29 09:15:49 EDT 2008 + config machine x86_64 +qpid: schema queue +Schema for class 'qpid.queue': + Element Type Unit Access Notes Description + =================================================================================================================== + vhostRef reference ReadCreate index + name short-string ReadCreate index + durable boolean ReadCreate + autoDelete boolean ReadCreate + exclusive boolean ReadCreate + arguments field-table ReadOnly Arguments supplied in queue.declare + storeRef reference ReadOnly Reference to persistent queue (if durable) + msgTotalEnqueues uint64 message Total messages enqueued + msgTotalDequeues uint64 message Total messages dequeued + msgTxnEnqueues uint64 message Transactional messages enqueued + msgTxnDequeues uint64 message Transactional messages dequeued + msgPersistEnqueues uint64 message Persistent messages enqueued + msgPersistDequeues uint64 message Persistent messages dequeued + msgDepth uint32 message Current size of queue in messages + msgDepthHigh uint32 message Current size of queue in messages (High) + msgDepthLow uint32 message Current size of queue in messages (Low) + byteTotalEnqueues uint64 octet Total messages enqueued + byteTotalDequeues uint64 octet Total messages dequeued + byteTxnEnqueues uint64 octet Transactional messages enqueued + byteTxnDequeues uint64 octet Transactional messages dequeued + bytePersistEnqueues uint64 octet Persistent messages enqueued + bytePersistDequeues uint64 octet Persistent messages dequeued + byteDepth uint32 octet Current size of queue in bytes + byteDepthHigh uint32 octet Current size of queue in bytes (High) + byteDepthLow uint32 octet Current size of queue in bytes (Low) + enqueueTxnStarts uint64 transaction Total enqueue transactions started + enqueueTxnCommits uint64 transaction Total enqueue transactions committed + enqueueTxnRejects uint64 transaction Total enqueue transactions rejected + enqueueTxnCount uint32 transaction Current pending enqueue transactions + enqueueTxnCountHigh uint32 transaction Current pending enqueue transactions (High) + enqueueTxnCountLow uint32 transaction Current pending enqueue transactions (Low) + dequeueTxnStarts uint64 transaction Total dequeue transactions started + dequeueTxnCommits uint64 transaction Total dequeue transactions committed + dequeueTxnRejects uint64 transaction Total dequeue transactions rejected + dequeueTxnCount uint32 transaction Current pending dequeue transactions + dequeueTxnCountHigh uint32 transaction Current pending dequeue transactions (High) + dequeueTxnCountLow uint32 transaction Current pending dequeue transactions (Low) + consumers uint32 consumer Current consumers on queue + consumersHigh uint32 consumer Current consumers on queue (High) + consumersLow uint32 consumer Current consumers on queue (Low) + bindings uint32 binding Current bindings + bindingsHigh uint32 binding Current bindings (High) + bindingsLow uint32 binding Current bindings (Low) + unackedMessages uint32 message Messages consumed but not yet acked + unackedMessagesHigh uint32 message Messages consumed but not yet acked (High) + unackedMessagesLow uint32 message Messages consumed but not yet acked (Low) + messageLatencySamples delta-time nanosecond Broker latency through this queue (Samples) + messageLatencyMin delta-time nanosecond Broker latency through this queue (Min) + messageLatencyMax delta-time nanosecond Broker latency through this queue (Max) + messageLatencyAverage delta-time nanosecond Broker latency through this queue (Average) +Method 'purge' Discard all messages on queue +qpid: list queue +Objects of type qpid.queue + ID Created Destroyed Index + =========================================================================== + 1012 21:08:13 - 1002.pub_start + 1014 21:08:13 - 1002.pub_done + 1016 21:08:13 - 1002.sub_ready + 1018 21:08:13 - 1002.sub_done + 1020 21:08:13 - 1002.perftest0 + 1038 21:09:08 - 1002.mgmt-3206ff16-fb29-4a30-82ea-e76f50dd7d15 + 1040 21:09:08 - 1002.repl-3206ff16-fb29-4a30-82ea-e76f50dd7d15 + 1046 21:09:32 - 1002.mgmt-df06c7a6-4ce7-426a-9f66-da91a2a6a837 + 1048 21:09:32 - 1002.repl-df06c7a6-4ce7-426a-9f66-da91a2a6a837 + 1054 21:10:01 - 1002.mgmt-c55915c2-2fda-43ee-9410-b1c1cbb3e4ae + 1056 21:10:01 - 1002.repl-c55915c2-2fda-43ee-9410-b1c1cbb3e4ae + 1063 21:26:00 - 1002.mgmt-8d621997-6356-48c3-acab-76a37081d0f3 + 1065 21:26:00 - 1002.repl-8d621997-6356-48c3-acab-76a37081d0f3 +qpid: list 1020 +Object of type qpid.queue: (last sample time: 21:26:02) + Type Element 1020 + ========================================================================== + config vhostRef 1002 + config name perftest0 + config durable False + config autoDelete False + config exclusive False + config arguments {'qpid.max_size': 0, 'qpid.max_count': 0} + config storeRef NULL + inst msgTotalEnqueues 500000 messages + inst msgTotalDequeues 500000 + inst msgTxnEnqueues 0 + inst msgTxnDequeues 0 + inst msgPersistEnqueues 0 + inst msgPersistDequeues 0 + inst msgDepth 0 + inst msgDepthHigh 0 + inst msgDepthLow 0 + inst byteTotalEnqueues 512000000 octets + inst byteTotalDequeues 512000000 + inst byteTxnEnqueues 0 + inst byteTxnDequeues 0 + inst bytePersistEnqueues 0 + inst bytePersistDequeues 0 + inst byteDepth 0 + inst byteDepthHigh 0 + inst byteDepthLow 0 + inst enqueueTxnStarts 0 transactions + inst enqueueTxnCommits 0 + inst enqueueTxnRejects 0 + inst enqueueTxnCount 0 + inst enqueueTxnCountHigh 0 + inst enqueueTxnCountLow 0 + inst dequeueTxnStarts 0 + inst dequeueTxnCommits 0 + inst dequeueTxnRejects 0 + inst dequeueTxnCount 0 + inst dequeueTxnCountHigh 0 + inst dequeueTxnCountLow 0 + inst consumers 0 consumers + inst consumersHigh 0 + inst consumersLow 0 + inst bindings 1 binding + inst bindingsHigh 1 + inst bindingsLow 1 + inst unackedMessages 0 messages + inst unackedMessagesHigh 0 + inst unackedMessagesLow 0 + inst messageLatencySamples 0 + inst messageLatencyMin 0 + inst messageLatencyMax 0 + inst messageLatencyAverage 0 +qpid: +</programlisting> +<!--h3--></section> + + <section role="h3" id="MgmtC-2B-2B-Usingqpidprintevents"><title> + Using + qpid-printevents + </title> + <para> + This utility connects to one or more brokers and collects events, + printing out a line per event. + </para> + <programlisting> +$ qpid-printevents --help +Usage: qpid-printevents [options] [broker-addr]... + +Collect and print events from one or more Qpid message brokers. If no broker- +addr is supplied, qpid-printevents will connect to 'localhost:5672'. broker- +addr is of the form: [username/password@] hostname | ip-address [:<port>] ex: +localhost, 10.1.1.7:10000, broker-host:10000, guest/guest@localhost + +Options: + -h, --help show this help message and exit +</programlisting> + <para> + You get the idea... have fun! + </para> +<!--h3--></section> +<section> + <title>Using qpid-ha</title> + <para>This utility lets you monitor and control the activity of the clustering behavior provided by the HA module. + </para> + <programlisting> + <![CDATA[ +qpid-ha --help +usage: qpid-ha <command> [<arguments>] + +Commands are: + + ready Test if a backup broker is ready. + query Print HA configuration settings. + set Set HA configuration settings. + promote Promote broker from backup to primary. + replicate Set up replication from <queue> on <remote-broker> to <queue> on the current broker. + +For help with a command type: qpid-ha <command> --help +]]> + </programlisting> +</section> +</section> diff --git a/qpid/cpp/docs/book/src/cpp-broker/QMF-Python-Console-Tutorial.xml b/qpid/cpp/docs/book/src/cpp-broker/QMF-Python-Console-Tutorial.xml new file mode 100644 index 0000000000..2cb802671b --- /dev/null +++ b/qpid/cpp/docs/book/src/cpp-broker/QMF-Python-Console-Tutorial.xml @@ -0,0 +1,894 @@ +<?xml version="1.0" encoding="utf-8"?> +<!-- + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +--> + +<section><title> + QMF Python Console Tutorial + </title> + <itemizedlist> + <listitem><para> + <xref linkend="QMFPythonConsoleTutorial-PrerequisiteInstallQpidMessaging"/> + </para></listitem> + <listitem><para> + <xref linkend="QMFPythonConsoleTutorial-SynchronousConsoleOperations"/> + </para></listitem> + <listitem><para> + <itemizedlist> + <listitem><para> + <xref linkend="QMFPythonConsoleTutorial-CreatingaQMFConsoleSessionandAttachingtoaBroker"/> + </para></listitem> + <listitem><para> + <xref linkend="QMFPythonConsoleTutorial-AccessingManagedObjects"/> + </para></listitem> + <listitem><para> + <itemizedlist> + <listitem><para> + <xref linkend="QMFPythonConsoleTutorial-ViewingPropertiesandStatisticsofanObject"/> + </para></listitem> + <listitem><para> + <xref linkend="QMFPythonConsoleTutorial-InvokingMethodsonanObject"/> + </para></listitem> + </itemizedlist> + </para></listitem> + </itemizedlist> + </para></listitem> + <listitem><para> + <xref linkend="QMFPythonConsoleTutorial-AsynchronousConsoleOperations"/> + </para></listitem> + <listitem><para> + <itemizedlist> + <listitem><para> + <xref linkend="QMFPythonConsoleTutorial-CreatingaConsoleClasstoReceiveAsynchronousData"/> + </para></listitem> + <listitem><para> + <xref linkend="QMFPythonConsoleTutorial-ReceivingEvents"/> + </para></listitem> + <listitem><para> + <xref linkend="QMFPythonConsoleTutorial-ReceivingObjects"/> + </para></listitem> + <listitem><para> + <xref linkend="QMFPythonConsoleTutorial-AsynchronousMethodCallsandMethodTimeouts"/> + </para></listitem> + </itemizedlist> + </para></listitem> + <listitem><para> + <xref linkend="QMFPythonConsoleTutorial-DiscoveringwhatKindsofObjectsareAvailable"/> + </para></listitem> + </itemizedlist> + <section role="h1" id="QMFPythonConsoleTutorial-PrerequisiteInstallQpidMessaging"><title> + Prerequisite + - Install Qpid Messaging + </title> + + <para> + QMF uses AMQP Messaging (QPid) as its means of communication. To + use QMF, Qpid messaging must be installed somewhere in the + network. Qpid can be downloaded as source from Apache, is + packaged with a number of Linux distributions, and can be + purchased from commercial vendors that use Qpid. Please see + <ulink url="http://qpid.apache.org">http://qpid.apache.org</ulink>for + information as to where to get Qpid Messaging. + </para><para> + Qpid Messaging includes a message broker (qpidd) which typically + runs as a daemon on a system. It also includes client bindings in + various programming languages. The Python-language client library + includes the QMF console libraries needed for this tutorial. + </para><para> + Please note that Qpid Messaging has two broker implementations. + One is implemented in C++ and the other in Java. At press time, + QMF is supported only by the C++ broker. + </para><para> + If the goal is to get the tutorial examples up and running as + quickly as possible, all of the Qpid components can be installed + on a single system (even a laptop). For more realistic + deployments, the broker can be deployed on a server and the + client/QMF libraries installed on other systems. + </para> +<!--h1--></section> + + <section role="h1" id="QMFPythonConsoleTutorial-SynchronousConsoleOperations"><title> + Synchronous + Console Operations + </title> + + <para> + The Python console API for QMF can be used in a synchronous + style, an asynchronous style, or a combination of both. + Synchronous operations are conceptually simple and are well + suited for user-interactive tasks. All operations are performed + in the context of a Python function call. If communication over + the message bus is required to complete an operation, the + function call blocks and waits for the expected result (or + timeout failure) before returning control to the caller. + </para><section role="h2" id="QMFPythonConsoleTutorial-CreatingaQMFConsoleSessionandAttachingtoaBroker"><title> + Creating a QMF Console Session and Attaching to a Broker + </title> + + <para> + For the purposes of this tutorial, code examples will be shown as + they are entered in an interactive python session. + </para> + <programlisting> +$ python +Python 2.5.2 (r252:60911, Sep 30 2008, 15:41:38) +[GCC 4.3.2 20080917 (Red Hat 4.3.2-4)] on linux2 +Type "help", "copyright", "credits" or "license" for more information. +>>> +</programlisting> + <para> + We will begin by importing the required libraries. If the Python + client is properly installed, these libraries will be found + normally by the Python interpreter. + </para> + <programlisting> +>>> from qmf.console import Session +</programlisting> + <para> + We must now create a <emphasis>Session</emphasis> object to manage this QMF + console session. + </para> + <programlisting> +>>> sess = Session() +</programlisting> + <para> + If no arguments are supplied to the creation of <emphasis>Session</emphasis>, + it defaults to synchronous-only operation. It also defaults to + user-management of connections. More on this in a moment. + </para><para> + We will now establish a connection to the messaging broker. If + the broker daemon is running on the local host, simply use the + following: + </para> + <programlisting> +>>> broker = sess.addBroker() +</programlisting> + <para> + If the messaging broker is on a remote host, supply the URL to + the broker in the <emphasis>addBroker</emphasis> function call. Here's how to + connect to a local broker using the URL. + </para> + <programlisting> +>>> broker = sess.addBroker("amqp://localhost") +</programlisting> + <para> + The call to <emphasis>addBroker</emphasis> is synchronous and will return + only after the connection has been successfully established or + has failed. If a failure occurs, <emphasis>addBroker</emphasis> will raise an + exception that can be handled by the console script. + </para> + <programlisting> +>>> try: +... broker = sess.addBroker("amqp://localhost:1000") +... except: +... print "Connection Failed" +... +Connection Failed +>>> +</programlisting> + <para> + This operation fails because there is no Qpid Messaging broker + listening on port 1000 (the default port for qpidd is 5672). + </para><para> + If preferred, the QMF session can manage the connection for you. + In this case, <emphasis>addBroker</emphasis> returns immediately and the + session attempts to establish the connection in the background. + This will be covered in detail in the section on asynchronous + operations. + </para> +<!--h2--></section> + + <section role="h2" id="QMFPythonConsoleTutorial-AccessingManagedObjects"><title> + Accessing + Managed Objects + </title> + + <para> + The Python console API provides access to remotely managed + objects via a <emphasis>proxy</emphasis> model. The API gives the client an + object that serves as a proxy representing the "real" object + being managed on the agent application. Operations performed on + the proxy result in the same operations on the real object. + </para><para> + The following examples assume prior knowledge of the kinds of + objects that are actually available to be managed. There is a + section later in this tutorial that describes how to discover + what is manageable on the QMF bus. + </para><para> + Proxy objects are obtained by calling the + <emphasis>Session.getObjects</emphasis> function. + </para><para> + To illustrate, we'll get a list of objects representing queues in + the message broker itself. + </para> + <programlisting> +>>> queues = sess.getObjects(_class="queue", _package="org.apache.qpid.broker") +</programlisting> + <para> + <emphasis>queues</emphasis> is an array of proxy objects representing real + queues on the message broker. A proxy object can be printed to + display a description of the object. + </para> + <programlisting> +>>> for q in queues: +... print q +... +org.apache.qpid.broker:queue[0-1537-1-0-58] 0-0-1-0-1152921504606846979:reply-localhost.localdomain.32004 +org.apache.qpid.broker:queue[0-1537-1-0-61] 0-0-1-0-1152921504606846979:topic-localhost.localdomain.32004 +>>> +</programlisting> + <section role="h3" id="QMFPythonConsoleTutorial-ViewingPropertiesandStatisticsofanObject"><title> + Viewing Properties and Statistics of an Object + </title> + + <para> + Let us now focus our attention on one of the queue objects. + </para> + <programlisting> +>>> queue = queues[0] +</programlisting> + <para> + The attributes of an object are partitioned into + <emphasis>properties</emphasis> and <emphasis>statistics</emphasis>. Though the + distinction is somewhat arbitrary, <emphasis>properties</emphasis> tend to + be fairly static and may also be large and <emphasis>statistics</emphasis> + tend to change rapidly and are relatively small (counters, etc.). + </para><para> + There are two ways to view the properties of an object. An array + of properties can be obtained using the <emphasis>getProperties</emphasis> + function: + </para> + <programlisting> +>>> props = queue.getProperties() +>>> for prop in props: +... print prop +... +(vhostRef, 0-0-1-0-1152921504606846979) +(name, u'reply-localhost.localdomain.32004') +(durable, False) +(autoDelete, True) +(exclusive, True) +(arguments, {}) +>>> +</programlisting> + <para> + The <emphasis>getProperties</emphasis> function returns an array of tuples. + Each tuple consists of the property descriptor and the property + value. + </para><para> + A more convenient way to access properties is by using the + attribute of the proxy object directly: + </para> + <programlisting> +>>> queue.autoDelete +True +>>> queue.name +u'reply-localhost.localdomain.32004' +>>> +</programlisting> + <para> + Statistics are accessed in the same way: + </para> + <programlisting> +>>> stats = queue.getStatistics() +>>> for stat in stats: +... print stat +... +(msgTotalEnqueues, 53) +(msgTotalDequeues, 53) +(msgTxnEnqueues, 0) +(msgTxnDequeues, 0) +(msgPersistEnqueues, 0) +(msgPersistDequeues, 0) +(msgDepth, 0) +(byteDepth, 0) +(byteTotalEnqueues, 19116) +(byteTotalDequeues, 19116) +(byteTxnEnqueues, 0) +(byteTxnDequeues, 0) +(bytePersistEnqueues, 0) +(bytePersistDequeues, 0) +(consumerCount, 1) +(consumerCountHigh, 1) +(consumerCountLow, 1) +(bindingCount, 2) +(bindingCountHigh, 2) +(bindingCountLow, 2) +(unackedMessages, 0) +(unackedMessagesHigh, 0) +(unackedMessagesLow, 0) +(messageLatencySamples, 0) +(messageLatencyMin, 0) +(messageLatencyMax, 0) +(messageLatencyAverage, 0) +>>> +</programlisting> + <para> + or alternatively: + </para> + <programlisting> +>>> queue.byteTotalEnqueues +19116 +>>> +</programlisting> + <para> + The proxy objects do not automatically track changes that occur + on the real objects. For example, if the real queue enqueues more + bytes, viewing the <emphasis>byteTotalEnqueues</emphasis> statistic will show + the same number as it did the first time. To get updated data on + a proxy object, use the <emphasis>update</emphasis> function call: + </para> + <programlisting> +>>> queue.update() +>>> queue.byteTotalEnqueues +19783 +>>> +</programlisting> + + <note><title>Be Advised</title> + <para> + The <emphasis>update</emphasis> method was added after the M4 release + of Qpid/Qmf. It may not be available in your + distribution. + </para> + </note> +<!--h3--></section> + + <section role="h3" id="QMFPythonConsoleTutorial-InvokingMethodsonanObject"><title> + Invoking + Methods on an Object + </title> + + <para> + Up to this point, we have used the QMF Console API to find + managed objects and view their attributes, a read-only activity. + The next topic to illustrate is how to invoke a method on a + managed object. Methods allow consoles to control the managed + agents by either triggering a one-time action or by changing the + values of attributes in an object. + </para><para> + First, we'll cover some background information about methods. A + <emphasis>QMF object class</emphasis> (of which a <emphasis>QMF object</emphasis> is an + instance), may have zero or more methods. To obtain a list of + methods available for an object, use the <emphasis>getMethods</emphasis> + function. + </para> + <programlisting> +>>> methodList = queue.getMethods() +</programlisting> + <para> + <emphasis>getMethods</emphasis> returns an array of method descriptors (of + type qmf.console.SchemaMethod). To get a summary of a method, you + can simply print it. The _<emphasis>repr</emphasis>_ function returns a + string that looks like a function prototype. + </para> + <programlisting> +>>> print methodList +[purge(request)] +>>> +</programlisting> + <para> + For the purposes of illustration, we'll use a more interesting + method available on the <emphasis>broker</emphasis> object which represents + the connected Qpid message broker. + </para> + <programlisting> +>>> br = sess.getObjects(_class="broker", _package="org.apache.qpid.broker")[0] +>>> mlist = br.getMethods() +>>> for m in mlist: +... print m +... +echo(sequence, body) +connect(host, port, durable, authMechanism, username, password, transport) +queueMoveMessages(srcQueue, destQueue, qty) +>>> +</programlisting> + <para> + We have just learned that the <emphasis>broker</emphasis> object has three + methods: <emphasis>echo</emphasis>, <emphasis>connect</emphasis>, and + <emphasis>queueMoveMessages</emphasis>. We'll use the <emphasis>echo</emphasis> method to + "ping" the broker. + </para> + <programlisting> +>>> result = br.echo(1, "Message Body") +>>> print result +OK (0) - {'body': u'Message Body', 'sequence': 1} +>>> print result.status +0 +>>> print result.text +OK +>>> print result.outArgs +{'body': u'Message Body', 'sequence': 1} +>>> +</programlisting> + <para> + In the above example, we have invoked the <emphasis>echo</emphasis> method on + the instance of the broker designated by the proxy "br" with a + sequence argument of 1 and a body argument of "Message Body". The + result indicates success and contains the output arguments (in + this case copies of the input arguments). + </para><para> + To be more precise... Calling <emphasis>echo</emphasis> on the proxy causes + the input arguments to be marshalled and sent to the remote agent + where the method is executed. Once the method execution + completes, the output arguments are marshalled and sent back to + the console to be stored in the method result. + </para><para> + You are probably wondering how you are supposed to know what + types the arguments are and which arguments are input, which are + output, or which are both. This will be addressed later in the + "Discovering what Kinds of Objects are Available" section. + </para> +<!--h3--></section> +<!--h2--></section> +<!--h1--></section> + + <section role="h1" id="QMFPythonConsoleTutorial-AsynchronousConsoleOperations"><title> + Asynchronous + Console Operations + </title> + + <para> + QMF is built on top of a middleware messaging layer (Qpid + Messaging). Because of this, QMF can use some communication + patterns that are difficult to implement using network transports + like UDP, TCP, or SSL. One of these patterns is called the + <emphasis>Publication and Subscription</emphasis> pattern (pub-sub for + short). In the pub-sub pattern, data sources <emphasis>publish</emphasis> + information without a particular destination in mind. Data sinks + (destinations) <emphasis>subscribe</emphasis> using a set of criteria that + describes what kind of data they are interested in receiving. + Data published by a source may be received by zero, one, or many + subscribers. + </para><para> + QMF uses the pub-sub pattern to distribute events, object + creation and deletion, and changes to properties and statistics. + A console application using the QMF Console API can receive these + asynchronous and unsolicited events and updates. This is useful + for applications that store and analyze events and/or statistics. + It is also useful for applications that react to certain events + or conditions. + </para><para> + Note that console applications may always use the synchronous + mechanisms. + </para> + + <section role="h2" id="QMFPythonConsoleTutorial-CreatingaConsoleClasstoReceiveAsynchronousData"><title> + Creating a Console Class to Receive Asynchronous Data + </title> + + <para> + Asynchronous API operation occurs when the console application + supplies a <emphasis>Console</emphasis> object to the session manager. The + <emphasis>Console</emphasis> object (which overrides the + <emphasis>qmf.console.Console</emphasis> class) handles all asynchronously + arriving data. The <emphasis>Console</emphasis> class has the following + methods. Any number of these methods may be overridden by the + console application. Any method that is not overridden defaults + to a null handler which takes no action when invoked. + </para><table><title>QMF Python Console Class Methods</title><tgroup cols="3"> + <tbody> + <row> + <entry> + Method + </entry> + <entry> + Arguments + </entry> + <entry> + Invoked when... + </entry> + </row> + <row> + <entry> + brokerConnected + </entry> + <entry> + broker + </entry> + <entry> + a connection to a broker is established + </entry> + </row> + <row> + <entry> + brokerDisconnected + </entry> + <entry> + broker + </entry> + <entry> + a connection to a broker is lost + </entry> + </row> + <row> + <entry> + newPackage + </entry> + <entry> + name + </entry> + <entry> + a new package is seen on the QMF bus + </entry> + </row> + <row> + <entry> + newClass + </entry> + <entry> + kind, classKey + </entry> + <entry> + a new class (event or object) is seen on the QMF bus + </entry> + </row> + <row> + <entry> + newAgent + </entry> + <entry> + agent + </entry> + <entry> + a new agent appears on the QMF bus + </entry> + </row> + <row> + <entry> + delAgent + </entry> + <entry> + agent + </entry> + <entry> + an agent disconnects from the QMF bus + </entry> + </row> + <row> + <entry> + objectProps + </entry> + <entry> + broker, object + </entry> + <entry> + the properties of an object are published + </entry> + </row> + <row> + <entry> + objectStats + </entry> + <entry> + broker, object + </entry> + <entry> + the statistics of an object are published + </entry> + </row> + <row> + <entry> + event + </entry> + <entry> + broker, event + </entry> + <entry> + an event is published + </entry> + </row> + <row> + <entry> + heartbeat + </entry> + <entry> + agent, timestamp + </entry> + <entry> + a heartbeat is published by an agent + </entry> + </row> + <row> + <entry> + brokerInfo + </entry> + <entry> + broker + </entry> + <entry> + information about a connected broker is available to be + queried + </entry> + </row> + <row> + <entry> + methodResponse + </entry> + <entry> + broker, seq, response + </entry> + <entry> + the result of an asynchronous method call is received + </entry> + </row> + </tbody> + </tgroup></table><para> + Supplied with the API is a class called <emphasis>DebugConsole</emphasis>. + This is a test <emphasis>Console</emphasis> instance that overrides all of + the methods such that arriving asynchronous data is printed to + the screen. This can be used to see all of the arriving + asynchronous data. + </para> + <!--h2--></section> + + <section role="h2" id="QMFPythonConsoleTutorial-ReceivingEvents"><title> + Receiving + Events + </title> + + <para> + We'll start the example from the beginning to illustrate the + reception and handling of events. In this example, we will create + a <emphasis>Console</emphasis> class that handles broker-connect, + broker-disconnect, and event messages. We will also allow the + session manager to manage the broker connection for us. + </para><para> + Begin by importing the necessary classes: + </para> + <programlisting> +>>> from qmf.console import Session, Console +</programlisting> + <para> + Now, create a subclass of <emphasis>Console</emphasis> that handles the three + message types: + </para> + <programlisting> +>>> class EventConsole(Console): +... def brokerConnected(self, broker): +... print "brokerConnected:", broker +... def brokerDisconnected(self, broker): +... print "brokerDisconnected:", broker +... def event(self, broker, event): +... print "event:", event +... +>>> +</programlisting> + <para> + Make an instance of the new class: + </para> + <programlisting> +>>> myConsole = EventConsole() +</programlisting> + <para> + Create a <emphasis>Session</emphasis> class using the console instance. In + addition, we shall request that the session manager do the + connection management for us. Notice also that we are requesting + that the session manager not receive objects or heartbeats. Since + this example is concerned only with events, we can optimize the + use of the messaging bus by telling the session manager not to + subscribe for object updates or heartbeats. + </para> + <programlisting> +>>> sess = Session(myConsole, manageConnections=True, rcvObjects=False, rcvHeartbeats=False) +>>> broker = sess.addBroker() +>>> +</programlisting> + <para> + Once the broker is added, we will begin to receive asynchronous + events (assuming there is a functioning broker available to + connect to). + </para> + <programlisting> +brokerConnected: Broker connected at: localhost:5672 +event: Thu Jan 29 19:53:19 2009 INFO org.apache.qpid.broker:bind broker=localhost:5672 ... +</programlisting> +<!--h2--></section> + + <section role="h2" id="QMFPythonConsoleTutorial-ReceivingObjects"><title> + Receiving + Objects + </title> + + <para> + To illustrate asynchronous handling of objects, a small console + program is supplied. The entire program is shown below for + convenience. We will then go through it part-by-part to explain + its design. + </para><para> + This console program receives object updates and displays a set + of statistics as they change. It focuses on broker queue objects. + </para> + <programlisting> +# Import needed classes +from qmf.console import Session, Console +from time import sleep + +# Declare a dictionary to map object-ids to queue names +queueMap = {} + +# Customize the Console class to receive object updates. +class MyConsole(Console): + + # Handle property updates + def objectProps(self, broker, record): + + # Verify that we have received a queue object. Exit otherwise. + classKey = record.getClassKey() + if classKey.getClassName() != "queue": + return + + # If this object has not been seen before, create a new mapping from objectID to name + oid = record.getObjectId() + if oid not in queueMap: + queueMap[oid] = record.name + + # Handle statistic updates + def objectStats(self, broker, record): + + # Ignore updates for objects that are not in the map + oid = record.getObjectId() + if oid not in queueMap: + return + + # Print the queue name and some statistics + print "%s: enqueues=%d dequeues=%d" % (queueMap[oid], record.msgTotalEnqueues, record.msgTotalDequeues) + + # if the delete-time is non-zero, this object has been deleted. Remove it from the map. + if record.getTimestamps()[2] > 0: + queueMap.pop(oid) + +# Create an instance of the QMF session manager. Set userBindings to True to allow +# this program to choose which objects classes it is interested in. +sess = Session(MyConsole(), manageConnections=True, rcvEvents=False, userBindings=True) + +# Register to receive updates for broker:queue objects. +sess.bindClass("org.apache.qpid.broker", "queue") +broker = sess.addBroker() + +# Suspend processing while the asynchronous operations proceed. +try: + while True: + sleep(1) +except: + pass + +# Disconnect the broker before exiting. +sess.delBroker(broker) +</programlisting> + <para> + Before going through the code in detail, it is important to + understand the differences between synchronous object access and + asynchronous object access. When objects are obtained + synchronously (using the <emphasis>getObjects</emphasis> function), the + resulting proxy contains all of the object's attributes, both + properties and statistics. When object data is published + asynchronously, the properties and statistics are sent separately + and only when the session first connects or when the content + changes. + </para><para> + The script wishes to print the queue name with the updated + statistics, but the queue name is only present with the + properties. For this reason, the program needs to keep some state + to correlate property updates with their corresponding statistic + updates. This can be done using the <emphasis>ObjectId</emphasis> that + uniquely identifies the object. + </para> + <programlisting> + # If this object has not been seen before, create a new mapping from objectID to name + oid = record.getObjectId() + if oid not in queueMap: + queueMap[oid] = record.name +</programlisting> + <para> + The above code fragment gets the object ID from the proxy and + checks to see if it is in the map (i.e. has been seen before). If + it is not in the map, a new map entry is inserted mapping the + object ID to the queue's name. + </para> + <programlisting> + # if the delete-time is non-zero, this object has been deleted. Remove it from the map. + if record.getTimestamps()[2] > 0: + queueMap.pop(oid) +</programlisting> + <para> + This code fragment detects the deletion of a managed object. + After reporting the statistics, it checks the timestamps of the + proxy. <emphasis>getTimestamps</emphasis> returns a list of timestamps in the + order: + </para><itemizedlist> + <listitem><para> + <emphasis>Current</emphasis> - The timestamp of the sending of this update. + </para></listitem> + <listitem><para> + <emphasis>Create</emphasis> - The time of the object's creation + </para></listitem> + <listitem><para> + <emphasis>Delete</emphasis> - The time of the object's deletion (or zero if + not deleted) + </para></listitem> + </itemizedlist><para> + This code structure is useful for getting information about + very-short-lived objects. It is possible that an object will be + created, used, and deleted within an update interval. In this + case, the property update will arrive first, followed by the + statistic update. Both will indicate that the object has been + deleted but a full accounting of the object's existence and final + state is reported. + </para> + <programlisting> +# Create an instance of the QMF session manager. Set userBindings to True to allow +# this program to choose which objects classes it is interested in. +sess = Session(MyConsole(), manageConnections=True, rcvEvents=False, userBindings=True) + +# Register to receive updates for broker:queue objects. +sess.bindClass("org.apache.qpid.broker", "queue") +</programlisting> + <para> + The above code is illustrative of the way a console application + can tune its use of the QMF bus. Note that <emphasis>rcvEvents</emphasis> is + set to False. This prevents the reception of events. Note also + the use of <emphasis>userBindings=True</emphasis> and the call to + <emphasis>sess.bindClass</emphasis>. If <emphasis>userBindings</emphasis> is set to False + (its default), the session will receive object updates for all + classes of object. In the case above, the application is only + interested in broker:queue objects and reduces its bus bandwidth + usage by requesting updates to only that class. + <emphasis>bindClass</emphasis> may be called as many times as desired to add + classes to the list of subscribed classes. + </para> +<!--h2--></section> + + <section role="h2" id="QMFPythonConsoleTutorial-AsynchronousMethodCallsandMethodTimeouts"><title> + Asynchronous Method Calls and Method Timeouts + </title> + + <para> + Method calls can also be invoked asynchronously. This is useful + if a large number of calls needs to be made in a short time + because the console application will not need to wait for the + complete round-trip delay for each call. + </para><para> + Method calls are synchronous by default. They can be made + asynchronous by adding the keyword-argument _<emphasis>async=True</emphasis> + to the method call. + </para><para> + In a synchronous method call, the return value is the method + result. When a method is called asynchronously, the return value + is a sequence number that can be used to correlate the eventual + result to the request. This sequence number is passed as an + argument to the <emphasis>methodResponse</emphasis> function in the + <emphasis>Console</emphasis> interface. + </para><para> + It is important to realize that the <emphasis>methodResponse</emphasis> + function may be invoked before the asynchronous call returns. + Make sure your code is written to handle this possibility. + </para> + <!--h2--></section> + <!--h1--></section> + + <section role="h1" id="QMFPythonConsoleTutorial-DiscoveringwhatKindsofObjectsareAvailable"><title> + Discovering what Kinds of Objects are Available + </title> + <para/> + <!--h1--></section> + + +</section> diff --git a/qpid/cpp/docs/book/src/cpp-broker/Qpid-Interoperability-Documentation.xml b/qpid/cpp/docs/book/src/cpp-broker/Qpid-Interoperability-Documentation.xml new file mode 100644 index 0000000000..74546693df --- /dev/null +++ b/qpid/cpp/docs/book/src/cpp-broker/Qpid-Interoperability-Documentation.xml @@ -0,0 +1,377 @@ +<?xml version="1.0" encoding="utf-8"?> +<!-- + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +--> + +<section role="h2" id="QpidInteroperabilityDocumentation-QpidInteroperabilityDocumentation"> + <title>Qpid Interoperability Documentation</title> + + <para> + This page documents the various interoperable features of the + Qpid clients. + </para> + <section role="h3" id="QpidInteroperabilityDocumentation-SASL"><title> + SASL + </title> + + <para> + + </para> + <section role="h4" id="QpidInteroperabilityDocumentation-StandardMechanisms"><title> + Standard + Mechanisms + </title> + + <para> + <ulink url="http://en.wikipedia.org/wiki/Simple_Authentication_and_Security_Layer#SASL_mechanisms"/> + </para><para> + This table list the various SASL mechanisms that each component + supports. The version listed shows when this + functionality was added to the product. + </para><table><title>SASL Mechanism Support</title><tgroup cols="7"> + <tbody> + <row> + <entry> + Component + </entry> + <entry> + ANONYMOUS + </entry> + <entry> + CRAM-MD5 + </entry> + <entry> + DIGEST-MD5 + </entry> + <entry> + EXTERNAL + </entry> + <entry> + GSSAPI/Kerberos + </entry> + <entry> + PLAIN + </entry> + </row> + <row> + <entry> + C++ Broker + </entry> + <entry> + M3[<xref linkend="QpidInteroperabilityDocumentation-1"/>] + </entry> + <entry> + M3[<xref linkend="QpidInteroperabilityDocumentation-1"/>,<xref linkend="QpidInteroperabilityDocumentation-2"/>] + </entry> + <entry> + + </entry> + <entry> + + </entry> + <entry> + M3[<xref linkend="QpidInteroperabilityDocumentation-1"/>,<xref linkend="QpidInteroperabilityDocumentation-2"/>] + </entry> + <entry> + M1 + </entry> + </row> + <row> + <entry> + C++ Client + </entry> + <entry> + M3[<xref linkend="QpidInteroperabilityDocumentation-1"/>] + </entry> + <entry> + + </entry> + <entry> + + </entry> + <entry> + + </entry> + <entry> + + </entry> + <entry> + M1 + </entry> + </row> + <row> + <entry> + Java Broker + </entry> + <entry> + + </entry> + <entry> + M1 + </entry> + <entry> + + </entry> + <entry> + + </entry> + <entry> + + </entry> + <entry> + M1 + </entry> + </row> + <row> + <entry> + Java Client + </entry> + <entry> + + </entry> + <entry> + M1 + </entry> + <entry> + + </entry> + <entry> + + </entry> + <entry> + + </entry> + <entry> + M1 + </entry> + </row> + <row> + <entry> + .Net Client + </entry> + <entry> + M2 + </entry> + <entry> + M2 + </entry> + <entry> + M2 + </entry> + <entry> + M2 + </entry> + <entry> + + </entry> + <entry> + M2 + </entry> + </row> + <row> + <entry> + Python Client + </entry> + <entry> + + </entry> + <entry> + + </entry> + <entry> + + </entry> + <entry> + + </entry> + <entry> + + </entry> + <entry> + ? + </entry> + </row> + <row> + <entry> + Ruby Client + </entry> + <entry> + + </entry> + <entry> + + </entry> + <entry> + + </entry> + <entry> + + </entry> + <entry> + + </entry> + <entry> + ? + </entry> + </row> + </tbody> + </tgroup></table> + + <para id="QpidInteroperabilityDocumentation-1"> + 1: Support for these will be in M3 (currently available on + trunk). + </para> + + <para id="QpidInteroperabilityDocumentation-2">2: C++ Broker uses <ulink url="http://freshmeat.net/projects/cyrussasl/">Cyrus SASL</ulink> which + supports CRAM-MD5 and GSSAPI but these have not been tested yet + </para> +<!--h4--></section> + + <section role="h4" id="QpidInteroperabilityDocumentation-CustomMechanisms"><title> + Custom + Mechanisms + </title> + + <para> + There have been some custom mechanisms added to our + implementations. + </para><table><title>SASL Custom Mechanisms</title><tgroup cols="3"> + <tbody> + <row> + <entry> + Component + </entry> + <entry> + AMQPLAIN + </entry> + <entry> + CRAM-MD5-HASHED + </entry> + </row> + <row> + <entry> + C++ Broker + </entry> + <entry> + + </entry> + <entry> + + </entry> + </row> + <row> + <entry> + C++ Client + </entry> + <entry> + + </entry> + <entry> + + </entry> + </row> + <row> + <entry> + Java Broker + </entry> + <entry> + M1 + </entry> + <entry> + M2 + </entry> + </row> + <row> + <entry> + Java Client + </entry> + <entry> + M1 + </entry> + <entry> + M2 + </entry> + </row> + <row> + <entry> + .Net Client + </entry> + <entry> + + </entry> + <entry> + + </entry> + </row> + <row> + <entry> + Python Client + </entry> + <entry> + M2 + </entry> + <entry> + + </entry> + </row> + <row> + <entry> + Ruby Client + </entry> + <entry> + M2 + </entry> + <entry> + + </entry> + </row> + </tbody> + </tgroup></table> + + <section><title>AMQPLAIN</title> + <para/> + </section> + + <section><title>CRAM-MD5-HASHED</title> + <para> + The Java SASL implementations require that you have the password + of the user to validate the incoming request. This then means + that the user's password must be stored on disk. For this to be + secure either the broker must encrypt the password file or the + need for the password being stored must be removed. + </para><para> + The CRAM-MD5-HASHED SASL plugin removes the need for the plain + text password to be stored on disk. The mechanism defers all + functionality to the build in CRAM-MD5 module the only change is + on the client side where it generates the hash of the password + and uses that value as the password. This means that the Java + Broker only need store the password hash on the file system. + While a one way hash is not very secure compared to other forms + of encryption in environments where the having the password in + plain text is unacceptable this will provide and additional layer + to protect the password. In particular this offers some + protection where the same password may be shared amongst many + systems. It offers no real extra protection against attacks on + the broker (the secret is now the hash rather than the password). + </para> + </section> +<!--h4--></section> +<!--h3--></section> +<!--h2--></section> diff --git a/qpid/cpp/docs/book/src/cpp-broker/Qpid-Management-Framework.xml b/qpid/cpp/docs/book/src/cpp-broker/Qpid-Management-Framework.xml new file mode 100644 index 0000000000..89bfe9d95e --- /dev/null +++ b/qpid/cpp/docs/book/src/cpp-broker/Qpid-Management-Framework.xml @@ -0,0 +1,944 @@ +<?xml version="1.0" encoding="utf-8"?> +<!-- + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +--> + +<section><title> + Qpid Management Framework + </title> + <itemizedlist> + <listitem><para> + <xref linkend="QpidManagementFramework-WhatIsQMF"/> + </para></listitem> + <listitem><para> + <xref linkend="QpidManagementFramework-GettingStartedwithQMF"/> + </para></listitem> + <listitem><para> + <xref linkend="QpidManagementFramework-QMFConcepts"/> + </para></listitem> + <listitem><para> + <itemizedlist> + <listitem><para> + <xref linkend="QpidManagementFramework-Console-2CAgent-2CandBroker"/> + </para></listitem> + <listitem><para> + <xref linkend="QpidManagementFramework-Schema"/> + </para></listitem> + <listitem><para> + <xref linkend="QpidManagementFramework-ClassKeysandClassVersioning"/> + </para></listitem> + </itemizedlist> + </para></listitem> + <listitem><para> + <xref linkend="QpidManagementFramework-TheQMFProtocol"/> + </para></listitem> + <listitem><para> + <xref linkend="QpidManagementFramework-HowtoWriteaQMFConsole"/> + </para></listitem> + <listitem><para> + <xref linkend="QpidManagementFramework-HowtoWriteaQMFAgent"/> + </para></listitem> + </itemizedlist> + + <para> + Please visit the <xref linkend="qpid_QMFv2-Project-Page"/> for information + about the future of QMF. + </para> + <section role="h1" id="QpidManagementFramework-WhatIsQMF"><title> + What Is QMF + </title> + + <para> + QMF (Qpid Management Framework) is a general-purpose management + bus built on Qpid Messaging. It takes advantage of the + scalability, security, and rich capabilities of Qpid to provide + flexible and easy-to-use manageability to a large set of + applications. + </para> + <!--h1--></section> + + <section role="h1" id="QpidManagementFramework-GettingStartedwithQMF"><title> + Getting + Started with QMF + </title> + + <para> + QMF is used through two primary APIs. The <emphasis>console</emphasis> API is + used for console applications that wish to access and manipulate + manageable components through QMF. The <emphasis>agent</emphasis> API is used + for application that wish to be managed through QMF. + </para><para> + The fastest way to get started with QMF is to work through the + "How To" tutorials for consoles and agents. For a deeper + understanding of what is happening in the tutorials, it is + recommended that you look at the <emphasis>Qmf Concepts</emphasis> section. + </para> + <!--h1--></section> + + <section role="h1" id="QpidManagementFramework-QMFConcepts"><title> + QMF Concepts + </title> + + <para> + This section introduces important concepts underlying QMF. + </para> + + + <section role="h2" id="QpidManagementFramework-Console-2CAgent-2CandBroker"><title> + Console, + Agent, and Broker + </title> + + <para> + The major architectural components of QMF are the Console, the + Agent, and the Broker. Console components are the "managing" + components of QMF and agent components are the "managed" parts. + The broker is a central (possibly distributed, clustered and + fault-tolerant) component that manages name spaces and caches + schema information. + </para><para> + A console application may be a command-line utility, a + three-tiered web-based GUI, a collection and storage device, a + specialized application that monitors and reacts to events and + conditions, or anything else somebody wishes to develop that uses + QMF management data. + </para><para> + An agent application is any application that has been enhanced to + allow itself to be managed via QMF. + </para> + <programlisting> + +-------------+ +---------+ +---------------+ +-------------------+ + | CLI utility | | Web app | | Audit storage | | Event correlation | + +-------------+ +---------+ +---------------+ +-------------------+ + ^ ^ ^ ^ | + | | | | | + v v v v v + +---------------------------------------------------------------------------------+ + | Qpid Messaging Bus (with QMF Broker capability) | + +---------------------------------------------------------------------------------+ + ^ ^ ^ + | | | + v v v + +----------------+ +----------------+ +----------------+ + | Manageable app | | Manageable app | | Manageable app | + +----------------+ +----------------+ +----------------+ +</programlisting> + <para> + In the above diagram, the <emphasis>Manageable apps</emphasis> are agents, + the <emphasis>CLI utility</emphasis>, <emphasis>Web app</emphasis>, and <emphasis>Audit + storage</emphasis> are consoles, and <emphasis>Event correlation</emphasis> is both + a console and an agent because it can create events based on the + aggregation of what it sees. + </para> +<!--h2--></section> + + <section role="h2" id="QpidManagementFramework-Schema"><title> + Schema + </title> + + <para> + A <emphasis>schema</emphasis> describes the structure of management data. + Each <emphasis>agent</emphasis> provides a schema that describes its + management model including the object classes, methods, events, + etc. that it provides. In the current QMF distribution, the + agent's schema is codified in an XML document. In the near + future, there will also be ways to programatically create QMF + schemata. + </para><section role="h3" id="QpidManagementFramework-Package"><title> + Package + </title> + + <para> + Each agent that exports a schema identifies itself using a + <emphasis>package</emphasis> name. The package provides a unique namespace + for the classes in the agent's schema that prevent collisions + with identically named classes in other agents' schemata. + </para><para> + Package names are in "reverse domain name" form with levels of + hierarchy separated by periods. For example, the Qpid messaging + broker uses package "org.apache.qpid.broker" and the Access + Control List plugin for the broker uses package + "org.apache.qpid.acl". In general, the package name should be the + reverse of the internet domain name assigned to the organization + that owns the agent software followed by identifiers to uniquely + identify the agent. + </para><para> + The XML document for a package's schema uses an enclosing + <schema> tag. For example: + </para> + <programlisting> +<schema package="org.apache.qpid.broker"> + +</schema> +</programlisting> +<!--h3--></section> + + <section role="h3" id="QpidManagementFramework-ObjectClasses"><title> + Object + Classes + </title> + + <para> + <emphasis>Object classes</emphasis> define types for manageable objects. The + agent may create and destroy objects which are instances of + object classes in the schema. An object class is defined in the + XML document using the <class> tag. An object class is + composed of properties, statistics, and methods. + </para> + <programlisting> + <class name="Exchange"> + <property name="vhostRef" type="objId" references="Vhost" access="RC" index="y" parentRef="y"/> + <property name="name" type="sstr" access="RC" index="y"/> + <property name="type" type="sstr" access="RO"/> + <property name="durable" type="bool" access="RC"/> + <property name="arguments" type="map" access="RO" desc="Arguments supplied in exchange.declare"/> + + <statistic name="producerCount" type="hilo32" desc="Current producers on exchange"/> + <statistic name="bindingCount" type="hilo32" desc="Current bindings"/> + <statistic name="msgReceives" type="count64" desc="Total messages received"/> + <statistic name="msgDrops" type="count64" desc="Total messages dropped (no matching key)"/> + <statistic name="msgRoutes" type="count64" desc="Total routed messages"/> + <statistic name="byteReceives" type="count64" desc="Total bytes received"/> + <statistic name="byteDrops" type="count64" desc="Total bytes dropped (no matching key)"/> + <statistic name="byteRoutes" type="count64" desc="Total routed bytes"/> + </class> +</programlisting> +<!--h3--></section> + + + <section role="h3" id="QpidManagementFramework-PropertiesandStatistics"><title> + Properties + and Statistics + </title> + + <para> + <property> and <statistic> tags must be placed within + <schema> and </schema> tags. + </para><para> + Properties, statistics, and methods are the building blocks of an + object class. Properties and statistics are both object + attributes, though they are treated differently. If an object + attribute is defining, seldom or never changes, or is large in + size, it should be defined as a <emphasis>property</emphasis>. If an + attribute is rapidly changing or is used to instrument the object + (counters, etc.), it should be defined as a <emphasis>statistic</emphasis>. + </para><para> + The XML syntax for <property> and <statistic> have + the following XML-attributes: + </para><table><title>XML Attributes for QMF Properties and Statistics</title><tgroup cols="4"> + <tbody> + <row> + <entry> + Attribute + </entry> + <entry> + <property> + </entry> + <entry> + <statistic> + </entry> + <entry> + Meaning + </entry> + </row> + <row> + <entry> + name + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + <entry> + The name of the attribute + </entry> + </row> + <row> + <entry> + type + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + <entry> + The data type of the attribute + </entry> + </row> + <row> + <entry> + unit + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + <entry> + Optional unit name - use the singular (i.e. MByte) + </entry> + </row> + <row> + <entry> + desc + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + <entry> + Description to annotate the attribute + </entry> + </row> + <row> + <entry> + references + </entry> + <entry> + Y + </entry> + <entry> + + </entry> + <entry> + If the type is "objId", names the referenced class + </entry> + </row> + <row> + <entry> + access + </entry> + <entry> + Y + </entry> + <entry> + + </entry> + <entry> + Access rights (RC, RW, RO) + </entry> + </row> + <row> + <entry> + index + </entry> + <entry> + Y + </entry> + <entry> + + </entry> + <entry> + "y" if this property is used to uniquely identify the + object. There may be more than one index property in a + class + </entry> + </row> + <row> + <entry> + parentRef + </entry> + <entry> + Y + </entry> + <entry> + + </entry> + <entry> + "y" if this property references an object in which this + object is in a child-parent relationship. + </entry> + </row> + <row> + <entry> + optional + </entry> + <entry> + Y + </entry> + <entry> + + </entry> + <entry> + "y" if this property is optional (i.e. may be + NULL/not-present) + </entry> + </row> + <row> + <entry> + min + </entry> + <entry> + Y + </entry> + <entry> + + </entry> + <entry> + Minimum value of a numeric attribute + </entry> + </row> + <row> + <entry> + max + </entry> + <entry> + Y + </entry> + <entry> + + </entry> + <entry> + Maximum value of a numeric attribute + </entry> + </row> + <row> + <entry> + maxLen + </entry> + <entry> + Y + </entry> + <entry> + + </entry> + <entry> + Maximum length of a string attribute + </entry> + </row> + </tbody> + </tgroup></table> +<!--h3--></section> + + + <section role="h3" id="QpidManagementFramework-Methods"><title> + Methods + </title> + + <para> + <method> tags must be placed within <schema> and + </schema> tags. + </para><para> + A <emphasis>method</emphasis> is an invokable function to be performed on + instances of the object class (i.e. a Remote Procedure Call). A + <method> tag has a name, an optional description, and + encloses zero or more arguments. Method arguments are defined by + the <arg> tag and have a name, a type, a direction, and an + optional description. The argument direction can be "I", "O", or + "IO" indicating input, output, and input/output respectively. An + example: + </para> + <programlisting> + <method name="echo" desc="Request a response to test the path to the management broker"> + <arg name="sequence" dir="IO" type="uint32"/> + <arg name="body" dir="IO" type="lstr"/> + </method> +</programlisting> +<!--h3--></section> + + <section role="h3" id="QpidManagementFramework-EventClasses"><title> + Event Classes + </title> + <para/> + <!--h3--></section> + + <section role="h3" id="QpidManagementFramework-DataTypes"><title> + Data Types + </title> + + <para> + Object attributes, method arguments, and event arguments have + data types. The data types are based on the rich data typing + system provided by the AMQP messaging protocol. The following + table describes the data types available for QMF: + </para><table><title>QMF Datatypes</title><tgroup cols="2"> + <tbody> + <row> + <entry> + QMF Type + </entry> + <entry> + Description + </entry> + </row> + <row> + <entry> + REF + </entry> + <entry> + QMF Object ID - Used to reference another QMF object. + </entry> + </row> + <row> + <entry> + U8 + </entry> + <entry> + 8-bit unsigned integer + </entry> + </row> + <row> + <entry> + U16 + </entry> + <entry> + 16-bit unsigned integer + </entry> + </row> + <row> + <entry> + U32 + </entry> + <entry> + 32-bit unsigned integer + </entry> + </row> + <row> + <entry> + U64 + </entry> + <entry> + 64-bit unsigned integer + </entry> + </row> + <row> + <entry> + S8 + </entry> + <entry> + 8-bit signed integer + </entry> + </row> + <row> + <entry> + S16 + </entry> + <entry> + 16-bit signed integer + </entry> + </row> + <row> + <entry> + S32 + </entry> + <entry> + 32-bit signed integer + </entry> + </row> + <row> + <entry> + S64 + </entry> + <entry> + 64-bit signed integer + </entry> + </row> + <row> + <entry> + BOOL + </entry> + <entry> + Boolean - True or False + </entry> + </row> + <row> + <entry> + SSTR + </entry> + <entry> + Short String - String of up to 255 bytes + </entry> + </row> + <row> + <entry> + LSTR + </entry> + <entry> + Long String - String of up to 65535 bytes + </entry> + </row> + <row> + <entry> + ABSTIME + </entry> + <entry> + Absolute time since the epoch in nanoseconds (64-bits) + </entry> + </row> + <row> + <entry> + DELTATIME + </entry> + <entry> + Delta time in nanoseconds (64-bits) + </entry> + </row> + <row> + <entry> + FLOAT + </entry> + <entry> + Single precision floating point number + </entry> + </row> + <row> + <entry> + DOUBLE + </entry> + <entry> + Double precision floating point number + </entry> + </row> + <row> + <entry> + UUID + </entry> + <entry> + UUID - 128 bits + </entry> + </row> + <row> + <entry> + FTABLE + </entry> + <entry> + Field-table - std::map in C++, dictionary in Python + </entry> + </row> + </tbody> + </tgroup></table><para> + In the XML schema definition, types go by different names and + there are a number of special cases. This is because the XML + schema is used in code-generation for the agent API. It provides + options that control what kind of accessors are generated for + attributes of different types. The following table enumerates the + types available in the XML format, which QMF types they map to, + and other special handling that occurs. + </para><table><title>XML Schema Mapping for QMF Types</title><tgroup cols="4"> + <tbody> + <row> + <entry> + XML Type + </entry> + <entry> + QMF Type + </entry> + <entry> + Accessor Style + </entry> + <entry> + Special Characteristics + </entry> + </row> + <row> + <entry> + objId + </entry> + <entry> + REF + </entry> + <entry> + Direct (get, set) + </entry> + <entry> + + </entry> + </row> + <row> + <entry> + uint8,16,32,64 + </entry> + <entry> + U8,16,32,64 + </entry> + <entry> + Direct (get, set) + </entry> + <entry> + + </entry> + </row> + <row> + <entry> + int8,16,32,64 + </entry> + <entry> + S8,16,32,64 + </entry> + <entry> + Direct (get, set) + </entry> + <entry> + + </entry> + </row> + <row> + <entry> + bool + </entry> + <entry> + BOOL + </entry> + <entry> + Direct (get, set) + </entry> + <entry> + + </entry> + </row> + <row> + <entry> + sstr + </entry> + <entry> + SSTR + </entry> + <entry> + Direct (get, set) + </entry> + <entry> + + </entry> + </row> + <row> + <entry> + lstr + </entry> + <entry> + LSTR + </entry> + <entry> + Direct (get, set) + </entry> + <entry> + + </entry> + </row> + <row> + <entry> + absTime + </entry> + <entry> + ABSTIME + </entry> + <entry> + Direct (get, set) + </entry> + <entry> + + </entry> + </row> + <row> + <entry> + deltaTime + </entry> + <entry> + DELTATIME + </entry> + <entry> + Direct (get, set) + </entry> + <entry> + + </entry> + </row> + <row> + <entry> + float + </entry> + <entry> + FLOAT + </entry> + <entry> + Direct (get, set) + </entry> + <entry> + + </entry> + </row> + <row> + <entry> + double + </entry> + <entry> + DOUBLE + </entry> + <entry> + Direct (get, set) + </entry> + <entry> + + </entry> + </row> + <row> + <entry> + uuid + </entry> + <entry> + UUID + </entry> + <entry> + Direct (get, set) + </entry> + <entry> + + </entry> + </row> + <row> + <entry> + map + </entry> + <entry> + FTABLE + </entry> + <entry> + Direct (get, set) + </entry> + <entry> + + </entry> + </row> + <row> + <entry> + hilo8,16,32,64 + </entry> + <entry> + U8,16,32,64 + </entry> + <entry> + Counter (inc, dec) + </entry> + <entry> + Generates value, valueMin, valueMax + </entry> + </row> + <row> + <entry> + count8,16,32,64 + </entry> + <entry> + U8,16,32,64 + </entry> + <entry> + Counter (inc, dec) + </entry> + <entry> + + </entry> + </row> + <row> + <entry> + mma32,64 + </entry> + <entry> + U32,64 + </entry> + <entry> + Direct + </entry> + <entry> + Generates valueMin, valueMax, valueAverage, valueSamples + </entry> + </row> + <row> + <entry> + mmaTime + </entry> + <entry> + DELTATIME + </entry> + <entry> + Direct + </entry> + <entry> + Generates valueMin, valueMax, valueAverage, valueSamples + </entry> + </row> + </tbody> + </tgroup></table> + + <note><title>Important</title> + <para> + When writing a schema using the XML format, types used in + <property> or <arg> must be types that have + <emphasis>Direct</emphasis> accessor style. Any type may be used in + <statistic> tags. + </para> + </note> + +<!--h3--></section> +<!--h2--></section> + + <section role="h2" id="QpidManagementFramework-ClassKeysandClassVersioning"><title> + Class + Keys and Class Versioning + </title> + + <para/> + +<!--h2--></section> + + <!--h1--></section> + + <section role="h1" id="QpidManagementFramework-TheQMFProtocol"><title> + The QMF + Protocol + </title> + + <para> + The QMF protocol defines the message formats and communication + patterns used by the different QMF components to communicate with + one another. + </para><para> + A description of the current version of the QMF protocol can be + found at <xref linkend="qpid_QMF-Protocol"/>. + </para><para> + A proposal for an updated protocol based on map-messages is in + progress and can be found at <xref linkend="qpid_QMF-Map-Message-Protocol"/>. + </para> +<!--h1--></section> + + <section role="h1" id="QpidManagementFramework-HowtoWriteaQMFConsole"><title> + How + to Write a QMF Console + </title> + + <para> + Please see the <xref linkend="qpid_QMF-Python-Console-Tutorial"/> for information about using the console API with + Python. + </para> +<!--h1--></section> + + <section role="h1" id="QpidManagementFramework-HowtoWriteaQMFAgent"><title> + How to + Write a QMF Agent + </title> + <para/> + <!--h1--></section> + + +</section> diff --git a/qpid/cpp/docs/book/src/cpp-broker/Running-CPP-Broker.xml b/qpid/cpp/docs/book/src/cpp-broker/Running-CPP-Broker.xml new file mode 100644 index 0000000000..7f76cbc354 --- /dev/null +++ b/qpid/cpp/docs/book/src/cpp-broker/Running-CPP-Broker.xml @@ -0,0 +1,846 @@ +<?xml version="1.0" encoding="utf-8"?> +<!-- + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +--> + +<section id="section-Running-a-Qpid-CPP-Broker"> + <title> + Running a Qpid C++ Broker + </title> + + <section role="h2" id="RASC-BuildingtheCppBrokerandClientLibraries"><title> + Building the + C++ Broker and Client Libraries + </title> + <para> + The root directory for the C++ distribution is named + qpidc-0.4. The README file in that directory gives + instructions for building the broker and client libraries. In + most cases you will do the following: + </para> + <programlisting> +[qpidc-0.4]$ ./configure +[qpidc-0.4]$ make +</programlisting> + <!--h2--></section> + <section role="h2" id="RASC-RunningtheCppBroker"><title> + Running the C++ Broker + </title> + <para> + Once you have built the broker and client libraries, you can + start the broker from the command line: + </para> + <programlisting> +[qpidc-0.4]$ src/qpidd +</programlisting> + <para> + Use the --daemon option to run the broker as a daemon + process: + </para> + <programlisting> +[qpidc-0.4]$ src/qpidd --daemon +</programlisting> + <para> + You can stop a running daemon with the --quit option: + </para> + <programlisting> +[qpidc-0.4]$ src/qpidd --quit +</programlisting> + <para> + You can see all available options with the --help option + </para> + <programlisting> +[qpidc-0.4]$ src/qpidd --help +</programlisting> + <!--h2--></section> + <section role="h2" id="RASC-Mostcommonquestionsgettingqpiddrunning"><title> + Most + common questions getting qpidd running + </title> + <section role="h3" id="RASC-Errorwhenstartingbroker-3A-22nodatadirectory-22"><title> + Error + when starting broker: "no data directory" + </title> + <para> + The C++ Broker requires you to set a data directory or specify + --no-data-dir (see help for more details). The data + directory is used for the journal, so it is important when + reliability counts. Make sure your process has write permission + to the data directory. + </para><para> + The default location is + </para> + <programlisting> +/lib/var/qpidd +</programlisting> + <para> + An alternate location can be set with --data-dir + </para> + <!--h3--></section> + <section role="h3" id="RASC-Errorwhenstartingbroker-3A-22thatprocessislocked-22"><title> + Error + when starting broker: "that process is locked" + </title> + <para> + Note that when qpidd starts it creates a lock file is data + directory are being used. If you have a un-controlled exit, + please mail + the trace from the core to the dev@qpid.apache.org mailing list. + To clear the lock run + </para> + <programlisting> +./qpidd -q +</programlisting> + <para> + It should also be noted that multiple brokers can be run on the + same host. To do so set alternate data directories for each qpidd + instance. + </para> + <!--h3--></section> + <section role="h3" id="RASC-Usingaconfigurationfile"><title> + Using a configuration + file + </title> + <para> + Each option that can be specified on the command line can also be + specified in a configuration file. To see available options, use + --help on the command line: + </para> + <programlisting> +./qpidd --help +</programlisting> + <para> + A configuration file uses name/value pairs, one on each line. To + convert a command line option to a configuration file entry: + </para><para> + a.) remove the '--' from the beginning of the option. + b.) place a '=' between the option and the value (use + <emphasis>yes</emphasis> or <emphasis>true</emphasis> to enable options that take no + value when specified on the command line). + c.) place one option per line. + </para><para> + For instance, the --daemon option takes no value, the + --log-to-syslog option takes the values yes or + no. The following configuration file sets these two + options: + </para> + <programlisting> +daemon=yes +log-to-syslog=yes +</programlisting> + <!--h3--></section> + <section role="h3" id="RASC-CanIuseanyLanguageclientwiththeCppBroker-3F"><title> + Can I use + any Language client with the C++ Broker? + </title> + <para> + Yes, all the clients work with the C++ broker; it is written in + C+<emphasis>, but uses the AMQP wire protocol. Any broker can be used + with any client that uses the same AMQP version. When running the + C</emphasis>+ broker, it is highly recommended to run AMQP 0-10. + </para><para> + Note that JMS also works with the C++ broker. + </para> + <!--h3--></section> + <!--h2--></section> + <section role="h2" id="RASC-Authentication"><title> + Authentication + </title> + <section role="h3" id="RASC-Linux"><title> + Linux + </title> + <para> + The PLAIN authentication is done on a username+password, which is + stored in the sasldb_path file. Usernames and passwords can be + added to the file using the command: + </para> + <programlisting> +saslpasswd2 -f /var/lib/qpidd/qpidd.sasldb -u <REALM> <USER> +</programlisting> + <para> + The REALM is important and should be the same as the + --auth-realm + option to the broker. This lets the broker properly find the user + in + the sasldb file. + </para><para> + Existing user accounts may be listed with: + </para> + <programlisting> +sasldblistusers2 -f /var/lib/qpidd/qpidd.sasldb +</programlisting> + <para> + NOTE: The sasldb file must be readable by the user running the + qpidd daemon, and should be readable only by that user. + </para> + <!--h3--></section> + <section role="h3" id="RASC-Windows"><title> + Windows + </title> + <para> + On Windows, the users are authenticated against the local + machine. You should add the appropriate users using the standard + Windows tools (Control Panel->User Accounts). To run many of + the examples, you will need to create a user "guest" with + password "guest". + </para><para> + If you cannot or do not want to create new users, you can run + without authentication by specifying the no-auth option to the + broker. + </para> + <!--h3--></section> + <!--h2--></section> + + <section role="h2" id="RASC-Slightlymorecomplexconfiguration"><title> + Slightly more + complex configuration + </title> + <para> + The easiest way to get a full listing of the broker's options are + to use the --help command, run it locally for the latest set of + options. These options can then be set in the conf file for + convenience (see above) + </para> + <programlisting> +./qpidd --help + +Usage: qpidd OPTIONS +Options: + -h [ --help ] Displays the help message + -v [ --version ] Displays version information + --config FILE (/etc/qpidd.conf) Reads configuration from FILE + +Module options: + --module-dir DIR (/usr/lib/qpidd) Load all .so modules in this directory + --load-module FILE Specifies additional module(s) to be loaded + --no-module-dir Don't load modules from module directory + +Broker Options: + --data-dir DIR (/var/lib/qpidd) Directory to contain persistent data generated by the broker + --no-data-dir Don't use a data directory. No persistent + configuration will be loaded or stored + -p [ --port ] PORT (5672) Tells the broker to listen on PORT + --worker-threads N (3) Sets the broker thread pool size + --max-connections N (500) Sets the maximum allowed connections + --connection-backlog N (10) Sets the connection backlog limit for the + server socket + --staging-threshold N (5000000) Stages messages over N bytes to disk + -m [ --mgmt-enable ] yes|no (1) Enable Management + --mgmt-pub-interval SECONDS (10) Management Publish Interval + --ack N (0) Send session.ack/solicit-ack at least every + N frames. 0 disables voluntary ack/solitict + -ack + +Daemon options: + -d [ --daemon ] Run as a daemon. + -w [ --wait ] SECONDS (10) Sets the maximum wait time to initialize the + daemon. If the daemon fails to initialize, prints + an error and returns 1 + -c [ --check ] Prints the daemon's process ID to stdout and + returns 0 if the daemon is running, otherwise + returns 1 + -q [ --quit ] Tells the daemon to shut down +Logging options: + -t [ --trace ] Enables all logging + --log-enable RULE (notice+) Enables logging for selected levels and components. + RULE is in the form 'LEVEL[+-][:PATTERN]' + LEVEL is one of: + trace debug info notice warning error critical + PATTERN is a logging category name, or a namespace-qualified + function name or name fragment. + Logging category names are: + Security Broker Management Protocol System HA Messaging Store + Network Test Client Model Unspecified + + For example: + '--log-enable warning+' + logs all warning, error and critical messages. + + '--log-enable trace+:Broker' + logs all category 'Broker' messages. + + '--log-enable debug:framing' + logs debug messages from all functions with 'framing' in + the namespace or function name. + + This option can be used multiple times + + --log-disable RULE Disables logging for selected levels and components. + RULE is in the form 'LEVEL[+-][:PATTERN]' + LEVEL is one of: + trace debug info notice warning error critical + PATTERN is a logging category name, or a namespace-qualified + function name or name fragment. + Logging category names are: + Security Broker Management Protocol System HA Messaging Store + Network Test Client Model Unspecified + + For example: + '--log-disable warning-' + disables logging all warning, notice, info, debug, and + trace messages. + + '--log-disable trace:Broker' + disables all category 'Broker' trace messages. + + '--log-disable debug-:qmf::' + disables logging debug and trace messages from all functions + with 'qmf::' in the namespace. + + This option can be used multiple times + + --log-time yes|no (1) Include time in log messages + --log-level yes|no (1) Include severity level in log messages + --log-source yes|no (0) Include source file:line in log + messages + --log-thread yes|no (0) Include thread ID in log messages + --log-function yes|no (0) Include function signature in log + messages + --log-hires-timestamp yes|no (0) Use hi-resolution timestamps in log + messages + --log-category yes|no (1) Include category in log messages + --log-prefix STRING Prefix to prepend to all log messages + +Logging sink options: + --log-to-stderr yes|no (1) Send logging output to stderr + --log-to-stdout yes|no (0) Send logging output to stdout + --log-to-file FILE Send log output to FILE. + --log-to-syslog yes|no (0) Send logging output to syslog; + customize using --syslog-name and + --syslog-facility + --syslog-name NAME (qpidd) Name to use in syslog messages + --syslog-facility LOG_XXX (LOG_DAEMON) + Facility to use in syslog messages + +</programlisting> + <!--h2--></section> + <section role="h2" id="RASC-Loadingextramodules"><title> + Loading extra modules + </title> + <para> + By default the broker will load all the modules in the module + directory, however it will NOT display options for modules that + are not loaded. So to see the options for extra modules loaded + you need to load the module and then add the help command like + this: + </para> + <programlisting> +./qpidd --load-module libbdbstore.so --help +Usage: qpidd OPTIONS +Options: + -h [ --help ] Displays the help message + -v [ --version ] Displays version information + --config FILE (/etc/qpidd.conf) Reads configuration from FILE + + + / .... non module options would be here ... / + + +Store Options: + --store-directory DIR Store directory location for persistence (overrides + --data-dir) + --store-async yes|no (1) Use async persistence storage - if store supports + it, enables AIO O_DIRECT. + --store-force yes|no (0) Force changing modes of store, will delete all + existing data if mode is changed. Be SURE you want + to do this! + --num-jfiles N (8) Number of files in persistence journal + --jfile-size-pgs N (24) Size of each journal file in multiples of read + pages (1 read page = 64kiB) +</programlisting> +<!--h2--></section> +<section role="h2" id="RASC-message-timestamps"> + <title>Timestamping Received Messages</title> + <para> + The AMQP 0-10 specification defines a <emphasis>timestamp</emphasis> message delivery + property. The timestamp delivery property is a <emphasis>datetime</emphasis> value + that is written to each message that arrives at the broker. See the description of + "message.delivery-properties" in the "Command Classes" section of the AMQP 0-10 + specification for more detail. + </para> + <para> + See the <emphasis>Programming in Apache Qpid</emphasis> documentation for + information regarding how clients may access the timestamp value in received + messages. + </para> + <para> + By default, this timestamping feature is disabled. To enable timestamping, use the + <emphasis>enable-timestamp</emphasis> broker configuration option. Setting the + enable-timestamp option to 'yes' will enable message timestamping: + </para> + <programlisting> +./qpidd --enable-timestamp yes + </programlisting> + <para> + Message timestamping can also be enabled (and disabled) without restarting the broker. + The QMF Broker management object defines two methods for accessing the timestamp + configuration: + </para> + <table> + <title>QMF Management - Broker Methods for Managing the Timestamp Configuration</title> + <tgroup cols="2"> + <thead> + <row> + <entry>Method</entry> + <entry>Description</entry> + </row> + </thead> + <tbody> + <row> + <entry>getTimestampConfig</entry> + <entry>Get the message timestamping configuration. Returns True if received messages are timestamped.</entry> + </row> + <row> + <entry>setTimestampConfig</entry> + <entry>Set the message timestamping configuration. Set True to enable timestamping received messages, False to disable timestamping.</entry> + </row> + </tbody> + </tgroup> + </table> + <example> + <title>Enabling Message Timestamping via QMF - Python</title> + <para> + The following code fragment uses these QMF method calls to enable message timestamping. + </para> + <programlisting lang="python"> +# get the state of the timestamp configuration +broker = self.qmf.getObjects(_class="broker")[0] +rc = broker.getTimestampConfig() +self.assertEqual(rc.status, 0) +self.assertEqual(rc.text, "OK") +print("The timestamp setting is %s" % str(rc.receive)) + +# try to enable it +rc = broker.setTimestampConfig(True) +self.assertEqual(rc.status, 0) +self.assertEqual(rc.text, "OK") + </programlisting> + </example> +<!--h2--></section> +<section role="h2" id="RASC-logging-options"> + <title>Logging Options</title> + <para> + The C++ Broker provides a rich set of logging options. To use logging effectively + a user must select a useful set of options to expose the log messages of interest. + This section introduces the logging options and how they are used in practice. + </para> + <section role="h3" id="RASC-LogConcepts"> + <title>Logging Concepts</title> + + <section role="h4" id="RASC-LogConcept-level"> + <title>Log Level</title> + <para> + The C++ Broker has a traditional set of log severity levels. The log levels + range from low frequency and high importance critical level + to high frequency and low importance trace level. + </para> + <table> + <title>C++ Broker Log Severity Levels</title> + <tgroup cols="2"> + <thead> + <row> + <entry>Name</entry> + <entry>Level</entry> + </row> + </thead> + <tbody> + <row><entry>critical</entry><entry>high</entry></row> + <row><entry>error</entry> <entry></entry> </row> + <row><entry>warning</entry> <entry></entry> </row> + <row><entry>notice</entry> <entry></entry> </row> + <row><entry>info</entry> <entry></entry> </row> + <row><entry>debug</entry> <entry></entry> </row> + <row><entry>trace</entry> <entry>low</entry> </row> + </tbody> + </tgroup> + </table> + <!--h4--></section> + + <section role="h4" id="RASC-LogConcept-category"> + <title>Log Category</title> + <para> + The C++ Broker groups log messages into categories. The log category + name may then be used to enable and disable groups of related messages + at varying log levels. + </para> + <table> + <title>C++ Broker Log Categories</title> + <tgroup cols="1"> + <thead> + <row> + <entry>Name</entry> + </row> + </thead> + <tbody> + <row><entry>Security</entry></row> + <row><entry>Broker</entry></row> + <row><entry>Management</entry></row> + <row><entry>Protocol</entry></row> + <row><entry>System</entry></row> + <row><entry>HA</entry></row> + <row><entry>Messaging</entry></row> + <row><entry>Store</entry></row> + <row><entry>Network</entry></row> + <row><entry>Test</entry></row> + <row><entry>Client</entry></row> + <row><entry>Model</entry></row> + <row><entry>Unspecified</entry></row> + </tbody> + </tgroup> + </table> + <para> + Generally speaking the log categories are groupings of messages from files + related by + thier placement in the source code directory structure. The + <emphasis>Model</emphasis> category is an exception. Debug log entries + identified by the Model category expose the creation, deletion, and usage + statistics for managed objects in the broker. Log messages in the Model + category are emitted by source files scattered throughout the source tree. + </para> + <!--h4--></section> + + <section role="h4" id="RASC-LogConcept-StatementAttributes"> + <title>Log Statement Attributes</title> + <para> + Every log statement in the C++ Broker has fixed attributes that may be + used in enabling or disabling log messages. + </para> + <table> + <title>C++ Broker Log Statement Attributes</title> + <tgroup cols="2"> + <thead> + <row> + <entry>Name</entry> + <entry>Description</entry> + </row> + </thead> + <tbody> + <row> + <entry>Level</entry> + <entry>Severity level</entry> + </row> + <row> + <entry>Category</entry> + <entry>Category</entry> + </row> + <row> + <entry>Function</entry> + <entry>Namespace-qualified source function name</entry> + </row> + </tbody> + </tgroup> + </table> + + <!--h4--></section> + + <!--h3--></section> + <section role="h3" id="RASC-LogRules-EnableDisable"> + <title>Enabling and Disabling Log Messages</title> + <para> + The Qpid C++ Broker has hundreds of log message statements in the source + code. Under typical conditions + most of the messages are deselected and never emitted as actual logs. + However, under some circumstances debug and trace messages must be enabled + to analyze broker behavior. This section discusses how the broker enables + and disables log messages. + </para> + <para> + At startup the broker processes command line and option file '--log-enable RULE' and + '--log-disable RULE' options using the following rule format: + </para> + <programlisting> + LEVEL[+-][:PATTERN} + </programlisting> + <table> + <title>C++ Broker Log Enable/Disable RULE Format</title> + <tgroup cols="2"> + <thead> + <row> + <entry>Name</entry> + <entry>Description</entry> + </row> + </thead> + <tbody> + <row> + <entry>LEVEL</entry> + <entry>Severity level</entry> + </row> + <row> + <entry>[+-]</entry> + <entry> + Option level modifiers. <emphasis>'+'</emphasis> indicates + <emphasis>this level and above</emphasis>. + <emphasis>'-'</emphasis> indicates <emphasis>this level and below</emphasis>. + </entry> + </row> + <row> + <entry>[:PATTERN]</entry> + <entry> + If PATTERN matches a Category name then the log option applies only + to log messages with the named category. Otherwise, the pattern is stored + as a function name match string. + </entry> + </row> + </tbody> + </tgroup> + </table> + <para> + As the options are procesed the results are aggregated into two pairs of tables. + </para> + <table> + <title>C++ Broker Log Enable/Disable Settings Tables</title> + <tgroup cols="2"> + <thead> + <row> + <entry>Name</entry> + <entry>Description</entry> + </row> + </thead> + <tbody> + <row> + <entry>Function Table</entry> + <entry> + A set of vectors of accumulated function name patterns. + There is a separate vector of name patterns for each log level. + </entry> + </row> + <row> + <entry>Category Table</entry> + <entry> + A simple two dimensional array of boolean values indexed by + [Level][Category] indicating + if all log statements are enabled for the Level and Category pair. + </entry> + </row> + </tbody> + </tgroup> + </table> + <para> + --log-enable statements and --log-disable statements are aggregated into dedicated + Function and Category tables. With this scheme multiple conflicting log enable and + disable commands may be processed in any order yet produce consistent patterns + of enabled broker log statements. + </para> + <!--h3--></section> + + <section role="h3" id="RASC-LogRules-RuleMatching"> + <title>Determining if a Log Statement is Enabled</title> + <para> + Function Table Lookups are simple string pattern matches where the searchable + text is the domain-name qualified function name from the log statement and the + search pattern is the set of Function Table entries for a given log level. + </para> + <para> + Category Table Lookups are boolean array queries where the Level and Category + indexes are from the log statement. + </para> + <para> + Each log statment sends its Level, Category, and FunctionName to the + Logger for evaluation. As a result the log statement is either visible or hidden. + </para> + <table> + <title>C++ Broker Log Statement Visibility Determination</title> + <tgroup cols="2"> + <thead> + <row> + <entry>Test</entry> + <entry>Description</entry> + </row> + </thead> + <tbody> + <row> + <entry>Disabled Function</entry> + <entry> + If the statement matches a Disabled Function pattern then the + statement is hidden. + </entry> + </row> + <row> + <entry>Disabled Category</entry> + <entry> + If the Disabled Category table for this [Level][Category] is true then the + statement is hidden. + </entry> + </row> + <row> + <entry>Enabled Function</entry> + <entry> + If the statement matches a Enabled Function pattern then the + statement is visible. + </entry> + </row> + <row> + <entry>Enabled Category</entry> + <entry> + If the Enabled Category table for this [Level][Category] is true then the + statement is visible. + </entry> + </row> + <row> + <entry>Unreferenced</entry> + <entry> + Log statements that are unreferenced by specific enable rules are by + default hidden. + </entry> + </row> + </tbody> + </tgroup> + </table> + <!--h3--></section> + + <section role="h3" id="RASC-LogRules-Reenabling"> + <title>Changing Log Enable/Disable Settings at Run Time</title> + <para> + The C++ Broker provides QMF management methods that allow users to query and to set + the log enable and disable settings while the broker is running. + </para> + <table> + <title>QMF Management - Broker Methods for Managing the Log Enable/Disable Settings</title> + <tgroup cols="2"> + <thead> + <row> + <entry>Method</entry> + <entry>Description</entry> + </row> + </thead> + <tbody> + <row> + <entry>getLogLevel</entry> + <entry>Get the log enable/disable settings.</entry> + </row> + <row> + <entry>setLogLevel</entry> + <entry>Set the log enable/disable settings.</entry> + </row> + </tbody> + </tgroup> + </table> + <para> + The management methods use a RULE format similar to the option RULE format: + </para> + <programlisting> + [!]LEVEL[+-][:PATTERN] + </programlisting> + <para> + The difference is the leading exclamation point that identifies disable rules. + </para> + <example> + <title> + Querying Log Settings via qpid-ctrl utility + </title> + <para> + At start up a C++ Broker may have the following options: + </para> + <programlisting> + --log-enable debug+ + --log-enable trace+:Protocol + --log-disable info-:Management + </programlisting> + <para> + The following command: + </para> + <programlisting> + qpid-ctrl getLogLevel + </programlisting> + <para> + will return the following result: + </para> + <programlisting> + level=debug+,trace+:Protocol,!info-:Management + </programlisting> + </example> + <example> + <title> + Setting Log Settings via qpid-ctrl utility + </title> + <para> + New broker log options may be set at any time using qpid-ctrl + </para> + <programlisting> + qpid-ctrl setLogLevel level='debug+:Broker !debug-:broker::Broker::ManagementMethod' + </programlisting> + </example> + <!--h3--></section> + + <section role="h3" id="RASC-LogRules-Explorer"> + <title>Discovering Log Sources</title> + <para> + A common condition for a user is being swamped by log messages that are not + interesting for some debug situation. Conversely, a particular log entry + may be of interest all the time but enabling all log levels just to see a + single log entry is too much. How can a user find and specify a pattern + to single out logs of interest? + </para> + <para> + The easiest way to hide messages it to disable logs at log level and + category combinations. This may not always work since using only these + coarse controls the log messages of interest may also be hidden. + To discover a more precise filter to specify the messages you want + to show or to hide you may temporarily enable the + <emphasis>"--log-function=yes"</emphasis> option. + The following log entries show a typical log message without and + with the log function names enabled: + </para> + <programlisting> + 2013-05-01 11:16:01 [Broker] notice Broker running + 2013-05-01 11:16:54 [Broker] notice qpid::broker::Broker::run: Broker running + </programlisting> + <para> + This log entry is emitted by function <emphasis>qpid::broker::Broker::run</emphasis> + and this is the function name pattern to be used in specific log enable and + disable rules. + For example, this log entry could be disabled with any of the following: + </para> + <programlisting> + --log-disable notice [1] + --log-disable notice:qpid:: [2] + --log-disable notice:Broker [3] + --log-disable notice-:Broker::run [4] + --log-disable notice:qpid::broker::Broker::run [5] + </programlisting> + <itemizedlist> + <listitem> + [1] Disables all messages at notice level. + </listitem> + <listitem> + [2] Disables all messages at notice level in qpid:: name space. This is + very broad and disables many log messages. + </listitem> + <listitem> + [3] Disables the category <emphasis>[Broker]</emphasis> and is not specific + to the function. Category names supercede function name fragments in + log option processing + </listitem> + <listitem> + [4] Disables the function. + </listitem> + <listitem> + [5] Disables the function. + </listitem> + </itemizedlist> + <para> + Remember that the log filter matching PATTERN strings are matched against the + domain-name qualified function names associated with the log statement + and not against the log message text itself. That is, in the previous example + log filters cannot be set on the log text <emphasis>Broker running</emphasis> + </para> + <!--h3--></section> +<!--h2--></section> +</section> diff --git a/qpid/cpp/docs/book/src/cpp-broker/Security.xml b/qpid/cpp/docs/book/src/cpp-broker/Security.xml new file mode 100644 index 0000000000..8f42e6abdf --- /dev/null +++ b/qpid/cpp/docs/book/src/cpp-broker/Security.xml @@ -0,0 +1,2516 @@ +<?xml version='1.0' encoding='utf-8' ?> +<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"> +<!-- + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +--> +<section id="chap-Messaging_User_Guide-Security"> + <title>Security</title> + <para> + This chapter describes how authentication, rule-based authorization, encryption, and digital signing can be accomplished using Qpid. Authentication is the process of verifying the identity of a user; in Qpid, this is done using the SASL framework. Rule-based authorization is a mechanism for specifying the actions that each user is allowed to perform; in Qpid, this is done using an Access Control List (ACL) that is part of the Qpid broker. Encryption is used to ensure that data is not transferred in a plain-text format that could be intercepted and read. Digital signatures provide proof that a given message was sent by a known sender. Encryption and signing are done using SSL (they can also be done using SASL, but SSL provides stronger encryption). + </para> + <section id="sect-Messaging_User_Guide-Security-User_Authentication"> + <title>User Authentication</title> + <para> + AMQP uses Simple Authentication and Security Layer (SASL) to authenticate client connections to the broker. SASL is a framework that supports a variety of authentication methods. For secure applications, we suggest <command>CRAM-MD5</command>, <command>DIGEST-MD5</command>, or <command>GSSAPI</command>. The <command>ANONYMOUS</command> method is not secure. The <command>PLAIN</command> method is secure only when used together with SSL. + </para> + <para> + Both the Qpid broker and Qpid clients use the <ulink url="http://cyrusimap.web.cmu.edu/">Cyrus SASL library</ulink>, a full-featured authentication framework, which offers many configuration options. This section shows how to configure users for authentication with SASL, which is sufficient when using <command>SASL PLAIN</command>. If you are not using SSL, you should configure SASL to use <command>CRAM-MD5</command>, <command>DIGEST-MD5</command>, or <command>GSSAPI</command> (which provides Kerberos authentication). For information on configuring these and other options in SASL, see the Cyrus SASL documentation<!-- at <filename>/usr/share/doc/cyrus-sasl-lib-2.1.22/index.html</filename> for &RHEL5; or <filename>/usr/share/doc/cyrus-sasl-2.1.19/index.html</filename> for &RHEL4;-->. + </para> + <important> + <title>Important</title> + <para> + The <command>SASL PLAIN</command> method sends passwords in cleartext, and is vulnerable to man-in-the-middle attacks unless SSL (Secure Socket Layer) is also used (see <xref linkend="sect-Messaging_User_Guide-Security-Encryption_using_SSL" />). + </para> + <para> + If you are not using SSL, we recommend that you disable <command>PLAIN</command> authentication in the broker. + </para> + + </important> + <para> + The Qpid broker uses the <command>auth yes|no</command> option to determine whether to use SASL authentication. Turn on authentication by setting <command>auth</command> to <command>yes</command> in <filename>/etc/qpidd.conf</filename>: + </para> + +<programlisting> +# /etc/qpidd.conf +# +# Set auth to 'yes' or 'no' + +auth=yes +</programlisting> + <section id="sect-Messaging_User_Guide-User_Authentication-Configuring_SASL"> + <title>Configuring SASL</title> + <para> + On Linux systems, the SASL configuration file is generally found in <filename>/etc/sasl2/qpidd.conf</filename> <!-- for &RHEL5; and-->or <filename>/usr/lib/sasl2/qpidd.conf</filename><!-- for &RHEL4;-->. + </para> + <para> + The SASL database contains user names and passwords for SASL. In SASL, a user may be associated with a <firstterm>realm</firstterm>. The Qpid broker authenticates users in the <command>QPID</command> realm by default, but it can be set to a different realm using the <command>realm</command> option: + </para> + +<programlisting> +# /etc/qpidd.conf +# +# Set the SASL realm using 'realm=' + +auth=yes +realm=QPID +</programlisting> + <para> + The SASL database is installed at <filename>/var/lib/qpidd/qpidd.sasldb</filename>; initially, it has one user named <command>guest</command> in the <command>QPID</command> realm, and the password for this user is <command>guest</command>. + </para> + <note> + <title>Note</title> + <para> + The user database is readable only by the <systemitem class="username">qpidd</systemitem> user. When run as a daemon, Qpid always runs as the <systemitem class="username">qpidd</systemitem> user. If you start the broker from a user other than the <systemitem class="username">qpidd</systemitem> user, you will need to either reconfigure SASL or turn authentication off. + </para> + + </note> + <important> + <title>Important</title> + <para> + The SASL database stores user names and passwords in plain text. If it is compromised so are all of the passwords that it stores. This is the reason that the <systemitem class="username">qpidd</systemitem> user is the only user that can read the database. If you modify permissions, be careful not to expose the SASL database. + </para> + + </important> + <para> + Add new users to the database by using the <command>saslpasswd2</command> command, which specifies a realm and a user ID. A user ID takes the form <command><replaceable>user-id</replaceable>@<replaceable>domain</replaceable>.</command>. + </para> + +<screen># saslpasswd2 -f /var/lib/qpidd/qpidd.sasldb -u <replaceable>realm</replaceable> <replaceable>new_user_name</replaceable></screen> + <para> + To list the users in the SASL database, use <command>sasldblistusers2</command>: + </para> + +<screen># sasldblistusers2 -f /var/lib/qpidd/qpidd.sasldb +</screen> + <para> + If you are using <command>PLAIN</command> authentication, users who are in the database can now connect with their user name and password. This is secure only if you are using SSL. If you are using a more secure form of authentication, please consult your SASL documentation for information on configuring the options you need. + </para> + + </section> + + <section id="sect-Messaging_User_Guide-User_Authentication-Kerberos"> + <title>Kerberos</title> + <para> + Both the Qpid broker and Qpid users are 'principals' of the Kerberos server, which means that they are both clients of the Kerberos authentication services. + </para> + <para> + To use Kerberos, both the Qpid broker and each Qpid user must be authenticated on the Kerberos server: + </para> + <procedure> + <step> + <para> + Install the Kerberos workstation software and Cyrus SASL GSSAPI on each machine that runs a qpidd broker or a qpidd messaging client: + </para> + +<screen>$ sudo yum install cyrus-sasl-gssapi krb5-workstation</screen> + + </step> + <step> + <para> + Make sure that the Qpid broker is registered in the Kerberos database. + </para> + <para> + Traditionally, a Kerberos principal is divided into three parts: the primary, the instance, and the realm. A typical Kerberos V5 has the format <literal>primary/instance@REALM</literal>. For a Qpid broker, the primary is <literal>qpidd</literal>, the instance is the fully qualified domain name, which you can obtain using <command>hostname --fqdn</command>, and the REALM is the Kerberos domain realm. By default, this realm is <literal>QPID</literal>, but a different realm can be specified in qpid.conf, e.g.: +<screen>realm=EXAMPLE.COM</screen> + + </para> + <para> + For instance, if the fully qualified domain name is <literal>dublduck.example.com</literal> and the Kerberos domain realm is <literal>EXAMPLE.COM</literal>, then the principal name is <literal>qpidd/dublduck.example.com@EXAMPLE.COM</literal>. + </para> + <para> + The following script creates a principal for qpidd: + </para> + +<programlisting> +FDQN=`hostname --fqdn` +REALM="EXAMPLE.COM" +kadmin -r $REALM -q "addprinc -randkey -clearpolicy qpidd/$FQDN" +</programlisting> + <para> + Now create a Kerberos keytab file for the Qpid broker. The Qpid broker must have read access to the keytab file. The following script creates a keytab file and allows the broker read access: + </para> + +<programlisting> +QPIDD_GROUP="qpidd" +kadmin -r $REALM -q "ktadd -k /etc/qpidd.keytab qpidd/$FQDN@$REALM" +chmod g+r /etc/qpidd.keytab +chgrp $QPIDD_GROUP /etc/qpidd.keytab +</programlisting> + <para> + The default location for the keytab file is <filename>/etc/krb5.keytab</filename>. If a different keytab file is used, the KRB5_KTNAME environment variable must contain the name of the file, e.g.: + </para> + +<programlisting> +export KRB5_KTNAME=/etc/qpidd.keytab +</programlisting> + <para> + If this is correctly configured, you can now enable kerberos support on the Qpid broker by setting the <varname>auth</varname> and <varname>realm</varname> options in <filename>/etc/qpidd.conf</filename>: + </para> + +<programlisting> +# /etc/qpidd.conf +auth=yes +realm=EXAMPLE.COM +</programlisting> + <para> + Restart the broker to activate these settings. + </para> + + </step> + <step> + <para> + Make sure that each Qpid user is registered in the Kerberos database, and that Kerberos is correctly configured on the client machine. The Qpid user is the account from which a Qpid messaging client is run. If it is correctly configured, the following command should succeed: + </para> + +<screen>$ kinit user@REALM.COM</screen> + + </step> + + </procedure> + + <para> + Java JMS clients require a few additional steps. + </para> + <procedure> + <step> + <para> + The Java JVM must be run with the following arguments: + </para> + <variablelist> + <varlistentry> + <term>-Djavax.security.auth.useSubjectCredsOnly=false</term> + <listitem> + <para> + Forces the SASL GASSPI client to obtain the kerberos credentials explicitly instead of obtaining from the "subject" that owns the current thread. + </para> + + </listitem> + + </varlistentry> + <varlistentry> + <term>-Djava.security.auth.login.config=myjas.conf</term> + <listitem> + <para> + Specifies the jass configuration file. Here is a sample JASS configuration file: + </para> + +<programlisting> +com.sun.security.jgss.initiate { + com.sun.security.auth.module.Krb5LoginModule required useTicketCache=true; +}; +</programlisting> + + </listitem> + + </varlistentry> + <varlistentry> + <term>-Dsun.security.krb5.debug=true</term> + <listitem> + <para> + Enables detailed debug info for troubleshooting + </para> + + </listitem> + + </varlistentry> + + </variablelist> + + </step> + <step> + <para> + The client's Connection URL must specify the following Kerberos-specific broker properties: + </para> + <itemizedlist> + <listitem> + <para> + <varname>sasl_mechs</varname> must be set to <literal>GSSAPI</literal>. + </para> + + </listitem> + <listitem> + <para> + <varname>sasl_protocol</varname> must be set to the principal for the qpidd broker, e.g. <literal>qpidd</literal>/ + </para> + + </listitem> + <listitem> + <para> + <varname>sasl_server</varname> must be set to the host for the SASL server, e.g. <literal>sasl.com</literal>. + </para> + + </listitem> + + </itemizedlist> + <para> + Here is a sample connection URL for a Kerberos connection: + </para> + +<screen>amqp://guest@clientid/testpath?brokerlist='tcp://localhost:5672?sasl_mechs='GSSAPI'&sasl_protocol='qpidd'&sasl_server='<server-host-name>''</screen> + + </step> + + </procedure> +<!-- + <para> + Please refer to the following documentation for more detail on using Kerberos: + </para> + <variablelist> + <varlistentry> + <term>RHEL5</term> + <listitem> + <para> + <ulink url="http://www.redhat.com/docs/manuals/enterprise/RHEL-5-manual/Deployment_Guide-en-US/ch-kerberos.html"> Red Hat Enterprise Linux 5: Deployment Guide </ulink> + </para> + + </listitem> + + </varlistentry> + <varlistentry> + <term>RHEL4</term> + <listitem> + <para> + <ulink url="http://www.redhat.com/docs/manuals/enterprise/RHEL-4-Manual/ref-guide/ch-kerberos.html"> Red Hat Enterprise Linux 4: Reference Guide </ulink> + </para> + + </listitem> + + </varlistentry> + <varlistentry> + <term>Java</term> + <listitem> + <para> + <ulink url="http://java.sun.com/j2se/1.5.0/docs/guide/security/jgss/tutorials/index.html"> Introduction to JAAS and Java GSS-API Tutorials </ulink> + </para> + + </listitem> + + </varlistentry> + + </variablelist> + --> + + </section> + + + </section> + + <!-- ################################################### --> <section id="sect-Messaging_User_Guide-Security-Authorization"> + <title>Authorization</title> + <para> + In Qpid, Authorization specifies which actions can be performed by each authenticated user using an Access Control List (ACL). + </para> + <para> + Use the <command>--acl-file</command> command to load the access control list. The filename should have a <filename>.acl</filename> extension: + </para> + +<screen> + $ qpidd --acl-file <replaceable>./aclfilename.acl</replaceable></screen> + <para> + Each line in an ACL file grants or denies specific rights to a user. If the last line in an ACL file is <literal>acl deny all all</literal>, the ACL uses <firstterm>deny mode</firstterm>, and only those rights that are explicitly allowed are granted: + </para> + +<programlisting> + acl allow rajith@QPID all all + acl deny all all +</programlisting> + <para> + On this server, <literal>rajith@QPID</literal> can perform any action, but nobody else can. Deny mode is the default, so the previous example is equivalent to the following ACL file: + </para> + +<programlisting> + acl allow rajith@QPID all all +</programlisting> + <para> + Alternatively the ACL file may use <firstterm>allow mode</firstterm> by placing: + </para> +<programlisting> + acl allow all all +</programlisting> + <para> + as the final line in the ACL file. In <emphasis>allow mode</emphasis> all actions by all users are allowed unless otherwise denied by specific ACL rules. + The ACL rule which selects <emphasis>deny mode</emphasis> or <emphasis>allow mode</emphasis> must be the last line in the ACL rule file. + </para> + <para> + ACL syntax allows fine-grained access rights for specific actions: + </para> + +<programlisting> + acl allow carlt@QPID create exchange name=carl.* + acl allow fred@QPID create all + acl allow all consume queue + acl allow all bind exchange + acl deny all all +</programlisting> + <para> + An ACL file can define user groups, and assign permissions to them: + </para> + +<programlisting> + group admin ted@QPID martin@QPID + acl allow admin create all + acl deny all all +</programlisting> + <para> + An ACL file can define per user connection and queue quotas: + </para> + +<programlisting> + group admin ted@QPID martin@QPID + group blacklist usera@qpid userb@qpid + quota connections 10 admin + quota connections 5 all + quota connections 0 blacklist + quota queues 50 admin + quota queues 5 all + quota queues 1 test@qpid +</programlisting> + + <para> + Performance Note: Most ACL queries are performed infrequently. The overhead associated with + ACL passing an allow or deny decision on the creation of a queue is negligible + compared to actually creating and using the queue. One notable exception is the <command>publish exchange</command> + query. ACL files with no <emphasis>publish exchange</emphasis> rules are noted and the broker short circuits the logic + associated with the per-messsage <emphasis>publish exchange</emphasis> ACL query. + However, if an ACL file has any <emphasis>publish exchange</emphasis> rules + then the broker is required to perform a <emphasis>publish exchange</emphasis> query for each message published. + Users with performance critical applications are encouraged to structure exchanges, queues, and bindings so that + the <emphasis>publish exchange</emphasis> ACL rules are unnecessary. + </para> + + <!-- ######## --> <section id="sect-Messaging_User_Guide-Authorization-ACL_Syntax"> + <title>ACL Syntax</title> + <para> + ACL rules follow this syntax: +<programlisting><![CDATA[ +aclline = ( comment | aclspec | groupspec | quotaspec ) + +comment = "#" [ STRING ] + +aclspec = "acl" permission ( groupname | name | "all" ) + ( action | "all" ) [ ( object | "all ) [ ( property "=" STRING )* ] ] + +groupspec = "group" groupname ( name )* [ "\" ] + +groupcontinuation = ( name )* [ "\" ] + +quotaspec = "quota" ( "connections" | "queues" ) NUMBER ( groupname | name | "all" )* + +name = ( ALPHANUMERIC | "-" | "_" | "." | "@" | "/" ) [ ( ALPHANUMERIC | "-" | "_" | "." | "@" | "/" )* ] + +groupname = ( ALPHANUMERIC | "-" | "_" ) [ ( ALPHANUMERIC | "-" | "_" )* ] + +permission = "allow" | "allow-log" | "deny" | "deny-log" + +action = "consume" | "publish" | "create" | "access" | + "bind" | "unbind" | "delete" | "purge" | + "update" + +object = "queue" | "exchange" | "broker" | "link" | + "method" | "query" | "connection" + +property = "name" | "durable" | "routingkey" | "autodelete" | + "exclusive" | "type" | "alternate" | "queuename" | + "exchangename" | "schemapackage" | "schemaclass" | + "policytype" | "paging" | + "queuemaxsizelowerlimit" | "queuemaxsizeupperlimit" | + "queuemaxcountlowerlimit" | "queuemaxcountupperlimit" | + "filemaxsizelowerlimit" | "filemaxsizeupperlimit" | + "filemaxcountlowerlimit" | "filemaxcountupperlimit" | + "pageslowerlimit" | "pagesupperlimit" | + "pagefactorlowerlimit" | "pagefactorupperlimit" +]]></programlisting> + + ACL rules can also include a single object name (or the keyword <parameter>all</parameter>) and one or more property name value pairs in the form <command>property=value</command> + </para> + <para> + The following tables show the possible values for <command>permission</command>, <command>action</command>, <command>object</command>, and <command>property</command> in an ACL rules file. + </para> + <table id="tabl-Messaging_User_Guide-ACL_Syntax-ACL_Rules_permission"> + <title>ACL Rules: permission</title> + <tgroup cols="2"> + <tbody> + <row> + <entry> + <command>allow</command> + </entry> + <entry> + <para> + Allow the action <!-- ### rule => the action --> + </para> + </entry> + </row> + <row> + <entry> + <command>allow-log</command> + </entry> + <entry> + <para> + Allow the action and log the action in the event log + </para> + </entry> + </row> + <row> + <entry> + <command>deny</command> + </entry> + <entry> + <para> + Deny the action + </para> + </entry> + </row> + <row> + <entry> + <command>deny-log</command> + </entry> + <entry> + <para> + Deny the action and log the action in the event log + </para> + </entry> + </row> + </tbody> + </tgroup> + </table> + <!-- Actions --> <table id="tabl-Messaging_User_Guide-ACL_Syntax-ACL_Rulesaction"> + <title>ACL Rules: action</title> + <tgroup cols="2"> + <tbody> + <row> + <entry> + <command>access</command> + </entry> + <entry> + <para> + Accessing or reading an object + </para> + </entry> + </row> + <row> + <entry> + <command>bind</command> + </entry> + <entry> + <para> + Associating a queue to an exchange with a routing key. + </para> + </entry> + </row> + <row> + <entry> + <command>consume</command> + </entry> + <entry> + <para> + Using an object + </para> + </entry> + </row> + <row> + <entry> + <command>create</command> + </entry> + <entry> + <para> + Creating an object. + </para> + </entry> + </row> + <row> + <entry> + <command>delete</command> + </entry> + <entry> + <para> + Deleting an object. + </para> + </entry> + </row> + <row> + <entry> + <command>move</command> + </entry> + <entry> + <para> + Moving messages between queues. + </para> + </entry> + </row> + <row> + <entry> + <command>publish</command> + </entry> + <entry> + <para> + Authenticating an incoming message. + </para> + </entry> + </row> + <row> + <entry> + <command>purge</command> + </entry> + <entry> + <para> + Purging a queue. + </para> + </entry> + </row> + <row> + <entry> + <command>redirect</command> + </entry> + <entry> + <para> + Redirecting messages between queues + </para> + </entry> + </row> + <row> + <entry> + <command>reroute</command> + </entry> + <entry> + <para> + Rerouting messages from a queue to an exchange + </para> + </entry> + </row> + <row> + <entry> + <command>unbind</command> + </entry> + <entry> + <para> + Disassociating a queue from an exchange with a routing key. + </para> + </entry> + </row> + <row> + <entry> + <command>update</command> + </entry> + <entry> + <para> + Changing a broker configuration setting. + </para> + </entry> + </row> + </tbody> + </tgroup> + </table> + <!-- object types --> <table id="tabl-Messaging_User_Guide-ACL_Syntax-ACL_Rulesobject"> + <title>ACL Rules:object</title> + <tgroup cols="2"> + <tbody> + <row> + <entry> + <command>broker</command> + </entry> + <entry> + <para> + </para> + </entry> + </row> + <row> + <entry> + <command>connection</command> + </entry> + <entry> + <para> + Incoming TCP/IP connection + </para> + </entry> + </row> + <row> + <entry> + <command>exchange</command> + </entry> + <entry> + <para> + </para> + </entry> + </row> + <row> + <entry> + <command>link</command> + </entry> + <entry> + <para> + A federation or inter-broker link + </para> + </entry> + </row> + <row> + <entry> + <command>method</command> + </entry> + <entry> + <para> + Management method + </para> + </entry> + </row> + <row> + <entry> + <command>query</command> + </entry> + <entry> + <para> + Management query of an object or class + </para> + </entry> + </row> + <row> + <entry> + <command>queue</command> + </entry> + <entry> + <para> + </para> + </entry> + </row> + </tbody> + </tgroup> + </table> + <!-- + <para> + Wild cards can be used on properties that are a string. The following rule properties are supported: --> + <table id="tabl-Messaging_User_Guide-ACL_Syntax-ACL_Rulesproperty"> + <title>ACL Rules: property</title> + <tgroup cols="4"> + <thead> + <row> + <entry>Property</entry> + <entry>Type</entry> + <entry>Description</entry> + <entry>Usage</entry> + </row> + </thead> + <tbody> + <row> + <entry> <command>name</command> </entry> + <entry>String</entry> + <entry>Rule refers to objects with this name. When 'name' is blank or absent then the rule + applies to all objects of the given type.</entry> + <entry></entry> + </row> + <row> + <entry> <command>alternate</command> </entry> + <entry>String</entry> + <entry>Name of an alternate exchange</entry> + <entry>CREATE QUEUE, CREATE EXCHANGE, ACCESS QUEUE, ACCESS EXCHANGE, DELETE QUEUE, DELETE EXCHANGE</entry> + </row> + <row> + <entry> <command>autodelete</command> </entry> + <entry>Boolean</entry> + <entry>Indicates whether or not the object gets deleted when the connection that created it is closed</entry> + <entry>CREATE QUEUE, CREATE EXCHANGE, ACCESS QUEUE, ACCESS EXCHANGE, DELETE QUEUE</entry> + </row> + <row> + <entry> <command>durable</command> </entry> + <entry>Boolean</entry> + <entry>Rule applies to durable objects</entry> + <entry>CREATE QUEUE, CREATE EXCHANGE, ACCESS QUEUE, ACCESS EXCHANGE, DELETE QUEUE, DELETE EXCHANGE</entry> + </row> + <row> + <entry> <command>exchangename</command> </entry> + <entry>String</entry> + <entry>Name of the exchange to which queue's entries are routed</entry> + <entry>REROUTE QUEUE</entry> + </row> + <row> + <entry> <command>filemaxcountlowerlimit</command> </entry> + <entry>Integer</entry> + <entry>Minimum value for file.max_count (files)</entry> + <entry>CREATE QUEUE</entry> + </row> + <row> + <entry> <command>filemaxcountupperlimit</command> </entry> + <entry>Integer</entry> + <entry>Maximum value for file.max_count (files)</entry> + <entry>CREATE QUEUE</entry> + </row> + <row> + <entry> <command>filemaxsizelowerlimit</command> </entry> + <entry>Integer</entry> + <entry>Minimum value for file.max_size (64kb pages)</entry> + <entry>CREATE QUEUE</entry> + </row> + <row> + <entry> <command>filemaxsizeupperlimit</command> </entry> + <entry>Integer</entry> + <entry>Maximum value for file.max_size (64kb pages)</entry> + <entry>CREATE QUEUE</entry> + </row> + <row> + <entry> <command>host</command> </entry> + <entry>String</entry> + <entry>Target TCP/IP host or host range for create connection rules</entry> + <entry>CREATE CONNECTION</entry> + </row> + <row> + <entry> <command>exclusive</command> </entry> + <entry>Boolean</entry> + <entry>Indicates the presence of an <parameter>exclusive</parameter> flag</entry> + <entry>CREATE QUEUE, ACCESS QUEUE, DELETE QUEUE</entry> + </row> + <row> + <entry> <command>pagefactorlowerlimit</command> </entry> + <entry>Integer</entry> + <entry>Minimum value for size of a page in paged queue</entry> + <entry>CREATE QUEUE</entry> + </row> + <row> + <entry> <command>pagefactorupperlimit</command> </entry> + <entry>Integer</entry> + <entry>Maximum value for size of a page in paged queue</entry> + <entry>CREATE QUEUE</entry> + </row> + <row> + <entry> <command>pageslowerlimit</command> </entry> + <entry>Integer</entry> + <entry>Minimum value for number of paged queue pages in memory</entry> + <entry>CREATE QUEUE</entry> + </row> + <row> + <entry> <command>pagesupperlimit</command> </entry> + <entry>Integer</entry> + <entry>Maximum value for number of paged queue pages in memory</entry> + <entry>CREATE QUEUE</entry> + </row> + <row> + <entry> <command>paging</command> </entry> + <entry>Boolean</entry> + <entry>Indicates if the queue is a paging queue</entry> + <entry>CREATE QUEUE</entry> + </row> + <row> + <entry> <command>policytype</command> </entry> + <entry>String</entry> + <entry>"ring", "self-destruct", "reject"</entry> + <entry>CREATE QUEUE, ACCESS QUEUE, DELETE QUEUE</entry> + </row> + <row> + <entry> <command>queuename</command> </entry> + <entry>String</entry> + <entry>Name of the target queue</entry> + <entry>ACCESS EXCHANGE, BIND EXCHANGE, MOVE QUEUE, UNBIND EXCHANGE</entry> + </row> + <row> + <entry> <command>queuemaxsizelowerlimit</command> </entry> + <entry>Integer</entry> + <entry>Minimum value for queue.max_size (memory bytes)</entry> + <entry>CREATE QUEUE, ACCESS QUEUE</entry> + </row> + <row> + <entry> <command>queuemaxsizeupperlimit</command> </entry> + <entry>Integer</entry> + <entry>Maximum value for queue.max_size (memory bytes)</entry> + <entry>CREATE QUEUE, ACCESS QUEUE</entry> + </row> + <row> + <entry> <command>queuemaxcountlowerlimit</command> </entry> + <entry>Integer</entry> + <entry>Minimum value for queue.max_count (messages)</entry> + <entry>CREATE QUEUE, ACCESS QUEUE</entry> + </row> + <row> + <entry> <command>queuemaxcountupperlimit</command> </entry> + <entry>Integer</entry> + <entry>Maximum value for queue.max_count (messages)</entry> + <entry>CREATE QUEUE, ACCESS QUEUE</entry> + </row> + <row> + <entry> <command>routingkey</command> </entry> + <entry>String</entry> + <entry>Specifies routing key</entry> + <entry>BIND EXCHANGE, UNBIND EXCHANGE, ACCESS EXCHANGE, PUBLISH EXCHANGE</entry> + </row> + <row> + <entry> <command>schemaclass</command> </entry> + <entry>String</entry> + <entry>QMF schema class name</entry> + <entry>ACCESS METHOD, ACCESS QUERY</entry> + </row> + <row> + <entry> <command>schemapackage</command> </entry> + <entry>String</entry> + <entry>QMF schema package name</entry> + <entry>ACCESS METHOD</entry> + </row> + <row> + <entry> <command>type</command> </entry> + <entry>String</entry> + <entry>Type of exchange, such as topic, fanout, or xml</entry> + <entry>CREATE EXCHANGE, ACCESS EXCHANGE, DELETE EXCHANGE</entry> + </row> + </tbody> + </tgroup> + </table> + + <section id="sect-Messaging_User_Guide-Authorization-ACL_ActionObjectPropertyTuples"> + <title>ACL Action-Object-Property Combinations</title> + <para> + Not every ACL action is applicable to every ACL object. Furthermore, not every property may be + specified for every action-object pair. The following table lists the broker events + that trigger ACL lookups. Then for each event it lists the action, object, and properties + allowed in the lookup. + </para> + <para> + User-specified ACL rules constrain property sets to those that match one or more of + the action and object pairs. For example these rules are allowed: + </para> +<programlisting> + acl allow all access exchange + acl allow all access exchange name=abc + acl allow all access exchange name=abc durable=true +</programlisting> + <para> + These rules could possibly match one or more of the broker lookups. However, this rule + is not allowed: + </para> +<programlisting> + acl allow all access exchange queuename=queue1 durable=true +</programlisting> + <para> + Properties <emphasis>queuename</emphasis> and <emphasis>durable</emphasis> + are not in the list of allowed properties for any 'access exchange' lookup. + This rule would never match a broker lookup query and would never contribute to an + allow or deny decision. + </para> + <para> + For more information about matching ACL rules please refer to + <link linkend="sect-Messaging_User_Guide-Authorization-ACL_Rule_Matching"> + ACL Rule Matching + </link> + </para> + + <table id="tabl-Messaging_User_Guide-ACL_Syntax-ACL_ActionObject_properties"> + <title>Broker Lookup Events With Allowed Action, Object, and Properties</title> + <tgroup cols="4"> + <thead> + <row> + <entry>Lookup Event</entry> + <entry>Action</entry> + <entry>Object</entry> + <entry>Properties</entry> + </row> + </thead> + <tbody> + <row> + <entry>User querying message timestamp setting </entry> + <entry>access</entry> + <entry>broker</entry> + <entry></entry> + </row> + <row> + <entry>AMQP 0-10 protocol received 'query' </entry> + <entry>access</entry> + <entry>exchange</entry> + <entry>name </entry> + </row> + <row> + <entry>AMQP 0-10 query binding </entry> + <entry>access</entry> + <entry>exchange</entry> + <entry>name queuename routingkey </entry> + </row> + <row> + <entry>AMQP 0-10 exchange declare </entry> + <entry>access</entry> + <entry>exchange</entry> + <entry>name type alternate durable autodelete </entry> + </row> + <row> + <entry>AMQP 1.0 exchange access </entry> + <entry>access</entry> + <entry>exchange</entry> + <entry>name type durable </entry> + </row> + <row> + <entry>AMQP 1.0 node resolution </entry> + <entry>access</entry> + <entry>exchange</entry> + <entry>name </entry> + </row> + <row> + <entry>Management method request </entry> + <entry>access</entry> + <entry>method</entry> + <entry>name schemapackage schemaclass </entry> + </row> + <row> + <entry>Management agent method request </entry> + <entry>access</entry> + <entry>method</entry> + <entry>name schemapackage schemaclass </entry> + </row> + <row> + <entry>Management agent query </entry> + <entry>access</entry> + <entry>query</entry> + <entry>name schemaclass </entry> + </row> + <row> + <entry>QMF 'query queue' method </entry> + <entry>access</entry> + <entry>queue</entry> + <entry>name </entry> + </row> + <row> + <entry>AMQP 0-10 query </entry> + <entry>access</entry> + <entry>queue</entry> + <entry>name </entry> + </row> + <row> + <entry>AMQP 0-10 queue declare </entry> + <entry>access</entry> + <entry>queue</entry> + <entry>name alternate durable exclusive autodelete policytype queuemaxcountlowerlimit queuemaxcountupperlimit queuemaxsizelowerlimit queuemaxsizeupperlimit </entry> + </row> + <row> + <entry>AMQP 1.0 queue access </entry> + <entry>access</entry> + <entry>queue</entry> + <entry>name alternate durable exclusive autodelete policytype queuemaxcountlowerlimit queuemaxcountupperlimit queuemaxsizelowerlimit queuemaxsizeupperlimit </entry> + </row> + <row> + <entry>AMQP 1.0 node resolution </entry> + <entry>access</entry> + <entry>queue</entry> + <entry>name </entry> + </row> + <row> + <entry>AMQP 0-10 or QMF bind request </entry> + <entry>bind</entry> + <entry>exchange</entry> + <entry>name queuename routingkey </entry> + </row> + <row> + <entry>AMQP 1.0 new outgoing link from exchange</entry> + <entry>bind</entry> + <entry>exchange</entry> + <entry>name queuename routingkey </entry> + </row> + <row> + <entry>AMQP 0-10 subscribe request </entry> + <entry>consume</entry> + <entry>queue</entry> + <entry>name </entry> + </row> + <row> + <entry>AMQP 1.0 new outgoing link from queue </entry> + <entry>consume</entry> + <entry>queue</entry> + <entry>name </entry> + </row> + <row> + <entry>TCP/IP connection creation </entry> + <entry>create</entry> + <entry>connection</entry> + <entry>host </entry> + </row> + <row> + <entry>Create exchange </entry> + <entry>create</entry> + <entry>exchange</entry> + <entry>name type alternate durable autodelete </entry> + </row> + <row> + <entry>Interbroker link creation </entry> + <entry>create</entry> + <entry>link</entry> + <entry></entry> + </row> + <row> + <entry>Interbroker link creation </entry> + <entry>create</entry> + <entry>link</entry> + <entry></entry> + </row> + <row> + <entry>Create queue </entry> + <entry>create</entry> + <entry>queue</entry> + <entry>name alternate durable exclusive autodelete policytype paging pageslowerlimit pagesupperlimit pagefactorlowerlimit pagefactorupperlimit queuemaxcountlowerlimit queuemaxcountupperlimit queuemaxsizelowerlimit queuemaxsizeupperlimit filemaxcountlowerlimit filemaxcountupperlimit filemaxsizelowerlimit filemaxsizeupperlimit </entry> + </row> + <row> + <entry>Delete exchange </entry> + <entry>delete</entry> + <entry>exchange</entry> + <entry>name type alternate durable </entry> + </row> + <row> + <entry>Delete queue </entry> + <entry>delete</entry> + <entry>queue</entry> + <entry>name alternate durable exclusive autodelete policytype </entry> + </row> + <row> + <entry>Management 'move queue' request </entry> + <entry>move</entry> + <entry>queue</entry> + <entry>name queuename </entry> + </row> + <row> + <entry>AMQP 0-10 received message processing </entry> + <entry>publish</entry> + <entry>exchange</entry> + <entry>name routingkey </entry> + </row> + <row> + <entry>AMQP 1.0 establish sender link to queue </entry> + <entry>publish</entry> + <entry>exchange</entry> + <entry>routingkey </entry> + </row> + <row> + <entry>AMQP 1.0 received message processing </entry> + <entry>publish</entry> + <entry>exchange</entry> + <entry>name routingkey </entry> + </row> + <row> + <entry>Management 'purge queue' request </entry> + <entry>purge</entry> + <entry>queue</entry> + <entry>name </entry> + </row> + <row> + <entry>Management 'purge queue' request </entry> + <entry>purge</entry> + <entry>queue</entry> + <entry>name </entry> + </row> + <row> + <entry>Management 'redirect queue' request </entry> + <entry>redirect</entry> + <entry>queue</entry> + <entry>name queuename </entry> + </row> + <row> + <entry>Management 'reroute queue' request </entry> + <entry>reroute</entry> + <entry>queue</entry> + <entry>name exchangename </entry> + </row> + <row> + <entry>Management 'unbind exchange' request </entry> + <entry>unbind</entry> + <entry>exchange</entry> + <entry>name queuename routingkey </entry> + </row> + <row> + <entry>User modifying message timestamp setting</entry> + <entry>update</entry> + <entry>broker</entry> + <entry></entry> + </row> + </tbody> + </tgroup> + </table> + </section> + </section> + + <section id="sect-Messaging_User_Guide-Authorization-ACL_Syntactic_Conventions"> + <title>ACL Syntactic Conventions</title> + <section id="sect-Messaging_User_Guide-Authorization-ACL_Syntactic_Conventions-comments"> + <title>Comments</title> + <para> + <itemizedlist> + <listitem> + <para> + A line starting with the <command>#</command> character is considered a comment and is ignored. + </para> + </listitem> + <listitem> + <para> + Embedded comments and trailing comments are not allowed. The <command>#</command> is commonly found in routing keys and other AMQP literals which occur naturally in ACL rule specifications. + </para> + </listitem> + </itemizedlist> + </para> + </section> + <section id="sect-Messaging_User_Guide-Authorization-ACL_Syntactic_Conventions-whitespace"> + <title>White Space</title> + <itemizedlist> + <listitem> + <para> + Empty lines and lines that contain only whitespace (' ', '\f', '\n', '\r', '\t', '\v') are ignored. + </para> + </listitem> + <listitem> + <para> + Additional whitespace between and after tokens is allowed. + </para> + </listitem> + <listitem> + <para> + Group and Acl definitions must start with <command>group</command> and <command>acl</command> respectively and with no preceding whitespace. + </para> + </listitem> + </itemizedlist> + </section> + <section id="sect-Messaging_User_Guide-Authorization-ACL_Syntactic_Conventions-characterset"> + <title>Character Set</title> + <itemizedlist> + <listitem> + <para> + ACL files use 7-bit ASCII characters only + </para> + </listitem> + <listitem> + <para> + Group names may contain only + <itemizedlist> + <listitem><command>[a-z]</command></listitem> + <listitem><command>[A-Z]</command></listitem> + <listitem><command>[0-9]</command></listitem> + <listitem><command>'-'</command> hyphen</listitem> + <listitem><command>'_'</command> underscore</listitem> + </itemizedlist> + </para> + </listitem> + <listitem> + <para> + Individual user names may contain only + <itemizedlist> + <listitem><command>[a-z]</command></listitem> + <listitem><command>[A-Z]</command></listitem> + <listitem><command>[0-9]</command></listitem> + <listitem><command>'-'</command> hyphen</listitem> + <listitem><command>'_'</command> underscore</listitem> + <listitem><command>'.'</command> period</listitem> + <listitem><command>'@'</command> ampersand</listitem> + <listitem><command>'/'</command> slash</listitem> + </itemizedlist> + </para> + </listitem> + </itemizedlist> + </section> + <section id="sect-Messaging_User_Guide-Authorization-ACL_Syntactic_Conventions-casesensitivity"> + <title>Case Sensitivity</title> + <itemizedlist> + <listitem> + <para> + All tokens are case sensitive. <parameter>name1</parameter> is not the same as <parameter>Name1</parameter> and <parameter>create</parameter> is not the same as <parameter>CREATE</parameter>. + </para> + </listitem> + </itemizedlist> + </section> + <section id="sect-Messaging_User_Guide-Authorization-ACL_Syntactic_Conventions-linecontinuation"> + <title>Line Continuation</title> + <itemizedlist> + <listitem> + <para> + Group lists can be extended to the following line by terminating the line with the <command>'\'</command> character. No other ACL file lines may be continued. + </para> + </listitem> + <listitem> + <para> + Group specification lines may be continued only after the group name or any of the user names included in the group. See example below. + </para> + </listitem> + <listitem> + <para> + Lines consisting solely of a <command>'\'</command> character are not permitted. + </para> + </listitem> + <listitem> + <para> + The <command>'\'</command> continuation character is recognized only if it is the last character in the line. Any characters after the <command>'\'</command> are not permitted. + </para> + </listitem> + </itemizedlist> +<programlisting><![CDATA[ + # + # Examples of extending group lists using a trailing '\' character + # + group group1 name1 name2 \ + name3 name4 \ + name5 + + group group2 \ + group1 \ + name6 + # + # The following are illegal: + # + # '\' must be after group name + # + group \ + group3 name7 name8 + # + # No empty extension line + # + group group4 name9 \ + \ + name10 +]]></programlisting> + + </section> + <section id="sect-Messaging_User_Guide-Authorization-ACL_Syntactic_Conventions-linelength"> + <title>Line Length</title> + <itemizedlist> + <listitem> + <para> + ACL file lines are limited to 1024 characters. + </para> + </listitem> + </itemizedlist> + </section> + + + <section id="sect-Messaging_User_Guide-Authorization-ACL_Syntactic_Conventions-keywords"> + <title>ACL File Keywords</title> + ACL reserves several words for convenience and for context sensitive substitution. + + <section id="sect-Messaging_User_Guide-Authorization-ACL_Syntactic_Conventions-keywords-all"> + <title>The <command>all</command> Keyword</title> + The keyword <command>all</command> is reserved. It may be used in ACL rules to match all individuals and groups, all actions, or all objects. + <itemizedlist> + <listitem>acl allow all create queue</listitem> + <listitem>acl allow bob@QPID all queue</listitem> + <listitem>acl allow bob@QPID create all</listitem> + </itemizedlist> + </section> + + <section id="sect-Messaging_User_Guide-Authorization-ACL_Syntactic_Conventions-keywords-userdomain"> + <title>User Name and Domain Name Keywords</title> + <para> + In the C++ Broker 0.20 a simple set of user name and domain name substitution variable keyword tokens is defined. This provides administrators with an easy way to describe private or shared resources. + </para> + <para> + Symbol substitution is allowed in the ACL file anywhere that text is supplied for a property value. + </para> + <para> + In the following table an authenticated user named bob.user@QPID.COM has his substitution keywords expanded. + + <table id="tabl-Messaging_User_Guide-ACL_Syntax-ACL_UsernameSubstitution"> + <title>ACL User Name and Domain Name Substitution Keywords</title> + <tgroup cols="2"> + <thead> + <row> + <entry>Keyword</entry> + <entry>Expansion</entry> + </row> + </thead> + <tbody> + <row> + <entry> <command>${userdomain}</command> </entry> + <entry>bob_user_QPID_COM</entry> + </row> + <row> + <entry> <command>${user}</command> </entry> + <entry>bob_user</entry> + </row> + <row> + <entry> <command>${domain}</command> </entry> + <entry>QPID_COM</entry> + </row> + </tbody> + </tgroup> + </table> + </para> + + <para> + <itemizedlist> + <listitem> + The original user name has the period “.” and ampersand “@” characters translated into underscore “_”. This allows substitution to work when the substitution keyword is used in a routingkey in the Acl file. + </listitem> + <listitem> + The Acl processing matches ${userdomain} before matching either ${user} or ${domain}. Rules that specify the combination ${user}_${domain} will never match. + </listitem> + </itemizedlist> + </para> + +<programlisting><![CDATA[ + # Example: + # + # Administrators can set up Acl rule files that allow every user to create a + # private exchange, a private queue, and a private binding between them. + # In this example the users are also allowed to create private backup exchanges, + # queues and bindings. This effectively provides limits to user's exchange, + # queue, and binding creation and guarantees that each user gets exclusive + # access to these resources. + # + # + # Create primary queue and exchange: + # + acl allow all create queue name=$\{user}-work alternate=$\{user}-work2 + acl deny all create queue name=$\{user}-work alternate=* + acl allow all create queue name=$\{user}-work + acl allow all create exchange name=$\{user}-work alternate=$\{user}-work2 + acl deny all create exchange name=$\{user}-work alternate=* + acl allow all create exchange name=$\{user}-work + # + # Create backup queue and exchange + # + acl deny all create queue name=$\{user}-work2 alternate=* + acl allow all create queue name=$\{user}-work2 + acl deny all create exchange name=$\{user}-work2 alternate=* + acl allow all create exchange name=$\{user}-work2 + # + # Bind/unbind primary exchange + # + acl allow all bind exchange name=$\{user}-work routingkey=$\{user} queuename=$\{user}-work + acl allow all unbind exchange name=$\{user}-work routingkey=$\{user} queuename=$\{user}-work + # + # Bind/unbind backup exchange + # + acl allow all bind exchange name=$\{user}-work2 routingkey=$\{user} queuename=$\{user}-work2 + acl allow all unbind exchange name=$\{user}-work2 routingkey=$\{user} queuename=$\{user}-work2 + # + # Access primary exchange + # + acl allow all access exchange name=$\{user}-work routingkey=$\{user} queuename=$\{user}-work + # + # Access backup exchange + # + acl allow all access exchange name=$\{user}-work2 routingkey=$\{user} queuename=$\{user}-work2 + # + # Publish primary exchange + # + acl allow all publish exchange name=$\{user}-work routingkey=$\{user} + # + # Publish backup exchange + # + acl allow all publish exchange name=$\{user}-work2 routingkey=$\{user} + # + # deny mode + # + acl deny all all +]]></programlisting> + </section> + + </section> + + <section id="sect-Messaging_User_Guide-Authorization-ACL_Syntatic_Conventions-wildcards"> + <title>Wildcards</title> + ACL privides two types of wildcard matching to provide flexibility in writing rules. + + <section id="sect-Messaging_User_Guide-Authorization-ACL_Syntatic_Conventions-wildcards-asterisk"> + <title>Property Value Wildcard</title> + <para> + Text specifying a property value may end with a single trailing <command>*</command> character. + This is a simple wildcard match indicating that strings which match up to that point are matches for the ACL property rule. + An ACL rule such as + </para> + <para> + <programlisting> acl allow bob@QPID create queue name=bob*</programlisting> + </para> + <para> + allow user bob@QPID to create queues named bob1, bob2, bobQueue3, and so on. + </para> + </section> + + <section id="sect-Messaging_User_Guide-Authorization-ACL_Syntatic_Conventions-wildcards-topickey"> + <title>Topic Routing Key Wildcard</title> + <para> + In the C++ Broker 0.20 the logic governing the ACL Match has changed for each ACL rule that contains a routingkey property. + The routingkey property is matched according to Topic Exchange match logic the broker uses when it distributes messages published to a topic exchange. + </para> + <para> + Routing keys are hierarchical where each level is separated by a period: + <itemizedlist> + <listitem>weather.usa</listitem> + <listitem>weather.europe.germany</listitem> + <listitem>weather.europe.germany.berlin</listitem> + <listitem>company.engineering.repository</listitem> + </itemizedlist> + </para> + <para> + Within the routing key hierarchy two wildcard characters are defined. + <itemizedlist> + <listitem><command>*</command> matches one field</listitem> + <listitem><command>#</command> matches zero or more fields</listitem> + </itemizedlist> + </para> + <para> + Suppose an ACL rule file is: + </para> + <para> + <programlisting> + acl allow-log uHash1@COMPANY publish exchange name=X routingkey=a.#.b + acl deny all all + </programlisting> + </para> + <para> + When user uHash1@COMPANY attempts to publish to exchange X the ACL will return these results: + + <table id="tabl-Messaging_User_Guide-ACL_Syntax-ACL_TopicExchangeMatch"> + <title>Topic Exchange Wildcard Match Examples</title> + <tgroup cols="2"> + <thead> + <row> + <entry>routingkey in publish to exchange X</entry> + <entry>result</entry> + </row> + </thead> + <tbody> + <row> + <entry> <command>a.b</command> </entry> + <entry>allow-log</entry> + </row> + <row> + <entry> <command>a.x.b</command> </entry> + <entry>allow-log</entry> + </row> + <row> + <entry> <command>a.x.y.zz.b</command> </entry> + <entry>allow-log</entry> + </row> + <row> + <entry> <command>a.b.</command> </entry> + <entry>deny</entry> + </row> + <row> + <entry> <command>q.x.b</command> </entry> + <entry>deny</entry> + </row> + </tbody> + </tgroup> + </table> + + </para> + </section> + + </section> + + + + </section> + + <section id="sect-Messaging_User_Guide-Authorization-ACL_Rule_Matching"> + <title>ACL Rule Matching</title> + <para> + The minimum matching criteria for ACL rules are: + <itemizedlist> + <listitem>An actor (individually named or group member)</listitem> + <listitem>An action</listitem> + <listitem>An object</listitem> + </itemizedlist> + </para> + <para> + If a rule does not match the minimum criteria then that rule does not control the ACL allow or deny decision. + </para> + <para> + ACL rules optionally specify object names and property name=value pairs. If an ACL rule specifies an object name or property values than all of them must match to cause the rule to match. + </para> + <para> + The following illustration shows how ACL rules are processed to find matching rules. +<programlisting><![CDATA[ + # Example of rule matching + # + # Using this ACL file content: + + (1) acl deny bob create exchange name=test durable=true passive=true + (2) acl deny bob create exchange name=myEx type=direct + (3) acl allow all all + + # + # Lookup 1. id:bob action:create objectType:exchange name=test + # {durable=false passive=false type=direct alternate=} + # + # ACL Match Processing: + # 1. Rule 1 passes minimum criteria with user bob, action create, + # and object exchange. + # 2. Rule 1 matches name=test. + # 3. Rule 1 does not match the rule's durable=true with the requested + # lookup of durable=false. + # 4. Rule 1 does not control the decision and processing continues + # to Rule 2. + # 5. Rule 2 passes minimum criteria with user bob, action create, + # and object exchange. + # 6. Rule 2 does not match the rule's name=myEx with the requested + # lookup of name=test. + # 7. Rule 2 does not control the decision and processing continues + # to Rule 3. + # 8. Rule 3 matches everything and the decision is 'allow'. + # + # Lookup 2. id:bob action:create objectType:exchange name=myEx + # {durable=true passive=true type=direct alternate=} + # + # ACL Match Processing: + # 1. Rule 1 passes minimum criteria with user bob, action create, + # and object exchange. + # 2. Rule 1 does not match the rule's name=test with the requested + # lookup of name=myEx. + # 3. Rule 1 does not control the decision and processing continues + # to Rule 2. + # 4. Rule 2 passes minimum criteria with user bob, action create, + # and object exchange. + # 5. Rule 2 matches name=myEx. + # 6. Rule 2 matches the rule's type=direct with the requested + # lookup of type=direct. + # 7. Rule 2 is the matching rule and the decision is 'deny'. + # +]]></programlisting> + </para> + <para> + Referring to <link linkend="tabl-Messaging_User_Guide-ACL_Syntax-ACL_ActionObject_properties">ACL Properties Allowed for each Action and Object table</link> observe that some Action/Object pairs have different sets of allowed properties. For example different broker ACL lookups for <emphasis>access exchange</emphasis> have different property subsets. + </para> + +<programlisting> + [1] access exchange name + [2] access exchange name type alternate durable autodelete + [3] access exchange name queuename routingkey + [4] access exchange name type durable +</programlisting> + + <para> + If an ACL rule specifies the <emphasis>autodelete</emphasis> property then it can possibly match only the second case above. It can never match cases 1, 3, and 4 because the broker calls to ACL will not present the autodelete property for matching. To get proper matching the ACL rule must have only the properties of the intended lookup case. + </para> + +<programlisting> + acl allow bob access exchange alternate=other ! may match pattern 2 only + acl allow bob access exchange queuename=other ! may match pattern 3 only + acl allow bob access exchange durable=true ! may match patterns 2 and 4 only + acl deny bob access exchange ! may match all patterns +</programlisting> + + </section> + + <section id="sect-Messaging_User_Guide-Authorization-Specifying_ACL_Permissions"> + <title>Specifying ACL Permissions</title> + <para> + Now that we have seen the ACL syntax, we will provide representative examples and guidelines for ACL files. + </para> + <para> + Most ACL files begin by defining groups: + </para> + +<programlisting> + group admin ted@QPID martin@QPID + group user-consume martin@QPID ted@QPID + group group2 kim@QPID user-consume rob@QPID + group publisher group2 \ + tom@QPID andrew@QPID debbie@QPID +</programlisting> + <para> + Rules in an ACL file grant or deny specific permissions to users or groups: + </para> + +<programlisting> + acl allow carlt@QPID create exchange name=carl.* + acl allow rob@QPID create queue + acl allow guest@QPID bind exchange name=amq.topic routingkey=stocks.rht.# + acl allow user-consume create queue name=tmp.* + + acl allow publisher publish all durable=false + acl allow publisher create queue name=RequestQueue + acl allow consumer consume queue durable=true + acl allow fred@QPID create all + acl allow bob@QPID all queue + acl allow admin all + acl allow all consume queue + acl allow all bind exchange + acl deny all all +</programlisting> + <para> + In the previous example, the last line, <literal>acl deny all all</literal>, denies all authorizations that have not been specifically granted. This is the default, but it is useful to include it explicitly on the last line for the sake of clarity. If you want to grant all rights by default, you can specify <literal>acl allow all all</literal> in the last line. + </para> + <para> + ACL allows specification of conflicting rules. Be sure to specify the most specific rules first followed by more general rules. Here is an example: + </para> + <para> +<programlisting> + group users alice@QPID bob@QPID charlie@QPID + acl deny charlie@QPID create queue + acl allow users create queue + acl deny all all +</programlisting> + </para> + <para> + In this example users alice and bob would be able to create queues due to their membership in the users group. However, user charlie is denied from creating a queue despite his membership in the users group because a deny rule for him is stated before the allow rule for the users group. + </para> + <para> + Do not allow <parameter>guest</parameter> to access and log QMF management methods that could cause security breaches: + </para> + +<programlisting> + group allUsers guest@QPID + ... + acl deny-log allUsers create link + acl deny-log allUsers access method name=connect + acl deny-log allUsers access method name=echo + acl allow all all +</programlisting> + + </section> + <section id="sect-Messaging_User_Guide-Authorization-Auditing_ACL_Settings"> + <title>Auditing ACL Settings</title> + <para> + The 0.30 C++ Broker ACL module provides a comprehensive set of run-time and debug logging checks. + The following example ACL file is used to illustrate working with the ACL module debugging features. + </para> +<programlisting> + group x a@QPID b@QPID b2@QPID b3@QPID + acl allow all delete broker + acl allow all create queue name=abc + acl allow all create queue exchangename=xyz + acl allow all create connection host=1.1.1.1 + acl allow all access exchange alternate=abc queuename=xyz + acl allow all access exchange queuename=xyz + acl allow all access exchange alternate=abc + acl allow a@qpid all all exchangename=123 + acl allow b@qpid all all + acl allow all all +</programlisting> + <para> + When this file is loaded it will show the following (truncated, formatted) Info-level log. + </para> +<programlisting> + notice ACL: Read file "/home/chug/acl/svn-acl.acl" + warning ACL rule ignored: Broker never checks for rules with + action: 'delete' and object: 'broker' + warning ACL rule ignored: Broker checks for rules with + action: 'create' and object: 'queue' + but will never match with property set: { exchangename=xyz } + warning ACL rule ignored: Broker checks for rules with + action: 'access' and object: 'exchange' + but will never match with property set: { alternate=abc queuename=xyz } + info ACL Plugin loaded +</programlisting> + <para> + Three of the rules are invalid. The first invalid rule is rejected because there are no rules + that specify 'delete broker' regardless of the properties. The other two rules are rejected + because the property sets in the ACL rule don't match any broker lookups. + </para> + <para> + The ACL module only issues a warning about these rules and continues to operate. Users upgrading + from previous versions should be concerned that these rules never had any effect and should fix + the rules to have the property sets needed to allow or deny the intended broker events. + </para> + <para> + The next illustration shows the Debug-level log. Debug log level includes information about + constructing the rule tables, expanding groups and keywords, connection and queue quotas, and + connection black and white lists. + </para> +<programlisting> + notice ACL: Read file "/home/chug/acl/svn-acl.acl" + debug ACL: Group list: 1 groups found: + debug ACL: "x": a@QPID b2@QPID b3@QPID b@QPID + debug ACL: name list: 7 names found: + debug ACL: * a@QPID a@qpid b2@QPID b3@QPID b@QPID b@qpid + debug ACL: Rule list: 10 ACL rules found: + debug ACL: 1 allow [*] delete broker + warning ACL rule ignored: Broker never checks for rules with + action: 'delete' and object: 'broker' + debug ACL: 2 allow [*] create queue name=abc + debug ACL: 3 allow [*] create queue exchangename=xyz + warning ACL rule ignored: Broker checks for rules with + action: 'create' and object: 'queue' + but will never match with property set: { exchangename=xyz } + debug ACL: 4 allow [*] create connection host=1.1.1.1 + debug ACL: 5 allow [*] access exchange alternate=abc queuename=xyz + warning ACL rule ignored: Broker checks for rules with + action: 'access' and object: 'exchange' + but will never match with property set: { alternate=abc queuename=xyz } + debug ACL: 6 allow [*] access exchange queuename=xyz + debug ACL: 7 allow [*] access exchange alternate=abc + debug ACL: 8 allow [a@qpid] * * exchangename=123 + debug ACL: 9 allow [b@qpid] * * + debug ACL: 10 allow [*] * + debug ACL: connections quota: 0 rules found: + debug ACL: queues quota: 0 rules found: + debug ACL: Load Rules + debug ACL: Processing 10 allow [*] * + debug ACL: FoundMode allow + debug ACL: Processing 9 allow [b@qpid] * * + debug ACL: Adding actions {access,bind,consume,create,delete,move,publish,purge, + redirect,reroute,unbind,update} + to objects {broker,connection,exchange,link,method,query,queue} + with props { } + for users {b@qpid} + debug ACL: Processing 8 allow [a@qpid] * * exchangename=123 + debug ACL: Adding actions {access,bind,consume,create,delete,move,publish,purge, + redirect,reroute,unbind,update} + to objects {broker,connection,exchange,link,method,query,queue} + with props { exchangename=123 } + for users {a@qpid} + debug ACL: Processing 7 allow [*] access exchange alternate=abc + debug ACL: Adding actions {access} + to objects {exchange} + with props { alternate=abc } + for users {*,a@QPID,a@qpid,b2@QPID,b3@QPID,b@QPID,b@qpid} + debug ACL: Processing 6 allow [*] access exchange queuename=xyz + debug ACL: Adding actions {access} + to objects {exchange} + with props { queuename=xyz } + for users {*,a@QPID,a@qpid,b2@QPID,b3@QPID,b@QPID,b@qpid} + debug ACL: Processing 5 allow [*] access exchange alternate=abc queuename=xyz + debug ACL: Processing 4 allow [*] create connection host=1.1.1.1 + debug ACL: Processing 3 allow [*] create queue exchangename=xyz + debug ACL: Processing 2 allow [*] create queue name=abc + debug ACL: Adding actions {create} + to objects {queue} + with props { name=abc } + for users {*,a@QPID,a@qpid,b2@QPID,b3@QPID,b@QPID,b@qpid} + debug ACL: Processing 1 allow [*] delete broker + debug ACL: global Connection Rule list : 1 rules found : + debug ACL: 1 [ruleMode = allow {(1.1.1.1,1.1.1.1)} + debug ACL: User Connection Rule lists : 0 user lists found : + debug ACL: Transfer ACL is Enabled! + info ACL Plugin loaded +</programlisting> + <para> + The previous illustration is interesting because it shows the settings as the <emphasis>all</emphasis> keywords are + being expanded. However, that does not show the information about what is actually going into the ACL lookup tables. + </para> + <para> + The next two illustrations show additional information provided by Trace-level logs for ACL startup. + The first shows a dump of the broker's internal + action/object/properties table. This table is authoratative. + </para> +<programlisting> + trace ACL: Definitions of action, object, (allowed properties) lookups + trace ACL: Lookup 1: "User querying message timestamp setting " + access broker () + trace ACL: Lookup 2: "AMQP 0-10 protocol received 'query' " + access exchange (name) + trace ACL: Lookup 3: "AMQP 0-10 query binding " + access exchange (name,routingkey,queuename) + trace ACL: Lookup 4: "AMQP 0-10 exchange declare " + access exchange (name,durable,autodelete,type,alternate) + trace ACL: Lookup 5: "AMQP 1.0 exchange access " + access exchange (name,durable,type) + trace ACL: Lookup 6: "AMQP 1.0 node resolution " + access exchange (name) + trace ACL: Lookup 7: "Management method request " + access method (name,schemapackage,schemaclass) + trace ACL: Lookup 8: "Management agent method request " + access method (name,schemapackage,schemaclass) + trace ACL: Lookup 9: "Management agent query " + access query (name,schemaclass) + trace ACL: Lookup 10: "QMF 'query queue' method " + access queue (name) + trace ACL: Lookup 11: "AMQP 0-10 query " + access queue (name) + trace ACL: Lookup 12: "AMQP 0-10 queue declare " + access queue (name,durable,autodelete,exclusive,alternate, + policytype,queuemaxsizelowerlimit,queuemaxsizeupperlimit, + queuemaxcountlowerlimit,queuemaxcountupperlimit) + trace ACL: Lookup 13: "AMQP 1.0 queue access " + access queue (name,durable,autodelete,exclusive,alternate, + policytype,queuemaxsizelowerlimit,queuemaxsizeupperlimit, + queuemaxcountlowerlimit,queuemaxcountupperlimit) + trace ACL: Lookup 14: "AMQP 1.0 node resolution " + access queue (name) + trace ACL: Lookup 15: "AMQP 0-10 or QMF bind request " + bind exchange (name,routingkey,queuename) + trace ACL: Lookup 16: "AMQP 1.0 new outgoing link from exchange " + bind exchange (name,routingkey,queuename) + trace ACL: Lookup 17: "AMQP 0-10 subscribe request " + consume queue (name) + trace ACL: Lookup 18: "AMQP 1.0 new outgoing link from queue " + consume queue (name) + trace ACL: Lookup 19: "TCP/IP connection creation " + create connection (host) + trace ACL: Lookup 20: "Create exchange " + create exchange (name,durable,autodelete,type,alternate) + trace ACL: Lookup 21: "Interbroker link creation " + create link () + trace ACL: Lookup 22: "Interbroker link creation " + create link () + trace ACL: Lookup 23: "Create queue " + create queue (name,durable,autodelete,exclusive, + alternate,policytype,paging, + queuemaxsizelowerlimit,queuemaxsizeupperlimit, + queuemaxcountlowerlimit,queuemaxcountupperlimit, + filemaxsizelowerlimit,filemaxsizeupperlimit, + filemaxcountlowerlimit,filemaxcountupperlimit, + pageslowerlimit,pagesupperlimit, + pagefactorlowerlimit,pagefactorupperlimit) + trace ACL: Lookup 24: "Delete exchange " + delete exchange (name,durable,type,alternate) + trace ACL: Lookup 25: "Delete queue " + delete queue (name,durable,autodelete,exclusive, + alternate,policytype) + trace ACL: Lookup 26: "Management 'move queue' request " + move queue (name,queuename) + trace ACL: Lookup 27: "AMQP 0-10 received message processing " + publish exchange (name,routingkey) + trace ACL: Lookup 28: "AMQP 1.0 establish sender link to queue " + publish exchange (routingkey) + trace ACL: Lookup 29: "AMQP 1.0 received message processing " + publish exchange (name,routingkey) + trace ACL: Lookup 30: "Management 'purge queue' request " + purge queue (name) + trace ACL: Lookup 31: "Management 'purge queue' request " + purge queue (name) + trace ACL: Lookup 32: "Management 'redirect queue' request " + redirect queue (name,queuename) + trace ACL: Lookup 33: "Management 'reroute queue' request " + reroute queue (name,exchangename) + trace ACL: Lookup 34: "Management 'unbind exchange' request " + unbind exchange (name,routingkey,queuename) + trace ACL: Lookup 35: "User modifying message timestamp setting " + update broker () +</programlisting> + <para> + The final illustration shows a dump of every rule for every user in the ACL database. + It includes the user name, action, object, original ACL rule number, allow or deny status, + and a cross reference indicating which Lookup Events the rule could possibly satisfy. + </para> + <para> + Note that rules identified by <emphasis>User: *</emphasis> are the rules in effect + for users otherwise unnamed in the ACL file. + </para> + +<programlisting> + trace ACL: Decision rule cross reference + trace ACL: User: b@qpid access broker + Rule: [rule 9 ruleMode = allow props{ }] + may match Lookups : (1) + trace ACL: User: * access exchange + Rule: [rule 6 ruleMode = allow props{ queuename=xyz }] + may match Lookups : (3) + trace ACL: User: * access exchange + Rule: [rule 7 ruleMode = allow props{ alternate=abc }] + may match Lookups : (4) + trace ACL: User: a@QPID access exchange + Rule: [rule 6 ruleMode = allow props{ queuename=xyz }] + may match Lookups : (3) + trace ACL: User: a@QPID access exchange + Rule: [rule 7 ruleMode = allow props{ alternate=abc }] + may match Lookups : (4) + trace ACL: User: a@qpid access exchange + Rule: [rule 6 ruleMode = allow props{ queuename=xyz }] + may match Lookups : (3) + trace ACL: User: a@qpid access exchange + Rule: [rule 7 ruleMode = allow props{ alternate=abc }] + may match Lookups : (4) + trace ACL: User: b2@QPID access exchange + Rule: [rule 6 ruleMode = allow props{ queuename=xyz }] + may match Lookups : (3) + trace ACL: User: b2@QPID access exchange + Rule: [rule 7 ruleMode = allow props{ alternate=abc }] + may match Lookups : (4) + trace ACL: User: b3@QPID access exchange + Rule: [rule 6 ruleMode = allow props{ queuename=xyz }] + may match Lookups : (3) + trace ACL: User: b3@QPID access exchange + Rule: [rule 7 ruleMode = allow props{ alternate=abc }] + may match Lookups : (4) + trace ACL: User: b@QPID access exchange + Rule: [rule 6 ruleMode = allow props{ queuename=xyz }] + may match Lookups : (3) + trace ACL: User: b@QPID access exchange + Rule: [rule 7 ruleMode = allow props{ alternate=abc }] + may match Lookups : (4) + trace ACL: User: b@qpid access exchange + Rule: [rule 6 ruleMode = allow props{ queuename=xyz }] + may match Lookups : (3) + trace ACL: User: b@qpid access exchange + Rule: [rule 7 ruleMode = allow props{ alternate=abc }] + may match Lookups : (4) + trace ACL: User: b@qpid access exchange + Rule: [rule 9 ruleMode = allow props{ }] + may match Lookups : (2,3,4,5,6) + trace ACL: User: b@qpid access method + Rule: [rule 9 ruleMode = allow props{ }] + may match Lookups : (7,8) + trace ACL: User: b@qpid access query + Rule: [rule 9 ruleMode = allow props{ }] + may match Lookups : (9) + trace ACL: User: b@qpid access queue + Rule: [rule 9 ruleMode = allow props{ }] + may match Lookups : (10,11,12,13,14) + trace ACL: User: b@qpid bind exchange + Rule: [rule 9 ruleMode = allow props{ }] + may match Lookups : (15,16) + trace ACL: User: b@qpid consume queue + Rule: [rule 9 ruleMode = allow props{ }] + may match Lookups : (17,18) + trace ACL: User: b@qpid create connection + Rule: [rule 9 ruleMode = allow props{ }] + may match Lookups : (19) + trace ACL: User: b@qpid create exchange + Rule: [rule 9 ruleMode = allow props{ }] + may match Lookups : (20) + trace ACL: User: b@qpid create link + Rule: [rule 9 ruleMode = allow props{ }] + may match Lookups : (21,22) + trace ACL: User: * create queue + Rule: [rule 2 ruleMode = allow props{ name=abc }] + may match Lookups : (23) + trace ACL: User: a@QPID create queue + Rule: [rule 2 ruleMode = allow props{ name=abc }] + may match Lookups : (23) + trace ACL: User: a@qpid create queue + Rule: [rule 2 ruleMode = allow props{ name=abc }] + may match Lookups : (23) + trace ACL: User: b2@QPID create queue + Rule: [rule 2 ruleMode = allow props{ name=abc }] + may match Lookups : (23) + trace ACL: User: b3@QPID create queue + Rule: [rule 2 ruleMode = allow props{ name=abc }] + may match Lookups : (23) + trace ACL: User: b@QPID create queue + Rule: [rule 2 ruleMode = allow props{ name=abc }] + may match Lookups : (23) + trace ACL: User: b@qpid create queue + Rule: [rule 2 ruleMode = allow props{ name=abc }] + may match Lookups : (23) + trace ACL: User: b@qpid create queue + Rule: [rule 9 ruleMode = allow props{ }] + may match Lookups : (23) + trace ACL: User: b@qpid delete exchange + Rule: [rule 9 ruleMode = allow props{ }] + may match Lookups : (24) + trace ACL: User: b@qpid delete queue + Rule: [rule 9 ruleMode = allow props{ }] + may match Lookups : (25) + trace ACL: User: b@qpid move queue + Rule: [rule 9 ruleMode = allow props{ }] + may match Lookups : (26) + trace ACL: User: b@qpid publish exchange + Rule: [rule 9 ruleMode = allow props{ }] + may match Lookups : (27,28,29) + trace ACL: User: b@qpid purge queue + Rule: [rule 9 ruleMode = allow props{ }] + may match Lookups : (30,31) + trace ACL: User: b@qpid redirect queue + Rule: [rule 9 ruleMode = allow props{ }] + may match Lookups : (32) + trace ACL: User: a@qpid reroute queue + Rule: [rule 8 ruleMode = allow props{ exchangename=123 }] + may match Lookups : (33) + trace ACL: User: b@qpid reroute queue + Rule: [rule 9 ruleMode = allow props{ }] + may match Lookups : (33) + trace ACL: User: b@qpid unbind exchange + Rule: [rule 9 ruleMode = allow props{ }] + may match Lookups : (34) + trace ACL: User: b@qpid update broker + Rule: [rule 9 ruleMode = allow props{ }] + may match Lookups : (35) +</programlisting> + + </section> + </section> + + <section id="sect-Messaging_User_Guide-Authorization-Specifying_ACL_Quotas"> + <title>User Connection and Queue Quotas</title> + <para> + The ACL module enforces various quotas and thereby limits user activity. + </para> + + <section id="sect-Messaging_User_Guide-Authorization-Specifying_ACL_Connection_Limits"> + <title>Connection Count Limits</title> + <para> + The ACL module creates broker command line switches that set limits on the number of concurrent connections allowed per user or per client host address. These settings are not specified in the ACL file. + </para> + <para> + <programlisting> + --max-connections N + --connection-limit-per-user N + --connection-limit-per-ip N + </programlisting> + </para> + <para> + <command>--max-connections</command> specifies an upper limit for all user connections. + </para> + <para> + <command>--connection-limit-per-user</command> specifies an upper limit for each user based on the authenticated user name. This limit is enforced regardless of the client IP address from which the connection originates. + </para> + <para> + <command>--connection-limit-per-ip</command> specifies an upper limit for connections for all users based on the originating client IP address. This limit is enforced regardless of the user credentials presented with the connection. + <itemizedlist> + <listitem> + Note that addresses using different transports are counted separately even though the originating host is actually the same physical machine. In the setting illustrated above a host would allow N_IP connections from [::1] IPv6 transport localhost and another N_IP connections from [127.0.0.1] IPv4 transport localhost. + </listitem> + <listitem> + The connection-limit-per-ip and connection-limit-per-user counts are active simultaneously. From a given client system users may be denied access to the broker by either connection limit. + </listitem> + </itemizedlist> + </para> + <para> + The 0.22 C++ Broker ACL module accepts fine grained per-user connection limits through quota rules in the ACL file. + </para> + <para> + <programlisting> + quota connections 10 admins userX@QPID + </programlisting> + </para> + <para> + <itemizedlist> + <listitem> + User <literal>all</literal> receives the value passed by the command line switch <literal>--connection-limit-per-user</literal>. + </listitem> + <listitem> + Values specified in the ACL rule for user <literal>all</literal> overwrite the value specified on the command line if any. + </listitem> + <listitem> + Connection quotas values are determined by first searching for the authenticated user name. If that user name is not specified then the value for user <literal>all</literal> + is used. If user <literal>all</literal> is not specified then the connection is denied. + </listitem> + <listitem> + The connection quota values range from 0..65530 inclusive. A value of zero disables connections from that user. + </listitem> + <listitem> + A user's quota may be specified many times in the ACL rule file. Only the last value specified is retained and enforced. + </listitem> + <listitem> + Per-user connection quotas are disabled when two conditions are true: 1) No --connection-limit-per-user command line switch and 2) No <literal>quota connections</literal> + rules in the ACL file. Per-user connections are always counted even if connection quotas are not enforced. This supports ACL file reloading that may subsequently + enable per-user connection quotas. + </listitem> + <listitem> + An ACL file reload may lower a user's connection quota value to a number lower than the user's current connection count. In that case the active connections + remain unaffected. New connections are denied until that user closes enough of his connections so that his count falls below the configured limit. + </listitem> + </itemizedlist> + </para> + </section> + + <section id="sect-Messaging_User_Guide-Authorization-Specifying_ACL_Connection_Host_Limits"> + <title>Connection Limits by Host Name</title> + <para> + The 0.30 C++ Broker ACL module adds the ability to create allow and deny lists of the TCP/IP hosts from which users may connect. The rule accepts these forms: + </para> + <para> + <programlisting> + acl allow user create connection host=host1 + acl allow user create connection host=host1,host2 + acl deny user create connection host=all + </programlisting> + </para> + <para> + Using the form <command>host=host1</command> specifies a single host. With a single host the name may resolve to multiple TCP/IP addresses. For example <emphasis>localhost</emphasis> resolves to both <emphasis>127.0.0.1</emphasis> and <emphasis>::1</emphasis> and possibly many other addresses. A connection from any of the addresses associated with this host matches the rule and the connection is allowed or denied accordingly. + </para> + <para> + Using the form <command>host=host1,host2</command> specifies a range of TCP/IP addresses. With a host range each host must resolve to a single TCP/IP address and the second address must be numerically larger than the first. A connection from any host where host >= host1 and host <= host2 match the rule and the connection is allowed or denied accordingly. + </para> + <para> + Using the form <command>host=all</command> specifies all TCP/IP addresses. A connection from any host matches the rule and the connection is allowed or denied accordingly. + </para> + <para> + Connection denial is only applied to incoming TCP/IP connections. Other socket types are not subjected to nor denied by range checks. + </para> + <para> + Connection creation rules are divided into three categories: + </para> + <procedure> + <step> + <para> + User = all, host != all + </para> + <para> + These define global rules and are applied before any specific user rules. + These rules may be used to reject connections before any AMPQ protocol is run and before + any user names have been negotiated. + </para> + </step> + <step> + <para> + User != all, host = any legal host or 'all' + </para> + <para> + These define user rules. These rules are applied after the global rules and + after the AMQP protocol has negotiated user identities. + </para> + </step> + <step> + <para> + User = all, host = all + </para> + <para> + This rule defines what to do if no other rule matches. The default value is "ALLOW". + Only one rule of this type may be defined. + </para> + </step> + </procedure> + + <para> + The following example illustrates how this feature can be used. + </para> + <para> + <programlisting> + group admins alice bob chuck + group Company1 c1_usera c1_userb + group Company2 c2_userx c2_usery c2_userz + acl allow admins create connection host=localhost + acl allow admins create connection host=10.0.0.0,10.255.255.255 + acl allow admins create connection host=192.168.0.0,192.168.255.255 + acl allow admins create connection host=[fc00::],[fc00::ff] + acl allow Company1 create connection host=company1.com + acl deny Company1 create connection host=all + acl allow Company2 create connection host=company2.com + acl deny Company2 create connection host=all + </programlisting> + </para> + <para> + In this example admins may connect from localhost or from any system on the 10.0.0.0/24, 192.168.0.0/16, and fc00::/7 subnets. Company1 users may connect only from company1.com and Company2 users may connect only from company2.com. + However, this example has a flaw. Although the admins group has specific hosts + from which it is allowed to make connections it is not blocked from connecting + from anywhere. The Company1 and Company2 groups are blocked appropriately. + This ACL file may be rewritten as follows: + </para> + <para> + <programlisting> + group admins alice bob chuck + group Company1 c1_usera c1_userb + group Company2 c2_userx c2_usery c2_userz + acl allow admins create connection host=localhost + acl allow admins create connection host=10.0.0.0,10.255.255.255 + acl allow admins create connection host=192.168.0.0,192.168.255.255 + acl allow admins create connection host=[fc00::],[fc00::ff] + acl allow Company1 create connection host=company1.com + acl allow Company2 create connection host=company2.com + acl deny all create connection host=all + </programlisting> + </para> + <para> + Now admins are blocked from connecting from anywhere but their allowed + hosts. + </para> + + </section> + + <section id="sect-Messaging_User_Guide-Authorization-Specifying_ACL_Queue_Limits"> + <title>Queue Limits</title> + <para> + The ACL module creates a broker command line switch that set limits on the number of queues each user is allowed to create. This settings is not specified in the ACL file. + </para> + <para> + <programlisting> + --max-queues-per-user N + </programlisting> + </para> + <para> + The queue limit is set for all users on the broker. + </para> + <para> + The 0.22 C++ Broker ACL module accepts fine grained per-user queue limits through quota rules in the ACL file. + </para> + <para> + <programlisting> + quota queues 10 admins userX@QPID + </programlisting> + </para> + <para> + <itemizedlist> + <listitem> + User <literal>all</literal> receives the value passed by the command line switch <literal>--max-queues-per-user</literal>. + </listitem> + <listitem> + Values specified in the ACL rule for user <literal>all</literal> overwrite the value specified on the command line if any. + </listitem> + <listitem> + Queue quotas values are determined by first searching for the authenticated user name. If that user name is not specified then the value for user <literal>all</literal> + is used. If user <literal>all</literal> is not specified then the queue creation is denied. + </listitem> + <listitem> + The queue quota values range from 0..65530 inclusive. A value of zero disables queue creation by that user. + </listitem> + <listitem> + A user's quota may be specified many times in the ACL rule file. Only the last value specified is retained and enforced. + </listitem> + <listitem> + Per-user queue quotas are disabled when two conditions are true: 1) No --queue-limit-per-user command line switch and 2) No <literal>quota queues</literal> + rules in the ACL file. Per-user queue creations are always counted even if queue quotas are not enforced. This supports ACL file reloading that may subsequently + enable per-user queue quotas. + </listitem> + <listitem> + An ACL file reload may lower a user's queue quota value to a number lower than the user's current queue count. In that case the active queues + remain unaffected. New queues are denied until that user closes enough of his queues so that his count falls below the configured limit. + </listitem> + </itemizedlist> + </para> + </section> + + </section> + + <!-- ########################### --> <section id="sect-Messaging_User_Guide-Security-Encryption_using_SSL"> + <title>Encryption using SSL</title> + <para> + Encryption and certificate management for <command>qpidd</command> is provided by Mozilla's Network Security Services Library (NSS). + </para> + <orderedlist id="orde-Messaging_User_Guide-Encryption_using_SSL-Enabling_SSL_for_the_RHM_broker"> + <title>Enabling SSL for the Qpid broker</title> + <listitem> + <para> + You will need a certificate that has been signed by a Certification Authority (CA). This certificate will also need to be trusted by your client. If you require client authentication in addition to server authentication, the client's certificate will also need to be signed by a CA and trusted by the broker. + </para> + <para> + In the broker, SSL is provided through the <command>ssl.so</command> module. This module is installed and loaded by default in Qpid. To enable the module, you need to specify the location of the database containing the certificate and key to use. This is done using the <command>ssl-cert-db</command> option. + </para> + <para> + The certificate database is created and managed by the Mozilla Network Security Services (NSS) <command>certutil</command> tool. Information on this utility can be found on the <ulink url="http://www.mozilla.org/projects/security/pki/nss/tools/certutil.html">Mozilla website</ulink>, including tutorials on setting up and testing SSL connections. The certificate database will generally be password protected. The safest way to specify the password is to place it in a protected file, use the password file when creating the database, and specify the password file with the <command>ssl-cert-password-file</command> option when starting the broker. + </para> + <para> + The following script shows how to create a certificate database using certutil: + </para> + <!-- TODO: improve description --> +<programlisting> +mkdir ${CERT_DIR} +certutil -N -d ${CERT_DIR} -f ${CERT_PW_FILE} +certutil -S -d ${CERT_DIR} -n ${NICKNAME} -s "CN=${NICKNAME}" -t "CT,," -x -f ${CERT_PW_FILE} -z /usr/bin/certutil +</programlisting> + <para> + When starting the broker, set <command>ssl-cert-password-file</command> to the value of <command>${CERT_PW_FILE}</command>, set <command>ssl-cert-db</command> to the value of <command>${CERT_DIR}</command>, and set <command>ssl-cert-name</command> to the value of <command>${NICKNAME}</command>. + </para> + + </listitem> + <!-- SSL options --> <listitem> + <para> + The following SSL options can be used when starting the broker: + <variablelist> + <varlistentry> + <term><command>--ssl-use-export-policy</command></term> + <listitem> + <para> + Use NSS export policy + </para> + + </listitem> + + </varlistentry> + <varlistentry> + <term><command>--ssl-cert-password-file <replaceable>PATH</replaceable></command></term> + <listitem> + <para> + Required. Plain-text file containing password to use for accessing certificate database. + </para> + + </listitem> + + </varlistentry> + <varlistentry> + <term><command>--ssl-cert-db <replaceable>PATH</replaceable></command></term> + <listitem> + <para> + Required. Path to directory containing certificate database. + </para> + + </listitem> + + </varlistentry> + <varlistentry> + <term><command>--ssl-cert-name <replaceable>NAME</replaceable></command></term> + <listitem> + <para> + Name of the certificate to use. Default is <literal>localhost.localdomain</literal>. + </para> + + </listitem> + + </varlistentry> + <varlistentry> + <term><command>--ssl-port <replaceable>NUMBER</replaceable></command></term> + <listitem> + <para> + Port on which to listen for SSL connections. If no port is specified, port 5671 is used. + </para> + + </listitem> + + </varlistentry> + <varlistentry> + <term><command>--ssl-require-client-authentication</command></term> + <listitem> + <para> + Require SSL client authentication (i.e. verification of a client certificate) during the SSL handshake. This occurs before SASL authentication, and is independent of SASL. + </para> + <para> + This option enables the <literal>EXTERNAL</literal> SASL mechanism for SSL connections. If the client chooses the <literal>EXTERNAL</literal> mechanism, the client's identity is taken from the validated SSL certificate, using the <literal>CN</literal>literal>, and appending any <literal>DC</literal>literal>s to create the domain. For instance, if the certificate contains the properties <literal>CN=bob</literal>, <literal>DC=acme</literal>, <literal>DC=com</literal>, the client's identity is <literal>bob@acme.com</literal>. + </para> + <para> + If the client chooses a different SASL mechanism, the identity take from the client certificate will be replaced by that negotiated during the SASL handshake. + </para> + + </listitem> + + </varlistentry> + <varlistentry> + <term><command>--ssl-sasl-no-dict</command></term> + <listitem> + <para> + Do not accept SASL mechanisms that can be compromised by dictionary attacks. This prevents a weaker mechanism being selected instead of <literal>EXTERNAL</literal>, which is not vulnerable to dictionary attacks. + </para> + + </listitem> + + </varlistentry> + + </variablelist> + Also relevant is the <command>--require-encryption</command> broker option. This will cause <command>qpidd</command> to only accept encrypted connections. + </para> + + </listitem> + + </orderedlist> + <!-- --> <variablelist id="vari-Messaging_User_Guide-Encryption_using_SSL-Enabling_SSL_in_Clients"> + <title>Enabling SSL in Clients</title> + <varlistentry> + <term>C++ clients:</term> + <listitem> + <para> + <orderedlist> + <listitem> + <para> + In C++ clients, SSL is implemented in the <command>sslconnector.so</command> module. This module is installed and loaded by default in Qpid. + </para> + <para> + The following options can be specified for C++ clients using environment variables: + </para> + <table frame="all" id="tabl-Messaging_User_Guide-Enabling_SSL_in_Clients-SSL_Client_Environment_Variables_for_C_clients"> + <title>SSL Client Environment Variables for C++ clients</title> + <tgroup align="left" cols="2" colsep="1" rowsep="1"> + <colspec colname="c1"></colspec> + <colspec colname="c2"></colspec> + <thead> + <row> + <entry align="center" nameend="c2" namest="c1"> + SSL Client Options for C++ clients + </entry> + + </row> + + </thead> + <tbody> + <row> + <entry> + <command>QPID_SSL_USE_EXPORT_POLICY</command> + </entry> + <entry> + Use NSS export policy + </entry> + + </row> + <row> + <entry> + <command>QPID_SSL_CERT_PASSWORD_FILE <replaceable>PATH</replaceable></command> + </entry> + <entry> + File containing password to use for accessing certificate database + </entry> + + </row> + <row> + <entry> + <command>QPID_SSL_CERT_DB <replaceable>PATH</replaceable></command> + </entry> + <entry> + Path to directory containing certificate database + </entry> + + </row> + <row> + <entry> + <command>QPID_SSL_CERT_NAME <replaceable>NAME</replaceable></command> + </entry> + <entry> + Name of the certificate to use. When SSL client authentication is enabled, a certificate name should normally be provided. + </entry> + + </row> + + </tbody> + + </tgroup> + + </table> + <!-- ######## --> + </listitem> + <listitem> + <para> + When using SSL connections, clients must specify the location of the certificate database, a directory that contains the client's certificate and the public key of the Certificate Authority. This can be done by setting the environment variable <command>QPID_SSL_CERT_DB</command> to the full pathname of the directory. If a connection uses SSL client authentication, the client's password is also needed—the password should be placed in a protected file, and the <command>QPID_SSL_CERT_PASSWORD_FILE</command> variable should be set to the location of the file containing this password. + </para> + + </listitem> + <listitem> + <para> + To open an SSL enabled connection in the Qpid Messaging API, set the <parameter>protocol</parameter> connection option to <parameter>ssl</parameter>. + </para> + + </listitem> + + </orderedlist> + + </para> + + </listitem> + + </varlistentry> + <varlistentry> + <term>Java clients:</term> + <listitem> + <para> + <orderedlist> + <listitem> + <para> + For both server and client authentication, import the trusted CA to your trust store and keystore and generate keys for them. Create a certificate request using the generated keys and then create a certificate using the request. You can then import the signed certificate into your keystore. Pass the following arguments to the Java JVM when starting your client: +<programlisting> +-Djavax.net.ssl.keyStore=/home/bob/ssl_test/keystore.jks +-Djavax.net.ssl.keyStorePassword=password +-Djavax.net.ssl.trustStore=/home/bob/ssl_test/certstore.jks +-Djavax.net.ssl.trustStorePassword=password +</programlisting> + + </para> + + </listitem> + <listitem> + <para> + For server side authentication only, import the trusted CA to your trust store and pass the following arguments to the Java JVM when starting your client: +<programlisting> +-Djavax.net.ssl.trustStore=/home/bob/ssl_test/certstore.jks +-Djavax.net.ssl.trustStorePassword=password +</programlisting> + + </para> + + </listitem> + <listitem> + <para> + Java clients must use the SSL option in the connection URL to enable SSL encryption, e.g. + </para> + +<programlisting>amqp://username:password@clientid/test?brokerlist='tcp://localhost:5672?ssl='true'' +</programlisting> + + </listitem> + <listitem> + <para> + If you need to debug problems in an SSL connection, enable Java's SSL debugging by passing the argument <literal>-Djavax.net.debug=ssl</literal> to the Java JVM when starting your client. + </para> + + </listitem> + + </orderedlist> + + </para> + + </listitem> + + </varlistentry> + + </variablelist> + + </section> + + +</section> diff --git a/qpid/cpp/docs/book/src/cpp-broker/Using-Broker-Federation.xml b/qpid/cpp/docs/book/src/cpp-broker/Using-Broker-Federation.xml new file mode 100644 index 0000000000..614c168704 --- /dev/null +++ b/qpid/cpp/docs/book/src/cpp-broker/Using-Broker-Federation.xml @@ -0,0 +1,715 @@ +<?xml version="1.0" encoding="utf-8"?> +<!-- + +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. + +--> + +<section id="chap-Messaging_User_Guide-Broker_Federation"> + <title>Broker Federation</title> + <para> + <firstterm>Broker Federation</firstterm> allows messaging networks to be defined by creating <firstterm>message routes</firstterm>, in which messages in one broker (the <firstterm>source broker</firstterm>) are automatically routed to another broker (the <firstterm>destination broker</firstterm>). These routes may be defined between exchanges in the two brokers (the <firstterm>source exchange</firstterm> and the <firstterm>destination exchange</firstterm>), or from a queue in the source broker (the <firstterm>source queue</firstterm>) to an exchange in the destination broker. Message routes are unidirectional; when bidirectional flow is needed, one route is created in each direction. Routes can be durable or transient. A durable route survives broker restarts, restoring a route as soon as both the source broker and the destination are available. If the connection to a destination is lost, messages associated with a durable route continue to accumulate on the source, so they can be retrieved when the connection is reestablished. + </para> + <para> + Broker Federation can be used to build large messaging networks, with many brokers, one route at a time. If network connectivity permits, an entire distributed messaging network can be configured from a single location. The rules used for routing can be changed dynamically as servers change, responsibilities change, at different times of day, or to reflect other changing conditions. + </para> + <para> + Broker Federation is useful in a wide variety of scenarios. Some of these have to do with functional organization; for instance, brokers may be organized by geography, service type, or priority. Here are some use cases for federation: + <itemizedlist> + <listitem> + <para> + Geography: Customer requests may be routed to a processing location close to the customer. + </para> + + </listitem> + <listitem> + <para> + Service Type: High value customers may be routed to more responsive servers. + </para> + + </listitem> + <listitem> + <para> + Load balancing: Routing among brokers may be changed dynamically to account for changes in actual or anticipated load. + </para> + + </listitem> + <listitem> + <para> + High Availability: Routing may be changed to a new broker if an existing broker becomes unavailable. + </para> + + </listitem> + <listitem> + <para> + WAN Connectivity: Federated routes may connect disparate locations across a wide area network, while clients connect to brokers on their own local area network. Each broker can provide persistent queues that can hold messages even if there are gaps in WAN connectivity. + </para> + + </listitem> + <listitem> + <para> + Functional Organization: The flow of messages among software subsystems can be configured to mirror the logical structure of a distributed application. + </para> + + </listitem> + <listitem> + <para> + Replicated Exchanges: High-function exchanges like the XML exchange can be replicated to scale performance. + </para> + + </listitem> + <listitem> + <para> + Interdepartmental Workflow: The flow of messages among brokers can be configured to mirror interdepartmental workflow at an organization. + </para> + + </listitem> + + </itemizedlist> + + </para> + <section id="sect-Messaging_User_Guide-Broker_Federation-Message_Routes"> + <title>Message Routes</title> + <para> + Broker Federation is done by creating message routes. The destination for a route is always an exchange on the destination broker. By default, a message route is created by configuring the destination broker, which then contacts the source broker to subscribe to the source queue. This is called a <firstterm>pull route</firstterm>. It is also possible to create a route by configuring the source broker, which then contacts the destination broker in order to send messages. This is called a <firstterm>push route</firstterm>, and is particularly useful when the destination broker may not be available at the time the messaging route is configured, or when a large number of routes are created with the same destination exchange. + </para> + <para> + The source for a route can be either an exchange or a queue on the source broker. If a route is between two exchanges, the routing criteria can be given explicitly, or the bindings of the destination exchange can be used to determine the routing criteria. To support this functionality, there are three kinds of message routes: queue routes, exchange routes, and dynamic exchange routes. + </para> + <section id="sect-Messaging_User_Guide-Message_Routes-Queue_Routes"> + <title>Queue Routes</title> + <para> + <firstterm>Queue Routes</firstterm> route all messages from a source queue to a destination exchange. If message acknowledgement is enabled, messages are removed from the queue when they have been received by the destination exchange; if message acknowledgement is off, messages are removed from the queue when sent. + </para> + + </section> + + <section id="sect-Messaging_User_Guide-Message_Routes-Exchange_Routes"> + <title>Exchange Routes</title> + <para> + <firstterm>Exchange routes</firstterm> route messages from a source exchange to a destination exchange, using a binding key (which is optional for a fanout exchange). + </para> + <para> + Internally, creating an exchange route creates a private queue (auto-delete, exclusive) on the source broker to hold messages that are to be routed to the destination broker, binds this private queue to the source broker exchange, and subscribes the destination broker to the queue. + </para> + + </section> + + <section id="sect-Messaging_User_Guide-Message_Routes-Dynamic_Exchange_Routes"> + <title>Dynamic Exchange Routes</title> + <para> + Dynamic exchange routes allow a client to create bindings to an exchange on one broker, and receive messages that satisfy the conditions of these bindings not only from the exchange to which the client created the binding, but also from other exchanges that are connected to it using dynamic exchange routes. If the client modifies the bindings for a given exchange, they are also modified for dynamic exchange routes associated with that exchange. + </para> + <para> + <firstterm>Dynamic exchange routes</firstterm> apply all the bindings of a destination exchange to a source exchange, so that any message that would match one of these bindings is routed to the destination exchange. If bindings are added or removed from the destination exchange, these changes are reflected in the dynamic exchange route -- when the destination broker creates a binding with a given binding key, this is reflected in the route, and when the destination broker drops a binding with a binding key, the route no longer incurs the overhead of transferring messages that match the binding key among brokers. If two exchanges have dynamic exchange routes to each other, then all bindings in each exchange are reflected in the dynamic exchange route of the other. In a dynamic exchange route, the source and destination exchanges must have the same exchange type, and they must have the same name; for instance, if the source exchange is a direct exchange, the destination exchange must also be a direct exchange, and the names must match. + </para> + <para> + Internally, dynamic exchange routes are implemented in the same way as exchange routes, except that the bindings used to implement dynamic exchange routes are modified if the bindings in the destination exchange change. + </para> + <para> + A dynamic exchange route is always a pull route. It can never be a push route. + </para> + + </section> + + + </section> + + <section id="sect-Messaging_User_Guide-Broker_Federation-Federation_Topologies"> + <title>Federation Topologies</title> + <para> + A federated network is generally a tree, star, or line, using bidirectional links (implemented as a pair of unidirectional links) between any two brokers. A ring topology is also possible, if only unidirectional links are used. + </para> + <para> + Every message transfer takes time. For better performance, you should minimize the number of brokers between the message origin and final destination. In most cases, tree or star topologies do this best. + </para> + <para> + For any pair of nodes A,B in a federated network, there should be only one path from A to B. If there is more than one path, message loops can cause duplicate message transmission and flood the federated network. The topologies discussed above do not have message loops. A ring topology with bidirectional links is one example of a topology that does cause this problem, because a given broker can receive the same message from two different brokers. Mesh topologies can also cause this problem. + </para> + + </section> + + <section id="sect-Messaging_User_Guide-Broker_Federation-Federation_among_High_Availability_Message_Clusters"> + <title>Federation among High Availability Message Clusters</title> + <para> + Federation is generally used together with High Availability Message Clusters, using clusters to provide high availability on each LAN, and federation to route messages among the clusters. Because message state is replicated within a cluster, it makes little sense to define message routes between brokers in the same cluster. + </para> + <para> + To create a message route between two clusters, simply create a route between any one broker in the first cluster and any one broker in the second cluster. Each broker in a given cluster can use message routes defined for another broker in the same cluster. If the broker for which a message route is defined should fail, another broker in the same cluster can restore the message route. + </para> + + </section> + + <section id="sect-Messaging_User_Guide-Broker_Federation-The_qpid_route_Utility"> + <title>The qpid-route Utility</title> + <para> + <command>qpid-route</command> is a command line utility used to configure federated networks of brokers and to view the status and topology of networks. It can be used to configure routes among any brokers that <command>qpid-route</command> can connect to. + </para> + <para> + The syntax of <command>qpid-route</command> is as follows: + </para> + + <screen> + qpid-route [OPTIONS] dynamic add <dest-broker> <src-broker> <exchange> + qpid-route [OPTIONS] dynamic del <dest-broker> <src-broker> <exchange> + + qpid-route [OPTIONS] route add <dest-broker> <src-broker> <exchange> <routing-key> + qpid-route [OPTIONS] route del <dest-broker> <src-broker> <exchange> <routing-key> + + qpid-route [OPTIONS] queue add <dest-broker> <src-broker> <dest-exchange> <src-queue> + qpid-route [OPTIONS] queue del <dest-broker> <src-broker> <dest-exchange> <src-queue> + + qpid-route [OPTIONS] list [<broker>] + qpid-route [OPTIONS] flush [<broker>] + qpid-route [OPTIONS] map [<broker>] + + <!-- qpid-route [OPTIONS] add connection <dest-broker> <src-broker> + qpid-route [OPTIONS] del connection <dest-broker> <src-broker> --> + qpid-route [OPTIONS] list connections [<broker>] + </screen> + <para> + The syntax for <command>broker</command>, <command>dest-broker</command>, and <command>src-broker</command> is as follows: + </para> + + <screen> + [username/password@] hostname | ip-address [:<port>] + </screen> + <para> + The following are all valid examples of the above syntax: <command>localhost</command>, <command>10.1.1.7:10000</command>, <command>broker-host:10000</command>, <command>guest/guest@localhost</command>. + </para> + <para> + These are the options for <command>qpid-route</command>: + </para> + <table frame="all" id="tabl-Messaging_User_Guide-The_qpid_route_Utility-qpid_route_options"> + <title><command>qpid-route</command> options</title> + <tgroup align="left" cols="2" colsep="1" rowsep="1"> + <colspec colname="c1"></colspec> + <colspec colname="c2"></colspec> + <!-- <thead> + <row> + <entry align="center" nameend="c2" namest="c1"> + Options for using <command>qpid-route</command> to Manage Federation + </entry> + </row> + </thead> + --> <tbody> + <row> + <entry> + <command>-v</command> + </entry> + <entry> + Verbose output. + </entry> + + </row> + <row> + <entry> + <command>-q</command> + </entry> + <entry> + Quiet output, will not print duplicate warnings. + </entry> + + </row> + <row> + <entry> + <command>-d</command> + </entry> + <entry> + Make the route durable. + </entry> + + </row> + <!-- + <row> + <entry> + <command>-e</command> + </entry> + <entry> + Delete link after deleting the last route on the link. + </entry> + </row> --> <row> + <entry> + <command> --timeout N</command> + </entry> + <entry> + Maximum time to wait when qpid-route connects to a broker, in seconds. Default is 10 seconds. + </entry> + + </row> + <row> + <entry> + <command>--ack N</command> + </entry> + <entry> + Acknowledge transfers of routed messages in batches of N. Default is 0 (no acknowledgements). Setting to 1 or greater enables acknowledgements; when using acknowledgements, values of N greater than 1 can significnantly improve performance, especially if there is significant network latency between the two brokers. + </entry> + + </row> + <row> + <entry> + <command>-s [ --src-local ]</command> + </entry> + <entry> + Configure the route in the source broker (create a push route). + </entry> + + </row> + <row> + <entry> + <command>-t <transport> [ --transport <transport>]</command> + </entry> + <entry> + Transport protocol to be used for the route. + <itemizedlist> + <listitem> + <para> + tcp (default) + </para> + + </listitem> + <listitem> + <para> + ssl + </para> + + </listitem> + <listitem> + <para> + rdma + </para> + + </listitem> + + </itemizedlist> + + </entry> + + </row> + + </tbody> + + </tgroup> + + </table> + <section id="sect-Messaging_User_Guide-The_qpid_route_Utility-Creating_and_Deleting_Queue_Routes"> + <title>Creating and Deleting Queue Routes</title> + <para> + The syntax for creating and deleting queue routes is as follows: + </para> + + <screen> + qpid-route [OPTIONS] queue add <dest-broker> <src-broker> <dest-exchange> <src-queue> + qpid-route [OPTIONS] queue del <dest-broker> <src-broker> <dest-exchange> <src-queue> + </screen> + <para> + For instance, the following creates a queue route that routes all messages from the queue named <command>public</command> on the source broker <command>localhost:10002</command> to the <command>amq.fanout</command> exchange on the destination broker <command>localhost:10001</command>: + </para> + + <screen> + $ qpid-route queue add localhost:10001 localhost:10002 amq.fanout public + </screen> + <para> + If the <command>-d</command> option is specified, this queue route is persistent, and will be restored if one or both of the brokers is restarted: + </para> + + <screen> + $ qpid-route -d queue add localhost:10001 localhost:10002 amq.fanout public + </screen> + <para> + The <command>del</command> command takes the same arguments as the <command>add</command> command. The following command deletes the queue route described above: + </para> + + <screen> + $ qpid-route queue del localhost:10001 localhost:10002 amq.fanout public + </screen> + + </section> + + <section id="sect-Messaging_User_Guide-The_qpid_route_Utility-Creating_and_Deleting_Exchange_Routes"> + <title>Creating and Deleting Exchange Routes</title> + <para> + The syntax for creating and deleting exchange routes is as follows: + </para> + + <screen> + qpid-route [OPTIONS] route add <dest-broker> <src-broker> <exchange> <routing-key> + qpid-route [OPTIONS] route del <dest-broker> <src-broker> <exchange> <routing-key> + qpid-route [OPTIONS] flush [<broker>] + </screen> + <para> + For instance, the following creates an exchange route that routes messages that match the binding key <command>global.#</command> from the <command>amq.topic</command> exchange on the source broker <command>localhost:10002</command> to the <command>amq.topic</command> exchange on the destination broker <command>localhost:10001</command>: + </para> + + <screen> + $ qpid-route route add localhost:10001 localhost:10002 amq.topic global.# + </screen> + <para> + In many applications, messages published to the destination exchange should also be routed to the source exchange. This is accomplished by creating a second exchange route, reversing the roles of the two exchanges: + </para> + + <screen> + $ qpid-route route add localhost:10002 localhost:10001 amq.topic global.# + </screen> + <para> + If the <command>-d</command> option is specified, the exchange route is persistent, and will be restored if one or both of the brokers is restarted: + </para> + + <screen> + $ qpid-route -d route add localhost:10001 localhost:10002 amq.fanout public + </screen> + <para> + The <command>del</command> command takes the same arguments as the <command>add</command> command. The following command deletes the first exchange route described above: + </para> + + <screen> + $ qpid-route route del localhost:10001 localhost:10002 amq.topic global.# + </screen> + + </section> + + <section id="sect-Messaging_User_Guide-The_qpid_route_Utility-Deleting_all_routes_for_a_broker"> + <title>Deleting all routes for a broker</title> + <para> + Use the <command>flush</command> command to delete all routes for a given broker: + </para> + + <screen> + qpid-route [OPTIONS] flush [<broker>] + </screen> + <para> + For instance, the following command deletes all routes for the broker <command>localhost:10001</command>: + </para> + + <screen> + $ qpid-route flush localhost:10001 + </screen> + + </section> + + <section id="sect-Messaging_User_Guide-The_qpid_route_Utility-Creating_and_Deleting_Dynamic_Exchange_Routes"> + <title>Creating and Deleting Dynamic Exchange Routes</title> + <para> + The syntax for creating and deleting dynamic exchange routes is as follows: + </para> + + <screen> + qpid-route [OPTIONS] dynamic add <dest-broker> <src-broker> <exchange> + qpid-route [OPTIONS] dynamic del <dest-broker> <src-broker> <exchange> + </screen> + <para> + In the following examples, we will route messages from a topic exchange. We will create a new topic exchange and federate it so that we are not affected by other all clients that use the built-in <command>amq.topic</command> exchange. The following commands create a new topic exchange on each of two brokers: + </para> + + <screen> + $ qpid-config -a localhost:10003 add exchange topic fed.topic + $ qpid-config -a localhost:10004 add exchange topic fed.topic + </screen> + <para> + Now let's create a dynamic exchange route that routes messages from the <command>fed.topic</command> exchange on the source broker <command>localhost:10004</command> to the <command>fed.topic</command> exchange on the destination broker <command>localhost:10003</command> if they match any binding on the destination broker's <command>fed.topic</command> exchange: + </para> + + <screen> + $ qpid-route dynamic add localhost:10003 localhost:10004 fed.topic + </screen> + <para> + Internally, this creates a private autodelete queue on the source broker, and binds that queue to the <command>fed.topic</command> exchange on the source broker, using each binding associated with the <command>fed.topic</command> exchange on the destination broker. + </para> + <para> + In many applications, messages published to the destination exchange should also be routed to the source exchange. This is accomplished by creating a second dynamic exchange route, reversing the roles of the two exchanges: + </para> + + <screen> + $ qpid-route dynamic add localhost:10004 localhost:10003 fed.topic + </screen> + <para> + If the <command>-d</command> option is specified, the exchange route is persistent, and will be restored if one or both of the brokers is restarted: + </para> + + <screen> + $ qpid-route -d dynamic add localhost:10004 localhost:10003 fed.topic + </screen> + <para> + When an exchange route is durable, the private queue used to store messages for the route on the source exchange is also durable. If the connection between the brokers is lost, messages for the destination exchange continue to accumulate until it can be restored. + </para> + <para> + The <command>del</command> command takes the same arguments as the <command>add</command> command. The following command deletes the first exchange route described above: + </para> + + <screen> + $ qpid-route dynamic del localhost:10004 localhost:10003 fed.topic + </screen> + <para> + Internally, this deletes the bindings on the source exchange for the the private queues associated with the message route. + </para> + + </section> + + <section id="sect-Messaging_User_Guide-The_qpid_route_Utility-Viewing_Routes"> + <title>Viewing Routes</title> + <para> + The <command>route list</command> command shows the routes associated with an individual broker. For instance, suppose we have created the following two routes: + </para> + + <screen> + $ qpid-route dynamic add localhost:10003 localhost:10004 fed.topic + $ qpid-route dynamic add localhost:10004 localhost:10003 fed.topic + </screen> + <para> + We can now use <command>route list</command> to show all routes for the broker <command>localhost:10003</command>: + </para> + + <screen> + $ qpid-route route list localhost:10003 + localhost:10003 localhost:10004 fed.topic <dynamic> + </screen> + <para> + Note that this shows only one of the two routes we created, the route for which <command>localhost:10003</command> is a destination. If we want to see the route for which <command>localhost:10004</command> is a destination, we need to do another route list: + </para> + + <screen> + $ qpid-route route list localhost:10004 + localhost:10004 localhost:10003 fed.topic <dynamic> + </screen> + <para> + The <command>route map</command> command shows all routes associated with a broker, and recursively displays all routes for brokers involved in federation relationships with the given broker. For instance, here is the output for the two brokers configured above: + </para> + + <screen> + $ qpid-route route map localhost:10003 + + Finding Linked Brokers: + localhost:10003... Ok + localhost:10004... Ok + + Dynamic Routes: + + Exchange fed.topic: + localhost:10004 <=> localhost:10003 + + Static Routes: + none found + </screen> + <para> + Note that the two dynamic exchange links are displayed as though they were one bidirectional link. The <command>route map</command> command is particularly helpful for larger, more complex networks. Let's configure a somewhat more complex network with 16 dynamic exchange routes: + </para> + + <screen> + qpid-route dynamic add localhost:10001 localhost:10002 fed.topic + qpid-route dynamic add localhost:10002 localhost:10001 fed.topic + + qpid-route dynamic add localhost:10003 localhost:10002 fed.topic + qpid-route dynamic add localhost:10002 localhost:10003 fed.topic + + qpid-route dynamic add localhost:10004 localhost:10002 fed.topic + qpid-route dynamic add localhost:10002 localhost:10004 fed.topic + + qpid-route dynamic add localhost:10002 localhost:10005 fed.topic + qpid-route dynamic add localhost:10005 localhost:10002 fed.topic + + qpid-route dynamic add localhost:10005 localhost:10006 fed.topic + qpid-route dynamic add localhost:10006 localhost:10005 fed.topic + + qpid-route dynamic add localhost:10006 localhost:10007 fed.topic + qpid-route dynamic add localhost:10007 localhost:10006 fed.topic + + qpid-route dynamic add localhost:10006 localhost:10008 fed.topic + qpid-route dynamic add localhost:10008 localhost:10006 fed.topic + </screen> + <para> + Now we can use <command>route map</command> starting with any one broker, and see the entire network: + </para> + + <screen> + $ ./qpid-route route map localhost:10001 + + Finding Linked Brokers: + localhost:10001... Ok + localhost:10002... Ok + localhost:10003... Ok + localhost:10004... Ok + localhost:10005... Ok + localhost:10006... Ok + localhost:10007... Ok + localhost:10008... Ok + + Dynamic Routes: + + Exchange fed.topic: + localhost:10002 <=> localhost:10001 + localhost:10003 <=> localhost:10002 + localhost:10004 <=> localhost:10002 + localhost:10005 <=> localhost:10002 + localhost:10006 <=> localhost:10005 + localhost:10007 <=> localhost:10006 + localhost:10008 <=> localhost:10006 + + Static Routes: + none found + </screen> + + </section> + + <section id="sect-Messaging_User_Guide-The_qpid_route_Utility-Resilient_Connections"> + <title>Resilient Connections</title> + <para> + When a broker route is created, or when a durable broker route is restored after broker restart, a connection is created between the source broker and the destination broker. The connections used between brokers are called <firstterm>resilient connections</firstterm>; if the connection fails due to a communication error, it attempts to reconnect. The retry interval begins at 2 seconds and, as more attempts are made, grows to 64 seconds, and continues to retry every 64 seconds thereafter. If the connection fails due to an authentication problem, it will not continue to retry. + </para> + <para> + The command <command>list connections</command> can be used to show the resilient connections for a broker: + </para> + + <screen> + $ qpid-route list connections localhost:10001 + + Host Port Transport Durable State Last Error + ============================================================================= + localhost 10002 tcp N Operational + localhost 10003 tcp N Operational + localhost 10009 tcp N Waiting Connection refused + </screen> + <para> + In the above output, <command>Last Error</command> contains the string representation of the last connection error received for the connection. <command>State</command> represents the state of the connection, and may be one of the following values: + </para> + <table id="tabl-Messaging_User_Guide-Resilient_Connections-State_values_in_qpid_route_list_connections"> + <title>State values in <command>$ qpid-route list connections</command></title> + <tgroup align="left" cols="2" colsep="1" rowsep="1"> + <tbody> + <row> + <entry> + Waiting + </entry> + <entry> + Waiting before attempting to reconnect. + </entry> + + </row> + <row> + <entry> + Connecting + </entry> + <entry> + Attempting to establish the connection. + </entry> + + </row> + <row> + <entry> + Operational + </entry> + <entry> + The connection has been established and can be used. + </entry> + + </row> + <row> + <entry> + Failed + </entry> + <entry> + The connection failed and will not retry (usually because authentication failed). + </entry> + + </row> + <row> + <entry> + Closed + </entry> + <entry> + The connection has been closed and will soon be deleted. + </entry> + + </row> + <row> + <entry> + Passive + </entry> + <entry> + If a cluster is federated to another cluster, only one of the nodes has an actual connection to remote node. Other nodes in the cluster have a passive connection. + </entry> + + </row> + + </tbody> + + </tgroup> + + </table> + + </section> + + </section> + + <section id="federation-broker-options"> + <title>Broker options affecting federation</title> + <para> + The following broker options affect federation: + <table frame="all" id="federation-broker-options-table"> + <title>Broker Options for Federation</title> + <tgroup align="left" cols="2" colsep="1" rowsep="1"> + <colspec colname="c1"/> + <colspec colname="c2"/> + <thead> + <row> + <entry align="center" nameend="c2" namest="c1"> + Options for Federation + </entry> + </row> + </thead> + <tbody> + <row> + <entry> + <literal>federation-tag <replaceable>NAME</replaceable></literal> + </entry> + <entry> + A unique name to identify this broker in federation network. + If not specified, the broker will generate a unique identifier. + </entry> + </row> + <row> + <entry> + <literal>link-maintenance-interval <replaceable>SECONDS</replaceable></literal> + <footnoteref linkend="ha-seconds-spec"/> + </entry> + <entry> + <para> + Interval to check if links need to be re-connected. Default 2 + seconds. Can be a sub-second interval for faster failover, + e.g. 0.1 seconds. + </para> + </entry> + </row> + <row> + <entry> + <literal>link-heartbeat-interval <replaceable>SECONDS</replaceable></literal> + <footnoteref linkend="ha-seconds-spec"/> + </entry> + <entry> + <para> + Heart-beat interval for federation links. If no heart-beat is + received for twice the interval the link is considered dead. + Default 120 seconds. + </para> + </entry> + </row> + </tbody> + </tgroup> + </table> + </para> + </section> +</section> diff --git a/qpid/cpp/docs/book/src/cpp-broker/Using-message-groups.xml b/qpid/cpp/docs/book/src/cpp-broker/Using-message-groups.xml new file mode 100644 index 0000000000..9b904d9f18 --- /dev/null +++ b/qpid/cpp/docs/book/src/cpp-broker/Using-message-groups.xml @@ -0,0 +1,295 @@ +<?xml version="1.0" encoding="utf-8"?> +<!-- + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +--> + +<section id="Using-message-groups"> + <title> + Using Message Groups + </title> + + <section role="h2" id="usingmessagegroups-Overview"> + <title> + Overview + </title> + <para> + The broker allows messaging applications to classify a set of related messages as + belonging to a group. This allows a message producer to indicate to the consumer + that a group of messages should be considered a single logical operation with + respect to the application. + </para> + <para> + The broker can use this group identification to enforce policies controlling how + messages from a given group can be distributed to consumers. For instance, the + broker can be configured to guarantee all the messages from a particular group are + processed in order across multiple consumers. + </para> + <para> + For example, assume we have a shopping application that manages items in a virtual + shopping cart. A user may add an item to their shopping cart, then change their + mind and remove it. If the application sends an <emphasis>add</emphasis> message to the broker, + immediately followed by a <emphasis>remove</emphasis> message, they will be queued in the proper + order - <emphasis>add</emphasis>, followed by <emphasis>remove</emphasis>. + </para> + <para> + However, if there are multiple consumers, it is possible that once a consumer + acquires the <emphasis>add</emphasis> message, a different consumer may acquire the + <emphasis>remove</emphasis> message. This allows both messages to be processed in parallel, + which could result in a "race" where the <emphasis>remove</emphasis> operation is incorrectly + performed before the <emphasis>add</emphasis> operation. + </para> + </section> + <!--h2--> + <section role="h2" id="usingmessagegroups-GroupingMessages"> + <title> + Grouping Messages + </title> + <para> + In order to group messages, the application would designate a particular + message header as containing a message's <emphasis>group identifier</emphasis>. The group + identifier stored in that header field would be a string value set by the message + producer. Messages from the same group would have the same group identifier + value. The key that identifies the header must also be known to the message + consumers. This allows the consumers to determine a message's assigned group. + </para> + <para> + The header that is used to hold the group identifier, as well as the values used + as group identifiers, are totally under control of the application. + </para> + </section> + <section role="h2" id="usingmessagegroups-BrokerRole"> + <title> + The Role of the Broker + </title> + <para> + The broker will apply the following processing on each grouped message: + <itemizedlist> + <listitem>Enqueue a received message on the destination queue.</listitem> + <listitem>Determine the message's group by examining the message's group identifier header.</listitem> + <listitem>Enforce <emphasis>consumption ordering</emphasis> among messages belonging to the same group.</listitem> + </itemizedlist> + <emphasis>Consumption ordering</emphasis> means that the broker will not allow outstanding + unacknowledged messages to <emphasis>more than one consumer for a given group</emphasis>. + </para> + <para> + This means that only one consumer can be processing messages from a particular + group at a given time. When the consumer acknowledges all of its acquired + messages, then the broker <emphasis>may</emphasis> pass the next pending message + from that group to a different consumer. + </para> + <para> + Specifically, for any given group the broker allows only the first N messages in + the group to be delivered to a consumer. The value of N would be determined by + the selected consumer's configured prefetch capacity. The broker blocks access by + any other consumer to any remaining undelivered messages in that group. Once the + receiving consumer has: + <itemizedlist> + <listitem>acknowledged,</listitem> + <listitem>released, or</listitem> + <listitem>rejected</listitem> + </itemizedlist> + all the delivered messages, the broker allows the next messages in the group to be + delivered. The next messages <emphasis>may</emphasis> be delivered to a different + consumer. + </para> + <para> + Note well that distinct message groups would not block each other from delivery. + For example, assume a queue contains messages from two different message groups - + say group "A" and group "B" - and they are enqueued such that "A"'s messages are + in front of "B". If the first message of group "A" is in the process of being + consumed by a client, then the remaining "A" messages are blocked, but the + messages of the "B" group are available for consumption by other consumers - even + though it is "behind" group "A" in the queue. + </para> + </section> + <section role="h2" id="usingmessagegroups-ConsumerGuide"> + <title> + Well Behaved Consumers + </title> + <para> + The broker can only enforce policy when delivering messages. To guarantee that + strict message ordering is preserved, the consuming application must adhere to the + following rules: + <itemizedlist> + <listitem>completely process the data in a received message before accepting + that message</listitem> + <listitem>acknowledge (or reject) messages in the same order as they are + received</listitem> + <listitem>avoid releasing messages (see below)</listitem> + </itemizedlist> + The term <emphasis>processed</emphasis> means that the consumer has finished + updating all application state affected by the message that has been received. + See section 2.6.2. Transfer of Responsibility, of the AMQP-0.10 specification for + more detail. + </para> + <note> + <title>Be Advised</title> + <para> + If a consumer does not adhere to the above rules, it may affect the ordering of + grouped messages even when the broker is enforcing consumption order. This can + be done by selectively acknowledging and releasing messages from the same group. + </para> + <para> + Assume a consumer has received two messages from group "A", "A-1" and "A-2", in + that order. If the consumer releases "A-1" then acknowledges "A-2", "A-1" will + be put back onto the queue and "A-2" will be removed from the queue. This + allows another consumer to acquire and process "A-1" <emphasis>after</emphasis> + "A-2" has been processed. + </para> + <para> + Under some application-defined circumstances, this may be acceptable behavior. + However, if order must be preserved, the client should either release + <emphasis>all</emphasis> currently held messages, or discard the target message + using reject. + </para> + </note> + </section> + <!--h2--> + <section role="h2" id="usingmessagegroups-BrokerConfig"> + <title> + Broker Configuration + </title> + <para> + In order for the broker to determine a message's group, the key for the header + that contains the group identifier must be provided to the broker via + configuration. This is done on a per-queue basis, when the queue is first + configured. + </para> + <para> + This means that message group classification is determined by the message's destination + queue. + </para> + <para> + Specifically, the queue "holds" the header key that is used to find the message's + group identifier. All messages arriving at the queue are expected to use the same + header key for holding the identifer. Once the message is enqueued, the broker + looks up the group identifier in the message's header, and classifies the message + by its group. + </para> + <para> + Message group support can be enabled on a queue using the + <command>qpid-config</command> command line tool. The following options should be + provided when adding a new queue: + <table> + <title>qpid-config options for creating message group queues</title> + <tgroup cols="2"> + <thead> + <colspec colnum="1" colwidth="1*"/> + <colspec colnum="2" colwidth="3*"/> + <row> + <entry>Option</entry><entry>Description</entry> + </row> + </thead> + <tbody> + <row> + <entry>--group-header=<replaceable>header-name</replaceable></entry> + <entry>Enable message group support for this queue. Specify name of application header that holds the group identifier.</entry> + </row> + <row> + <entry>--shared-groups</entry> + <entry>Enforce ordered message group consumption across multiple consumers.</entry> + </row> + </tbody> + </tgroup> + </table> + </para> + <para> + Message group support may also be specified in the + <command>queue.declare</command> method via the <command>arguments</command> + parameter map, or using the messaging address syntax. The following keys must be + provided in the arguments map to enable message group support on a queue: + </para> + <table> + <title>Queue Declare/Address Syntax Message Group Configuration Arguments</title> + <tgroup cols="2"> + <thead> + <row> + <entry>Key</entry> + <entry>Value</entry> + </row> + </thead> + <tbody> + <row> + <entry>qpid.group_header_key</entry> + <entry>string - key for message header that holds the group identifier value</entry> + </row> + <row> + <entry>qpid.shared_msg_group</entry> + <entry>1 - enforce ordering across multiple consumers</entry> + </row> + </tbody> + </tgroup> + </table> + <para> + It is important to note that there is no need to provide the actual group + identifer values that will be used. The broker learns this values as messages are + recieved. Also, there is no practical limit - aside from resource limitations - + to the number of different groups that the broker can track at run time. + </para> + <note> + <title>Restrictions</title> + <para> + Message grouping is not supported on LVQ or Priority queues. + </para> + </note> + <example> + <title>Creating a message group queue via qpid-config</title> + <para> + This example uses the qpid-config tool to create a message group queue called + "MyMsgQueue". The message header that contains the group identifier will use + the key "GROUP_KEY". + </para> + <programlisting> +qpid-config add queue MyMsgQueue --group-header="GROUP_KEY" --shared-groups + </programlisting> + </example> + <example> + <title>Creating a message group queue using address syntax (C++)</title> + <para> + This example uses the messaging address syntax to create a message group queue + with the same configuration as the previous example. + </para> + <programlisting> +sender = session.createSender("MyMsgQueue;" + " {create:always, delete:receiver," + " node: {x-declare: {arguments:" + " {'qpid.group_header_key':'GROUP_KEY'," + " 'qpid.shared_msg_group':1}}}}") + </programlisting> + </example> + <section role="h3" id="usingmessagegroups-DefaultGroup"> + <title> + Default Group + </title> + <para> + Should a message without a group identifier arrive at a queue configured for message grouping, the broker assigns the message to the default group. Therefore, all such "unidentified" messages are considered by the broker as part of the same group. The name of the default group is <command>"qpid.no-group"</command>. This default can be overridden by suppling a different value to the broker configuration item <command>"default-message-group"</command>: + <example> + <title>Overriding the default message group identifier for the broker</title> + <programlisting> +qpidd --default-msg-group "EMPTY-GROUP" + </programlisting> + </example> + </para> + </section> + </section> + </section> + + + diff --git a/qpid/cpp/docs/book/src/cpp-broker/producer-flow-control.xml b/qpid/cpp/docs/book/src/cpp-broker/producer-flow-control.xml new file mode 100644 index 0000000000..fd44f51e81 --- /dev/null +++ b/qpid/cpp/docs/book/src/cpp-broker/producer-flow-control.xml @@ -0,0 +1,351 @@ +<?xml version="1.0" encoding="utf-8"?> +<!-- + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +--> + +<section id="producer-flow-control"> + <title> + Producer Flow Control + </title> + + <section role="h2" id="producerflowcontrol-Overview"> + <title> + Overview + </title> + <para> + As of release 0.10, the C++ broker supports the use of flow control to + throttle back message producers that are at risk of overflowing a + destination queue. + </para> + + <para> + Each queue in the C++ broker has two threshold values associated with it: + </para> + + <para> + Flow Stop Threshold: this is the level of queue resource + utilization above which flow control will be enabled. Once this + threshold is crossed, the queue is considered in danger of overflow. + </para> + + <para> + Flow Resume Threshold - this is the level of queue resource utilization + below which flow control will be disabled. Once this threshold is + crossed, the queue is no longer considered in danger of overflow. + </para> + + <para> + In the above description, queue resource utilization may be + defined as the total count of messages currently enqueued, or the total + sum of all message content in bytes. + </para> + + <para> + The value for a queue's Flow Stop Threshold must be greater than or + equal to the value of the queue's Flow Resume Threshold. + </para> + + <section role="h3" id="producerflowcontrol-QueueThresholdsExample"> + <title> + Example + </title> + + <para> + Let's consider a queue with a maximum limit set on the total number of + messages that may be enqueued to that queue. Assume this maximum + message limit is 1000 messages. Assume also that the user configures a + Flow Stop Threshold of 900 messages, and a Flow Resume Threshold of 500 + messages. Then the following holds: + </para> + + <para> + The queue's initial flow control state is "OFF". + </para> + + <para> + While the total number of enqueued messages is less than or equal to + 900, the queue's flow control state remains "OFF". + </para> + + <para> + When the total number of enqueued messages is greater than 900, the + queue's flow control state transitions to "ON". + </para> + + <para> + When the queue's flow control state is "ON", it remains "ON" until the + total number of enqueued messages is less than 500. At that point, the queue's + flow control state transitions to "OFF". + </para> + + <para> + A similar example using total enqueued content bytes as the threshold + units are permitted. + </para> + </section> + + <para> + Thresholds may be set using both total message counts and total byte + counts. In this case, the following rules apply: + </para> + + <para> + 1) Flow control is "ON" when either stop threshold value is crossed. + </para> + <para> + 2) Flow control remains "ON" until both resume thresholds are satisfied. + </para> + + <section role="h3" id="producerflowcontro-MultiThresholdExample"> + <title> + Example + </title> + + <para> + Let's consider a queue with a maximum size limit of 10K bytes, and 5000 + messages. A user may assign a Flow Stop Threshold based on a total + message count of 4000 messages. They may also assigne a Flow Stop + Threshold of 8K bytes. The queue's flow control state transitions to + "ON" if either threshold is crossed: (total-msgs greater-than 4000 OR total-bytes + greater-than 8K). + </para> + + <para> + Assume the user has assigned Flow Resume threshold's of 3000 messages and + 6K bytes. Then the queue's flow control will remain active until both + thresholds are satified: (total-msg less-than 3000 AND total-bytes less-than 6K). + </para> + </section> + + <para> + The Broker enforces flow control by delaying the completion of the + Message.Transfer command that causes a message to be delivered to a queue + with active flow control. The completion of the Message.Transfer command + is held off until flow control state transitions to "OFF" for all queues + that are a destination for that command. + </para> + + <para> + A message producing client is permitted to have a finite number of + commands pending completion. When the total number of these outstanding + commands reaches the limit, the client must not issue further commands + until one or more of the outstanding commands have completed. This + window of outstanding commands is considered the sender's "capacity". + This allows any given producer to have a "capacity's" worth of messages + blocked due to flow control before the sender must stop sending further + messages. + </para> + + <para> + This capacity window must be considered when determining a suitable + flow stop threshold for a given queue, as a producer may send its + capacity worth of messages _after_ a queue has reached the flow stop + threshold. Therefore, a flow stop threshould should be set such that + the queue can accomodate more messages without overflowing. + </para> + + <para> + For example, assume two clients, C1 and C2, are producing messages to + one particular destination queue. Assume client C1 has a configured + capacity of 50 messages, and client C2's capacity is 15 messages. In + this example, assume C1 and C2 are the only clients queuing messages to + a given queue. If this queue has a Flow Stop Threshold of 100 + messages, then, worst-case, the queue may receive up to 165 messages + before clients C1 and C2 are blocked from sending further messages. + This is due to the fact that the queue will enable flow control on + receipt of its 101'st message - preventing the completion of the + Message.Transfer command that carried the 101'st message. However, C1 + and C2 are allowed to have a total of 65 (50 for C1 and 15 for C2) + messages pending completion of Message.Transfer before they will stop + producing messages. Thus, up to 65 messages may be enqueued beyond the + flow stop threshold before the producers will be blocked. + </para> + </section> + + <section role="h2" id="producerflowcontrol-UserInterface"> + <title> + User Interface + </title> + + <para> + By default, the C++ broker assigns a queue's flow stop and flow resume + thresholds when the queue is created. The C++ broker also allows the + user to manually specify the flow control thresholds on a per queue + basis. + </para> + + <para> + However, queues that have been configured with a Limit Policy of type + RING or RING-STRICT do NOT have queue flow thresholds enabled by + default. The nature of a RING queue defines its behavior when its + capacity is reach: replace the oldest message. + </para> + + <para> + The flow control state of a queue can be determined by the "flowState" + boolean in the queue's QMF management object. The queue's management + object also contains a counter that increments each time flow control + becomes active for the queue. + </para> + + <para> + The broker applies a threshold ratio to compute a queue's default flow + control configuration. These thresholds are expressed as a percentage + of a queue's maximum capacity. There is one value for determining the + stop threshold, and another for determining the resume threshold. The + user may configure these percentages using the following broker + configuration options: + </para> + + <programlisting> + --default-flow-stop-threshold ("Queue capacity level at which flow control is activated.") + --default-flow-resume-threshold ("Queue capacity level at which flow control is de-activated.") + </programlisting> + + <para> + For example: + </para> + + <programlisting> + qpidd --default-flow-stop-threshold=90 --default-flow-resume-threshold=75 + </programlisting> + + <para> + Sets the default flow stop threshold to 90% of a queue's maximum + capacity and the flow resume threshold to 75% of the maximum capacity. + If a queue is created with a default-queue-limit of 10000 bytes, then + the default flow stop threshold would be 90% of 10000 = 9000 bytes and + the flow resume threshold would be 75% of 10000 = 7500. The same + computation is performed should a queue be created with a maximum size + expressed as a message count instead of a byte count. + </para> + + <para> + If not overridden by the user, the value of the + default-flow-stop-threshold is 80% and the value of the + default-flow-resume-threshold is 70%. + </para> + + <para> + The user may disable default queue flow control broker-wide by + specifying the value 0 for both of these configuration options. Note + that flow control may still be applied manually on a per-queue basis in + this case. + </para> + + <para> + The user may manually set the flow thresholds when creating a queue. + The following options may be provided when adding a queue using the + <command>qpid-config</command> command line tool: + </para> + + <programlisting> + --flow-stop-size=<replaceable>N</replaceable> Sets the queue's flow stop threshold to <replaceable>N</replaceable> total bytes. + --flow-resume-size=<replaceable>N</replaceable> Sets the queue's flow resume threshold to <replaceable>N</replaceable> total bytes. + --flow-stop-count=<replaceable>N</replaceable> Sets the queue's flow stop threshold to <replaceable>N</replaceable> total messages. + --flow-resume-count=<replaceable>N</replaceable> Sets the queue's flow resume threshold to <replaceable>N</replaceable> total messages. + </programlisting> + + <para> + Flow thresholds may also be specified in the + <command>queue.declare</command> method, via the + <command>arguments</command> parameter map. The following keys can be + provided in the arguments map for setting flow thresholds: + </para> + + <table> + <title>Queue Declare Method Flow Control Arguments</title> + <tgroup cols="2"> + <thead> + <row> + <entry>Key</entry> + <entry>Value</entry> + </row> + </thead> + <tbody> + <row> + <entry>qpid.flow_stop_size</entry> + <entry>integer - queue's flow stop threshold value in bytes</entry> + </row> + <row> + <entry>qpid.flow_resume_size</entry> + <entry>integer - queue's flow resume threshold value in bytes</entry> + </row> + <row> + <entry>qpid.flow_stop_count</entry> + <entry>integer - queue's flow stop threshold value as a message count</entry> + </row> + <row> + <entry>qpid.flow_resume_count</entry> + <entry>integer - queue's flow resume threshold value as a message count</entry> + </row> + </tbody> + </tgroup> + </table> + + <para> + The user may disable flow control on a per queue basis by setting + the flow-stop-size and flow-stop-count to zero for the queue. + </para> + + <para> + The current state of flow control for a given queue can be + determined by the "flowStopped" statistic. This statistic is + available in the queue's QMF management object. The value of + flowStopped is True when the queue's capacity has exceeded the + flow stop threshold. The value of flowStopped is False when the + queue is no longer blocking due to flow control. + </para> + + <para> + A queue will also track the number of times flow control has been + activated. The "flowStoppedCount" statistic is incremented each time + the queue's capacity exceeds a flow stop threshold. This statistic can + be used to monitor the activity of flow control for any given queue + over time. + </para> + + <table> + <title>Flow Control Statistics available in Queue's QMF Class</title> + <tgroup cols="3"> + <thead> + <row> + <entry>Statistic Name</entry> + <entry>Type</entry> + <entry>Description</entry> + </row> + </thead> + <tbody> + <row> + <entry>flowStopped</entry> + <entry>Boolean</entry> + <entry>If true, producers are blocked by flow control.</entry> + </row> + <row> + <entry>flowStoppedCount</entry> + <entry>count32</entry> + <entry>Number of times flow control was activated for this queue</entry> + </row> + </tbody> + </tgroup> + </table> + </section> + <!--h2--> + </section> diff --git a/qpid/cpp/docs/book/src/cpp-broker/queue-state-replication.xml b/qpid/cpp/docs/book/src/cpp-broker/queue-state-replication.xml new file mode 100644 index 0000000000..3ffac805eb --- /dev/null +++ b/qpid/cpp/docs/book/src/cpp-broker/queue-state-replication.xml @@ -0,0 +1,333 @@ +<?xml version="1.0" encoding="utf-8"?> +<!-- + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +--> + +<section id="queue-state-replication"> + <title> + Queue State Replication + </title> + + <section role="h2" id="queuestatereplication-AsynchronousReplicationofQueueState"> + <title> + Asynchronous + Replication of Queue State + </title> + + <section role="h3" id="queuestatereplication-Overview"> + <title> + Overview + </title> + <para> + There is support in qpidd for selective asynchronous replication + of queue state. This is achieved by: + </para> + <para> + (a) enabling event generation for the queues in question + </para> + <para> + (b) loading a plugin on the 'source' broker to encode those + events as messages on a replication queue (this plugin is + called + replicating_listener.so) + </para> + <para> + (c) loading a custom exchange plugin on the 'backup' broker (this + plugin is called replication_exchange.so) + </para> + <para> + (d) creating an instance of the replication exchange type on the + backup broker + </para> + <para> + (e) establishing a federation bridge between the replication + queue on the source broker and the replication exchange on the + backup broker + </para> + <para> + The bridge established between the source and backup brokers for + replication (step (e) above) should have acknowledgements turned + on (this may be done through the --ack N option to qpid-route). + This ensures that replication events are not lost if the bridge + fails. + </para> + <para> + The replication protocol will also eliminate duplicates to ensure + reliably replicated state. Note though that only one bridge per + replication exchange is supported. If clients try to publish to + the replication exchange or if more than a the single required + bridge from the replication queue on the source broker is + created, replication will be corrupted. (Access control may be + used to restrict access and help prevent this). + </para> + <para> + The replicating event listener plugin (step (b) above) has the + following options: + </para> + <programlisting> +Queue Replication Options: + --replication-queue QUEUE Queue on which events for + other queues are recorded + --replication-listener-name NAME (replicator) name by which to register the + replicating event listener + --create-replication-queue if set, the replication will + be created if it does not + exist + </programlisting> + <para> + The name of the queue is required. It can either point to a + durable queue whose definition has been previously recorded, or + the --create-replication-queue option can be specified in which + case the queue will be created a simple non-durable queue if it + does not already exist. + </para> + <!--h3--> + </section> + + <section role="h3" id="queuestatereplication-UsewithClustering"> + <title> + Use with + Clustering + </title> + <para> + The source and/or backup brokers may also be clustered brokers. + In this case the federated bridge will be re-established between + replicas should either of the originally connected nodes fail. + There are however the following limitations at present: + </para> + <itemizedlist> + <listitem> + <para>The backup site does not process membership updates after it + establishes the first connection. In order for newly added + members on a source cluster to be eligible as failover targets, + the bridge must be recreated after those members have been added + to the source cluster. + </para> + </listitem> + </itemizedlist> + <itemizedlist> + <listitem> + <para>New members added to a backup cluster will not receive + information about currently established bridges. Therefore in + order to allow the bridge to be re-established from these members + in the event of failure of older nodes, the bridge must be + recreated after the new members have joined. + </para> + </listitem> + </itemizedlist> + <itemizedlist> + <listitem> + <para>Only a single URL can be passed to create the initial link + from backup site to the primary site. this means that at the time + of creating the initial connection the initial node in the + primary site to which the connection is made needs to be running. + Once connected the backup site will receive a membership update + of all the nodes in the primary site, and if the initial + connection node in the primary fails, the link will be + re-established on the next node that was started (time) on the + primary site. + </para> + </listitem> + </itemizedlist> + <para> + Due to the acknowledged transfer of events over the bridge (see + note above) manual recreation of the bridge and automatic + re-establishment of te bridge after connection failure (including + failover where either or both ends are clustered brokers) will + not result in event loss. + </para> + <!--h3--> + </section> + + <section role="h3" id="queuestatereplication-OperationsonBackupQueues"> + <title> + Operations + on Backup Queues + </title> + <para> + When replicating the state of a queue to a backup broker it is + important to recognise that any other operations performed + directly on the backup queue may break the replication. + </para> + <para> + If the backup queue is to be an active (i.e. accessed by clients + while replication is on) only enqueues should be selected + for + replication. In this mode, any message enqueued on the source + brokers copy of the queue will also be enqueued on the backup + brokers copy. However not attempt will be made to remove messages + from the backup queue in response to removal of messages from the + source queue. + </para> + <!--h3--> + </section> + + <section role="h3" id="queuestatereplication-SelectingQueuesforReplication"> + <title> + Selecting + Queues for Replication + </title> + <para> + Queues are selected for replication by specifying the types of + events they should generate (it is from these events that the + replicating plugin constructs messages which are then pulled and + processed by the backup site). This is done through options + passed to the initial queue-declare command that creates the + queue and may be done either through qpid-config or similar + tools, or by the application. + </para> + <para> + With qpid-config, the --generate-queue-events options is used: + </para> + <programlisting> + --generate-queue-events N + If set to 1, every enqueue will generate an event that can be processed by + registered listeners (e.g. for replication). If set to 2, events will be + generated for enqueues and dequeues + </programlisting> + <para> + From an application, the arguments field of the queue-declare + AMQP command is used to convey this information. An entry should + be added to the map with key 'qpid.queue_event_generation' and an + integer value of 1 (to replicate only enqueue events) or 2 (to + replicate both enqueue and dequeue events). + </para> + <para> + Applications written using the c++ client API may fine the + qpid::client::QueueOptions class convenient. This has a + enableQueueEvents() method on it that can be used to set the + option (the instance of QueueOptions is then passed as the value + of the arguments field in the queue-declare command. The boolean + option to that method should be set to true if only enequeue + events should be replicated; by default it is false meaning that + both enqueues and dequeues will be replicated. E.g. + </para> + <programlisting> + QueueOptions options; + options.enableQueueEvents(false); + session.queueDeclare(arg::queue="my-queue", arg::arguments=options); + </programlisting> + <!--h3--> + </section> + + <section role="h3" id="queuestatereplication-Example"> + <title> + Example + </title> + <para> + Lets assume we will run the primary broker on host1 and the + backup on host2, have installed qpidd on both and have the + replicating_listener and replication_exchange plugins in qpidd's + module directory(*1). + </para> + <para> + On host1 we start the source broker and specifcy that a queue + called 'replication' should be used for storing the events until + consumed by the backup. We also request that this queue be + created (as transient) if not already specified: + </para> + <programlisting> + qpidd --replication-queue replication-queue --create-replication-queue true --log-enable info+ + </programlisting> + <para> + On host2 we start up the backup broker ensuring that the + replication exchange module is loaded: + </para> + <programlisting> + qpidd + </programlisting> + <para> + We can then create the instance of that replication exchange that + we will use to process the events: + </para> + <programlisting> + qpid-config -a host2 add exchange replication replication-exchange + </programlisting> + <para> + If this fails with the message "Exchange type not implemented: + replication", it means the replication exchange module was + not + loaded. Check that the module is installed on your system and if + necessary provide the full path to the library. + </para> + <para> + We then connect the replication queue on the source broker with + the replication exchange on the backup broker using the + qpid-route command: + </para> + <programlisting> + qpid-route --ack 50 queue add host2 host1 replication-exchange replication-queue +</programlisting> + <para> + The example above configures the bridge to acknowledge messages + in batches of 50. + </para> + <para> + Now create two queues (on both source and backup brokers), one + replicating both enqueues and dequeues (queue-a) and the + other + replicating only dequeues (queue-b): + </para> + <programlisting> + qpid-config -a host1 add queue queue-a --generate-queue-events 2 + qpid-config -a host1 add queue queue-b --generate-queue-events 1 + + qpid-config -a host2 add queue queue-a + qpid-config -a host2 add queue queue-b + </programlisting> + <para> + We are now ready to use the queues and see the replication. + </para> + <para> + Any message enqueued on queue-a will be replicated to the backup + broker. When the message is acknowledged by a client connected to + host1 (and thus dequeued), that message will be removed from the + copy of the queue on host2. The state of queue-a on host2 will + thus mirror that of the equivalent queue on host1, albeit with a + small lag. (Note + however that we must not have clients connected to host2 publish + to-or consume from- queue-a or the state will fail to replicate + correctly due to conflicts). + </para> + <para> + Any message enqueued on queue-b on host1 will also be enqueued on + the equivalent queue on host2. However the acknowledgement and + consequent dequeuing of messages from queue-b on host1 will have + no effect on the state of queue-b on host2. + </para> + <para> + (*1) If not the paths in the above may need to be modified. E.g. + if using modules built from a qpid svn checkout, the following + would be added to the command line used to start qpidd on host1: + </para> + <programlisting> + --load-module <path-to-qpid-dir>/src/.libs/replicating_listener.so + </programlisting> + <para> + and the following for the equivalent command line on host2: + </para> + <programlisting> + --load-module <path-to-qpid-dir>/src/.libs/replication_exchange.so + </programlisting> + <!--h3--> + </section> + <!--h2--> + </section> +</section> |