diff options
| author | Jonathan Robie <jonathan@apache.org> | 2010-01-12 14:05:42 +0000 |
|---|---|---|
| committer | Jonathan Robie <jonathan@apache.org> | 2010-01-12 14:05:42 +0000 |
| commit | f80cbbcb302565d2a2ef321b5e2ad87c2485390f (patch) | |
| tree | 7285ce8eb96b8889d437d1f83a67720f10f54f9a | |
| parent | df9f26495b4d1823601cccc2680384c2a23427a8 (diff) | |
| download | qpid-python-f80cbbcb302565d2a2ef321b5e2ad87c2485390f.tar.gz | |
Content from Confluence Wiki. This version uses confluence-specific XSLT stylesheet for conversion, with section boundaries set by hand in the editor.
git-svn-id: https://svn.apache.org/repos/asf/qpid/trunk@898329 13f79535-47bb-0310-9956-ffa450edef68
31 files changed, 8847 insertions, 8 deletions
diff --git a/qpid/doc/book/src/82403.xml b/qpid/doc/book/src/82403.xml new file mode 100644 index 0000000000..5987eb271d --- /dev/null +++ b/qpid/doc/book/src/82403.xml @@ -0,0 +1,446 @@ +<?xml version="1.0" encoding="utf-8"?> +<chapter xmlns:html="http://www.w3.org/1999/xhtml"><title> + Apache Qpid : MgmtC++ + </title><section role="h2" id="MgmtC-2B-2B-ManagingtheC-5CBroker"><title> + Managing the C++ + Broker + </title> + <para> + There are quite a few ways to interact with the C++ broker. The + command line tools + include: + </para><itemizedlist> + <listitem><para>qpid-route - used to configure federation (a set of federated + brokers) + </para></listitem> + <listitem><para>qpid-config - used to configure queues, exchanges, bindings + and list them etc + </para></listitem> + <listitem><para>qpid-tool - used to view management information/statistics + and call any management actions on the broker + </para></listitem> + <listitem><para>qpid-printevents - used to receive and print QMF events + </para></listitem> + </itemizedlist> + + <section role="h3" id="MgmtC-2B-2B-Usingqpidconfig"><title> + Using qpid-config + </title> + <para> + This utility can be used to create queues exchanges and bindings, + both durable and transient. Always check for latest options by + running --help command. + </para> + <programlisting> +$ qpid-config --help +Usage: qpid-config [OPTIONS] + qpid-config [OPTIONS] exchanges [filter-string] + qpid-config [OPTIONS] queues [filter-string] + qpid-config [OPTIONS] add exchange <type> <name> [AddExchangeOptions] + qpid-config [OPTIONS] del exchange <name> + qpid-config [OPTIONS] add queue <name> [AddQueueOptions] + qpid-config [OPTIONS] del queue <name> + qpid-config [OPTIONS] bind <exchange-name> <queue-name> [binding-key] + qpid-config [OPTIONS] unbind <exchange-name> <queue-name> [binding-key] + +Options: + -b [ --bindings ] Show bindings in queue or exchange list + -a [ --broker-addr ] Address (localhost) Address of qpidd broker + broker-addr is in the form: [username/password@] hostname | ip-address [:<port>] + ex: localhost, 10.1.1.7:10000, broker-host:10000, guest/guest@localhost + +Add Queue Options: + --durable Queue is durable + --cluster-durable Queue becomes durable if there is only one functioning cluster node + --file-count N (8) Number of files in queue's persistence journal + --file-size N (24) File size in pages (64Kib/page) + --max-queue-size N Maximum in-memory queue size as bytes + --max-queue-count N Maximum in-memory queue size as a number of messages + --limit-policy [none | reject | flow-to-disk | ring | ring-strict] + Action taken when queue limit is reached: + none (default) - Use broker's default policy + reject - Reject enqueued messages + flow-to-disk - Page messages to disk + ring - Replace oldest unacquired message with new + ring-strict - Replace oldest message, reject if oldest is acquired + --order [fifo | lvq | lvq-no-browse] + Set queue ordering policy: + fifo (default) - First in, first out + lvq - Last Value Queue ordering, allows queue browsing + lvq-no-browse - Last Value Queue ordering, browsing clients may lose data + --generate-queue-events N + If set to 1, every enqueue will generate an event that can be processed by + registered listeners (e.g. for replication). If set to 2, events will be + generated for enqueues and dequeues + +Add Exchange Options: + --durable Exchange is durable + --sequence Exchange will insert a 'qpid.msg_sequence' field in the message header + with a value that increments for each message forwarded. + --ive Exchange will behave as an 'initial-value-exchange', keeping a reference + to the last message forwarded and enqueuing that message to newly bound + queues. +</programlisting> + <para> + Get the summary page + </para> + <programlisting> +$ qpid-config +Total Exchanges: 6 + topic: 2 + headers: 1 + fanout: 1 + direct: 2 + Total Queues: 7 + durable: 0 + non-durable: 7 +</programlisting> + <para> + List the queues + </para> + <programlisting> +$ qpid-config queues +Queue Name Attributes +================================================================= +pub_start +pub_done +sub_ready +sub_done +perftest0 --durable +reply-dhcp-100-18-254.bos.redhat.com.20713 auto-del excl +topic-dhcp-100-18-254.bos.redhat.com.20713 auto-del excl + +</programlisting> + <para> + List the exchanges with bindings + </para> + <programlisting> +$ ./qpid-config -b exchanges +Exchange '' (direct) + bind pub_start => pub_start + bind pub_done => pub_done + bind sub_ready => sub_ready + bind sub_done => sub_done + bind perftest0 => perftest0 + bind mgmt-3206ff16-fb29-4a30-82ea-e76f50dd7d15 => mgmt-3206ff16-fb29-4a30-82ea-e76f50dd7d15 + bind repl-3206ff16-fb29-4a30-82ea-e76f50dd7d15 => repl-3206ff16-fb29-4a30-82ea-e76f50dd7d15 +Exchange 'amq.direct' (direct) + bind repl-3206ff16-fb29-4a30-82ea-e76f50dd7d15 => repl-3206ff16-fb29-4a30-82ea-e76f50dd7d15 + bind repl-df06c7a6-4ce7-426a-9f66-da91a2a6a837 => repl-df06c7a6-4ce7-426a-9f66-da91a2a6a837 + bind repl-c55915c2-2fda-43ee-9410-b1c1cbb3e4ae => repl-c55915c2-2fda-43ee-9410-b1c1cbb3e4ae +Exchange 'amq.topic' (topic) +Exchange 'amq.fanout' (fanout) +Exchange 'amq.match' (headers) +Exchange 'qpid.management' (topic) + bind mgmt.# => mgmt-3206ff16-fb29-4a30-82ea-e76f50dd7d15 +</programlisting> +<!--h3--></section> + + <section role="h3" id="MgmtC-2B-2B-Usingqpidroute"><title> + Using qpid-route + </title> + <para> + This utility is to create federated networks of brokers, This + allows you for forward messages between brokers in a network. + Messages can be routed statically (using "qpid-route route add") + where the bindings that control message forwarding are supplied + in the route. Message routing can also be dynamic (using + "qpid-route dynamic add") where the messages are automatically + forwarded to clients based on their bindings to the local broker. + </para> + <programlisting> +$ qpid-route +Usage: qpid-route [OPTIONS] dynamic add <dest-broker> <src-broker> <exchange> [tag] [exclude-list] + qpid-route [OPTIONS] dynamic del <dest-broker> <src-broker> <exchange> + + qpid-route [OPTIONS] route add <dest-broker> <src-broker> <exchange> <routing-key> [tag] [exclude-list] + qpid-route [OPTIONS] route del <dest-broker> <src-broker> <exchange> <routing-key> + qpid-route [OPTIONS] queue add <dest-broker> <src-broker> <exchange> <queue> + qpid-route [OPTIONS] queue del <dest-broker> <src-broker> <exchange> <queue> + qpid-route [OPTIONS] route list [<dest-broker>] + qpid-route [OPTIONS] route flush [<dest-broker>] + qpid-route [OPTIONS] route map [<broker>] + + qpid-route [OPTIONS] link add <dest-broker> <src-broker> + qpid-route [OPTIONS] link del <dest-broker> <src-broker> + qpid-route [OPTIONS] link list [<dest-broker>] + +Options: + -v [ --verbose ] Verbose output + -q [ --quiet ] Quiet output, don't print duplicate warnings + -d [ --durable ] Added configuration shall be durable + -e [ --del-empty-link ] Delete link after deleting last route on the link + -s [ --src-local ] Make connection to source broker (push route) + -t <transport> [ --transport <transport>] + Specify transport to use for links, defaults to tcp + + dest-broker and src-broker are in the form: [username/password@] hostname | ip-address [:<port>] + ex: localhost, 10.1.1.7:10000, broker-host:10000, guest/guest@localhost +</programlisting> + <para> + A few examples: + </para> + <programlisting> +qpid-route dynamic add host1 host2 fed.topic +qpid-route dynamic add host2 host1 fed.topic + +qpid-route -v route add host1 host2 hub1.topic hub2.topic.stock.buy +qpid-route -v route add host1 host2 hub1.topic hub2.topic.stock.sell +qpid-route -v route add host1 host2 hub1.topic 'hub2.topic.stock.#' +qpid-route -v route add host1 host2 hub1.topic 'hub2.#' +qpid-route -v route add host1 host2 hub1.topic 'hub2.topic.#' +qpid-route -v route add host1 host2 hub1.topic 'hub2.global.#' +</programlisting> + <para> + The link map feature can be used to display the entire federated + network configuration by supplying a single broker as an entry + point: + </para> + <programlisting> +$ qpid-route route map localhost:10001 + +Finding Linked Brokers: + localhost:10001... Ok + localhost:10002... Ok + localhost:10003... Ok + localhost:10004... Ok + localhost:10005... Ok + localhost:10006... Ok + localhost:10007... Ok + localhost:10008... Ok + +Dynamic Routes: + + Exchange fed.topic: + localhost:10002 <=> localhost:10001 + localhost:10003 <=> localhost:10002 + localhost:10004 <=> localhost:10002 + localhost:10005 <=> localhost:10002 + localhost:10006 <=> localhost:10005 + localhost:10007 <=> localhost:10006 + localhost:10008 <=> localhost:10006 + + Exchange fed.direct: + localhost:10002 => localhost:10001 + localhost:10004 => localhost:10003 + localhost:10003 => localhost:10002 + localhost:10001 => localhost:10004 + +Static Routes: + + localhost:10003(ex=amq.direct) <= localhost:10005(ex=amq.direct) key=rkey + localhost:10003(ex=amq.direct) <= localhost:10005(ex=amq.direct) key=rkey2 +</programlisting> +<!--h3--></section> + + <section role="h3" id="MgmtC-2B-2B-Usingqpidtool"><title> + Using qpid-tool + </title> + <para> + This utility provided a telnet style interface to be able to + view, list all stats and action + all the methods. Simple capture below. Best to just play with it + and mail the list if you have + questions or want features added. + </para> + <programlisting> +qpid: +qpid: help +Management Tool for QPID +Commands: + list - Print summary of existing objects by class + list <className> - Print list of objects of the specified class + list <className> all - Print contents of all objects of specified class + list <className> active - Print contents of all non-deleted objects of specified class + list <list-of-IDs> - Print contents of one or more objects (infer className) + list <className> <list-of-IDs> - Print contents of one or more objects + list is space-separated, ranges may be specified (i.e. 1004-1010) + call <ID> <methodName> <args> - Invoke a method on an object + schema - Print summary of object classes seen on the target + schema <className> - Print details of an object class + set time-format short - Select short timestamp format (default) + set time-format long - Select long timestamp format + quit or ^D - Exit the program +qpid: list +Management Object Types: + ObjectType Active Deleted + ================================ + qpid.binding 21 0 + qpid.broker 1 0 + qpid.client 1 0 + qpid.exchange 6 0 + qpid.queue 13 0 + qpid.session 4 0 + qpid.system 1 0 + qpid.vhost 1 0 +qpid: list qpid.system +Objects of type qpid.system + ID Created Destroyed Index + ================================== + 1000 21:00:02 - host +qpid: list 1000 +Object of type qpid.system: (last sample time: 21:26:02) + Type Element 1000 + ======================================================= + config sysId host + config osName Linux + config nodeName localhost.localdomain + config release 2.6.24.4-64.fc8 + config version #1 SMP Sat Mar 29 09:15:49 EDT 2008 + config machine x86_64 +qpid: schema queue +Schema for class 'qpid.queue': + Element Type Unit Access Notes Description + =================================================================================================================== + vhostRef reference ReadCreate index + name short-string ReadCreate index + durable boolean ReadCreate + autoDelete boolean ReadCreate + exclusive boolean ReadCreate + arguments field-table ReadOnly Arguments supplied in queue.declare + storeRef reference ReadOnly Reference to persistent queue (if durable) + msgTotalEnqueues uint64 message Total messages enqueued + msgTotalDequeues uint64 message Total messages dequeued + msgTxnEnqueues uint64 message Transactional messages enqueued + msgTxnDequeues uint64 message Transactional messages dequeued + msgPersistEnqueues uint64 message Persistent messages enqueued + msgPersistDequeues uint64 message Persistent messages dequeued + msgDepth uint32 message Current size of queue in messages + msgDepthHigh uint32 message Current size of queue in messages (High) + msgDepthLow uint32 message Current size of queue in messages (Low) + byteTotalEnqueues uint64 octet Total messages enqueued + byteTotalDequeues uint64 octet Total messages dequeued + byteTxnEnqueues uint64 octet Transactional messages enqueued + byteTxnDequeues uint64 octet Transactional messages dequeued + bytePersistEnqueues uint64 octet Persistent messages enqueued + bytePersistDequeues uint64 octet Persistent messages dequeued + byteDepth uint32 octet Current size of queue in bytes + byteDepthHigh uint32 octet Current size of queue in bytes (High) + byteDepthLow uint32 octet Current size of queue in bytes (Low) + enqueueTxnStarts uint64 transaction Total enqueue transactions started + enqueueTxnCommits uint64 transaction Total enqueue transactions committed + enqueueTxnRejects uint64 transaction Total enqueue transactions rejected + enqueueTxnCount uint32 transaction Current pending enqueue transactions + enqueueTxnCountHigh uint32 transaction Current pending enqueue transactions (High) + enqueueTxnCountLow uint32 transaction Current pending enqueue transactions (Low) + dequeueTxnStarts uint64 transaction Total dequeue transactions started + dequeueTxnCommits uint64 transaction Total dequeue transactions committed + dequeueTxnRejects uint64 transaction Total dequeue transactions rejected + dequeueTxnCount uint32 transaction Current pending dequeue transactions + dequeueTxnCountHigh uint32 transaction Current pending dequeue transactions (High) + dequeueTxnCountLow uint32 transaction Current pending dequeue transactions (Low) + consumers uint32 consumer Current consumers on queue + consumersHigh uint32 consumer Current consumers on queue (High) + consumersLow uint32 consumer Current consumers on queue (Low) + bindings uint32 binding Current bindings + bindingsHigh uint32 binding Current bindings (High) + bindingsLow uint32 binding Current bindings (Low) + unackedMessages uint32 message Messages consumed but not yet acked + unackedMessagesHigh uint32 message Messages consumed but not yet acked (High) + unackedMessagesLow uint32 message Messages consumed but not yet acked (Low) + messageLatencySamples delta-time nanosecond Broker latency through this queue (Samples) + messageLatencyMin delta-time nanosecond Broker latency through this queue (Min) + messageLatencyMax delta-time nanosecond Broker latency through this queue (Max) + messageLatencyAverage delta-time nanosecond Broker latency through this queue (Average) +Method 'purge' Discard all messages on queue +qpid: list queue +Objects of type qpid.queue + ID Created Destroyed Index + =========================================================================== + 1012 21:08:13 - 1002.pub_start + 1014 21:08:13 - 1002.pub_done + 1016 21:08:13 - 1002.sub_ready + 1018 21:08:13 - 1002.sub_done + 1020 21:08:13 - 1002.perftest0 + 1038 21:09:08 - 1002.mgmt-3206ff16-fb29-4a30-82ea-e76f50dd7d15 + 1040 21:09:08 - 1002.repl-3206ff16-fb29-4a30-82ea-e76f50dd7d15 + 1046 21:09:32 - 1002.mgmt-df06c7a6-4ce7-426a-9f66-da91a2a6a837 + 1048 21:09:32 - 1002.repl-df06c7a6-4ce7-426a-9f66-da91a2a6a837 + 1054 21:10:01 - 1002.mgmt-c55915c2-2fda-43ee-9410-b1c1cbb3e4ae + 1056 21:10:01 - 1002.repl-c55915c2-2fda-43ee-9410-b1c1cbb3e4ae + 1063 21:26:00 - 1002.mgmt-8d621997-6356-48c3-acab-76a37081d0f3 + 1065 21:26:00 - 1002.repl-8d621997-6356-48c3-acab-76a37081d0f3 +qpid: list 1020 +Object of type qpid.queue: (last sample time: 21:26:02) + Type Element 1020 + ========================================================================== + config vhostRef 1002 + config name perftest0 + config durable False + config autoDelete False + config exclusive False + config arguments {'qpid.max_size': 0, 'qpid.max_count': 0} + config storeRef NULL + inst msgTotalEnqueues 500000 messages + inst msgTotalDequeues 500000 + inst msgTxnEnqueues 0 + inst msgTxnDequeues 0 + inst msgPersistEnqueues 0 + inst msgPersistDequeues 0 + inst msgDepth 0 + inst msgDepthHigh 0 + inst msgDepthLow 0 + inst byteTotalEnqueues 512000000 octets + inst byteTotalDequeues 512000000 + inst byteTxnEnqueues 0 + inst byteTxnDequeues 0 + inst bytePersistEnqueues 0 + inst bytePersistDequeues 0 + inst byteDepth 0 + inst byteDepthHigh 0 + inst byteDepthLow 0 + inst enqueueTxnStarts 0 transactions + inst enqueueTxnCommits 0 + inst enqueueTxnRejects 0 + inst enqueueTxnCount 0 + inst enqueueTxnCountHigh 0 + inst enqueueTxnCountLow 0 + inst dequeueTxnStarts 0 + inst dequeueTxnCommits 0 + inst dequeueTxnRejects 0 + inst dequeueTxnCount 0 + inst dequeueTxnCountHigh 0 + inst dequeueTxnCountLow 0 + inst consumers 0 consumers + inst consumersHigh 0 + inst consumersLow 0 + inst bindings 1 binding + inst bindingsHigh 1 + inst bindingsLow 1 + inst unackedMessages 0 messages + inst unackedMessagesHigh 0 + inst unackedMessagesLow 0 + inst messageLatencySamples 0 + inst messageLatencyMin 0 + inst messageLatencyMax 0 + inst messageLatencyAverage 0 +qpid: +</programlisting> +<!--h3--></section> + + <section role="h3" id="MgmtC-2B-2B-Usingqpidprintevents"><title> + Using + qpid-printevents + </title> + <para> + This utility connects to one or more brokers and collects events, + printing out a line per event. + </para> + <programlisting> +$ qpid-printevents --help +Usage: qpid-printevents [options] [broker-addr]... + +Collect and print events from one or more Qpid message brokers. If no broker- +addr is supplied, qpid-printevents will connect to 'localhost:5672'. broker- +addr is of the form: [username/password@] hostname | ip-address [:<port>] ex: +localhost, 10.1.1.7:10000, broker-host:10000, guest/guest@localhost + +Options: + -h, --help show this help message and exit +</programlisting> + <para> + You get the idea... have fun! + </para> +<!--h3--></section> +<!--h2--></section> +</chapter> diff --git a/qpid/doc/book/src/ACL.xml b/qpid/doc/book/src/ACL.xml new file mode 100644 index 0000000000..01be375983 --- /dev/null +++ b/qpid/doc/book/src/ACL.xml @@ -0,0 +1,771 @@ +<?xml version="1.0" encoding="utf-8"?> +<chapter xmlns:html="http://www.w3.org/1999/xhtml"><title> + Apache Qpid : ACL + </title><section role="h2" id="ACL-v2ACLfileformatforbrokers"><title> + v2 ACL file format + for brokers + </title> + <para> + This new ACL implementation has been designed for implementation + and interoperability on all Qpid brokers. It is currently + supported in the following brokers: + </para> + + <table><title/><tgroup cols="2"> + <thead> + <row> + <entry> + Broker + </entry> + <entry> + Version + </entry> + </row> + </thead> + <tbody> + <row> + <entry> + C++ + </entry> + <entry> + M4 onward + </entry> + </row> + <row> + <entry> + Java + </entry> + <entry> + M5 anticipated + </entry> + </row> + </tbody> + </tgroup></table> + + <para> + Contents + </para> + <itemizedlist> + <listitem><para> + <xref linkend="ACL-v2ACLfileformatforbrokers"/> + </para></listitem> + <listitem><para> + <itemizedlist> + <listitem><para> + <xref linkend="ACL-Specification"/> + </para></listitem> + <listitem><para> + <xref linkend="ACL-Validation"/> + </para></listitem> + <listitem><para> + <xref linkend="ACL-Examplefile-3A"/> + </para></listitem> + </itemizedlist> + </para></listitem> + <listitem><para> + <xref linkend="ACL-DesignDocumentation"/> + </para></listitem> + <listitem><para> + <itemizedlist> + <listitem><para> + <xref linkend="ACL-MappingofACLtrapstoactionandtype"/> + </para></listitem> + </itemizedlist> + </para></listitem> + <listitem><para> + <xref linkend="ACL-v2ACLUserGuide"/> + </para></listitem> + <listitem><para> + <itemizedlist> + <listitem><para> + <xref linkend="ACL-WritingGood-2FFastACL"/> + </para></listitem> + <listitem><para> + <xref linkend="ACL-GettingACLtoLog"/> + </para></listitem> + <listitem><para> + <xref linkend="ACL-UserId-2FdomainsrunningwithCbroker"/> + </para></listitem> + </itemizedlist> + </para></listitem> + </itemizedlist> + <para> + + </para><section role="h3" id="ACL-Specification"><title> + Specification + </title> + <para> + Â Notes on file formats + </para><itemizedlist> + <listitem><para>A line starting with the character '#' will be considered a + comment, and are ignored. + </para></listitem> + <listitem><para>Since the '#' char (and others that are commonly used for + comments) are commonly found in routing keys and other AMQP + literals, it is simpler (for now) to hold off on allowing + trailing comments (ie comments in which everything following a + '#' is considered a comment). This could be reviewed later once + the rest of the format is finalized. + </para></listitem> + <listitem><para>Empty lines ("") and lines that contain only whitespace (any + combination of ' ', '\f', '\n', '\r', '\t', '\v') are ignored. + </para></listitem> + <listitem><para>All tokens are case sensitive. "name1" != "Name1" and + "create" != "CREATE". + </para></listitem> + <listitem><para>Group lists may be extended to the following line by + terminating the line with the '\' character. However, this may + only occur after the group name or any of the names following the + group name. Empty extension lines (ie just a '\' character) are + not permitted. + + <programlisting> +# Examples of extending group lists using a trailing '\' character + +group group1 name1 name2 \ + name3 name4 \ + name5 + +group group2 \ + group1 \ + name6 + +# The following are illegal: + +# '\' must be after group name +group \ + group3 name7 name8 + +# No empty extension lines +group group4 name9 \ + \ + name10 +</programlisting> + + </para></listitem> + <listitem><para>Additional whitespace (ie more than one whitespace char) + between and after tokens is ignored. However group and acl + definitions must start with "group" or "acl" respectively and + with no preceding whitespace. + </para></listitem> + <listitem><para>All acl rules are limited to a single line. + </para></listitem> + <listitem><para>Rules are interpreted from the top of the file down until the + name match is obtained; at which point processing stops. + </para></listitem> + <listitem><para>The keyword "all" is reserved, and matches all individuals, + groups and actions. It may be used in place of a group or + individual name and/or an action - eg "acl allow all all", "acl + deny all all" or "acl deny user1 all". + </para></listitem> + <listitem><para>The last line of the file (whether present or not) will be + assumed to be "acl deny all all". If present in the file, any + lines below this one are ignored. + </para></listitem> + <listitem><para>Names and group names may contain only a-z, A-Z, 0-9, + '-','_'. + </para></listitem> + <listitem><para>Rules must be preceded by any group definitions they may use; + any name not previously defined as a group will be assumed to be + that of an individual. + </para></listitem> + <listitem><para>ACL rules must have the following tokens in order on a single + line: + <itemizedlist> + <listitem><para>The string literal "acl"; + </para></listitem> + <listitem><para>The permission; + </para></listitem> + <listitem><para>The name of a single group or individual or the keyword + "all"; + </para></listitem> + <listitem><para>The name of an action or the keyword "all"; + </para></listitem> + <listitem><para>Optionally, a single object name or the keyword "all"; + </para></listitem> + <listitem><para>If the object is present, then optionally one or more + property name-value pair(s) (in the form property=value). + </para></listitem> + </itemizedlist> + </para></listitem> + </itemizedlist> + <programlisting> +user = username[@domain[/realm]] +user-list = user1 user2 user3 ... +group-name-list = group1 group2 group3 ... + +group <group-name> = [user-list] [group-name-list] + + +permission = [allow|allow-log|deny|deny-log] +action = [consume|publish|create|access|bind|unbind|delete|purge|update] +object = [virtualhost|queue|exchange|broker|link|route|method] +property = [name|durable|owner|routingkey|passive|autodelete|exclusive|type|alternate|queuename|schemapackage|schemaclass] + +acl permission {<group-name>|<user-name>|"all"} {action|"all"} [object|"all"] [property=<property-value>] +</programlisting> +<!--h3--></section> + <section role="h3" id="ACL-Validation"><title> + Validation + </title> + <para> + The new ACL file format needs to perform validation on the acl + rules. The validation should be performed depending on the set + value: + </para><para> + strict-acl-validation=none + The default setting should be 'warn' + </para><para> + On validation of this acl the following checks would be expected: + </para> + <programlisting> +acl allow client publish routingkey=exampleQueue exchange=amq.direct +</programlisting> + <orderedlist> + <listitem><para>The If the user 'client' cannot be found, if the + authentication mechanism cannot be queried then a 'user' value + should be added to the file. + </para></listitem> + <listitem><para>There is an exchange called 'amq.direct' + </para></listitem> + <listitem><para>There is a queue bound to 'exampleQueue' on 'amq.direct' + </para></listitem> + </orderedlist><para> + Each of these checks that fail will result in a log statement + being generated. + </para><para> + In the case of a fatal logging the full file will be validated + before the broker shuts down. + </para> +<!--h3--></section> + + + <section role="h3" id="ACL-Examplefile-3A"><title> + Example file: + </title> + + <programlisting> + +# Some groups +group admin ted@QPID martin@QPID +group user-consume martin@QPID ted@QPID +group group2 kim@QPID user-consume rob@QPID +group publisher group2 \ + tom@QPID andrew@QPID debbie@QPID + +# Some rules +acl allow carlt@QPID create exchange name=carl.* +acl deny rob@QPID create queue +acl allow guest@QPID bind exchange name=amq.topic routingkey=stocks.ibm.# owner=self +acl allow user-consume create queue name=tmp.* + +acl allow publisher publish all durable=false +acl allow publisher create queue name=RequestQueue +acl allow consumer consume queue durable=true +acl allow fred@QPID create all +acl allow bob@QPID all queue +acl allow admin all +acl deny kim@QPID all +acl allow all consume queue owner=self +acl allow all bind exchange owner=self + +# Last (default) rule +acl deny all all +</programlisting> + +<!--h3--></section> + <!--h2--></section> + + + <section role="h2" id="ACL-DesignDocumentation"><title> + Design Documentation + </title> + <section role="h3" id="ACL-MappingofACLtrapstoactionandtype"><title> + Mapping of ACL + traps to action and type + </title> + <para> + The C++ broker maps the ACL traps in the follow way for AMQP + 0-10: + The Java broker currently only performs ACLs on the AMQP + connection not on management functions: + </para> + + <table><title/><tgroup cols="5"> + <thead> + <row> + <entry> + Object + </entry> + <entry> + Action + </entry> + <entry> + Properties + </entry> + <entry> + Trap C++ + </entry> + <entry> + Trap Java + </entry> + </row> + </thead> + <tbody> + <row> + <entry> + Exchange + </entry> + <entry> + Create + </entry> + <entry> + name type alternate passive durable + </entry> + <entry> + ExchangeHandlerImpl::declare + </entry> + <entry> + ExchangeDeclareHandler + </entry> + </row> + <row> + <entry> + Exchange + </entry> + <entry> + Delete + </entry> + <entry> + name + </entry> + <entry> + ExchangeHandlerImpl::delete + </entry> + <entry> + ExchangeDeleteHandler + </entry> + </row> + <row> + <entry> + Exchange + </entry> + <entry> + Access + </entry> + <entry> + name + </entry> + <entry> + ExchangeHandlerImpl::query + </entry> + <entry> + + </entry> + </row> + <row> + <entry> + Exchange + </entry> + <entry> + Bind + </entry> + <entry> + name routingkey queuename owner + </entry> + <entry> + ExchangeHandlerImpl::bind + </entry> + <entry> + QueueBindHandler + </entry> + </row> + <row> + <entry> + Exchange + </entry> + <entry> + Unbind + </entry> + <entry> + name routingkey + </entry> + <entry> + ExchangeHandlerImpl::unbind + </entry> + <entry> + ExchangeUnbindHandler + </entry> + </row> + <row> + <entry> + Exchange + </entry> + <entry> + Access + </entry> + <entry> + name queuename routingkey + </entry> + <entry> + ExchangeHandlerImpl::bound + </entry> + <entry> + + </entry> + </row> + <row> + <entry> + Exchange + </entry> + <entry> + Publish + </entry> + <entry> + name routingKey + </entry> + <entry> + SemanticState::route + </entry> + <entry> + BasicPublishMethodHandler + </entry> + </row> + <row> + <entry> + Queue + </entry> + <entry> + Access + </entry> + <entry> + name + </entry> + <entry> + QueueHandlerImpl::query + </entry> + <entry> + + </entry> + </row> + <row> + <entry> + Queue + </entry> + <entry> + Create + </entry> + <entry> + name alternate passive durable exclusive autodelete + </entry> + <entry> + QueueHandlerImpl::declare + </entry> + <entry> + QueueDeclareHandler + </entry> + </row> + <row> + <entry> + Queue + </entry> + <entry> + Purge + </entry> + <entry> + name + </entry> + <entry> + QueueHandlerImpl::purge + </entry> + <entry> + QueuePurgeHandler + </entry> + </row> + <row> + <entry> + Queue + </entry> + <entry> + Purge + </entry> + <entry> + name + </entry> + <entry> + Management::Queue::purge + </entry> + <entry> + + </entry> + </row> + <row> + <entry> + Queue + </entry> + <entry> + Delete + </entry> + <entry> + name + </entry> + <entry> + QueueHandlerImpl::delete + </entry> + <entry> + QueueDeleteHandler + </entry> + </row> + <row> + <entry> + Queue + </entry> + <entry> + Consume + </entry> + <entry> + name (possibly add in future?) + </entry> + <entry> + MessageHandlerImpl::subscribe + </entry> + <entry> + BasicConsumeMethodHandler + BasicGetMethodHandler + </entry> + </row> + <row> + <entry> + <Object> + </entry> + <entry> + Update + </entry> + <entry> + Â + </entry> + <entry> + ManagementProperty::set + </entry> + <entry> + + </entry> + </row> + <row> + <entry> + <Object> + </entry> + <entry> + Access + </entry> + <entry> + Â + </entry> + <entry> + ManagementProperty::read + </entry> + <entry> + + </entry> + </row> + <row> + <entry> + Link + </entry> + <entry> + Create + </entry> + <entry> + Â + </entry> + <entry> + Management::connect + </entry> + <entry> + + </entry> + </row> + <row> + <entry> + Route + </entry> + <entry> + Create + </entry> + <entry> + Â + </entry> + <entry> + Management:: -createFederationRoute- + </entry> + <entry> + + </entry> + </row> + <row> + <entry> + Route + </entry> + <entry> + Delete + </entry> + <entry> + Â + </entry> + <entry> + Management:: -deleteFederationRoute- + </entry> + <entry> + + </entry> + </row> + <row> + <entry> + Virtualhost + </entry> + <entry> + Access + </entry> + <entry> + name + </entry> + <entry> + TBD + </entry> + <entry> + ConnectionOpenMethodHandler + </entry> + </row> + </tbody> + </tgroup></table><para> + Management actions that are not explicitly given a name property + it will default the name property to management method name, if + the action is 'W' Action will be 'Update', if 'R' Action will be + 'Access'. + </para><para> + for example, if the mgnt method 'joinCluster' was not mapped in + schema it will be mapped in ACL file as follows + </para> + + <table><title/><tgroup cols="3"> + <thead> + <row> + <entry> + Object + </entry> + <entry> + Action + </entry> + <entry> + Property + </entry> + </row> + </thead> + <tbody> + <row> + <entry> + Broker + </entry> + <entry> + Update + </entry> + <entry> + name=joinCluster + </entry> + </row> + </tbody> + </tgroup></table> + + <!--h3--></section> + <!--h2--></section> + + + <section role="h2" id="ACL-v2ACLUserGuide"><title> + v2 ACL User Guide + </title> + <section role="h3" id="ACL-WritingGood-2FFastACL"><title> + Writing Good/Fast ACL + </title> + <para> + The file gets read top down and rule get passed based on the + first match. In the following example the first rule is a dead + rule. I.e. the second rule is wider than the first rule. DON'T do + this, it will force extra analysis, worst case if the parser does + not kill the dead rule you might get a false deny. + </para> + <programlisting> +allow peter@QPID create queue name=tmp <-- dead rule!! +allow peter@QPID create queue +deny all all +</programlisting> + <para> + By default files end with + </para> + <programlisting> +deny all all +</programlisting> + <para> + the mode of the ACL engine can be swapped to be allow based by + putting the following at the end of the file + </para> + <programlisting> +allow all all +</programlisting> + <para> + Note that 'allow' based file will be a LOT faster for message + transfer. This is because the AMQP specification does not allow + for creating subscribes on publish, so the ACL is executed on + every message transfer. Also, ACL's rules using less properties + on publish will in general be faster. + </para> + <!--h3--></section> + + <section role="h3" id="ACL-GettingACLtoLog"><title> + Getting ACL to Log + </title> + <para> + In order to get log messages from ACL actions use allow-log and + deny-log for example + </para> + <programlisting> +allow-log john@QPID all all +deny-log guest@QPID all all +</programlisting> + <!--h3--></section> + + <section role="h3" id="ACL-UserId-2FdomainsrunningwithCbroker"><title> + User Id / + domains running with C++ broker + </title> + <para> + The user-id used for ACL is taken from the connection user-id. + Thus in order to use ACL the broker authentication has to be + setup. i.e. (if --auth no is used in combination with ACL the + broker will deny everything) + </para><para> + The user id in the ACL file is of the form + <user-id>@<domain> The Domain is configured via the + SASL configuration for the broker, and the domain/realm for qpidd + is set using --realm and default to 'QPID'. + </para><para> + To load the ACL module use, load the acl module cmd line or via + the config file + </para> + <programlisting> +./src/qpidd --load-module src/.libs/acl.so +</programlisting> + <para> + The ACL plugin provides the following option '--acl-file'. If do + ACL file is supplied the broker will not enforce ACL. If an ACL + file name is supplied, and the file does not exist or is invalid + the broker will not start. + </para> + <programlisting> +ACL Options: + --acl-file FILE The policy file to load from, loaded from data dir +</programlisting> + + <!--h3--></section> + <!--h2--></section> + +</chapter> diff --git a/qpid/doc/book/src/AMQP%20Messaging%20Clients.xml b/qpid/doc/book/src/AMQP%20Messaging%20Clients.xml new file mode 100644 index 0000000000..2fcd4ea33f --- /dev/null +++ b/qpid/doc/book/src/AMQP%20Messaging%20Clients.xml @@ -0,0 +1,11 @@ +<?xml version="1.0"?> +<section> + <title/> + <para> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="AMQP%20Java%20JMS%20Messaging%20Client.html"/> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="4589059.html"/> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="AMQP%20.NET%20Messaging%20Client.html"/> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="AMQP%20Python%20Messaging%20Client.html"/> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="AMQP%20Ruby%20Messaging%20Client.html"/> + </para> +</section> diff --git a/qpid/doc/book/src/AMQP-Messaging-Broker-CPP.xml b/qpid/doc/book/src/AMQP-Messaging-Broker-CPP.xml new file mode 100644 index 0000000000..3cd8be8665 --- /dev/null +++ b/qpid/doc/book/src/AMQP-Messaging-Broker-CPP.xml @@ -0,0 +1,35 @@ +<?xml version="1.0"?> +<chapter> + <title>AMQP Messaging Broker (implemented in C++)</title> + <section> + <title> + Running the AMQP Messaging Broker + </title> + + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="RASC.xml"/> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="Cheat%20Sheet%20for%20configuring%20Queue%20Options.xml"/> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="Cheat%20Sheet%20for%20configuring%20Exchange%20Options.xml"/> + + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="Using%20Broker%20Federation.xml"/> + + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="SSL.xml"/> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="LVQ.xml"/> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="queue%20state%20replication.xml"/> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="Starting%20a%20cluster.xml"/> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="ACL.xml"/> + + </section> + + <section> + <title> + Managing the AMQP Messaging Broker + </title> + + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="82403.xml"/> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="QMan%20-%20Qpid%20Management%20bridge.xml"/> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="Qpid%20Management%20Framework.xml"/> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="Management%20Design%20notes.xml"/> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="QMF%20Python%20Console%20Tutorial.xml"/> + + </section> +</chapter> diff --git a/qpid/doc/book/src/AMQP-Messaging-Broker-Java.xml b/qpid/doc/book/src/AMQP-Messaging-Broker-Java.xml new file mode 100644 index 0000000000..15e37ed64e --- /dev/null +++ b/qpid/doc/book/src/AMQP-Messaging-Broker-Java.xml @@ -0,0 +1,132 @@ +<?xml version="1.0"?> +<section> + <title> + General + User Guides + </title> + <itemizedlist> + <listitem> + <para> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="Java%20Broker%20Feature%20Guide.html"/> + </para> + </listitem> + <listitem> + <para> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="Qpid%20Java%20FAQ.html"/> + </para> + </listitem> + <listitem> + <para> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="Getting%20Started%20Guide.html"/> + </para> + </listitem> + <listitem> + <para> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="Java%20Environment%20Variables.html"/> + </para> + </listitem> + <listitem> + <para> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="Qpid%20Troubleshooting%20Guide.html"/> + </para> + </listitem> + </itemizedlist> + <bridgehead id="AMQPMessagingBroker%28implementedinJava%29-HowTos"><anchor id="AMQPMessagingBroker%28implementedinJava%29-HowTos"/>How Tos + </bridgehead> + <itemizedlist> + <listitem> + <para> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="Add%20New%20Users.html"/> + </para> + </listitem> + <listitem> + <para> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="Configure%20ACLs.html"/> + </para> + </listitem> + <listitem> + <para> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="Configure%20Java%20Qpid%20to%20use%20a%20SSL%20connection..html"/> + </para> + </listitem> + <listitem> + <para> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="Configure%20Log4j%20CompositeRolling%20Appender.html"/> + </para> + </listitem> + <listitem> + <para> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="Configure%20the%20Broker%20via%20config.xml.html"/> + </para> + </listitem> + <listitem> + <para> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="Configure%20the%20Virtual%20Hosts%20via%20virtualhosts.xml.html"/> + </para> + </listitem> + <listitem> + <para> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="Debug%20using%20log4j.html"/> + </para> + </listitem> + <listitem> + <para> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="How%20to%20Tune%20M3%20Java%20Broker%20Performance.html"/> + </para> + </listitem> + <listitem> + <para> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="Qpid%20Java%20Build%20How%20To.html"/> + </para> + </listitem> + <listitem> + <para> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="Use%20Priority%20Queues.html"/> + </para> + </listitem> + </itemizedlist> + <bridgehead id="AMQPMessagingBroker%28implementedinJava%29-ManagementTools"><anchor id="AMQPMessagingBroker%28implementedinJava%29-ManagementTools"/>Management + Tools + </bridgehead> + <itemizedlist> + <listitem> + <para> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="Qpid%20JMX%20Management%20Console.html"/> + </para> + </listitem> + <listitem> + <para> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="MessageStore%20Tool.html"/> + </para> + </listitem> + <listitem> + <para> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="Qpid%20Java%20Broker%20Management%20CLI.html"/> + </para> + </listitem> + <listitem> + <para> + <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="Management%20Design%20notes.html"/> + </para> + </listitem> + </itemizedlist> + <informaltable> + <tgroup cols="1"> + <colspec colnum="1" colname="col1"/> + <thead> + <row> + <entry> + <anchor id="comment-5144724"/> + <para> + Replaces <ulink url="http://cwiki.apache.org/confluence/display/qpid/Java+Broker"> + http://cwiki.apache.org/confluence/display/qpid/Java+Broker</ulink></para> + <para> + Identical content, changing title and location. + </para> + </entry> + </row> + </thead> + <tbody/> + </tgroup> + </informaltable> +</section> diff --git a/qpid/doc/book/src/AMQP.xml b/qpid/doc/book/src/AMQP.xml new file mode 100644 index 0000000000..b9127401c0 --- /dev/null +++ b/qpid/doc/book/src/AMQP.xml @@ -0,0 +1,75 @@ +<chapter> +<title>AMQP (Advanced Message Queueing Protocol</title> + + +<para>AMQP <ulink url="http://www.amqp.org/" >Advanced +Message Queuing Protocol</ulink> is an open standard designed to +support reliable, high-performance messaging over the Internet. +AMQP can be used for any distributed or business application, and +supports common messaging paradigms like point-to-point, fanout, +publish-subscribe, and request-response.</para> +<para>Apache Qpid implements AMQP, including transaction management, +queuing, clustering, federation, security, management and +multi-platform support.</para> +<para>Apache Qpid implements the latest AMQP specification, providing +transaction management, queuing, distribution, security, +management, clustering, federation and heterogeneous multi-platform +support and a lot more.</para> +<para>Apache Qpid is highly optimized, and <ulink url= +"amqp-compatibility.html">aims to be 100% AMQP Compliant</ulink>.</para> + +<section> +<title>Download the AMQP Specifications</title> + +<itemizedlist> +<title>AMQP version 0-10</title> +<listitem><para><ulink url="https://jira.amqp.org/confluence/download/attachments/720900/amqp.0-10.pdf?version=1" +>AMQP 0-10 Specification (PDF)</ulink></para></listitem> +<listitem><para><ulink> url="https://jira.amqp.org/confluence/download/attachments/720900/amqp.0-10.xml?version=1" +>AMQP 0-10 Protocol Definition XML</ulink></para></listitem> +<listitem><para><ulink> url="https://jira.amqp.org/confluence/download/attachments/720900/amqp.0-10.dfd?version=1" +>AMQP 0-10 Protocol Definition DTD</ulink></para></listitem> +</itemizedlist> + + +<itemizedlist> +<title>AMQP version 0-9-1</title> +<listitem><para><ulink url= +"https://jira.amqp.org/confluence/download/attachments/720900/amqp0-9-1.pdf?version=1" +>AMQP 0-9-1 Specification (PDF)</ulink></para></listitem> +<listitem><para><ulink url= +"https://jira.amqp.org/confluence/download/attachments/720900/amqp0-9-1.xml?version=1" +>AMQP 0-9-1 Protocol Documentation (PDF)</ulink></para></listitem> +<listitem><para><ulink url= +"https://jira.amqp.org/confluence/download/attachments/720900/amqp0-9-1.dtd?version=1" +>AMQP 0-9-1 Protocol Definitions (XML)</ulink></para></listitem> +</itemizedlist> + + +<itemizedlist> +<title>AMQP version 0-9</title> +<listitem><para><ulink url= +"https://jira.amqp.org/confluence/download/attachments/720900/amqp0-9.pdf?version=1" +>AMQP 0-9 Specification (PDF)</ulink></para></listitem> +<listitem><para><ulink url= +"https://jira.amqp.org/confluence/download/attachments/720900/amqp0-9.xml?version=1" +>AMQP 0-9 Protocol Documentation (PDF)</ulink></para></listitem> +<listitem><para><ulink url= +"https://jira.amqp.org/confluence/download/attachments/720900/amqp0-9.dtd?version=1" +>AMQP 0-9 Protocol Definitions (XML)</ulink></para></listitem> +</itemizedlist> + + +<itemizedlist> +<title>AMQP version 0-8</title> +<listitem><para><ulink url= +"https://jira.amqp.org/confluence/download/attachments/720900/amqp0-8.pdf?version=1" +>AMQP 0-8 Specification (PDF)</ulink></para></listitem> +<listitem><para><ulink url="https://jira.amqp.org/confluence/download/attachments/720900/amqp0-8.dtd?version=1" +>AMQP 0-8 Protocol Documentation (PDF)</ulink></para></listitem> +<listitem><para><ulink url="https://jira.amqp.org/confluence/download/attachments/720900/amqp0-8.xml?version=1" +>AMQP 0-8 Protocol Definitions (XML)</ulink></para></listitem> +</itemizedlist> +</section> + +</chapter>
\ No newline at end of file diff --git a/qpid/doc/book/src/AMQP_Compatibility.xml b/qpid/doc/book/src/AMQP_Compatibility.xml new file mode 100644 index 0000000000..595855f1eb --- /dev/null +++ b/qpid/doc/book/src/AMQP_Compatibility.xml @@ -0,0 +1,64 @@ +http://qpid.apache.org/amqp-compatibility.html + +*Qpid provides the most complete and compatible implementation of AMQP. And is the most aggressive in implementing the latest version of the specification. Qpid can be* *[*downloaded here*|Download]* + +There are two brokers: + +C+\+ with support for AMQP 0-10 +Java with support for AMQP 0-8 and 0-9 (0-10 planned) + +There are client libraries for C++, Java (JMS), .Net (written in C#), python and ruby. +* All clients support 0-10 and interoperate with the C+\+ broker. + +* The JMS client supports 0-8, 0-9 and 0-10 and interoperates with both brokers. + +* The python and ruby clients will also support all versions, but the API is dynamically driven by the specification used and so differs between versions. To work with the Java broker you must use 0-8 or 0-9, to work with the C+\+ broker you must use 0-10. + +* There are two separate C# clients, one for 0-8 that interoperates with the Java broker, one for 0-10 that inteoperates with the C+\+ broker. + +QMF Management is supported in Ruby, Python, C++, and via QMan for Java JMX & WS-DM. + +h3. AMQP Compatibility of Qpid releases: + +Qpid implements the AMQP Specification, and as the specification has progressed Qpid is keeping up with the updates. This means that different Qpid versions support different versions of AMQP. Here is a simple guide on what use. + +Here is a matrix that describes the different versions supported by each release +Y = supported +N = unsupported +IP = in progress +P = planned +|| Component || Spec || || || || || +| | || M2.1 || M3 || M4 || 0.5 | +|| java client | 0-10 | | Y | Y | Y || +| | 0-9 | Y | Y | Y | Y | +| | 0-8 | Y | Y | Y | Y | +|| java broker | 0-10 | | | | P || +| | 0-9 | Y | Y | Y | Y | +| | 0-8 | Y | Y | Y | Y | +|| c+\+ client/broker | 0-10 | | Y | Y | Y || +| | 0-9 | Y | | | | +|| python client | 0-10 | | Y | Y | Y || +| | 0-9 | Y | Y | Y | Y | +| | 0-8 | Y | Y | Y | Y | +|| ruby client | 0-10 | | | Y | Y || +| | 0-8 | Y | Y | Y | Y | +|| C# client | 0-10 | | | Y | Y || +| | 0-8 | Y | Y | Y | Y | + +h3. Interop table by AMQP specification version + +Above table represented in another format. +| | release | 0-8 | 0-9 | 0-10 | +| java client | M3 M4 0.5 | Y | Y | Y | +| java client | M2.1 | Y | Y | N | +| java broker | M3 M4 0.5 | Y | Y | N | +| java broker | trunk | Y | Y | P | +| java broker | M2.1 | Y | Y | N | +| c+\+ client/broker | M3 M4 0.5 | N | N | Y | +| c+\+ client/broker | M2.1 | N | Y | N | +| python client | M3 M4 0.5 | Y | Y | Y | +| python client | M2.1 | Y | Y | N | +| ruby client | M3 M4 0.5 | Y | Y | N | +| ruby client | trunk | Y | Y | P | +| C# client | M3 M4 0.5 | Y | N | N | +| C# client | trunk | Y | N | Y | diff --git a/qpid/doc/book/src/Book.xml b/qpid/doc/book/src/Book.xml index 52154810ee..ae590433ed 100644 --- a/qpid/doc/book/src/Book.xml +++ b/qpid/doc/book/src/Book.xml @@ -4,14 +4,23 @@ <book xmlns:xi="http://www.w3.org/2001/XInclude"> <xi:include href="Book_Info.xml"/> - <xi:include href="Introduction.xml"/> - <xi:include href="Getting_Started.xml"/> - <xi:include href="Download.xml"/> - <xi:include href="AMQP.xml"/> - <xi:include href="Brokers.xml"/> - <xi:include href="Clients.xml"/> + <part> + <title>Basics</title> + <xi:include href="Introduction.xml"/> + <xi:include href="AMQP.xml"/> + <xi:include href="Getting_Started.xml"/> + <xi:include href="Download.xml"/> + </part> + <xi:include href="Brokers.xml"/> + <part> + <title>Clients</title> + <xi:include href="Clients.xml"/> + </part> + <part> + <title>Appendices</title> <xi:include href="Interoperability.xml"/> <!-- <xi:include href="FAQ.xml"/> --> + </part> </book> <!-- diff --git a/qpid/doc/book/src/Broker_CPP.xml b/qpid/doc/book/src/Broker_CPP.xml new file mode 100644 index 0000000000..cf3b3416e8 --- /dev/null +++ b/qpid/doc/book/src/Broker_CPP.xml @@ -0,0 +1,20 @@ +h1. Running the AMQP Messaging Broker + +* [Running an AMQP 0-10 C++ broker | RASC] +* [Configuring Queue Options|Cheat Sheet for configuring Queue Options] +* [Configuring Exchange Options|Cheat Sheet for configuring Exchange Options] +* [Using Broker Federation] +* [How to use SSL | SSL] +* [Understanding Last Value Queues (LVQ) |LVQ] +* [Queue State Replication|queue state replication] +* [Getting Started] +* [Starting a cluster] +* [Understanding Access Control Lists|Qpid ACLs] + +h2. Management + +* [Managing| MgmtC++] the C++ Broker +* [QMan - Qpid Management bridge] +* [Qpid Management Framework] +* [Qpid Management Framework (QMF) Protocol|management design notes] +* [Manage anything with Qpid - QMF Python Console Tutorial|QMF Python Console Tutorial] diff --git a/qpid/doc/book/src/Broker_Java.xml b/qpid/doc/book/src/Broker_Java.xml new file mode 100644 index 0000000000..aa7e89f9b8 --- /dev/null +++ b/qpid/doc/book/src/Broker_Java.xml @@ -0,0 +1,25 @@ +h3. General User Guides + +* [Feature Guide| Java Broker Feature Guide] +* [FAQ|Qpid Java FAQ] +* [Getting Started Guide] +* [Broker Environment Variables|Java Environment Variables] +* [Troubleshooting Guide|Qpid Troubleshooting Guide] + +h3. How Tos +* [Add New Users] +* [Configure ACLs] +* [Configure Java Qpid to use a SSL connection.] +* [Configure Log4j CompositeRolling Appender] +* [Configure the Broker via config.xml] +* [Configure the Virtual Hosts via virtualhosts.xml] +* [Debug using log4j] +* [How to Tune M3 Java Broker Performance] +* [Qpid Java Build How To] +* [Use Priority Queues] + +h3. Management Tools +* [Qpid JMX Management Console] +* [MessageStore Tool] +* [Qpid Java Broker Management CLI] +* [Management Design notes] diff --git a/qpid/doc/book/src/Brokers.xml b/qpid/doc/book/src/Brokers.xml new file mode 100644 index 0000000000..8854f21234 --- /dev/null +++ b/qpid/doc/book/src/Brokers.xml @@ -0,0 +1,19 @@ +<part> + <title>AMQP Messaging Brokers</title> + + <partintro> + <para>Qpid provides two AMQP messaging brokers:</para> + + <itemizedlist> + <listitem><para>Implemented in C++ - high performance, low latency, and RDMA support.</para></listitem> + <listitem><para>Implemented in Java - Fully JMS compliant, runs on any Java platform.</para></listitem> + </itemizedlist> + + <para>Both AMQP messaging brokers support clients in multiple languages, as long as the messaging client and the messaging broker use the same version of AMQP. See <link linkend="###AMQP-Compatibility"/> to see which messaging clients work with each broker.</para> + </partintro> + + <xi:include href="AMQP-Messaging-Broker-CPP.xml" xmlns:xi="http://www.w3.org/2001/XInclude"/> + + <xi:include href="AMQP-Messaging-Broker-Java.xml" xmlns:xi="http://www.w3.org/2001/XInclude"/> + +</part> diff --git a/qpid/doc/book/src/Cheat Sheet for configuring Exchange Options.xml b/qpid/doc/book/src/Cheat Sheet for configuring Exchange Options.xml new file mode 100644 index 0000000000..bc918336f9 --- /dev/null +++ b/qpid/doc/book/src/Cheat Sheet for configuring Exchange Options.xml @@ -0,0 +1,118 @@ +<?xml version="1.0" encoding="utf-8"?> +<chapter xmlns:html="http://www.w3.org/1999/xhtml"><title> + Apache Qpid : Cheat Sheet for configuring Exchange Options + </title><section role="h2" id="CheatSheetforconfiguringExchangeOptions-ConfiguringExchangeOptions"><title> + Configuring Exchange Options + </title> + <para> + The C++ Broker M4 or later supports the following additional + Exchange options in addition to the standard AMQP define options + </para><itemizedlist> + <listitem><para>Exchange Level Message sequencing + </para></listitem> + <listitem><para>Initial Value Exchange + </para></listitem> + </itemizedlist><para> + Note that these features can be used on any exchange type, that + has been declared with the options set. + </para><para> + It also supports an additional option to the bind operation on a + direct exchange + </para><itemizedlist> + <listitem><para>Exclusive binding for key + </para></listitem> + </itemizedlist> + + <section role="h3" id="CheatSheetforconfiguringExchangeOptions-ExchangeLevelMessagesequencing"><title> + Exchange Level Message sequencing + </title> + <para> + This feature can be used to place a sequence number into each + message's headers, based on the order they pass through an + exchange. The sequencing starts at 0 and then wraps in an AMQP + int64 type. + </para><para> + The field name used is "qpid.msg_sequence" + </para><para> + To use this feature an exchange needs to be declared specifying + this option in the declare + </para> + <programlisting> +.... + FieldTable args; + args.setInt("qpid.msg_sequence",1); + +... + // now declare the exchange + session.exchangeDeclare(arg::exchange="direct", arg::arguments=args); +</programlisting> + <para> + Then each message passing through that exchange will be numbers + in the application headers. + </para> + <programlisting> + unit64_t seqNo; + //after message transfer + seqNo = message.getHeaders().getAsInt64("qpid.msg_sequence"); +</programlisting> + <!--h3--></section> + <section role="h3" id="CheatSheetforconfiguringExchangeOptions-InitialValueExchange"><title> + Initial + Value Exchange + </title> + <para> + This feature caches a last message sent to an exchange. When a + new binding is created onto the exchange it will then attempt to + route this cached messaged to the queue, based on the binding. + This allows for topics or the creation of configurations where a + new consumer can receive the last message sent to the broker, + with matching routing. + </para><para> + To use this feature an exchange needs to be declared specifying + this option in the declare + </para> + <programlisting> +.... + FieldTable args; + args.setInt("qpid.ive",1); + +... + // now declare the exchange + session.exchangeDeclare(arg::exchange="direct", arg::arguments=args); +</programlisting> + <para> + now use the exchange in the same way you would use any other + exchange. + </para> + <!--h3--></section> + + <section role="h3" id="CheatSheetforconfiguringExchangeOptions-Exclusivebindingforkey"><title> + Exclusive + binding for key + </title> + <para> + Direct exchanges in qpidd support a qpid.exclusive-binding option + on the bind operation that causes the binding specified to be the + only one for the given key. I.e. if there is already a binding at + this exchange with this key it will be atomically updated to bind + the new queue. This means that the binding can be changed + concurrently with an incoming stream of messages and each message + will be routed to exactly one queue. + </para> + <programlisting> +.... + FieldTable args; + args.setInt("qpid.exclusive-binding",1); + + //the following will cause the only binding from amq.direct with 'my-key' + //to be the one to 'my-queue'; if there were any previous bindings for that + //key they will be removed. This is atomic w.r.t message routing through the + //exchange. + session.exchangeBind(arg::exchange="amq.direct", arg::queue="my-queue", + arg::bindingKey="my-key", arg::arguments=args); + +... +</programlisting> +<!--h3--></section> +<!--h2--></section> +</chapter> diff --git a/qpid/doc/book/src/Cheat Sheet for configuring Queue Options.xml b/qpid/doc/book/src/Cheat Sheet for configuring Queue Options.xml new file mode 100644 index 0000000000..115e75a03c --- /dev/null +++ b/qpid/doc/book/src/Cheat Sheet for configuring Queue Options.xml @@ -0,0 +1,227 @@ +<?xml version="1.0" encoding="utf-8"?> +<chapter xmlns:html="http://www.w3.org/1999/xhtml"><title> + Apache Qpid : Cheat Sheet for configuring Queue Options + </title> + + <section role="h2" id="CheatSheetforconfiguringQueueOptions-ConfiguringQueueOptions"><title> + Configuring + Queue Options + </title> + + <para> + The C++ Broker M4 or later supports the following additional + Queue constraints. + </para> + <itemizedlist> + <listitem><para> + <xref linkend="CheatSheetforconfiguringQueueOptions-ConfiguringQueueOptions"/> + </para></listitem> + <listitem><para> + <itemizedlist> + <listitem><para> + <xref linkend="CheatSheetforconfiguringQueueOptions-ApplyingQueueSizingConstraints"/> + </para></listitem> + <listitem><para> + <xref linkend="CheatSheetforconfiguringQueueOptions-ChangingtheQueueorderingBehaviors-28FIFO-2FLVQ-29"/> + </para></listitem> + <listitem><para> + <xref linkend="CheatSheetforconfiguringQueueOptions-Settingadditionalbehaviors"/> + </para></listitem> + <listitem><para> + <itemizedlist> + <listitem><para> + <xref linkend="CheatSheetforconfiguringQueueOptions-PersistLastNode"/> + </para></listitem> + <listitem><para> + <xref linkend="CheatSheetforconfiguringQueueOptions-Queueeventgeneration"/> + </para></listitem> + </itemizedlist> + </para></listitem> + <listitem><para> + <xref linkend="CheatSheetforconfiguringQueueOptions-OtherClients"/> + </para></listitem> + </itemizedlist> + </para></listitem> + </itemizedlist> + <section role="h3" id="CheatSheetforconfiguringQueueOptions-ApplyingQueueSizingConstraints"><title> + Applying Queue Sizing Constraints + </title> + + <para> + This allows to specify how to size a queue and what to do when + the sizing constraints have been reached. The queue size can be + limited by the number messages (message depth) or byte depth on + the queue. + </para><para> + Once the Queue meets/ exceeds these constraints the follow + policies can be applied + </para><itemizedlist> + <listitem><para>REJECT - Reject the published message + </para></listitem> + <listitem><para>FLOW_TO_DISK - Flow the messages to disk, to preserve memory + </para></listitem> + <listitem><para>RING - start overwriting messages in a ring based on sizing. + If head meets tail, advance head + </para></listitem> + <listitem><para>RING_STRICT - start overwriting messages in a ring based on + sizing. If head meets tail, AND the consumer has the tail message + acquired it will reject + </para></listitem> + </itemizedlist><para> + Examples: + </para><para> + Create a queue an auto delete queue that will support 100 000 + bytes, and then REJECT + </para> + <programlisting> +#include "qpid/client/QueueOptions.h" + + QueueOptions qo; + qo.setSizePolicy(REJECT,100000,0); + + session.queueDeclare(arg::queue=queue, arg::autoDelete=true, arg::arguments=qo); +</programlisting> + <para> + Create a queue that will support 1000 messages into a RING buffer + </para> + <programlisting> +#include "qpid/client/QueueOptions.h" + + QueueOptions qo; + qo.setSizePolicy(RING,0,1000); + + session.queueDeclare(arg::queue=queue, arg::arguments=qo); +</programlisting> + <!--h3--></section> + <section role="h3" id="CheatSheetforconfiguringQueueOptions-ChangingtheQueueorderingBehaviors-28FIFO-2FLVQ-29"><title> + Changing the Queue ordering Behaviors (FIFO/LVQ) + </title> + <para> + The default ordering in a queue in Qpid is FIFO. However + additional ordering semantics can be used namely LVQ (Last Value + Queue). Last Value Queue is define as follows. + </para><para> + If I publish symbols RHT, IBM, JAVA, MSFT, and then publish RHT + before the consumer is able to consume RHT, that message will be + over written in the queue and the consumer will receive the last + published value for RHT. + </para><para> + Example: + </para> + <programlisting> +#include "qpid/client/QueueOptions.h" + + QueueOptions qo; + qo.setOrdering(LVQ); + + session.queueDeclare(arg::queue=queue, arg::arguments=qo); + + ..... + string key; + qo.getLVQKey(key); + + .... + for each message, set the into application headers before transfer + message.getHeaders().setString(key,"RHT"); + +</programlisting> + <para> + Notes: + </para><itemizedlist> + <listitem><para>Messages that are dequeued and the re-queued will have the + following exceptions. a.) if a new message has been queued with + the same key, the re-queue from the consumer, will combine these + two messages. b.) If an update happens for a message of the same + key, after the re-queue, it will not update the re-queued + message. This is done to protect a client from being able to + adversely manipulate the queue. + </para></listitem> + <listitem><para>Acquire: When a message is acquired from the queue, no matter + it's position, it will behave the same as a dequeue + </para></listitem> + <listitem><para>LVQ does not support durable messages. If the queue or + messages are declared durable on an LVQ, the durability will be + ignored. + </para></listitem> + </itemizedlist><para> + A fully worked <xref linkend="qpid_LVQ-20Example"/> can be found here + </para> + <!--h3--></section> + <section role="h3" id="CheatSheetforconfiguringQueueOptions-Settingadditionalbehaviors"><title> + Setting additional behaviors + </title> + <section role="h4" id="CheatSheetforconfiguringQueueOptions-PersistLastNode"><title> + Persist + Last Node + </title> + <para> + This option is used in conjunction with clustering. It allows for + a queue configured with this option to persist transient messages + if the cluster fails down to the last node. If additional nodes + in the cluster are restored it will stop persisting transient + messages. + </para><para> + Note + </para><itemizedlist> + <listitem><para>if a cluster is started with only one active node, this mode + will not be triggered. It is only triggered the first time the + cluster fails down to 1 node. + </para></listitem> + <listitem><para>The queue MUST be configured durable + </para></listitem> + </itemizedlist><para> + Example: + </para> + <programlisting> +#include "qpid/client/QueueOptions.h" + + QueueOptions qo; + qo.clearPersistLastNode(); + + session.queueDeclare(arg::queue=queue, arg::durable=true, arg::arguments=qo); +</programlisting> + <!--h4--></section> + <section role="h4" id="CheatSheetforconfiguringQueueOptions-Queueeventgeneration"><title> + Queue + event generation + </title> + <para> + This option is used to determine whether enqueue/dequeue events + representing changes made to queue state are generated. These + events can then be processed by plugins such as that used for + <xref linkend="qpid_queue-20state-20replication"/>. + </para><para> + Example: + </para> + <programlisting> +#include "qpid/client/QueueOptions.h" + + QueueOptions options; + options.enableQueueEvents(1); + session.queueDeclare(arg::queue="my-queue", arg::arguments=options); +</programlisting> + <para> + The boolean option indicates whether only enqueue events should + be generated. The key set by this is + 'qpid.queue_event_generation' and the value is and integer value + of 1 (to replicate only enqueue events) or 2 (to replicate both + enqueue and dequeue events). + </para> + <!--h3--></section> + <!--h4--></section> + <section role="h3" id="CheatSheetforconfiguringQueueOptions-OtherClients"><title> + Other + Clients + </title> + <para> + Note that these options can be set from any client. QueueOptions + just correctly formats the arguments passed to the QueueDeclare() + method. + </para> + + <!--h3--></section> + + <!--h2--></section> + + +</chapter> diff --git a/qpid/doc/book/src/Clients.xml b/qpid/doc/book/src/Clients.xml new file mode 100644 index 0000000000..3519d7d7ad --- /dev/null +++ b/qpid/doc/book/src/Clients.xml @@ -0,0 +1,5 @@ +[AMQP Java JMS Messaging Client] +[AMQP C++ Messaging Client] +[AMQP .NET Messaging Client] +[AMQP Python Messaging Client] +[AMQP Ruby Messaging Client] diff --git a/qpid/doc/book/src/Download.xml b/qpid/doc/book/src/Download.xml index 24f5c80eb1..86ad4b65d8 100644 --- a/qpid/doc/book/src/Download.xml +++ b/qpid/doc/book/src/Download.xml @@ -23,7 +23,8 @@ <section><title>Multiple Component Packages</title> -<table><thead><tr><td> Component </td><td> Download </td><td> AMQP 0-10 </td><td> AMQP 0-8/0-9 </td></tr></thead> +<table> +<thead><tr><td> Component </td><td> Download </td><td> AMQP 0-10 </td><td> AMQP 0-8/0-9 </td></tr></thead> <tbody> <tr><td> Full release & keys </td><td> <ulink url="http://www.apache.org/dist/qpid/0.5/">http://www.apache.org/dist/qpid/0.5/</ulink> </td><td> Y </td><td> Y </td></tr> <tr><td> C++ broker & client </td><td> <ulink url="http://www.apache.org/dist/qpid/0.5/qpid-cpp-0.5.tar.gz">http://www.apache.org/dist/qpid/0.5/qpid-cpp-0.5.tar.gz</ulink> </td><td> Y </td><td> </td></tr> @@ -137,6 +138,5 @@ </para> </section> -<!-- ##################### --> </chapter> diff --git a/qpid/doc/book/src/FAQ.xml b/qpid/doc/book/src/FAQ.xml new file mode 100644 index 0000000000..27db7c31a8 --- /dev/null +++ b/qpid/doc/book/src/FAQ.xml @@ -0,0 +1,504 @@ +{toc} + + +*This page is a collection of FAQ and How to-s for Qpid. If you have a question, post it to the users list and we will place the answer here to build out our FAQ/ How to.* + +h1. FAQ + +h2. About AMQP + +h3. What is AMQP? + +AMQP is a wire-level protocol and model for high performance enterprise messaging. + +[From the AMQP website:|http://www.amqp.org] + + AMQP is an Open Standard for Messaging Middleware. + + By complying to the AMQP standard, middleware products written for different platforms and in different languages can send messages to one another. AMQP addresses the problem of transporting value-bearing messages across and between organizations in a timely manner. + + AMQP enables complete interoperability for messaging middleware; both the networking protocol and the semantics of broker services are defined in AMQP. + +h3. Where did AMQP come from + +AMQP was born out from Frustration by John O'Hara at JPMC. He started a project internally to create commodity messaging that was easy to use. Carl Trieloff from Red Hat had started a project to build messaging for both users and for use in infrastructure, while looking around spoke to John about his work. Out of these discussion was born the AMQP working Group with 6 initial members, under an agreement that it will be eternally be licensed for everyone to use. + +Since then the Working Group has had many join, and has been making solid progress working on revisions of the specification. [For more details see.|http://jira.amqp.org/confluence/display/AMQP/About+AMQP] + +h3. Why use AMQP? + +AMQP is has been designed to be able to handle the hardest workloads, scale to the largest systems, but also deal with reduction of change and maintenance costs by doing a refresh on many aged practices. The specification is also not language specific allowing the freedom from language and platform lock in, without compromise on user experience, security, scalability and consistently excellent performance. + +[Text mostly taken from|http://jira.amqp.org/confluence/display/AMQP/About+AMQP] + +h2. Qpid & AMQP + + +h3. Is Qpid AMQP Compliant? + +Yes, Apache Qpid implements the latest AMQP specifications, providing transaction management, queuing, distribution, security, management, clustering, federation and heterogeneous multi-platform support and a lot more. And Apache Qpid is extremely fast. [Apache Qpid aims to be 100% AMQP Compliant|AMQP compatibility]. + +h3. What Client support does Qpid have? + +Apache Qpid provides AMQP Client APIs for the following languages: +* C+\+ +* C# .NET, using WCF +* Ruby +* Python +* Java JMS, fully conformant with Java CTS1.1 + +If you need another client, join the lists and ask or feel free to contribute one. + +h3. What messaging topologies are supported by AMQP and Qpid? + +AMQP provides the ability to do Point-to-Point, Peer-to-Peer, Pub-Sub, and Eventing. This allows many patterns to be craeted: + ++*Point-to-point*+ + +This is one of the simplest use-cases. AMQP allows for this in a few ways. + a.) A client can create a named queue allowing the producer to publish the message to the direct exchange with the key mapping the queue name. This will route the message to that queue. + b.) The above pattern can be extended by specifying a reply-to address in the published messages allowing for the consumer to reply the producer without knowing who it was send from prior to receiving the message. + ++*One-to-many*+ + +There are a few patterns that can be used. + + a.) AMQP provides a 'fanout' exchange which will send a message to all the queues that have been bound to it. Different domains or topics are created with the 'fanout' exchange by declaring different named fan-out exchanges. + + b.) A 'topic' or 'headers' exchange can also be used. in this case the pattern match is used to send the message to all the bound queues. It can be thought of as a filter allowing you to create just about any One-to-many routing patterns. + ++*Pub-Sub*+ + +Topic can be created with the 'topic' or other 'direct' exchange to allow consumer to bind to into the steams of data they care about. This pattern combined with the use of reply-to and Alternate-routing is the staple of what most people use messaging for today. + ++*FAST Reliable Messaging*+ + +AMQP 0-10 allows for fully reliable transfers between any two peers. This means that you can publish or subscribe to the broker fully reliable without requiring the need for transactions. This can all be done in async mode with the C++ broker allowing for high throughput while running entirely reliable. + ++*Transactional*+ + +AMQP supports two types of transactions in AMQP 0-10, TX and DTX. This allows for local (1PC), and 2PC transaction and the ability to coordinate with a TM (Transaction Manager). The Java broker supports TX, the C++ broker support TX, DTX, XA, JTA for fully ACID transactions. This allows you to commit a single unit of work with may contain enqueues & dequeues either locally on the broker, or in coordination with other transactional resource like RDBMS. + ++*Transient message delivery*+ + +By default messages are transient. Transient message can be sent to queues that are durable. They will not be safe stored or recovered, and will perform as any other transient message - fast! + ++*Durable message delivery*+ + +There is a header on each message where the message properties are specified, one of these is durability. Messages that are marked as durable and published to a durable queue will be safe stored. Durable messages will survive restart of the broker or cluster. + ++*Federation (Hub-spoke, Trees, graphs)*+ + +As AMQP 0-10 is symmetric for peer-to-peer communication all the building block are in place for creating networks of brokers. The C++ broker allows you to link the brokers together using 'qpid-route' and then create routes between the brokers either statically or with dynamic routes. + +This allows for a message to be published to one broker and consumed from another broker in the federated broker network. This feature is great to create data-center, or project isolation, but allow cross communication. It also allows networks to be created to scaled. [For more details see|Using Broker Federation] + ++*And many others, including custom pattern*+ + ++*Message Reply, Rings, Initial Value Caches, Last Value Messaging*+ + +All the above cases can be constructed using the AMQP and features of Qpid. For example reply can be constructed using message browsing and setting TTL on the messages. The C++ broker also support ring queues, last value queues, initial value caches on exchanges. With a bit of throught many additional patterns can be constructed. + +*+Store-and-forward+* + +Store-and-forward can be achieved by publishing to well know durable queues, that are not marked with auto delete. Consumers will be able to 'came back' to consume then at any time, even after restarts. + +h3. What AMQP and other exchanges does Qpid support? + +Both brokers support: + * Direct Exchange + * Topic Exchange + * Fanout Exchange + * Headers Exchange + +In additional the C++ broker support + * XML Exchange - Query routing + * Custom exchange via plug-in. + +Custom exchanges allow you to provide your own custom routing logic and algorithms via a plug-in. If you build an interesting exchange, please feel free to contribute it back to the Qpid project. + +h2. Security + +h3. What encryption does Qpid support? + + * Qpid support SSL/TSL as per the AMQP specification. + * In addition the C++ broker supports Kerberos encryption of messages independent on which transport is used. Support in not yet included in all clients for this but is in process. + +h3. What authentication does Qpid support? + +SASL Authentication is supported. All Clients support PLAIN, and Kerberos support if being added to all the clients. The C++ broker support Kerberbos authentication. + +h3. What authorization does Qpid support? + +Full ACL is supported in the brokers. [For details on configuring ACL see|Qpid ACLs]. + +ACL supports realms and allows for granular permission to be set on all the broker actions including management on an user or group basis. + +h3. How to setup Kerberos with the Java client + +You could force the java client to use kerberos auth by specifying it in the connection URL as follows. +{code} +amqp://guest:guest@clientid/testpath?brokerlist='tcp://localhost:5672?'&sasl_mechs='GSSAPI' +{code} + +You would then need to pass in the following jvm arguments +{code} +-Djavax.security.auth.useSubjectCrehttp://code.google.com/p/confluence-el/dsOnly=false +# (This will force the SASL GASSPI client to obtain the kerberos credentials explicitly instead of obtaining from the "subject" that owns the currents thread) +-Djava.security.auth.login.config=myjas.conf (this specifies the jass config file) +-Dsun.security.krb5.debug=true (to enable detailed debug info for troubleshooting) +{code} + +Before running the java client you would need to do kinit and grab a kerberos ticket. Alternative you could set useTicketCache=false and when the client loads, it will prompt you for the user/pass and will obtain the ticket +(You would also need to setup your kerberos environment properly -refer to doc links below). + +Sample JASS Config file +{code} +com.sun.security.jgss.initiate { + com.sun.security.auth.module.Krb5LoginModule required useTicketCache=true; +}; +{code} + + +h2. Semantics of Exclusive + +h3. I want to be able to have an exclusive consumer, but when it dies I want another to be able to pick up the queue and then block others, can this be done? + +Yes, Declare you queue exclusive. this will prevent anyone else from connecting to the queue. If the consumer dies the next consumer can attach to the queue by redeclaring it using the exclusive flag. Make sure not to set auto delete. Any consumer trying to declare, while a consumer is attached to the queue will receive an exception. + +h3. When will the queue become free for a re-declare + +Once the session that held the consumer is closed. + + +h2. Performance + +h3. Does Qpid Perform (Latency/Throughput)? + +Yes, The Qpid C++ broker has been achieved great benchmark results in published papers by those that redistribute it. [Red Hat MRG|http://www.redhat.com/mrg] product build on Qpid has shown 760,000msg/sec ingress on an 8 way box or 6,000,000msg/sec OPRA messages. + +Latencies have been recored as low as 180-250us (.18ms-.3ms) for TCP round trip and 60-80us for RDMA round trip using the C++ broker. + +h3. How do I measure throughput? + +There is a great resource supplied in the C++ broker test directory called perftest. If allows you to create load on a broker for all the exchanges, multiple queues, multiple connection, coordinate multiple publishing and consuming processes, beachmark transactions and much much more such as acquire mode, txn size, message size. + +For all the options +{code} +./perftest --help +{code} + +h3. How do I measure latency? + +There is a great resource supplied in the C++ broker test directory called latencytest. It is a loopback test that produces messages by count or at a rate, time stamps them and then consumes them back and record the latency. It supports many of the Qpid options, including the ability to vary things like frame-size. + +Latencies to expect round trip: + * 1G TCP ~ .3ms -.5ms + * 10G TCP - .18ms - .22ms + * RDMA transport - 40us - 80us + +Don't forget to set tune the machine and set --tcp-nodelay on both the C++ broker & client. + +For all the options +{code} +./latencytest --help +{code} + +h3. How do I measure performance with Java clients? + +In Java we provide a utility called QpidBench. It allows you to test the performance of the native AMQP API in Java for 0-10 and the JMS API against both brokers. + +h3. Can I run my Java client with JAVA-RT? + +Yes, recently a thread abstraction layer has been added to the Java client allowing it to be used with both the SUN and IBM RT JVMs. This increases the determinism of latency when using the Java client. + +h3. Does Qpid support flow control? + +yes, AMQP 0-10 allows for flow control on the consumer and producer. + +h3. How do I configure producer side flow control + +from qpidd --help + +set the following in the config file on via cmd line options. +{code} + --max-session-rate MESSAGES/S (0) Sets the maximum message rate per session (0=unlimited) +{code} + +h2. Management + +h3. What Management does Qpid support + +The Java broker supports JMX and provides an Eclipse plug-in and command line tool to manage via JMX. The C++ broker has far more extensive management support via QMF which will be added to the Java broker in a future release. + +The C++ Broker supports a layered management protocol over AMQP called QMF. This allows for the management of resource either in the broker or connected to the broker via the AMQP fabric. This management includes statistics, control, eventing, and reporting/updating properties. + +h3. How do I manage a broker? + +A set of tools are provided to manage the C++ broker, they include + * qpid-tool - telnet type tool to access data, view schema, issue command an and QMF resource + * qpid-config - tool to configure queues, exchanges, etc. all the details on the AMQP model + * qpid-route - tool to configure broker federation + * qpid-events - utility that will print to cmd line or syslog event from a broker like, userconnected, user crested/deleted a queue. + * qpid-stats - utility that will print out queue statistics to the cmd line or syslog like rate and message depth. + +Then you can also access all thsi information via JMX or WS-DM (work in progress) using QMan. + +h3. What logging tracing and events does Qpid support? + +Qpid support the ability to output events from any the broker or any managed object via QMF, or to do a variety of logging from the broker & clients. for tracing options run qpidd --help. + +Multiple levels of of logging are supported in the C++ broker from debug, warning, error, info, etc -- all of which can be filtered. + +h3. Can I get to all the management data from a client? + +yes, All the management data is just AMQP messages on specially named queues. An API is provided for working with the management data called QMFC + +h3. What is QMF + +QMF is the layered Management protocol used to manage the C++ broker. For details on teh protocol see the Development pages. + +QMF allows you to manage any resource and provides the following infrsstructure: + * Properties + * Statistics + * Commands + * Events + * Schema for resources and versioning + * tools for creating agents and consuming QMF data. + +h3. What are QMF Agents, and what do they do for me? + +An Agent is any client (producer or consumer) that generates a QMF schema and registers itself to be management by QMF. + +A great use case of this is a consumer that is processing order from a queue can reference itself to that queue and for example provide a schema for the number or successful orders process and a method to suspend processing. Now it becomes possible to use qpid-tool to connect to the broker, see which order processors are on queue via the reference and the via the stats of the order processor client. It is also possible to issue a command to the client via qpid-tool to suspend processing. ACL in the broker can be applied to all these actions if desired. + +h3. What is QMFC and what does it do for mr? + +QMFC is the API used to consume QMF data, event and issue commands to QMF agents from an AMQP client. + +h3. What is QMan + +Qman is a tool that dynamically reads the QMF Schema information and creates JMX objects that consumed by any JMX console or application server to manage Qpid. QMan is also adding support for WS-DM management of QMF resources. + + +h2. Clustering, Federation and Disaster Recovery + +h3. Does Qpid provide Fault Tolerance for the cluster? + +The C++ broker has plug-ins for Active-Active clustering which keep all the nodes of the cluster in sync. This means that any action that is performed on one of the brokers on the cluster is performed on all of them at the same time. New nodes can be added to the cluster at any time, and removed at any time with no consequences, exept for the extra multi-cast load created for the sync on joining. + +h3. What does the cluster guarantee? + +Everything! All configuration, all messages and all actions are replicated in a cluster. This means that two consumers can be connected to different nodes in the cluster and they will behave EXACTLY the same as if they where on a single broker. + +h3. Do clients get notified members joining or leaving the cluster? + +yes, All clients are updated with the addresses of node add/removed as supported by the AMQP 0-10 specification. This means that the client can dynamically track the nodes in the cluster and reconnect as required. + +h3. Can I specify more than one host to connect initially to the cluster to avoid single point of failure? + +yes, the AMQP address is multi-honed and more than one IP address can be specified at the initial connection. The client will then iterate through the host until it makes a successful connection. This feature can also be used in none clustered brokers. + +h3. How does Clustering work? + +When C++ brokers are configured into a cluster, the nodes communicate with each other over a mulitcast protocol called AIS, an open Telco multicast protocol that provides all the quorum and group services. + +Every action that is performed on any node of the cluster is then sequenced via totem and then performed on each node of the cluster in sync. As the cluster backbone is multicast, a separate network can be used for cluster communication and there is little impact adding additional nodes to the cluster with-in reason. + +h3. What is Federation? + +Federation provides the ability to create networks of brokers that communicate with each other in all types of typologies. This allows a producer to publish messages to one broker and someone to consume the messages from another broke somewhere on the broker federated network. + +[For more details see|Using Broker Federation] + +h3. Disater recover features are in process, Q&A will be added once they are complete. + +h2. Heartbeats + +Heartbeat can be configured to allow clients to detect when a broker has failed and connect to another broker or cluster member.Heartbeats are sent by the broker at a client specified, per-connection frequency. If the client does not receive a heartbeat or any other traffic for two heartbeat intervals, the connection will be made to fail. + +h3. What would happen when there is a no heartbeat within a predefined interval? + +If there is no traffic for two heartbeat intervals, the client will fail the connection. The application will see the exact same response as when the connection is killed. + +h3. What happens if the broker is unable to send heartbeat? + +As above, if there is no other traffic the client will eventually kill the connection. + +h3. Does the client retry? + +You can control the heartbeat interval on the client through the heartbeat member of ConnectionSettings (it is measured in seconds). Some of the options on policies do vary for different clients. + +h3. Failover taking too long... + +First check to make sure a heartbeat has been specified in the connection properties for the connection. + +Then make sure that the interfaces on each broker are reachable from the host you run my clients, else it will take a long time for the socket to timeout until it gets to one that can be reached. + +Make sure the list of URL's on the cient are the ones you want tht client to try + +Make sure that the broker is only exporting URL's that the client can connect to, use the --cluster-url option on the broker to specify this. + +h2. Threading + +h3. Could someone provide a brief description of the worker thread duties in the current Qpid release? + +The broker uses IO threads for all the work it does. This means that when work is signalled via an event (socket, RDMA, timer) an IO thread is scheduled and it runs until it completes the work and then returns back to the IO thread pool. This allows the CPUs to be utilized efficiently. The general rule is that we allocate 1 thread per core +1. So on a 8 way machine you see worker-threads default to 9. On a 4 way it will be 5. Sometimes it if work changing the default allocation if: + +a.) you run on high core count machine >8 to a lower number +b.) if you taskset, then set to the cores allocated +1 + +h3. Why was the number X chosen as the default number of worker threads? + +Qpidd defaults to cores + 1 + +h3. What happens in parallel? + +Concurrency in the broker is at the session level. So yes. If you want more concurrency, create another session on the same connection. + +h3. How are worker threads allocated to individual client sessions if there are more clients than threads in the pool? + +They are not allocated to a specific client + +h2. Persistence + +h3. Does Qpid support persistence (durability)? + +Yes, there are third-party (non-Apache) modules for both C++ and Java. Historically, BDB has been used to provide persistence for both C++ and Java. However, this has created a licensing conflict with Apache, and thus the store modules are maintained off-site. + +The Java broker includes a fully Apache licensed persistent store that uses Derby DB. + +The terms _durable_ and _persistent_ are used interchangeably in this FAQ. + +h3. Where do I get the 3rd party persistence store modules? + +The 3rd party persistence store modules may be obtained through anonymous subversion at the following locations: + +C++: http://anonsvn.jboss.org/repos/rhmessaging/store/trunk/cpp +Java: http://anonsvn.jboss.org/repos/rhmessaging/store/trunk/java/bdbstore + +For further details see [3rd Party Libraries] + +h3. How do I build the persistence store module from subversion checkouts? + ++*C\+\+*+ +The README file contains detailed instructions, but here is a summary: + # Make sure that both the db4-devel and libaio-devel packages are installed prior to building. + # Make sure that qpid is built and you know the location of the qpid directory (ie the top-level directory containing the python and cpp sub-directories). + # In the store directory, run: +{code} +./bootstrap +./configure --with-qpid-checkout=/abs/path/to/qpid/dir +make +{code} +# When built, the store library *msgstore.so* will be located in the *lib/.libs* directory. + ++*Java*+ +TODO + +h3. How do I use the persistence store module? + ++*C\+\+*+ + # Start the broker making sure that the store module is loaded, ie +{code} +qpidd --load-module=/path/to/msgstore.so --data-dir=/path/to/store-files ... +{code} + # Make sure that queues that will handle persistent messages are set durable. +{note:title=Note: Existing non-persistent queues cannot be made persistent} +If a queue has been declared without persistence, doing so again with persistence enabled while the old queue still exists in the broker will be ignored. Make sure that when a queue is declared persistent, there is no non-persistent queue of the same name in existence. +{note} + # For each message sent to a durable queue, make sure that it is set durable. + ++*Java*+ +TODO + +h3. How do I configure the persistence store? + ++*C\+\+*+ + +The broker loads help information from each module. To see the help options for the store, load the store module and specify help: +{code} +qpidd --load-module /abs/path/to/store/lib/.libs/msgstore.so --help +{code} + +Note that a set of journal files will be created for each queue declared and marked persistent. Each persistent queue has its own private journal. These are stored in the data directory by default (ie it uses the broker's *\-\-data\-dir* setting) or can be overridden with the *\-\-store\-dir* option. Note that if the broker is started with the *\-\-no\-data\-dir* option, then no store default exists, and the *\-\-store\-dir* option MUST be specified. + +The store file details - or "store geometry" - can be set with command-line options. These include the size and number of files that make up the journal for each queue. The *\-\-num\-jfiles* options sets the number of files to use (between 4 and 64) and the *\-\-jfile\-size\-pgs* sets the size of the file in 64kiB blocks. + +The size of the pages in the write page cache is set with the *\-\-wcache\-page-size* option, and sets a size in KiB. (Legal values are powers of 2, ie: 1, 2, 4, 8, 16, 32, 64, 128). Typically small page sizes give improved latency (especially for small messages), but are bad for message throughput, while large page sizes improve throughput but may cause some messages to have higher latencies. + ++*Java*+ ++Derby Store+ +For details of configuring the Derby Store see [here|Derby Store Plugin] + ++3rd Party Stores+ + +For details of using the 3rd party persistent modules see [here|3rd Party Libraries] + + +h3. \[C++ store\] What is a RHM_IORES_ENQCAPTHRESH error? + +The journal ran out of space (ENQueue CAPacity THRESHold). The journal is a circular file buffer of fixed capacity set by the journal file size and number of files. When an attempt to write a record causes the journal to exceed an approx. 80% threshold, then the enqueue is rejected with this error code. Dequeues (a written record of a consumed message) may continue, however, as these free up space in the journal. Once space has been freed up, enqueues may continue as normal. + +This error may be caused by: +# The journal is too small for the size and number of messages being stored. The journal must be made large enough to hold all of the messages you expect to be on the queue at any one moment (a worst-case scenario). Make the journal capacity larger through the use of the *\-\-num\-jfiles* and *\-\-jfile\-size\-pgs* parameters. +{info:title=Rule of thumb for sizing the journal} +Make the journal twice the size of all the messages you need to store at any one moment in time. +{info} +# Messages are not being dequeued (consumed) as expected. Since the store is a circular file buffer, if one un-dequeued (not consumed) message remains, it can eventually "block" the storage of new messages as the buffer gets overwritten. + +h3. \[C++ store\] What is the TPL? What are the \-\-tpl\-\* options for? +The TPL stands for *Transaction Prepared List*. The store creates a single instance of a store for storing transaction boundaries called the Transaction Prepared List. Because the TPL is frequently flushed and has very different usage patterns to a normal store, it has been provided with its own set of configuration parameters: +* *\-\-tpl\-num\-jfiles:* The number of files in the TPL journal +* *\-\-tpl\-jfile\-size-pgs:* The file size in 64kiB blocks of the TPL journal. +* *\-\-tpl\-wcache\-page-size:* The size of the write cache in the TPL in KiB, which is typically set a lot smaller than the average message store. + +h1. How To + +h2. C++ + +h3. How to use RDMA with Qpid + +The RDMA plugin uses native OFED1.3 and puts AMQP directly onto the DMA. When using the RDMA plug-in for Qpid note the following +* IP over IB or Fibre needs to be setup for the initial negociation +* You need to make sure you have enough memory to pin for DMA use ulimit \-l something large +* you might need to edit /etc/security/limits.conf first then log in again + +Once you have it up and running, use latencytest to make sure it is working. You should see latencies between 50 and 80us round trip. + +h3. Message TTL, auto expire + +I need to be able to set time for a message that I send to be removed from the queue if it is not read by my subscriber. For example: I enqueue a message and I want it to be automatically dequeued after a certain amount of time has passed.Is there a feature like this in qpid? + +yes, the TTL can be set in the message headers and the messages get dequeued if TTL expires + +E.g. from c++: +{code} +Message m("Hello World!"); + m.getDeliveryProperties().setTtl(500); +{code} +Sets a 500 millisecond timeout. + +h3. How to install the qpid-tools for c++ broker? + +I see +{code} +[commands]$ ./qpid-queue-stats + Traceback (most recent call last): + File "./qpid-queue-stats", line 29, in + from qmf.console import Session, Console + ImportError: No module named qmf.console +{code} + +This problem occurs because the PYTHONPATH environment variable does not include the location of the qpid python files. If you are running from the SVN checkout, add <path>/qpid/python to PYTHONPATH (where <path> is the location of your SVN tree). If you are installing from source, make sure you configure with the same prefix where Python is installed. This is most likely: + +{code} +# configure --prefix=/usr +# make +# make install +{code} +If you are running from vendor RPMs, this should work automatically. + + +h2. Java +{children:page=Qpid Java How To} diff --git a/qpid/doc/book/src/LVQ.xml b/qpid/doc/book/src/LVQ.xml new file mode 100644 index 0000000000..5b7b4a2c9a --- /dev/null +++ b/qpid/doc/book/src/LVQ.xml @@ -0,0 +1,341 @@ +<?xml version="1.0" encoding="utf-8"?> +<chapter xmlns:html="http://www.w3.org/1999/xhtml"><title> + Apache Qpid : LVQ + </title> + + <section role="h2" id="LVQ-UnderstandingLVQ"><title> + Understanding LVQ + </title> + <para> + Last Value Queues are useful youUser Documentation are only + interested in the latest value entered into a queue. LVQ + semantics are typically used for things like stock symbol updates + when all you care about is the latest value for example. + </para><para> + Qpid C++ M4 or later supports two types of LVQ semantics: + </para><itemizedlist> + <listitem><para>LVQ + </para></listitem> + <listitem><para>LVQ_NO_BROWSE + </para></listitem> + </itemizedlist> + <!--h2--></section> + + + <section role="h2" id="LVQ-LVQsemantics-3A"><title> + LVQ semantics: + </title> + <para> + LVQ uses a header for a key, if the key matches it replaces the + message in-place in the queue except + a.) if the message with the matching key has been acquired + b.) if the message with the matching key has been browsed + In these two cases the message is placed into the queue in FIFO, + if another message with the same key is received it will the + 'un-accessed' message with the same key will be replaced + </para><para> + These two exceptions protect the consumer from missing the last + update where a consumer or browser accesses a message and an + update comes with the same key. + </para><para> + An example + </para> + <programlisting> +[localhost tests]$ ./lvqtest --mode create_lvq +[localhost tests]$ ./lvqtest --mode write +Sending Data: key1=key1.0x7fffdf3f3180 +Sending Data: key2=key2.0x7fffdf3f3180 +Sending Data: key3=key3.0x7fffdf3f3180 +Sending Data: key1=key1.0x7fffdf3f3180 +Sending Data: last=last +[localhost tests]$ ./lvqtest --mode browse +Receiving Data:key1.0x7fffdf3f3180 +Receiving Data:key2.0x7fffdf3f3180 +Receiving Data:key3.0x7fffdf3f3180 +Receiving Data:last +[localhost tests]$ ./lvqtest --mode write +Sending Data: key1=key1.0x7fffe4c7fa10 +Sending Data: key2=key2.0x7fffe4c7fa10 +Sending Data: key3=key3.0x7fffe4c7fa10 +Sending Data: key1=key1.0x7fffe4c7fa10 +Sending Data: last=last +[localhost tests]$ ./lvqtest --mode browse +Receiving Data:key1.0x7fffe4c7fa10 +Receiving Data:key2.0x7fffe4c7fa10 +Receiving Data:key3.0x7fffe4c7fa10 +Receiving Data:last +[localhost tests]$ ./lvqtest --mode consume +Receiving Data:key1.0x7fffdf3f3180 +Receiving Data:key2.0x7fffdf3f3180 +Receiving Data:key3.0x7fffdf3f3180 +Receiving Data:last +Receiving Data:key1.0x7fffe4c7fa10 +Receiving Data:key2.0x7fffe4c7fa10 +Receiving Data:key3.0x7fffe4c7fa10 +Receiving Data:last +</programlisting> +<!--h2--></section> + <section role="h2" id="LVQ-LVQNOBROWSEsemantics-3A"><title> + LVQ_NO_BROWSE + semantics: + </title> + <para> + LVQ uses a header for a key, if the key matches it replaces the + message in-place in the queue except + a.) if the message with the matching key has been acquired + In these two cases the message is placed into the queue in FIFO, + if another message with the same key is received it will the + 'un-accessed' message with the same key will be replaced + </para><para> + Note, in this case browsed messaged are not invalidated, so + updates can be missed. + </para><para> + An example + </para> + <programlisting> +[localhost tests]$ ./lvqtest --mode create_lvq_no_browse +[localhost tests]$ ./lvqtest --mode write +Sending Data: key1=key1.0x7fffce5fb390 +Sending Data: key2=key2.0x7fffce5fb390 +Sending Data: key3=key3.0x7fffce5fb390 +Sending Data: key1=key1.0x7fffce5fb390 +Sending Data: last=last +[localhost tests]$ ./lvqtest --mode write +Sending Data: key1=key1.0x7fff346ae440 +Sending Data: key2=key2.0x7fff346ae440 +Sending Data: key3=key3.0x7fff346ae440 +Sending Data: key1=key1.0x7fff346ae440 +Sending Data: last=last +[localhost tests]$ ./lvqtest --mode browse +Receiving Data:key1.0x7fff346ae440 +Receiving Data:key2.0x7fff346ae440 +Receiving Data:key3.0x7fff346ae440 +Receiving Data:last +[localhost tests]$ ./lvqtest --mode browse +Receiving Data:key1.0x7fff346ae440 +Receiving Data:key2.0x7fff346ae440 +Receiving Data:key3.0x7fff346ae440 +Receiving Data:last +[localhost tests]$ ./lvqtest --mode write +Sending Data: key1=key1.0x7fff606583e0 +Sending Data: key2=key2.0x7fff606583e0 +Sending Data: key3=key3.0x7fff606583e0 +Sending Data: key1=key1.0x7fff606583e0 +Sending Data: last=last +[localhost tests]$ ./lvqtest --mode consume +Receiving Data:key1.0x7fff606583e0 +Receiving Data:key2.0x7fff606583e0 +Receiving Data:key3.0x7fff606583e0 +Receiving Data:last +[localhost tests]$ + +</programlisting> + <!--h2--></section> + <section role="h2" id="LVQ-Examplesource"><title> + Example source + </title> + + <programlisting> + +/* + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + * + */ + + +#include <qpid/client/AsyncSession.h> +#include <qpid/client/Connection.h> +#include <qpid/client/SubscriptionManager.h> +#include <qpid/client/Session.h> +#include <qpid/client/Message.h> +#include <qpid/client/MessageListener.h> +#include <qpid/client/QueueOptions.h> + +#include <iostream> + +using namespace qpid::client; +using namespace qpid::framing; +using namespace qpid::sys; +using namespace qpid; +using namespace std; + + +enum Mode { CREATE_LVQ, CREATE_LVQ_NO_BROWSE, WRITE, BROWSE, CONSUME}; +const char* modeNames[] = { "create_lvq","create_lvq_no_browse","write","browse","consume" }; + +// istream/ostream ops so Options can read/display Mode. +istream& operator>>(istream& in, Mode& mode) { + string s; + in >> s; + int i = find(modeNames, modeNames+5, s) - modeNames; + if (i >= 5) throw Exception("Invalid mode: "+s); + mode = Mode(i); + return in; +} + +ostream& operator<<(ostream& out, Mode mode) { + return out << modeNames[mode]; +} + +struct Args : public qpid::Options, + public qpid::client::ConnectionSettings +{ + bool help; + Mode mode; + + Args() : qpid::Options("Simple latency test optins"), help(false), mode(BROWSE) + { + using namespace qpid; + addOptions() + ("help", optValue(help), "Print this usage statement") + ("broker,b", optValue(host, "HOST"), "Broker host to connect to") + ("port,p", optValue(port, "PORT"), "Broker port to connect to") + ("username", optValue(username, "USER"), "user name for broker log in.") + ("password", optValue(password, "PASSWORD"), "password for broker log in.") + ("mechanism", optValue(mechanism, "MECH"), "SASL mechanism to use when authenticating.") + ("tcp-nodelay", optValue(tcpNoDelay), "Turn on tcp-nodelay") + ("mode", optValue(mode, "'see below'"), "Action mode." + "\ncreate_lvq: create a new queue of type lvq.\n" + "\ncreate_lvq_no_browse: create a new queue of type lvq with no lvq on browse.\n" + "\nwrite: write a bunch of data & keys.\n" + "\nbrowse: browse the queue.\n" + "\nconsume: consume from the queue.\n"); + } +}; + +class Listener : public MessageListener +{ + private: + Session session; + SubscriptionManager subscriptions; + std::string queue; + Message request; + QueueOptions args; + public: + Listener(Session& session); + void setup(bool browse); + void send(std::string kv); + void received(Message& message); + void browse(); + void consume(); +}; + +Listener::Listener(Session& s) : + session(s), subscriptions(s), + queue("LVQtester") +{} + +void Listener::setup(bool browse) +{ + // set queue mode + args.setOrdering(browse?LVQ_NO_BROWSE:LVQ); + + session.queueDeclare(arg::queue=queue, arg::exclusive=false, arg::autoDelete=false, arg::arguments=args); + +} + +void Listener::browse() +{ + subscriptions.subscribe(*this, queue, SubscriptionSettings(FlowControl::unlimited(), ACCEPT_MODE_NONE, ACQUIRE_MODE_NOT_ACQUIRED)); + subscriptions.run(); +} + +void Listener::consume() +{ + subscriptions.subscribe(*this, queue, SubscriptionSettings(FlowControl::unlimited(), ACCEPT_MODE_NONE, ACQUIRE_MODE_PRE_ACQUIRED)); + subscriptions.run(); +} + +void Listener::send(std::string kv) +{ + request.getDeliveryProperties().setRoutingKey(queue); + + std::string key; + args.getLVQKey(key); + request.getHeaders().setString(key, kv); + + std::ostringstream data; + data << kv; + if (kv != "last") data << "." << hex << this; + request.setData(data.str()); + + cout << "Sending Data: " << kv << "=" << data.str() << std::endl; + async(session).messageTransfer(arg::content=request); + +} + +void Listener::received(Message& response) +{ + + cout << "Receiving Data:" << response.getData() << std::endl; +/* if (response.getData() == "last"){ + subscriptions.cancel(queue); + } +*/ +} + +int main(int argc, char** argv) +{ + Args opts; + opts.parse(argc, argv); + + if (opts.help) { + std::cout << opts << std::endl; + return 0; + } + + Connection connection; + try { + connection.open(opts); + Session session = connection.newSession(); + Listener listener(session); + + switch (opts.mode) + { + case CONSUME: + listener.consume(); + break; + case BROWSE: + listener.browse(); + break; + case CREATE_LVQ: + listener.setup(false); + break; + case CREATE_LVQ_NO_BROWSE: + listener.setup(true); + break; + case WRITE: + listener.send("key1"); + listener.send("key2"); + listener.send("key3"); + listener.send("key1"); + listener.send("last"); + break; + } + connection.close(); + return 0; + } catch(const std::exception& error) { + std::cout << error.what() << std::endl; + } + return 1; +} + +</programlisting> +<!--h2--></section> +</chapter> diff --git a/qpid/doc/book/src/Management Design notes.xml b/qpid/doc/book/src/Management Design notes.xml new file mode 100644 index 0000000000..a2eb28e4cb --- /dev/null +++ b/qpid/doc/book/src/Management Design notes.xml @@ -0,0 +1,2115 @@ +<?xml version="1.0" encoding="utf-8"?> +<chapter xmlns:html="http://www.w3.org/1999/xhtml"><title> + Apache Qpid : Management Design notes + </title><section role="h2" id="ManagementDesignnotes-StatusofThisDocument"><title> + Status + of This Document + </title> + + <para> + This document does not track any current development activity. It + is the specification of the management framework implemented in + the M3 release of the C++ broker and will be left here for user + and developer reference. + </para><para> + Development continues on the Qpid Management Framework (QMF) for + M4. If you are using M3, this is the document you need. If you + are using the SVN trunk, please refer to <xref linkend="qpid_Qpid-20Management-20Framework"/> for + up-to-date information. + </para> + <!--h2--></section> + + <section role="h2" id="ManagementDesignnotes-Introduction"><title> + Introduction + </title> + + <para> + This document describes the management features that are used in + the QPID C++ broker as of the M3 milestone. These features do not + appear in earlier milestones nor are they implemented in the Java + broker. + </para><para> + This specification is <emphasis>not</emphasis> a standard and is not endorsed + by the AMQP working group. When such a standard is adopted, the + QPID implementation will be brought into compliance with that + standard. + </para> + <!--h2--></section> + + <section role="h2" id="ManagementDesignnotes-Links"><title> + Links + </title> + + <itemizedlist> + <listitem><para>The schema is checked into <xref linkend="qpid_management-schema.xml"/>. + </para></listitem> + </itemizedlist><section role="h3" id="ManagementDesignnotes-DesignnoteforgettinginfoinandoutviaJMX"><title> + Design + note for getting info in and out via JMX + </title> + + <para> + <xref linkend="qpid_JMX-20Gateway"/> + </para> + <!--h3--></section> + + <!--h2--></section> + + <section role="h2" id="ManagementDesignnotes-ManagementRequirements"><title> + Management + Requirements + </title> + + <itemizedlist> + <listitem><para>Must operate from a formally defined management schema. + </para></listitem> + <listitem><para>Must natively use the AMQP protocol and its type system. + </para></listitem> + <listitem><para>Must support the following operations + <itemizedlist> + <listitem><para>SET operation on configurable (persistent) aspects of + objects + </para></listitem> + <listitem><para>GET operation on all aspects of objects + </para></listitem> + <listitem><para>METHOD invocation on schema-defined object-specific + methods + </para></listitem> + <listitem><para>Distribution of unsolicited periodic updates of + instrumentation data + <itemizedlist> + <listitem><para>Data updates shall carry an accurate sample timestamp + for rate calculation + </para></listitem> + <listitem><para>Updates shall carry object create/delete timestamps. + </para></listitem> + <listitem><para>Transient objects shall be fully accounted for via + updates. Note that short-lived transient objects may come + and go within a single update interval. All of the + information pertaining to such an object must be captured + and transmitted. + </para></listitem> + </itemizedlist> + </para></listitem> + <listitem><para>Distribution of unsolicited event and/or alert + indications (schema defined) + </para></listitem> + </itemizedlist> + </para></listitem> + <listitem><para>Role-based access control at object, operation, and method + granularity + </para></listitem> + <listitem><para>End-to-end encryption and signing of management content + </para></listitem> + <listitem><para>Schema must be self-describing so the management client need + not have prior knowledge of the management model of the system + under management. + </para></listitem> + <listitem><para>Must be extensible to support the management of objects + beyond the QPID component set. This allows AMQP to be used as a + general-purpose management protocol. + </para></listitem> + </itemizedlist> + <!--h2--></section> + + <section role="h2" id="ManagementDesignnotes-DefinitionofTerms"><title> + Definition + of Terms + </title> + + <table><title/><tgroup cols="2"> + <tbody> + <row> + <entry> + class + </entry> + <entry> + A type definition for a manageable object. + </entry> + </row> + <row> + <entry> + package + </entry> + <entry> + A grouping of class definitions that are related to a + single software component. The package concept is used to + extend the management schema beyond just the QPID software + components. + </entry> + </row> + <row> + <entry> + object + </entry> + <entry> + Also "manageable object". An instantiation of a class. An + object represents a physical or logical component in the + core function of the system under management. + </entry> + </row> + <row> + <entry> + property + </entry> + <entry> + A typed member of a class which represents a configurable + attribute of the class. In general, properties don't change + frequently or may not change at all. + </entry> + </row> + <row> + <entry> + statistic + </entry> + <entry> + A typed member of a class which represents an + instrumentation attribute of the class. Statistics are + always read-only in nature and tend to change rapidly. + </entry> + </row> + <row> + <entry> + method + </entry> + <entry> + A member of a class which represents a callable procedure + on an object of the class. Methods may have an arbitrary + set of typed arguments and may supply a return code. + Methods typically have side effects on the associated + object. + </entry> + </row> + <row> + <entry> + event + </entry> + <entry> + A member of a class which represents the occurence of an + event of interest within the system under management. + </entry> + </row> + <row> + <entry> + management broker + </entry> + <entry> + A software component built into the messaging broker that + handles management traffic and distributes management data. + </entry> + </row> + <row> + <entry> + management agent + </entry> + <entry> + A software component that is separate from the messaging + broker, connected to the management broker via an AMQP + connection, which allows any software component to be + managed remotely by QPID. + </entry> + </row> + </tbody> + </tgroup></table> + <!--h2--></section> + + + <section role="h2" id="ManagementDesignnotes-OperationalScenarios-3ABasicvs.Extended"><title> + Operational + Scenarios: Basic vs. Extended + </title> + + <para> + The extensibility requirement introduces complexity to the + management protocol that is unnecessary and undesirable for the + user/developer that wishes only to manage QPID message brokers. + For this reason, the protocol is partitioned into two parts: The + <emphasis>basic protocol</emphasis>, which contains only the capability to + manage a single broker; and the <emphasis>extended protocol</emphasis>, which + provides the hooks for managing an extended set of components. A + management console can be implemented using only the basic + protocol if the extended capabilities are not needed. + </para> +<!--h2--></section> + + + <section role="h2" id="ManagementDesignnotes-ArchitecturalFramework"><title> + Architectural + Framework + </title> + + <para> + <xref linkend="qpid_qmf_architecture"/> + </para> + <!--h2--></section> + + <section role="h2" id="ManagementDesignnotes-TheManagementExchange"><title> + The + Management Exchange + </title> + + <para> + The management exchange (called "qpid.management" currently) is a + special type of exchange used for remote management access to the + Qpid broker. The management exchange is an extension of the + standard "Topic" exchange. It behaves like a topic exchange with + the following exceptions: + </para><orderedlist> + <listitem><para>When a queue is successfully bound to the exchange, a method + is invoked on the broker's management agent to notify it of the + presence of a new remote managment client. + </para></listitem> + <listitem><para>When messages arrive at the exchange for routing, the + exchange examines the message's routing key and if the key + represents a management command or method, it routes it directly + to the management agent rather than routing it to queues using + the topic algorithm. + The management exchange is used by the management agent to + distribute unsolicited management data. Such data is classified + by the routing key allowing management clients to register for + only the data they need. + </para></listitem> + </orderedlist><section role="h3" id="ManagementDesignnotes-RoutingKeyStructure"><title> + Routing + Key Structure + </title> + + <para> + As noted above, the structure of the binding and routing keys + used on the management exchange is important to the function of + the management architecture. The routing key of a management + message determines: + </para><orderedlist> + <listitem><para>The type of message (i.e. operation request or unsolicited + update). + </para></listitem> + <listitem><para>The class of the object that the message pertains to. + </para></listitem> + <listitem><para>The specific operation or update type. + </para></listitem> + <listitem><para>The namespace in which the class belongs. This allows for + plug-in expansion of the management schema for manageable objects + that are outside of the broker itself. + </para></listitem> + </orderedlist><para> + Placing this information in the routing key provides the ability + to enforce access control at class, operation, and method + granularity. It also separates the command structure from the + content of the management message (i.e. element values) allowing + the content to be encrypted and signed end-to-end while still + allowing access control at the message-transport level. This + means that special access control code need not be written for + the management agent. + There are two general types of routing/binding key: + </para><itemizedlist> + <listitem><para> + <emphasis>Command</emphasis> messages use the key: + agent.<bank#> or broker + </para></listitem> + <listitem><para> + <emphasis>Unsolicited</emphasis> keys have the structure: + mgmt.<agent>.<type>.<package>.<class>.<severity> + where + <itemizedlist> + <listitem><para> + <agent> is the uuid of the originating + management agent, + </para></listitem> + <listitem><para> + <type> is one of "schema", "prop", "stat", + or "event", + </para></listitem> + <listitem><para> + <package> is the namespace in which the + <class> name is valid, and + </para></listitem> + <listitem><para> + <class> is the name of the class as defined + in the schema. + </para></listitem> + <listitem><para> + <severity> is relevant for events only. It + is one of "critical", "error", "warning", or "info". + </para></listitem> + </itemizedlist> + </para></listitem> + </itemizedlist><para> + In both cases, the content of the message (i.e. method arguments, + element values, etc.) is carried in the body segment of the + message. + </para><para> + The <package> namespace allows this management + framework to be extended with the addition of other software + packages. + </para> + <!--h3--></section> + <!--h2--></section> + + <section role="h2" id="ManagementDesignnotes-TheProtocol"><title> + The Protocol + </title> + + <section role="h3" id="ManagementDesignnotes-ProtocolHeader"><title> + Protocol + Header + </title> + + <para> + The body segments of management messages are composed of + sequences of binary-encoded data fields, in a manner consistent + with the 0-10 version of the AMQP specification. + </para><para> + All management messages begin with a message header: + </para> + <programlisting> + octet 0 1 2 3 4 5 6 7 + +---------+---------+---------+---------+---------+---------+---------+---------+ + | 'A' | 'M' | '1' | op-code | sequence | + +---------+---------+---------+---------+---------+---------+---------+---------+ +</programlisting> + <para> + The first three octets contain the protocol <emphasis>magic number</emphasis> + "AM1" which is used to identify the type and version of the + message. + </para><para> + The <emphasis>opcode</emphasis> field identifies the operation represented by + the message + </para> + <!--h3--></section> + + <section role="h3" id="ManagementDesignnotes-ProtocolExchangePatterns"><title> + Protocol + Exchange Patterns + </title> + + <para> + The following patterns are followed in the design of the + protocol: + </para><itemizedlist> + <listitem><para>Request-Response + </para></listitem> + <listitem><para>Query-Indication + </para></listitem> + <listitem><para>Unsolicited Indication + </para></listitem> + </itemizedlist><section role="h4" id="ManagementDesignnotes-TheRequestResponsePattern"><title> + The + Request-Response Pattern + </title> + + <para> + In the request-response pattern, a requestor sends a + <emphasis>request</emphasis> message to one of its peers. The peer then does + one of two things: If the request can be successfully processed, + a single <emphasis>response</emphasis> message is sent back to the requestor. + This response contains the requested results and serves as the + positive acknowledgement that the request was successfully + completed. + </para><para> + If the request cannot be successfully completed, the peer sends a + <emphasis>command complete</emphasis> message back to the requestor with an + error code and error text describing what went wrong. + </para><para> + The sequence number in the <emphasis>response</emphasis> or <emphasis>command + complete</emphasis> message is the same as the sequence number in the + <emphasis>request</emphasis>. + </para> + <programlisting> + Requestor Peer + | | + | --- Request (seq) ------------------------------------------> | + | | + | <----------------------------------------- Response (seq) --- | + | | +</programlisting> + + <programlisting> + Requestor Peer + | | + | --- Request (seq) ------------------------------------------> | + | | + | <-------------------------- Command Complete (seq, error) --- | + | | +</programlisting> +<!--h4--></section> + + <section role="h4" id="ManagementDesignnotes-TheQueryIndicationPattern"><title> + The + Query-Indication Pattern + </title> + + <para> + The query-indication pattern is used when there may be zero or + more answers to a question. In this case, the requestor sends a + <emphasis>query</emphasis> message to its peer. The peer processes the query, + sending as many <emphasis>indication</emphasis> messages as needed back to the + requestor (zero or more). Once the last <emphasis>indication</emphasis> has + been sent, the peer then sends a <emphasis>command complete</emphasis> message + with a success code indicating that the query is complete. + </para><para> + If there is an error in the <emphasis>query</emphasis>, the peer may reply with + a <emphasis>command complete</emphasis> message containg an error code. In this + case, no <emphasis>indication</emphasis> messages may be sent. + </para><para> + All <emphasis>indication</emphasis> and <emphasis>command complete</emphasis> messages shall + have the same sequence number that appeared in the <emphasis>query</emphasis> + message. + </para> + <programlisting> + Requestor Peer + | | + | --- Query (seq) --------------------------------------------> | + | | + | <--------------------------------------- Indication (seq) --- | + | <--------------------------------------- Indication (seq) --- | + | <--------------------------------------- Indication (seq) --- | + | <--------------------------------------- Indication (seq) --- | + | <--------------------------------------- Indication (seq) --- | + | | + | <------------------------ Command Complete (seq, success) --- | + | | +</programlisting> + + <programlisting> + Requestor Peer + | | + | --- Query (seq) --------------------------------------------> | + | | + | <-------------------------- Command Complete (seq, error) --- | + | | +</programlisting> +<!--h4--></section> + + <section role="h4" id="ManagementDesignnotes-TheUnsolicitedIndicationPattern"><title> + The + Unsolicited-Indication Pattern + </title> + + <para> + The unsolicited-indication pattern is used when one peer needs to + send unsolicited information to another peer, or to broadcast + information to multiple peers via a topic exchange. In this case, + indication messages are sent with the sequence number field set + to zero. + </para> + <programlisting> + Peer Peer + | | + | <----------------------------------- Indication (seq = 0) --- | + | <----------------------------------- Indication (seq = 0) --- | + | <----------------------------------- Indication (seq = 0) --- | + | <----------------------------------- Indication (seq = 0) --- | + | | +</programlisting> +<!--h4--></section> +<!--h3--></section> + + <section role="h3" id="ManagementDesignnotes-ObjectIdentifiers"><title> + Object + Identifiers + </title> + + <para> + Manageable objects are tagged with a unique 64-bit object + identifier. The object identifier space is owned and managed by + the management broker. Objects managed by a single management + broker shall have unique object identifiers. Objects managed by + separate management brokers may have the same object identifier. + </para><para> + If a management console is designed to manage multiple management + brokers, it must use the broker identifier as well as the object + identifier to ensure global uniqueness. + </para> + <programlisting> + 62 48 47 24 23 0 + +-+-------------+-----------------------+-----------------------+ + |0| sequence | bank | object | + +-+-------------+-----------------------+-----------------------+ + + bit 63 - reserved, must be zero + bits 63 .. 48 - broker boot sequence (32K) + bits 47 .. 24 - bank (16M) + bits 23 .. 0 - object (16M) +</programlisting> + <itemizedlist> + <listitem><para>For persistent IDs, boot-sequence is zero + </para></listitem> + <listitem><para>For non-persistent IDs, boot sequence is a constant number + which increments each time the management broker is restarted. + </para></listitem> + <listitem><para>Bank number: + <itemizedlist> + <listitem><para>0 - reserved + </para></listitem> + <listitem><para>1 - broker-persistent objects + </para></listitem> + <listitem><para>2..4 - store-persistent objects + </para></listitem> + <listitem><para>> 4 - transient objects + </para></listitem> + </itemizedlist> + </para></listitem> + </itemizedlist> +<!--h3--></section> + + + <section role="h3" id="ManagementDesignnotes-EstablishingCommunicationBetweenClientandAgent"><title> + Establishing Communication Between Client and Agent + </title> + + <para> + Communication is established between the management client and + management agent using normal AMQP procedures. The client creates + a connection to the broker and then establishes a session with + its corresponding channel. + </para><para> + Two private queues are then declared (only one if method + invocation is not needed). A management queue is declared and + bound to the qpid.management exchange. If the binding key is + "mgmt.#", all management-related messages sent to the exchange + will be received by this client. A more specific binding key will + result in a more restricted set of messages being received (see + the section on Routing Key Structure below). + </para><para> + If methods are going to be invoked on managed objects, a second + private queue must be declared so the client can receive method + replies. This queue is bound to the amq.direct exchange using a + routing key equal to the name of the queue. + </para><para> + When a client successfully binds to the qpid.management exchange, + the management agent schedules a schema broadcast to be sent to + the exchange. The agent will publish, via the exchange, a + description of the schema for all manageable objects in its + control. + </para> + <programlisting> + Client Broker + | | + | --- AMQP Connection and Session Setup ----------------------> | + | | + | --- Queue.declare (private data queue) ---------------------> | + | --- Bind queue to exchange 'qpid.management' key 'mgmt.#' --> | + | | + | --- Queue.declare (private method-reply queue) -------------> | + | --- Bind queue to exchange 'amq.direct' --------------------> | + | | + | --- Broker Request -----------------------------------------> | + | <---------------------------------------- Broker Response --- | + | | + | | + | | + | <------- Management schema via exchange 'qpid.management' --- | + | | +</programlisting> +<!--h3--></section> + + <section role="h3" id="ManagementDesignnotes-BroadcastofConfigurationandInstrumentationUpdates"><title> + Broadcast of Configuration and Instrumentation Updates + </title> + + <para> + The management agent will periodically publish updates to the + configuration and instrumentation of management objects under its + control. Under normal circumstances, these updates are published + only if they have changed since the last time they were + published. Configuration updates are only published if + configuration has changed and instrumentation updates are only + published if instrumentation has changed. The exception to this + rule is that after a management client binds to the + qpid.management exchange, all configuration and instrumentation + records are published as though they had changed whether or not + they actually did. + </para> + <programlisting> + Client Broker + | | + | <------------------ Object properties via 'mgmt.*.prop.#' --- | | + | <------------------ Object statistics via 'mgmt.*.stat.#' --- | | + | | | + | | | Publish Interval + | | | + | | | + | | V + | <------------------ Object properties via 'mgmt.*.prop.#' --- | + | <------------------ Object statistics via 'mgmt.*.stat.#' --- | + | | +</programlisting> +<!--h3--></section> + + <section role="h3" id="ManagementDesignnotes-InvokingaMethodonaManagedObject"><title> + Invoking + a Method on a Managed Object + </title> + + <para> + When the management client wishes to invoke a method on a managed + object, it sends a method request message to the qpid.management + exchange. The routing key contains the object class and method + name (refer to Routing Key Structure below). The method request + must have a header entry (reply-to) that contains the name of the + method-reply queue so that the method response can be properly + routed back to the requestor. + </para><para> + The method request contains a sequence number that is copied to + the method reply. This number is opaque to the management agent + and may be used by the management client to correlate the reply + to the request. The asynchronous nature of requests and replies + allows any number of methods to be in-flight at a time. Note that + there is no guarantee that methods will be replied to in the + order in which they were requested. + </para> + <programlisting> + Client Broker + | | + | --- Method Request (to exchange 'qpid.management') ---------> | + | | + | | + | <--------------- Method Reply (via exchange 'amq.direct') --- | + | | +</programlisting> +<!--h3--></section> + + + <section role="h3" id="ManagementDesignnotes-MessagesfortheBasicScenario"><title> + Messages + for the Basic Scenario + </title> + + <para> + The principals in a management exchange are the <emphasis>management + client</emphasis> and the <emphasis>management agent</emphasis>. The management + agent is integrated into the QPID broker and the management + client is a remote entity. A management agent may be managed by + zero or more management clients at any given time. Additionally, + a management client may manage multiple management agents at the + same time. + </para><para> + For authentication and access control, management relies on the + mechanisms supplied by the AMQP protocol. + </para><section role="h4" id="ManagementDesignnotes-BasicOpcodes"><title> + Basic Opcodes + </title> + + <table><title/><tgroup cols="3"> + <tbody> + <row> + <entry> + <emphasis>opcode</emphasis> + </entry> + <entry> + <emphasis>message</emphasis> + </entry> + <entry> + <emphasis>description</emphasis> + </entry> + </row> + <row> + <entry> + 'B' + </entry> + <entry> + Broker Request + </entry> + <entry> + This message contains a broker request, sent from the + management console to the broker to initiate a management + session. + </entry> + </row> + <row> + <entry> + 'b' + </entry> + <entry> + Broker Response + </entry> + <entry> + This message contains a broker response, sent from the + broker in response to a broker request message. + </entry> + </row> + <row> + <entry> + 'z' + </entry> + <entry> + Command Completion + </entry> + <entry> + This message is sent to indicate the completion of a + request. + </entry> + </row> + <row> + <entry> + 'Q' + </entry> + <entry> + Class Query + </entry> + <entry> + Class query messages are used by a management console to + request a list of schema classes that are known by the + management broker. + </entry> + </row> + <row> + <entry> + 'q' + </entry> + <entry> + Class Indication + </entry> + <entry> + Sent by the management broker, a class indication notifies + the peer of the existence of a schema class. + </entry> + </row> + <row> + <entry> + 'S' + </entry> + <entry> + Schema Request + </entry> + <entry> + Schema request messages are used to request the full schema + details for a class. + </entry> + </row> + <row> + <entry> + 's' + </entry> + <entry> + Schema Response + </entry> + <entry> + Schema response message contain a full description of the + schema for a class. + </entry> + </row> + <row> + <entry> + 'h' + </entry> + <entry> + Heartbeat Indication + </entry> + <entry> + This message is published once per publish-interval. It can + be used by a client to positively determine which objects + did not change during the interval (since updates are not + published for objects with no changes). + </entry> + </row> + <row> + <entry> + 'c', 'i', 'g' + </entry> + <entry> + Content Indication + </entry> + <entry> + This message contains a content record. Content records + contain the values of all properties or statistics in an + object. Such records are broadcast on a periodic interval + if 1) a change has been made in the value of one of the + elements, or 2) if a new management client has bound a + queue to the management exchange. + </entry> + </row> + <row> + <entry> + 'G' + </entry> + <entry> + Get Query + </entry> + <entry> + Sent by a management console, a get query requests that the + management broker provide content indications for all + objects that match the query criteria. + </entry> + </row> + <row> + <entry> + 'M' + </entry> + <entry> + Method Request + </entry> + <entry> + This message contains a method request. + </entry> + </row> + <row> + <entry> + 'm' + </entry> + <entry> + Method Response + </entry> + <entry> + This message contains a method result. + </entry> + </row> + </tbody> + </tgroup></table> +<!--h4--></section> + + <section role="h4" id="ManagementDesignnotes-BrokerRequestMessage"><title> + Broker + Request Message + </title> + + <para> + When a management client first establishes contact with the + broker, it sends a Hello message to initiate the exchange. + </para> + <programlisting> + +-----+-----+-----+-----+-----------------------+ + | 'A' | 'M' | '1' | 'B' | 0 | + +-----+-----+-----+-----+-----------------------+ +</programlisting> + <para> + The Broker Request message has no payload. + </para> +<!--h4--></section> + + <section role="h4" id="ManagementDesignnotes-BrokerResponseMessage"><title> + Broker + Response Message + </title> + + <para> + When the broker receives a Broker Request message, it responds + with a Broker Response message. This message contains an + identifier unique to the broker. + </para> + <programlisting> + +-----+-----+-----+-----+-----------------------+ + | 'A' | 'M' | '1' | 'b' | 0 | + +-----+-----+-----+-----+-----------------------+----------------------------+ + | brokerId (uuid) | + +----------------------------------------------------------------------------+ +</programlisting> +<!--h4--></section> + + <section role="h4" id="ManagementDesignnotes-CommandCompletionMessage"><title> + Command + Completion Message + </title> + + + <programlisting> + +-----+-----+-----+-----+-----------------------+ + | 'A' | 'M' | '1' | 'z' | seq | + +-----+-----+-----+-----+-----------------------+ + | Completion Code | + +-----------------------+-----------------------------------------+ + | Completion Text | + +-----------------------------------------------------------------+ +</programlisting> +<!--h4--></section> + + + <section role="h4" id="ManagementDesignnotes-ClassQuery"><title> + Class Query + </title> + + <programlisting> + +-----+-----+-----+-----+-----------------------+ + | 'A' | 'M' | '1' | 'Q' | seq | + +-----+-----+-----+-----+-----------------------+----------+ + | package name (str8) | + +----------------------------------------------------------+ +</programlisting> +<!--h4--></section> + + + <section role="h4" id="ManagementDesignnotes-ClassIndication"><title> + Class + Indication + </title> + + + <programlisting> + +-----+-----+-----+-----+-----------------------+ + | 'A' | 'M' | '1' | 'q' | seq | + +-----+-----+-----+-----+-----------------------+----------+ + | package name (str8) | + +----------------------------------------------------------+ + | class name (str8) | + +----------------------------------------------------------+ + | schema hash (bin128) | + +----------------------------------------------------------+ +</programlisting> +<!--h4--></section> + + + <section role="h4" id="ManagementDesignnotes-SchemaRequest"><title> + Schema Request + </title> + + + <programlisting> + +-----+-----+-----+-----+-----------------------+ + | 'A' | 'M' | '1' | 'S' | seq | + +-----+-----+-----+-----+-----------------------+----------+ + | packageName (str8) | + +----------------------------------------------------------+ + | className (str8) | + +----------------------------------------------------------+ + | schema-hash (bin128) | + +----------------------------------------------------------+ +</programlisting> +<!--h4--></section> + + <section role="h4" id="ManagementDesignnotes-SchemaResponse"><title> + Schema + Response + </title> + + <programlisting> + +-----+-----+-----+-----+-----------------------+ + | 'A' | 'M' | '1' | 's' | seq | + +-----+-----+-----+-----+-----------------------+----------+ + | packageName (str8) | + +----------------------------------------------------------+ + | className (str8) | + +----------------------------------------------------------+ + | schema-hash (bin128) | + +-----------+-----------+-----------+-----------+----------+ + | propCnt | statCnt | methodCnt | eventCnt | + +-----------+-----------+-----------+-----------+----------------------------+ + | propCnt property records | + +----------------------------------------------------------------------------+ + | statCnt statistic records | + +----------------------------------------------------------------------------+ + | methodCnt method records | + +----------------------------------------------------------------------------+ + | eventCnt event records | + +----------------------------------------------------------------------------+ +</programlisting> + <para> + Each <emphasis>property</emphasis> record is an AMQP map with the following + fields. Optional fields may optionally be omitted from the map. + </para><table><title/><tgroup cols="3"> + <tbody> + <row> + <entry> + <emphasis>field name</emphasis> + </entry> + <entry> + <emphasis>optional</emphasis> + </entry> + <entry> + <emphasis>description</emphasis> + </entry> + </row> + <row> + <entry> + name + </entry> + <entry> + no + </entry> + <entry> + Name of the property + </entry> + </row> + <row> + <entry> + type + </entry> + <entry> + no + </entry> + <entry> + Type code for the property + </entry> + </row> + <row> + <entry> + access + </entry> + <entry> + no + </entry> + <entry> + Access code for the property + </entry> + </row> + <row> + <entry> + index + </entry> + <entry> + no + </entry> + <entry> + 1 = index element, 0 = not an index element + </entry> + </row> + <row> + <entry> + optional + </entry> + <entry> + no + </entry> + <entry> + 1 = optional element (may be not present), 0 = mandatory + (always present) + </entry> + </row> + <row> + <entry> + unit + </entry> + <entry> + yes + </entry> + <entry> + Units for numeric values (i.e. seconds, bytes, etc.) + </entry> + </row> + <row> + <entry> + min + </entry> + <entry> + yes + </entry> + <entry> + Minimum value for numerics + </entry> + </row> + <row> + <entry> + max + </entry> + <entry> + yes + </entry> + <entry> + Maximum value for numerics + </entry> + </row> + <row> + <entry> + maxlen + </entry> + <entry> + yes + </entry> + <entry> + Maximum length for strings + </entry> + </row> + <row> + <entry> + desc + </entry> + <entry> + yes + </entry> + <entry> + Description of the property + </entry> + </row> + </tbody> + </tgroup></table><para> + Each <emphasis>statistic</emphasis> record is an AMQP map with the following + fields: + </para><table><title/><tgroup cols="3"> + <tbody> + <row> + <entry> + <emphasis>field name</emphasis> + </entry> + <entry> + <emphasis>optional</emphasis> + </entry> + <entry> + <emphasis>description</emphasis> + </entry> + </row> + <row> + <entry> + name + </entry> + <entry> + no + </entry> + <entry> + Name of the statistic + </entry> + </row> + <row> + <entry> + type + </entry> + <entry> + no + </entry> + <entry> + Type code for the statistic + </entry> + </row> + <row> + <entry> + unit + </entry> + <entry> + yes + </entry> + <entry> + Units for numeric values (i.e. seconds, bytes, etc.) + </entry> + </row> + <row> + <entry> + desc + </entry> + <entry> + yes + </entry> + <entry> + Description of the statistic + </entry> + </row> + </tbody> + </tgroup></table><para> + <emphasis>method</emphasis> and <emphasis>event</emphasis> records contain a main map that + describes the method or header followed by zero or more maps + describing arguments. The main map contains the following fields: + </para><table><title/><tgroup cols="3"> + <tbody> + <row> + <entry> + <emphasis>field name</emphasis> + </entry> + <entry> + <emphasis>optional</emphasis> + </entry> + <entry> + <emphasis>description</emphasis> + </entry> + </row> + <row> + <entry> + name + </entry> + <entry> + no + </entry> + <entry> + Name of the method or event + </entry> + </row> + <row> + <entry> + argCount + </entry> + <entry> + no + </entry> + <entry> + Number of argument records to follow + </entry> + </row> + <row> + <entry> + desc + </entry> + <entry> + yes + </entry> + <entry> + Description of the method or event + </entry> + </row> + </tbody> + </tgroup></table><para> + Argument maps contain the following fields: + </para><table><title/><tgroup cols="5"> + <tbody> + <row> + <entry> + <emphasis>field name</emphasis> + </entry> + <entry> + <emphasis>method</emphasis> + </entry> + <entry> + <emphasis>event</emphasis> + </entry> + <entry> + <emphasis>optional</emphasis> + </entry> + <entry> + <emphasis>description</emphasis> + </entry> + </row> + <row> + <entry> + name + </entry> + <entry> + yes + </entry> + <entry> + yes + </entry> + <entry> + no + </entry> + <entry> + Argument name + </entry> + </row> + <row> + <entry> + type + </entry> + <entry> + yes + </entry> + <entry> + yes + </entry> + <entry> + no + </entry> + <entry> + Type code for the argument + </entry> + </row> + <row> + <entry> + dir + </entry> + <entry> + yes + </entry> + <entry> + no + </entry> + <entry> + yes + </entry> + <entry> + Direction code for method arguments + </entry> + </row> + <row> + <entry> + unit + </entry> + <entry> + yes + </entry> + <entry> + yes + </entry> + <entry> + yes + </entry> + <entry> + Units for numeric values (i.e. seconds, bytes, etc.) + </entry> + </row> + <row> + <entry> + min + </entry> + <entry> + yes + </entry> + <entry> + no + </entry> + <entry> + yes + </entry> + <entry> + Minimum value for numerics + </entry> + </row> + <row> + <entry> + max + </entry> + <entry> + yes + </entry> + <entry> + no + </entry> + <entry> + yes + </entry> + <entry> + Maximum value for numerics + </entry> + </row> + <row> + <entry> + maxlen + </entry> + <entry> + yes + </entry> + <entry> + no + </entry> + <entry> + yes + </entry> + <entry> + Maximum length for strings + </entry> + </row> + <row> + <entry> + desc + </entry> + <entry> + yes + </entry> + <entry> + yes + </entry> + <entry> + yes + </entry> + <entry> + Description of the argument + </entry> + </row> + <row> + <entry> + default + </entry> + <entry> + yes + </entry> + <entry> + no + </entry> + <entry> + yes + </entry> + <entry> + Default value for the argument + </entry> + </row> + </tbody> + </tgroup></table><para> + <emphasis>type codes</emphasis> are numerics with the following values: + </para><table><title/><tgroup cols="2"> + <tbody> + <row> + <entry> + <emphasis>value</emphasis> + </entry> + <entry> + <emphasis>type</emphasis> + </entry> + </row> + <row> + <entry> + 1 + </entry> + <entry> + uint8 + </entry> + </row> + <row> + <entry> + 2 + </entry> + <entry> + uint16 + </entry> + </row> + <row> + <entry> + 3 + </entry> + <entry> + uint32 + </entry> + </row> + <row> + <entry> + 4 + </entry> + <entry> + uint64 + </entry> + </row> + <row> + <entry> + 6 + </entry> + <entry> + str8 + </entry> + </row> + <row> + <entry> + 7 + </entry> + <entry> + str16 + </entry> + </row> + <row> + <entry> + 8 + </entry> + <entry> + absTime(uint64) + </entry> + </row> + <row> + <entry> + 9 + </entry> + <entry> + deltaTime(uint64) + </entry> + </row> + <row> + <entry> + 10 + </entry> + <entry> + objectReference(uint64) + </entry> + </row> + <row> + <entry> + 11 + </entry> + <entry> + boolean(uint8) + </entry> + </row> + <row> + <entry> + 12 + </entry> + <entry> + float + </entry> + </row> + <row> + <entry> + 13 + </entry> + <entry> + double + </entry> + </row> + <row> + <entry> + 14 + </entry> + <entry> + uuid + </entry> + </row> + <row> + <entry> + 15 + </entry> + <entry> + map + </entry> + </row> + <row> + <entry> + 16 + </entry> + <entry> + int8 + </entry> + </row> + <row> + <entry> + 17 + </entry> + <entry> + int16 + </entry> + </row> + <row> + <entry> + 18 + </entry> + <entry> + int32 + </entry> + </row> + <row> + <entry> + 19 + </entry> + <entry> + int64 + </entry> + </row> + </tbody> + </tgroup></table><para> + <emphasis>access codes</emphasis> are numerics with the following values: + </para><table><title/><tgroup cols="2"> + <tbody> + <row> + <entry> + <emphasis>value</emphasis> + </entry> + <entry> + <emphasis>access</emphasis> + </entry> + </row> + <row> + <entry> + 1 + </entry> + <entry> + Read-Create access + </entry> + </row> + <row> + <entry> + 2 + </entry> + <entry> + Read-Write access + </entry> + </row> + <row> + <entry> + 3 + </entry> + <entry> + Read-Only access + </entry> + </row> + </tbody> + </tgroup></table><para> + <emphasis>direction codes</emphasis> are numerics with the following values: + </para><table><title/><tgroup cols="2"> + <tbody> + <row> + <entry> + <emphasis>value</emphasis> + </entry> + <entry> + <emphasis>direction</emphasis> + </entry> + </row> + <row> + <entry> + 1 + </entry> + <entry> + Input (from client to broker) + </entry> + </row> + <row> + <entry> + 2 + </entry> + <entry> + Output (from broker to client) + </entry> + </row> + <row> + <entry> + 3 + </entry> + <entry> + IO (bidirectional) + </entry> + </row> + </tbody> + </tgroup></table> +<!--h4--></section> + + + <section role="h4" id="ManagementDesignnotes-HeartbeatIndication"><title> + Heartbeat + Indication + </title> + + + <programlisting> + +-----+-----+-----+-----+-----------------------+ + | 'A' | 'M' | '1' | 'h' | 0 | + +-----+-----+-----+-----+-----------------------+ + | timestamp of current interval (datetime) | + +-----------------------------------------------+ +</programlisting> +<!--h4--></section> + + <section role="h4" id="ManagementDesignnotes-ConfigurationandInstrumentationContentMessages"><title> + Configuration and Instrumentation Content Messages + </title> + + <para> + Content messages are published when changes are made to the + values of properties or statistics or when new management clients + bind a queue to the management exchange. + </para> + <programlisting> + +-----+-----+-----+-------+-----------------------+ + | 'A' | 'M' | '1' |'g/c/i'| seq | + +-----+-----+-----+-------+-----------------------+--------+ + | packageName (str8) | + +----------------------------------------------------------+ + | className (str8) | + +----------------------------------------------------------+ + | class hash (bin128) | + +-----+-----+-----+-----+-----+-----+-----+-----+----------+ + | timestamp of current sample (datetime) | + +-----+-----+-----+-----+-----+-----+-----+-----+ + | time object was created (datetime) | + +-----+-----+-----+-----+-----+-----+-----+-----+ + | time object was deleted (datetime) | + +-----+-----+-----+-----+-----+-----+-----+-----+ + | objectId (uint64) | + +-----+-----+-----+-----+-----+-----+-----+-----+ + | presence bitmasks (0 or more uint8 fields) | + +-----+-----+-----+-----+-----+-----+-----+-----+------------------------+ + | config/inst values (in schema order) | + +------------------------------------------------------------------------+ +</programlisting> + <para> + All timestamps are uint64 values representing nanoseconds since + the epoch (January 1, 1970). The objectId is a uint64 value that + uniquely identifies this object instance. + </para><para> + If any of the properties in the object are defined as optional, + there will be 1 or more "presence bitmask" octets. There are as + many octets as are needed to provide one bit per optional + property. The bits are assigned to the optional properties in + schema order (first octet first, lowest order bit first). + </para><para> + For example: If there are two optional properties in the schema + called "option1" and "option2" (defined in that order), there + will be one presence bitmask octet and the bits will be assigned + as bit 0 controls option1 and bit 1 controls option2. + </para><para> + If the bit for a particular optional property is set (1), the + property will be encoded normally in the "values" portion of the + message. If the bit is clear (0), the property will be omitted + from the list of encoded values and will be considered "NULL" or + "not present". + </para><para> + The element values are encoded by their type into the message in + the order in which they appeared in the schema message. + </para> +<!--h4--></section> + + <section role="h4" id="ManagementDesignnotes-GetQueryMessage"><title> + Get Query + Message + </title> + + <para> + A Get Request may be sent by the management console to cause a + management agent to immediately send content information for + objects of a class. + </para> + <programlisting> + +-----+-----+-----+-----+-----------------------+ + | 'A' | 'M' | '1' | 'G' | seq | + +-----+-----+-----+-----+-----------------------+----------+ + | Get request field table | + +----------------------------------------------------------+ +</programlisting> + <para> + The content of a get request is a field table that specifies what + objects are being requested. Most of the fields are optional and + are available for use in more extensive deployments. + </para><table><title/><tgroup cols="4"> + <tbody> + <row> + <entry> + <emphasis>Field Key</emphasis> + </entry> + <entry> + <emphasis>Mandatory</emphasis> + </entry> + <entry> + <emphasis>Type</emphasis> + </entry> + <entry> + <emphasis>Description</emphasis> + </entry> + </row> + <row> + <entry> + "_class" + </entry> + <entry> + yes + </entry> + <entry> + short-string + </entry> + <entry> + The name of the class of objects being requested. + </entry> + </row> + <row> + <entry> + "_package" + </entry> + <entry> + no + </entry> + <entry> + short-string + </entry> + <entry> + The name of the extension package the class belongs to. If + omitted, the package defaults to "qpid" for access to + objects in the connected broker. + </entry> + </row> + <row> + <entry> + "_agent" + </entry> + <entry> + no + </entry> + <entry> + uuid + </entry> + <entry> + The management agent that is the target of the request. If + omitted, agent defaults to the connected broker. + </entry> + </row> + </tbody> + </tgroup></table><para> + When the management agent receives a get request, it sends + content messages describing the requested objects. Once the last + content message is sent, it then sends a Command Completion + message with the same sequence number supplied in the request to + indicate to the requestor that there are no more messages coming. + </para> +<!--h4--></section> + + + <section role="h4" id="ManagementDesignnotes-MethodRequest"><title> + Method Request + </title> + + <para> + Method request messages have the following structure. The + sequence number is opaque to the management agent. It is returned + unchanged in the method reply so the calling client can correctly + associate the reply to the request. The objectId is the unique ID + of the object on which the method is to be executed. + </para> + <programlisting> + +-----+-----+-----+-----+-----------------------+ + | 'A' | 'M' | '1' | 'M' | seq | + +-----+-----+-----+-----+-----------------------+ + | objectId (uint64) | + +-----------------------------------------------+ + | methodName (str8) | + +-----------------------------------------------+------------------------+ + | input and bidirectional argument values (in schema order) | + +------------------------------------------------------------------------+ +</programlisting> +<!--h4--></section> + + <section role="h4" id="ManagementDesignnotes-MethodResponse"><title> + Method + Response + </title> + + <para> + Method reply messages have the following structure. The sequence + number is identical to that supplied in the method request. The + status code (and text) indicate whether or not the method was + successful and if not, what the error was. Output and + bidirectional arguments are only included if the status code was + 0 (STATUS_OK). + </para> + <programlisting> + +-----+-----+-----+-----+-----------------------+ + | 'A' | 'M' | '1' | 'm' | seq | + +-----+-----+-----+-----+-----------------------+ + | status code | + +-----------------------+----------------------------------+ + | status text (str8) | + +-----------------------+----------------------------------+-------------+ + | output and bidirectional argument values (in schema order) | + +------------------------------------------------------------------------+ +</programlisting> + <para> + <emphasis>status code</emphasis> values are: + </para><table><title/><tgroup cols="2"> + <tbody> + <row> + <entry> + <emphasis>value</emphasis> + </entry> + <entry> + <emphasis>description</emphasis> + </entry> + </row> + <row> + <entry> + 0 + </entry> + <entry> + STATUS_OK - successful completion + </entry> + </row> + <row> + <entry> + 1 + </entry> + <entry> + STATUS_UNKNOWN_OBJECT - objectId not found in the agent + </entry> + </row> + <row> + <entry> + 2 + </entry> + <entry> + STATUS_UNKNOWN_METHOD - method is not known by the object + type + </entry> + </row> + <row> + <entry> + 3 + </entry> + <entry> + STATUS_NOT_IMPLEMENTED - method is not currently + implemented + </entry> + </row> + </tbody> + </tgroup></table> +<!--h4--></section> +<!--h3--></section> + + <section role="h3" id="ManagementDesignnotes-MessagesforExtendedScenario"><title> + Messages + for Extended Scenario + </title> + + <section role="h4" id="ManagementDesignnotes-ExtendedManagementProtocol"><title> + Extended + Management Protocol + </title> + + <para> + Qpid supports management extensions that allow the management + broker to be a central point for the management of multiple + external entities with their own management schemas. + </para> + <programlisting> + Broker Remote Agent + | | + | <----------------------------------------- Attach Request --- | + | --- Attach Response ----------------------------------------> | + | | + | <------------------------------------- Package Indication --- | + | <------------------------------------- Package Indication --- | + | | + | <--------------------------------------- Class Indication --- | + | <--------------------------------------- Class Indication --- | + | <--------------------------------------- Class Indication --- | + | <--------------------------------------- Class Indication --- | + | <--------------------------------------- Class Indication --- | + | | + | --- Schema Request (class key) -----------------------------> | + | <---------------------------------------- Schema Response --- | + | | + | --- Schema Request (class key) -----------------------------> | + | <---------------------------------------- Schema Response --- | + | | + | | +</programlisting> +<!--h4--></section> + + <section role="h4" id="ManagementDesignnotes-ExtendedOpcodes"><title> + Extended + Opcodes + </title> + + <table><title/><tgroup cols="3"> + <tbody> + <row> + <entry> + <emphasis>opcode</emphasis> + </entry> + <entry> + <emphasis>message</emphasis> + </entry> + <entry> + <emphasis>description</emphasis> + </entry> + </row> + <row> + <entry> + 'P' + </entry> + <entry> + Package Query + </entry> + <entry> + This message contains a schema package query request, + requesting that the broker dump the list of known packages + </entry> + </row> + <row> + <entry> + 'p' + </entry> + <entry> + Package Indication + </entry> + <entry> + This message contains a schema package indication, + identifying a package known by the broker + </entry> + </row> + <row> + <entry> + 'A' + </entry> + <entry> + Agent Attach Request + </entry> + <entry> + This message is sent by a remote agent when it wishes to + attach to a management broker + </entry> + </row> + <row> + <entry> + 'a' + </entry> + <entry> + Agent Attach Response + </entry> + <entry> + The management broker sends this response if an attaching + remote agent is permitted to join + </entry> + </row> + <row> + <entry> + 'x' + </entry> + <entry> + Console Added Indication + </entry> + <entry> + This message is sent to all remote agents by the management + broker when a new console binds to the management exchange + </entry> + </row> + </tbody> + </tgroup></table> +<!--h4--></section> + + + <section role="h4" id="ManagementDesignnotes-PackageQuery"><title> + Package Query + </title> + + + <programlisting> + +-----+-----+-----+-----+-----------------------+ + | 'A' | 'M' | '1' | 'P' | seq | + +-----+-----+-----+-----+-----------------------+ +</programlisting> +<!--h4--></section> + + <section role="h4" id="ManagementDesignnotes-PackageIndication"><title> + Package + Indication + </title> + + + <programlisting> + +-----+-----+-----+-----+-----------------------+ + | 'A' | 'M' | '1' | 'p' | seq | + +-----+-----+-----+-----+-----------------------+----------+ + | package name (str8) | + +----------------------------------------------------------+ +</programlisting> +<!--h4--></section> + + <section role="h4" id="ManagementDesignnotes-AttachRequest"><title> + Attach Request + </title> + + <programlisting> + +-----+-----+-----+-----+-----------------------+ + | 'A' | 'M' | '1' | 'A' | seq | + +-----+-----+-----+-----+-----------------------+----------+ + | label (str8) | + +-----------------------+----------------------------------+ + | system-id (uuid) | + +-----------------------+----------------------------------+ + | requested objId bank | + +-----------------------+ +</programlisting> +<!--h4--></section> + + <section role="h4" id="ManagementDesignnotes-AttachResponse-28success-29"><title> + Attach + Response (success) + </title> + + + <programlisting> + +-----+-----+-----+-----+-----------------------+ + | 'A' | 'M' | '1' | 'a' | seq | + +-----+-----+-----+-----+-----------------------+ + | assigned broker bank | + +-----------------------+ + | assigned objId bank | + +-----------------------+ +</programlisting> +<!--h4--></section> + + <section role="h4" id="ManagementDesignnotes-ConsoleAddedIndication"><title> + Console Added + Indication + </title> + + + <programlisting> + +-----+-----+-----+-----+-----------------------+ + | 'A' | 'M' | '1' | 'x' | seq | + +-----+-----+-----+-----+-----------------------+ +</programlisting> + + + + <!--h4--></section> + <!--h3--></section> + <!--h2--></section> + + +</chapter> +
\ No newline at end of file diff --git a/qpid/doc/book/src/QMF Python Console Tutorial.xml b/qpid/doc/book/src/QMF Python Console Tutorial.xml new file mode 100644 index 0000000000..1fd8bf86d3 --- /dev/null +++ b/qpid/doc/book/src/QMF Python Console Tutorial.xml @@ -0,0 +1,873 @@ +<?xml version="1.0" encoding="utf-8"?> +<chapter xmlns:html="http://www.w3.org/1999/xhtml"><title> + Apache Qpid : QMF Python Console Tutorial + </title> + <itemizedlist> + <listitem><para> + <xref linkend="QMFPythonConsoleTutorial-PrerequisiteInstallQpidMessaging"/> + </para></listitem> + <listitem><para> + <xref linkend="QMFPythonConsoleTutorial-SynchronousConsoleOperations"/> + </para></listitem> + <listitem><para> + <itemizedlist> + <listitem><para> + <xref linkend="QMFPythonConsoleTutorial-CreatingaQMFConsoleSessionandAttachingtoaBroker"/> + </para></listitem> + <listitem><para> + <xref linkend="QMFPythonConsoleTutorial-AccessingManagedObjects"/> + </para></listitem> + <listitem><para> + <itemizedlist> + <listitem><para> + <xref linkend="QMFPythonConsoleTutorial-ViewingPropertiesandStatisticsofanObject"/> + </para></listitem> + <listitem><para> + <xref linkend="QMFPythonConsoleTutorial-InvokingMethodsonanObject"/> + </para></listitem> + </itemizedlist> + </para></listitem> + </itemizedlist> + </para></listitem> + <listitem><para> + <xref linkend="QMFPythonConsoleTutorial-AsynchronousConsoleOperations"/> + </para></listitem> + <listitem><para> + <itemizedlist> + <listitem><para> + <xref linkend="QMFPythonConsoleTutorial-CreatingaConsoleClasstoReceiveAsynchronousData"/> + </para></listitem> + <listitem><para> + <xref linkend="QMFPythonConsoleTutorial-ReceivingEvents"/> + </para></listitem> + <listitem><para> + <xref linkend="QMFPythonConsoleTutorial-ReceivingObjects"/> + </para></listitem> + <listitem><para> + <xref linkend="QMFPythonConsoleTutorial-AsynchronousMethodCallsandMethodTimeouts"/> + </para></listitem> + </itemizedlist> + </para></listitem> + <listitem><para> + <xref linkend="QMFPythonConsoleTutorial-DiscoveringwhatKindsofObjectsareAvailable"/> + </para></listitem> + </itemizedlist> + <section role="h1" id="QMFPythonConsoleTutorial-PrerequisiteInstallQpidMessaging"><title> + Prerequisite + - Install Qpid Messaging + </title> + + <para> + QMF uses AMQP Messaging (QPid) as its means of communication. To + use QMF, Qpid messaging must be installed somewhere in the + network. Qpid can be downloaded as source from Apache, is + packaged with a number of Linux distributions, and can be + purchased from commercial vendors that use Qpid. Please see + <xref linkend="qpid_Download"/> for + information as to where to get Qpid Messaging. + </para><para> + Qpid Messaging includes a message broker (qpidd) which typically + runs as a daemon on a system. It also includes client bindings in + various programming languages. The Python-language client library + includes the QMF console libraries needed for this tutorial. + </para><para> + Please note that Qpid Messaging has two broker implementations. + One is implemented in C++ and the other in Java. At press time, + QMF is supported only by the C++ broker. + </para><para> + If the goal is to get the tutorial examples up and running as + quickly as possible, all of the Qpid components can be installed + on a single system (even a laptop). For more realistic + deployments, the broker can be deployed on a server and the + client/QMF libraries installed on other systems. + </para> +<!--h1--></section> + + <section role="h1" id="QMFPythonConsoleTutorial-SynchronousConsoleOperations"><title> + Synchronous + Console Operations + </title> + + <para> + The Python console API for QMF can be used in a synchronous + style, an asynchronous style, or a combination of both. + Synchronous operations are conceptually simple and are well + suited for user-interactive tasks. All operations are performed + in the context of a Python function call. If communication over + the message bus is required to complete an operation, the + function call blocks and waits for the expected result (or + timeout failure) before returning control to the caller. + </para><section role="h2" id="QMFPythonConsoleTutorial-CreatingaQMFConsoleSessionandAttachingtoaBroker"><title> + Creating a QMF Console Session and Attaching to a Broker + </title> + + <para> + For the purposes of this tutorial, code examples will be shown as + they are entered in an interactive python session. + </para> + <programlisting> +$ python +Python 2.5.2 (r252:60911, Sep 30 2008, 15:41:38) +[GCC 4.3.2 20080917 (Red Hat 4.3.2-4)] on linux2 +Type "help", "copyright", "credits" or "license" for more information. +>>> +</programlisting> + <para> + We will begin by importing the required libraries. If the Python + client is properly installed, these libraries will be found + normally by the Python interpreter. + </para> + <programlisting> +>>> from qmf.console import Session +</programlisting> + <para> + We must now create a <emphasis>Session</emphasis> object to manage this QMF + console session. + </para> + <programlisting> +>>> sess = Session() +</programlisting> + <para> + If no arguments are supplied to the creation of <emphasis>Session</emphasis>, + it defaults to synchronous-only operation. It also defaults to + user-management of connections. More on this in a moment. + </para><para> + We will now establish a connection to the messaging broker. If + the broker daemon is running on the local host, simply use the + following: + </para> + <programlisting> +>>> broker = sess.addBroker() +</programlisting> + <para> + If the messaging broker is on a remote host, supply the URL to + the broker in the <emphasis>addBroker</emphasis> function call. Here's how to + connect to a local broker using the URL. + </para> + <programlisting> +>>> broker = sess.addBroker("amqp://localhost") +</programlisting> + <para> + The call to <emphasis>addBroker</emphasis> is synchronous and will return + only after the connection has been successfully established or + has failed. If a failure occurs, <emphasis>addBroker</emphasis> will raise an + exception that can be handled by the console script. + </para> + <programlisting> +>>> try: +... broker = sess.addBroker("amqp://localhost:1000") +... except: +... print "Connection Failed" +... +Connection Failed +>>> +</programlisting> + <para> + This operation fails because there is no Qpid Messaging broker + listening on port 1000 (the default port for qpidd is 5672). + </para><para> + If preferred, the QMF session can manage the connection for you. + In this case, <emphasis>addBroker</emphasis> returns immediately and the + session attempts to establish the connection in the background. + This will be covered in detail in the section on asynchronous + operations. + </para> +<!--h2--></section> + + <section role="h2" id="QMFPythonConsoleTutorial-AccessingManagedObjects"><title> + Accessing + Managed Objects + </title> + + <para> + The Python console API provides access to remotely managed + objects via a <emphasis>proxy</emphasis> model. The API gives the client an + object that serves as a proxy representing the "real" object + being managed on the agent application. Operations performed on + the proxy result in the same operations on the real object. + </para><para> + The following examples assume prior knowledge of the kinds of + objects that are actually available to be managed. There is a + section later in this tutorial that describes how to discover + what is manageable on the QMF bus. + </para><para> + Proxy objects are obtained by calling the + <emphasis>Session.getObjects</emphasis> function. + </para><para> + To illustrate, we'll get a list of objects representing queues in + the message broker itself. + </para> + <programlisting> +>>> queues = sess.getObjects(_class="queue", _package="org.apache.qpid.broker") +</programlisting> + <para> + <emphasis>queues</emphasis> is an array of proxy objects representing real + queues on the message broker. A proxy object can be printed to + display a description of the object. + </para> + <programlisting> +>>> for q in queues: +... print q +... +org.apache.qpid.broker:queue[0-1537-1-0-58] 0-0-1-0-1152921504606846979:reply-localhost.localdomain.32004 +org.apache.qpid.broker:queue[0-1537-1-0-61] 0-0-1-0-1152921504606846979:topic-localhost.localdomain.32004 +>>> +</programlisting> + <section role="h3" id="QMFPythonConsoleTutorial-ViewingPropertiesandStatisticsofanObject"><title> + Viewing Properties and Statistics of an Object + </title> + + <para> + Let us now focus our attention on one of the queue objects. + </para> + <programlisting> +>>> queue = queues[0] +</programlisting> + <para> + The attributes of an object are partitioned into + <emphasis>properties</emphasis> and <emphasis>statistics</emphasis>. Though the + distinction is somewhat arbitrary, <emphasis>properties</emphasis> tend to + be fairly static and may also be large and <emphasis>statistics</emphasis> + tend to change rapidly and are relatively small (counters, etc.). + </para><para> + There are two ways to view the properties of an object. An array + of properties can be obtained using the <emphasis>getProperties</emphasis> + function: + </para> + <programlisting> +>>> props = queue.getProperties() +>>> for prop in props: +... print prop +... +(vhostRef, 0-0-1-0-1152921504606846979) +(name, u'reply-localhost.localdomain.32004') +(durable, False) +(autoDelete, True) +(exclusive, True) +(arguments, {}) +>>> +</programlisting> + <para> + The <emphasis>getProperties</emphasis> function returns an array of tuples. + Each tuple consists of the property descriptor and the property + value. + </para><para> + A more convenient way to access properties is by using the + attribute of the proxy object directly: + </para> + <programlisting> +>>> queue.autoDelete +True +>>> queue.name +u'reply-localhost.localdomain.32004' +>>> +</programlisting> + <para> + Statistics are accessed in the same way: + </para> + <programlisting> +>>> stats = queue.getStatistics() +>>> for stat in stats: +... print stat +... +(msgTotalEnqueues, 53) +(msgTotalDequeues, 53) +(msgTxnEnqueues, 0) +(msgTxnDequeues, 0) +(msgPersistEnqueues, 0) +(msgPersistDequeues, 0) +(msgDepth, 0) +(byteDepth, 0) +(byteTotalEnqueues, 19116) +(byteTotalDequeues, 19116) +(byteTxnEnqueues, 0) +(byteTxnDequeues, 0) +(bytePersistEnqueues, 0) +(bytePersistDequeues, 0) +(consumerCount, 1) +(consumerCountHigh, 1) +(consumerCountLow, 1) +(bindingCount, 2) +(bindingCountHigh, 2) +(bindingCountLow, 2) +(unackedMessages, 0) +(unackedMessagesHigh, 0) +(unackedMessagesLow, 0) +(messageLatencySamples, 0) +(messageLatencyMin, 0) +(messageLatencyMax, 0) +(messageLatencyAverage, 0) +>>> +</programlisting> + <para> + or alternatively: + </para> + <programlisting> +>>> queue.byteTotalEnqueues +19116 +>>> +</programlisting> + <para> + The proxy objects do not automatically track changes that occur + on the real objects. For example, if the real queue enqueues more + bytes, viewing the <emphasis>byteTotalEnqueues</emphasis> statistic will show + the same number as it did the first time. To get updated data on + a proxy object, use the <emphasis>update</emphasis> function call: + </para> + <programlisting> +>>> queue.update() +>>> queue.byteTotalEnqueues +19783 +>>> +</programlisting> + + <note><title>Be Advised</title> + <para> + The <emphasis>update</emphasis> method was added after the M4 release + of Qpid/Qmf. It may not be available in your + distribution. + </para> + </note> +<!--h3--></section> + + <section role="h3" id="QMFPythonConsoleTutorial-InvokingMethodsonanObject"><title> + Invoking + Methods on an Object + </title> + + <para> + Up to this point, we have used the QMF Console API to find + managed objects and view their attributes, a read-only activity. + The next topic to illustrate is how to invoke a method on a + managed object. Methods allow consoles to control the managed + agents by either triggering a one-time action or by changing the + values of attributes in an object. + </para><para> + First, we'll cover some background information about methods. A + <emphasis>QMF object class</emphasis> (of which a <emphasis>QMF object</emphasis> is an + instance), may have zero or more methods. To obtain a list of + methods available for an object, use the <emphasis>getMethods</emphasis> + function. + </para> + <programlisting> +>>> methodList = queue.getMethods() +</programlisting> + <para> + <emphasis>getMethods</emphasis> returns an array of method descriptors (of + type qmf.console.SchemaMethod). To get a summary of a method, you + can simply print it. The _<emphasis>repr</emphasis>_ function returns a + string that looks like a function prototype. + </para> + <programlisting> +>>> print methodList +[purge(request)] +>>> +</programlisting> + <para> + For the purposes of illustration, we'll use a more interesting + method available on the <emphasis>broker</emphasis> object which represents + the connected Qpid message broker. + </para> + <programlisting> +>>> br = sess.getObjects(_class="broker", _package="org.apache.qpid.broker")[0] +>>> mlist = br.getMethods() +>>> for m in mlist: +... print m +... +echo(sequence, body) +connect(host, port, durable, authMechanism, username, password, transport) +queueMoveMessages(srcQueue, destQueue, qty) +>>> +</programlisting> + <para> + We have just learned that the <emphasis>broker</emphasis> object has three + methods: <emphasis>echo</emphasis>, <emphasis>connect</emphasis>, and + <emphasis>queueMoveMessages</emphasis>. We'll use the <emphasis>echo</emphasis> method to + "ping" the broker. + </para> + <programlisting> +>>> result = br.echo(1, "Message Body") +>>> print result +OK (0) - {'body': u'Message Body', 'sequence': 1} +>>> print result.status +0 +>>> print result.text +OK +>>> print result.outArgs +{'body': u'Message Body', 'sequence': 1} +>>> +</programlisting> + <para> + In the above example, we have invoked the <emphasis>echo</emphasis> method on + the instance of the broker designated by the proxy "br" with a + sequence argument of 1 and a body argument of "Message Body". The + result indicates success and contains the output arguments (in + this case copies of the input arguments). + </para><para> + To be more precise... Calling <emphasis>echo</emphasis> on the proxy causes + the input arguments to be marshalled and sent to the remote agent + where the method is executed. Once the method execution + completes, the output arguments are marshalled and sent back to + the console to be stored in the method result. + </para><para> + You are probably wondering how you are supposed to know what + types the arguments are and which arguments are input, which are + output, or which are both. This will be addressed later in the + "Discovering what Kinds of Objects are Available" section. + </para> +<!--h3--></section> +<!--h2--></section> +<!--h1--></section> + + <section role="h1" id="QMFPythonConsoleTutorial-AsynchronousConsoleOperations"><title> + Asynchronous + Console Operations + </title> + + <para> + QMF is built on top of a middleware messaging layer (Qpid + Messaging). Because of this, QMF can use some communication + patterns that are difficult to implement using network transports + like UDP, TCP, or SSL. One of these patterns is called the + <emphasis>Publication and Subscription</emphasis> pattern (pub-sub for + short). In the pub-sub pattern, data sources <emphasis>publish</emphasis> + information without a particular destination in mind. Data sinks + (destinations) <emphasis>subscribe</emphasis> using a set of criteria that + describes what kind of data they are interested in receiving. + Data published by a source may be received by zero, one, or many + subscribers. + </para><para> + QMF uses the pub-sub pattern to distribute events, object + creation and deletion, and changes to properties and statistics. + A console application using the QMF Console API can receive these + asynchronous and unsolicited events and updates. This is useful + for applications that store and analyze events and/or statistics. + It is also useful for applications that react to certain events + or conditions. + </para><para> + Note that console applications may always use the synchronous + mechanisms. + </para> + + <section role="h2" id="QMFPythonConsoleTutorial-CreatingaConsoleClasstoReceiveAsynchronousData"><title> + Creating a Console Class to Receive Asynchronous Data + </title> + + <para> + Asynchronous API operation occurs when the console application + supplies a <emphasis>Console</emphasis> object to the session manager. The + <emphasis>Console</emphasis> object (which overrides the + <emphasis>qmf.console.Console</emphasis> class) handles all asynchronously + arriving data. The <emphasis>Console</emphasis> class has the following + methods. Any number of these methods may be overridden by the + console application. Any method that is not overridden defaults + to a null handler which takes no action when invoked. + </para><table><title/><tgroup cols="3"> + <tbody> + <row> + <entry> + Method + </entry> + <entry> + Arguments + </entry> + <entry> + Invoked when... + </entry> + </row> + <row> + <entry> + brokerConnected + </entry> + <entry> + broker + </entry> + <entry> + a connection to a broker is established + </entry> + </row> + <row> + <entry> + brokerDisconnected + </entry> + <entry> + broker + </entry> + <entry> + a connection to a broker is lost + </entry> + </row> + <row> + <entry> + newPackage + </entry> + <entry> + name + </entry> + <entry> + a new package is seen on the QMF bus + </entry> + </row> + <row> + <entry> + newClass + </entry> + <entry> + kind, classKey + </entry> + <entry> + a new class (event or object) is seen on the QMF bus + </entry> + </row> + <row> + <entry> + newAgent + </entry> + <entry> + agent + </entry> + <entry> + a new agent appears on the QMF bus + </entry> + </row> + <row> + <entry> + delAgent + </entry> + <entry> + agent + </entry> + <entry> + an agent disconnects from the QMF bus + </entry> + </row> + <row> + <entry> + objectProps + </entry> + <entry> + broker, object + </entry> + <entry> + the properties of an object are published + </entry> + </row> + <row> + <entry> + objectStats + </entry> + <entry> + broker, object + </entry> + <entry> + the statistics of an object are published + </entry> + </row> + <row> + <entry> + event + </entry> + <entry> + broker, event + </entry> + <entry> + an event is published + </entry> + </row> + <row> + <entry> + heartbeat + </entry> + <entry> + agent, timestamp + </entry> + <entry> + a heartbeat is published by an agent + </entry> + </row> + <row> + <entry> + brokerInfo + </entry> + <entry> + broker + </entry> + <entry> + information about a connected broker is available to be + queried + </entry> + </row> + <row> + <entry> + methodResponse + </entry> + <entry> + broker, seq, response + </entry> + <entry> + the result of an asynchronous method call is received + </entry> + </row> + </tbody> + </tgroup></table><para> + Supplied with the API is a class called <emphasis>DebugConsole</emphasis>. + This is a test <emphasis>Console</emphasis> instance that overrides all of + the methods such that arriving asynchronous data is printed to + the screen. This can be used to see all of the arriving + asynchronous data. + </para> + <!--h2--></section> + + <section role="h2" id="QMFPythonConsoleTutorial-ReceivingEvents"><title> + Receiving + Events + </title> + + <para> + We'll start the example from the beginning to illustrate the + reception and handling of events. In this example, we will create + a <emphasis>Console</emphasis> class that handles broker-connect, + broker-disconnect, and event messages. We will also allow the + session manager to manage the broker connection for us. + </para><para> + Begin by importing the necessary classes: + </para> + <programlisting> +>>> from qmf.console import Session, Console +</programlisting> + <para> + Now, create a subclass of <emphasis>Console</emphasis> that handles the three + message types: + </para> + <programlisting> +>>> class EventConsole(Console): +... def brokerConnected(self, broker): +... print "brokerConnected:", broker +... def brokerDisconnected(self, broker): +... print "brokerDisconnected:", broker +... def event(self, broker, event): +... print "event:", event +... +>>> +</programlisting> + <para> + Make an instance of the new class: + </para> + <programlisting> +>>> myConsole = EventConsole() +</programlisting> + <para> + Create a <emphasis>Session</emphasis> class using the console instance. In + addition, we shall request that the session manager do the + connection management for us. Notice also that we are requesting + that the session manager not receive objects or heartbeats. Since + this example is concerned only with events, we can optimize the + use of the messaging bus by telling the session manager not to + subscribe for object updates or heartbeats. + </para> + <programlisting> +>>> sess = Session(myConsole, manageConnections=True, rcvObjects=False, rcvHeartbeats=False) +>>> broker = sess.addBroker() +>>> +</programlisting> + <para> + Once the broker is added, we will begin to receive asynchronous + events (assuming there is a functioning broker available to + connect to). + </para> + <programlisting> +brokerConnected: Broker connected at: localhost:5672 +event: Thu Jan 29 19:53:19 2009 INFO org.apache.qpid.broker:bind broker=localhost:5672 ... +</programlisting> +<!--h2--></section> + + <section role="h2" id="QMFPythonConsoleTutorial-ReceivingObjects"><title> + Receiving + Objects + </title> + + <para> + To illustrate asynchronous handling of objects, a small console + program is supplied. The entire program is shown below for + convenience. We will then go through it part-by-part to explain + its design. + </para><para> + This console program receives object updates and displays a set + of statistics as they change. It focuses on broker queue objects. + </para> + <programlisting> +# Import needed classes +from qmf.console import Session, Console +from time import sleep + +# Declare a dictionary to map object-ids to queue names +queueMap = {} + +# Customize the Console class to receive object updates. +class MyConsole(Console): + + # Handle property updates + def objectProps(self, broker, record): + + # Verify that we have received a queue object. Exit otherwise. + classKey = record.getClassKey() + if classKey.getClassName() != "queue": + return + + # If this object has not been seen before, create a new mapping from objectID to name + oid = record.getObjectId() + if oid not in queueMap: + queueMap[oid] = record.name + + # Handle statistic updates + def objectStats(self, broker, record): + + # Ignore updates for objects that are not in the map + oid = record.getObjectId() + if oid not in queueMap: + return + + # Print the queue name and some statistics + print "%s: enqueues=%d dequeues=%d" % (queueMap[oid], record.msgTotalEnqueues, record.msgTotalDequeues) + + # if the delete-time is non-zero, this object has been deleted. Remove it from the map. + if record.getTimestamps()[2] > 0: + queueMap.pop(oid) + +# Create an instance of the QMF session manager. Set userBindings to True to allow +# this program to choose which objects classes it is interested in. +sess = Session(MyConsole(), manageConnections=True, rcvEvents=False, userBindings=True) + +# Register to receive updates for broker:queue objects. +sess.bindClass("org.apache.qpid.broker", "queue") +broker = sess.addBroker() + +# Suspend processing while the asynchronous operations proceed. +try: + while True: + sleep(1) +except: + pass + +# Disconnect the broker before exiting. +sess.delBroker(broker) +</programlisting> + <para> + Before going through the code in detail, it is important to + understand the differences between synchronous object access and + asynchronous object access. When objects are obtained + synchronously (using the <emphasis>getObjects</emphasis> function), the + resulting proxy contains all of the object's attributes, both + properties and statistics. When object data is published + asynchronously, the properties and statistics are sent separately + and only when the session first connects or when the content + changes. + </para><para> + The script wishes to print the queue name with the updated + statistics, but the queue name is only present with the + properties. For this reason, the program needs to keep some state + to correlate property updates with their corresponding statistic + updates. This can be done using the <emphasis>ObjectId</emphasis> that + uniquely identifies the object. + </para> + <programlisting> + # If this object has not been seen before, create a new mapping from objectID to name + oid = record.getObjectId() + if oid not in queueMap: + queueMap[oid] = record.name +</programlisting> + <para> + The above code fragment gets the object ID from the proxy and + checks to see if it is in the map (i.e. has been seen before). If + it is not in the map, a new map entry is inserted mapping the + object ID to the queue's name. + </para> + <programlisting> + # if the delete-time is non-zero, this object has been deleted. Remove it from the map. + if record.getTimestamps()[2] > 0: + queueMap.pop(oid) +</programlisting> + <para> + This code fragment detects the deletion of a managed object. + After reporting the statistics, it checks the timestamps of the + proxy. <emphasis>getTimestamps</emphasis> returns a list of timestamps in the + order: + </para><itemizedlist> + <listitem><para> + <emphasis>Current</emphasis> - The timestamp of the sending of this update. + </para></listitem> + <listitem><para> + <emphasis>Create</emphasis> - The time of the object's creation + </para></listitem> + <listitem><para> + <emphasis>Delete</emphasis> - The time of the object's deletion (or zero if + not deleted) + </para></listitem> + </itemizedlist><para> + This code structure is useful for getting information about + very-short-lived objects. It is possible that an object will be + created, used, and deleted within an update interval. In this + case, the property update will arrive first, followed by the + statistic update. Both will indicate that the object has been + deleted but a full accounting of the object's existence and final + state is reported. + </para> + <programlisting> +# Create an instance of the QMF session manager. Set userBindings to True to allow +# this program to choose which objects classes it is interested in. +sess = Session(MyConsole(), manageConnections=True, rcvEvents=False, userBindings=True) + +# Register to receive updates for broker:queue objects. +sess.bindClass("org.apache.qpid.broker", "queue") +</programlisting> + <para> + The above code is illustrative of the way a console application + can tune its use of the QMF bus. Note that <emphasis>rcvEvents</emphasis> is + set to False. This prevents the reception of events. Note also + the use of <emphasis>userBindings=True</emphasis> and the call to + <emphasis>sess.bindClass</emphasis>. If <emphasis>userBindings</emphasis> is set to False + (its default), the session will receive object updates for all + classes of object. In the case above, the application is only + interested in broker:queue objects and reduces its bus bandwidth + usage by requesting updates to only that class. + <emphasis>bindClass</emphasis> may be called as many times as desired to add + classes to the list of subscribed classes. + </para> +<!--h2--></section> + + <section role="h2" id="QMFPythonConsoleTutorial-AsynchronousMethodCallsandMethodTimeouts"><title> + Asynchronous Method Calls and Method Timeouts + </title> + + <para> + Method calls can also be invoked asynchronously. This is useful + if a large number of calls needs to be made in a short time + because the console application will not need to wait for the + complete round-trip delay for each call. + </para><para> + Method calls are synchronous by default. They can be made + asynchronous by adding the keyword-argument _<emphasis>async=True</emphasis> + to the method call. + </para><para> + In a synchronous method call, the return value is the method + result. When a method is called asynchronously, the return value + is a sequence number that can be used to correlate the eventual + result to the request. This sequence number is passed as an + argument to the <emphasis>methodResponse</emphasis> function in the + <emphasis>Console</emphasis> interface. + </para><para> + It is important to realize that the <emphasis>methodResponse</emphasis> + function may be invoked before the asynchronous call returns. + Make sure your code is written to handle this possibility. + </para> + <!--h2--></section> + <!--h1--></section> + + <section role="h1" id="QMFPythonConsoleTutorial-DiscoveringwhatKindsofObjectsareAvailable"><title> + Discovering what Kinds of Objects are Available + </title> + <para/> + <!--h1--></section> + + +</chapter> diff --git a/qpid/doc/book/src/QMan - Qpid Management bridge.xml b/qpid/doc/book/src/QMan - Qpid Management bridge.xml new file mode 100644 index 0000000000..fa700bbca7 --- /dev/null +++ b/qpid/doc/book/src/QMan - Qpid Management bridge.xml @@ -0,0 +1,145 @@ +<?xml version="1.0" encoding="utf-8"?> +<chapter xmlns:html="http://www.w3.org/1999/xhtml"><title> + Apache Qpid : QMan - Qpid Management bridge + </title><section role="h1" id="QMan-QpidManagementbridge-QMan-3AQpidManagementBridge"><title> + QMan + : Qpid Management Bridge + </title> + <para> + QMan is a management bridge for Qpid. It allows external clients + to manage and monitor one or more Qpid brokers. + </para> + <para> + Please note: All WS-DM related concerns have to be considered + part of M5 release. + </para><para> + QMan exposes the broker management interfaces using Java + Management Extensions (JMX) and / or OASIS Web Services + Distributed Management (WSDM). While the first one is supposed to + be used by java based clients only the latter is an interoperable + protocol that enables management clients to access and receive + notifications of management-enabled resources using Web Services. + </para><para> + QMan can be easily integrated in your preexisting system in + different ways : + </para><itemizedlist> + <listitem><para>As a standalone application : in this case it runs as a + server. More specifically it enables communication via RMI (for + JMX) or via HTTP (for WS-DM); Note that when the WS-DM adapter is + used the JMX interface is not exposed; + </para></listitem> + <listitem><para>As a deployable unit : it is also available as a standard + Java web application (war); This is useful when there's a + preexisting Application Server in your environment and you don't + want start another additional server in order to run QMan. + </para></listitem> + </itemizedlist> + + <section role="h2" id="QMan-QpidManagementbridge-UserDocumentation"><title> + User + Documentation + </title> + <para> + With "User Documentation" we mean all information that you need + to know in order to use QMan from a user perspective. Those + information include : + </para><table><title/><tgroup cols="2"> + <tbody> + <row> + <entry> + Section + </entry> + <entry> + Description + </entry> + </row> + <row> + <entry> + <xref linkend="qpid_Get-20me-20up-20and-20running"/> + </entry> + <entry> + How to install & start QMan. + </entry> + </row> + <row> + <entry> + <xref linkend="qpid_QMan-20User-20Guide"/> + </entry> + <entry> + QMan (WS-DM version only) Administration Console. + </entry> + </row> + <row> + <entry> + <xref linkend="qpid_JMX-20Interface-20Specification"/> + </entry> + <entry> + Describes each JMX interface exposed by QMan. + </entry> + </row> + <row> + <entry> + <xref linkend="qpid_WS-DM-20Interface-20Specification"/> + </entry> + <entry> + Describes each WS-DM interface exposed by QMan. + </entry> + </row> + <row> + <entry> + <xref linkend="qpid_QMan-20Messages-20Catalogue"/> + </entry> + <entry> + Informational / Debug / Error / Warning messages catalogue. + </entry> + </row> + </tbody> + </tgroup></table> + <!--h2--></section> + + + <section role="h2" id="QMan-QpidManagementbridge-TechnicalDocumentation"><title> + Technical + Documentation + </title> + <para> + If you are interested in technical details about QMan and related + technologies this is a good starting point. In general this + section provides information about QMan design, interfaces, + patterns and so on... + </para> + + <table><title/><tgroup cols="2"> + <tbody> + <row> + <entry> + Section + </entry> + <entry> + Description + </entry> + </row> + <row> + <entry> + <xref linkend="qpid_QMan-20System-20Overview"/> + </entry> + <entry> + A short introduction about QMan deployment context. + </entry> + </row> + <row> + <entry> + <xref linkend="qpid_QMan-20Components-20View"/> + </entry> + <entry> + Describes QMan components, their interactions and + responsibilities. + </entry> + </row> + </tbody> + </tgroup></table> + +<!--h2--></section> +<!--h1--></section> + +</chapter> diff --git a/qpid/doc/book/src/Qpid ACLs.xml b/qpid/doc/book/src/Qpid ACLs.xml new file mode 100644 index 0000000000..5aa0773fe5 --- /dev/null +++ b/qpid/doc/book/src/Qpid ACLs.xml @@ -0,0 +1,20 @@ +<?xml version="1.0"?> +<section><title> + ACL Formats + </title>Apache Qpid : Qpid ACLs + This page last changed on Nov 18, 2008 by ritchiem. + <para> + The Qpid project has two ACL implementations. An initial version + of ACLs was added to the Java Broker for M2.1 that uses XML + configuration. For M4 a new format was designed to be implemented + by both C++ and Java brokers. M4 release includes the initial C++ + implementation and M5 is expected to include the Java + implementation. + </para><bridgehead id="QpidACLs-Specifications"><anchor id="QpidACLs-Specifications"/>Specifications + </bridgehead><para> + The specifications for each of the ACL formats are linked here: + </para><para><xref linkend="JavaXMLACLs-specification"/><xref linkend="ACL-specification"/></para><bridgehead id="QpidACLs-UserGuides"><anchor id="QpidACLs-UserGuides"/>User Guides + </bridgehead><para> + To aid users in defining their ACLs we have a user guide for each + of the ACL formats. + </para><para><xref linkend="JavaXMLACLs-userguide"/><xref linkend="ACL-userguide"/></para></section> diff --git a/qpid/doc/book/src/Qpid Management Framework.xml b/qpid/doc/book/src/Qpid Management Framework.xml new file mode 100644 index 0000000000..4482ece768 --- /dev/null +++ b/qpid/doc/book/src/Qpid Management Framework.xml @@ -0,0 +1,923 @@ +<?xml version="1.0" encoding="utf-8"?> +<chapter xmlns:html="http://www.w3.org/1999/xhtml"><title> + Apache Qpid : Qpid Management Framework + </title> + <itemizedlist> + <listitem><para> + <xref linkend="QpidManagementFramework-WhatIsQMF"/> + </para></listitem> + <listitem><para> + <xref linkend="QpidManagementFramework-GettingStartedwithQMF"/> + </para></listitem> + <listitem><para> + <xref linkend="QpidManagementFramework-QMFConcepts"/> + </para></listitem> + <listitem><para> + <itemizedlist> + <listitem><para> + <xref linkend="QpidManagementFramework-Console-2CAgent-2CandBroker"/> + </para></listitem> + <listitem><para> + <xref linkend="QpidManagementFramework-Schema"/> + </para></listitem> + <listitem><para> + <xref linkend="QpidManagementFramework-ClassKeysandClassVersioning"/> + </para></listitem> + </itemizedlist> + </para></listitem> + <listitem><para> + <xref linkend="QpidManagementFramework-TheQMFProtocol"/> + </para></listitem> + <listitem><para> + <xref linkend="QpidManagementFramework-HowtoWriteaQMFConsole"/> + </para></listitem> + <listitem><para> + <xref linkend="QpidManagementFramework-HowtoWriteaQMFAgent"/> + </para></listitem> + </itemizedlist> + + <para> + Please visit the <xref linkend="qpid_QMFv2-20Project-20Page"/> for information + about the future of QMF. + </para> + <section role="h1" id="QpidManagementFramework-WhatIsQMF"><title> + What Is QMF + </title> + + <para> + QMF (Qpid Management Framework) is a general-purpose management + bus built on Qpid Messaging. It takes advantage of the + scalability, security, and rich capabilities of Qpid to provide + flexible and easy-to-use manageability to a large set of + applications. + </para> + <!--h1--></section> + + <section role="h1" id="QpidManagementFramework-GettingStartedwithQMF"><title> + Getting + Started with QMF + </title> + + <para> + QMF is used through two primary APIs. The <emphasis>console</emphasis> API is + used for console applications that wish to access and manipulate + manageable components through QMF. The <emphasis>agent</emphasis> API is used + for application that wish to be managed through QMF. + </para><para> + The fastest way to get started with QMF is to work through the + "How To" tutorials for consoles and agents. For a deeper + understanding of what is happening in the tutorials, it is + recommended that you look at the <emphasis>Qmf Concepts</emphasis> section. + </para> + <!--h1--></section> + + <section role="h1" id="QpidManagementFramework-QMFConcepts"><title> + QMF Concepts + </title> + + <para> + This section introduces important concepts underlying QMF. + </para> + + + <section role="h2" id="QpidManagementFramework-Console-2CAgent-2CandBroker"><title> + Console, + Agent, and Broker + </title> + + <para> + The major architectural components of QMF are the Console, the + Agent, and the Broker. Console components are the "managing" + components of QMF and agent components are the "managed" parts. + The broker is a central (possibly distributed, clustered and + fault-tolerant) component that manages name spaces and caches + schema information. + </para><para> + A console application may be a command-line utility, a + three-tiered web-based GUI, a collection and storage device, a + specialized application that monitors and reacts to events and + conditions, or anything else somebody wishes to develop that uses + QMF management data. + </para><para> + An agent application is any application that has been enhanced to + allow itself to be managed via QMF. + </para> + <programlisting> + +-------------+ +---------+ +---------------+ +-------------------+ + | CLI utility | | Web app | | Audit storage | | Event correlation | + +-------------+ +---------+ +---------------+ +-------------------+ + ^ ^ ^ ^ | + | | | | | + v v v v v + +---------------------------------------------------------------------------------+ + | Qpid Messaging Bus (with QMF Broker capability) | + +---------------------------------------------------------------------------------+ + ^ ^ ^ + | | | + v v v + +----------------+ +----------------+ +----------------+ + | Manageable app | | Manageable app | | Manageable app | + +----------------+ +----------------+ +----------------+ +</programlisting> + <para> + In the above diagram, the <emphasis>Manageable apps</emphasis> are agents, + the <emphasis>CLI utility</emphasis>, <emphasis>Web app</emphasis>, and <emphasis>Audit + storage</emphasis> are consoles, and <emphasis>Event correlation</emphasis> is both + a console and an agent because it can create events based on the + aggregation of what it sees. + </para> +<!--h2--></section> + + <section role="h2" id="QpidManagementFramework-Schema"><title> + Schema + </title> + + <para> + A <emphasis>schema</emphasis> describes the structure of management data. + Each <emphasis>agent</emphasis> provides a schema that describes its + management model including the object classes, methods, events, + etc. that it provides. In the current QMF distribution, the + agent's schema is codified in an XML document. In the near + future, there will also be ways to programatically create QMF + schemata. + </para><section role="h3" id="QpidManagementFramework-Package"><title> + Package + </title> + + <para> + Each agent that exports a schema identifies itself using a + <emphasis>package</emphasis> name. The package provides a unique namespace + for the classes in the agent's schema that prevent collisions + with identically named classes in other agents' schemata. + </para><para> + Package names are in "reverse domain name" form with levels of + hierarchy separated by periods. For example, the Qpid messaging + broker uses package "org.apache.qpid.broker" and the Access + Control List plugin for the broker uses package + "org.apache.qpid.acl". In general, the package name should be the + reverse of the internet domain name assigned to the organization + that owns the agent software followed by identifiers to uniquely + identify the agent. + </para><para> + The XML document for a package's schema uses an enclosing + <schema> tag. For example: + </para> + <programlisting> +<schema package="org.apache.qpid.broker"> + +</schema> +</programlisting> +<!--h3--></section> + + <section role="h3" id="QpidManagementFramework-ObjectClasses"><title> + Object + Classes + </title> + + <para> + <emphasis>Object classes</emphasis> define types for manageable objects. The + agent may create and destroy objects which are instances of + object classes in the schema. An object class is defined in the + XML document using the <class> tag. An object class is + composed of properties, statistics, and methods. + </para> + <programlisting> + <class name="Exchange"> + <property name="vhostRef" type="objId" references="Vhost" access="RC" index="y" parentRef="y"/> + <property name="name" type="sstr" access="RC" index="y"/> + <property name="type" type="sstr" access="RO"/> + <property name="durable" type="bool" access="RC"/> + <property name="arguments" type="map" access="RO" desc="Arguments supplied in exchange.declare"/> + + <statistic name="producerCount" type="hilo32" desc="Current producers on exchange"/> + <statistic name="bindingCount" type="hilo32" desc="Current bindings"/> + <statistic name="msgReceives" type="count64" desc="Total messages received"/> + <statistic name="msgDrops" type="count64" desc="Total messages dropped (no matching key)"/> + <statistic name="msgRoutes" type="count64" desc="Total routed messages"/> + <statistic name="byteReceives" type="count64" desc="Total bytes received"/> + <statistic name="byteDrops" type="count64" desc="Total bytes dropped (no matching key)"/> + <statistic name="byteRoutes" type="count64" desc="Total routed bytes"/> + </class> +</programlisting> +<!--h3--></section> + + + <section role="h3" id="QpidManagementFramework-PropertiesandStatistics"><title> + Properties + and Statistics + </title> + + <para> + <property> and <statistic> tags must be placed within + <schema> and </schema> tags. + </para><para> + Properties, statistics, and methods are the building blocks of an + object class. Properties and statistics are both object + attributes, though they are treated differently. If an object + attribute is defining, seldom or never changes, or is large in + size, it should be defined as a <emphasis>property</emphasis>. If an + attribute is rapidly changing or is used to instrument the object + (counters, etc.), it should be defined as a <emphasis>statistic</emphasis>. + </para><para> + The XML syntax for <property> and <statistic> have + the following XML-attributes: + </para><table><title/><tgroup cols="4"> + <tbody> + <row> + <entry> + Attribute + </entry> + <entry> + <property> + </entry> + <entry> + <statistic> + </entry> + <entry> + Meaning + </entry> + </row> + <row> + <entry> + name + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + <entry> + The name of the attribute + </entry> + </row> + <row> + <entry> + type + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + <entry> + The data type of the attribute + </entry> + </row> + <row> + <entry> + unit + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + <entry> + Optional unit name - use the singular (i.e. MByte) + </entry> + </row> + <row> + <entry> + desc + </entry> + <entry> + Y + </entry> + <entry> + Y + </entry> + <entry> + Description to annotate the attribute + </entry> + </row> + <row> + <entry> + references + </entry> + <entry> + Y + </entry> + <entry> +  + </entry> + <entry> + If the type is "objId", names the referenced class + </entry> + </row> + <row> + <entry> + access + </entry> + <entry> + Y + </entry> + <entry> +  + </entry> + <entry> + Access rights (RC, RW, RO) + </entry> + </row> + <row> + <entry> + index + </entry> + <entry> + Y + </entry> + <entry> +  + </entry> + <entry> + "y" if this property is used to uniquely identify the + object. There may be more than one index property in a + class + </entry> + </row> + <row> + <entry> + parentRef + </entry> + <entry> + Y + </entry> + <entry> +  + </entry> + <entry> + "y" if this property references an object in which this + object is in a child-parent relationship. + </entry> + </row> + <row> + <entry> + optional + </entry> + <entry> + Y + </entry> + <entry> +  + </entry> + <entry> + "y" if this property is optional (i.e. may be + NULL/not-present) + </entry> + </row> + <row> + <entry> + min + </entry> + <entry> + Y + </entry> + <entry> +  + </entry> + <entry> + Minimum value of a numeric attribute + </entry> + </row> + <row> + <entry> + max + </entry> + <entry> + Y + </entry> + <entry> +  + </entry> + <entry> + Maximum value of a numeric attribute + </entry> + </row> + <row> + <entry> + maxLen + </entry> + <entry> + Y + </entry> + <entry> +  + </entry> + <entry> + Maximum length of a string attribute + </entry> + </row> + </tbody> + </tgroup></table> +<!--h3--></section> + + + <section role="h3" id="QpidManagementFramework-Methods"><title> + Methods + </title> + + <para> + <method> tags must be placed within <schema> and + </schema> tags. + </para><para> + A <emphasis>method</emphasis> is an invokable function to be performed on + instances of the object class (i.e. a Remote Procedure Call). A + <method> tag has a name, an optional description, and + encloses zero or more arguments. Method arguments are defined by + the <arg> tag and have a name, a type, a direction, and an + optional description. The argument direction can be "I", "O", or + "IO" indicating input, output, and input/output respectively. An + example: + </para> + <programlisting> + <method name="echo" desc="Request a response to test the path to the management broker"> + <arg name="sequence" dir="IO" type="uint32"/> + <arg name="body" dir="IO" type="lstr"/> + </method> +</programlisting> +<!--h3--></section> + + <section role="h3" id="QpidManagementFramework-EventClasses"><title> + Event Classes + </title> + <para/> + <!--h3--></section> + + <section role="h3" id="QpidManagementFramework-DataTypes"><title> + Data Types + </title> + + <para> + Object attributes, method arguments, and event arguments have + data types. The data types are based on the rich data typing + system provided by the AMQP messaging protocol. The following + table describes the data types available for QMF: + </para><table><title/><tgroup cols="2"> + <tbody> + <row> + <entry> + QMF Type + </entry> + <entry> + Description + </entry> + </row> + <row> + <entry> + REF + </entry> + <entry> + QMF Object ID - Used to reference another QMF object. + </entry> + </row> + <row> + <entry> + U8 + </entry> + <entry> + 8-bit unsigned integer + </entry> + </row> + <row> + <entry> + U16 + </entry> + <entry> + 16-bit unsigned integer + </entry> + </row> + <row> + <entry> + U32 + </entry> + <entry> + 32-bit unsigned integer + </entry> + </row> + <row> + <entry> + U64 + </entry> + <entry> + 64-bit unsigned integer + </entry> + </row> + <row> + <entry> + S8 + </entry> + <entry> + 8-bit signed integer + </entry> + </row> + <row> + <entry> + S16 + </entry> + <entry> + 16-bit signed integer + </entry> + </row> + <row> + <entry> + S32 + </entry> + <entry> + 32-bit signed integer + </entry> + </row> + <row> + <entry> + S64 + </entry> + <entry> + 64-bit signed integer + </entry> + </row> + <row> + <entry> + BOOL + </entry> + <entry> + Boolean - True or False + </entry> + </row> + <row> + <entry> + SSTR + </entry> + <entry> + Short String - String of up to 255 bytes + </entry> + </row> + <row> + <entry> + LSTR + </entry> + <entry> + Long String - String of up to 65535 bytes + </entry> + </row> + <row> + <entry> + ABSTIME + </entry> + <entry> + Absolute time since the epoch in nanoseconds (64-bits) + </entry> + </row> + <row> + <entry> + DELTATIME + </entry> + <entry> + Delta time in nanoseconds (64-bits) + </entry> + </row> + <row> + <entry> + FLOAT + </entry> + <entry> + Single precision floating point number + </entry> + </row> + <row> + <entry> + DOUBLE + </entry> + <entry> + Double precision floating point number + </entry> + </row> + <row> + <entry> + UUID + </entry> + <entry> + UUID - 128 bits + </entry> + </row> + <row> + <entry> + FTABLE + </entry> + <entry> + Field-table - std::map in C++, dictionary in Python + </entry> + </row> + </tbody> + </tgroup></table><para> + In the XML schema definition, types go by different names and + there are a number of special cases. This is because the XML + schema is used in code-generation for the agent API. It provides + options that control what kind of accessors are generated for + attributes of different types. The following table enumerates the + types available in the XML format, which QMF types they map to, + and other special handling that occurs. + </para><table><title/><tgroup cols="4"> + <tbody> + <row> + <entry> + XML Type + </entry> + <entry> + QMF Type + </entry> + <entry> + Accessor Style + </entry> + <entry> + Special Characteristics + </entry> + </row> + <row> + <entry> + objId + </entry> + <entry> + REF + </entry> + <entry> + Direct (get, set) + </entry> + <entry> +  + </entry> + </row> + <row> + <entry> + uint8,16,32,64 + </entry> + <entry> + U8,16,32,64 + </entry> + <entry> + Direct (get, set) + </entry> + <entry> +  + </entry> + </row> + <row> + <entry> + int8,16,32,64 + </entry> + <entry> + S8,16,32,64 + </entry> + <entry> + Direct (get, set) + </entry> + <entry> +  + </entry> + </row> + <row> + <entry> + bool + </entry> + <entry> + BOOL + </entry> + <entry> + Direct (get, set) + </entry> + <entry> +  + </entry> + </row> + <row> + <entry> + sstr + </entry> + <entry> + SSTR + </entry> + <entry> + Direct (get, set) + </entry> + <entry> +  + </entry> + </row> + <row> + <entry> + lstr + </entry> + <entry> + LSTR + </entry> + <entry> + Direct (get, set) + </entry> + <entry> +  + </entry> + </row> + <row> + <entry> + absTime + </entry> + <entry> + ABSTIME + </entry> + <entry> + Direct (get, set) + </entry> + <entry> +  + </entry> + </row> + <row> + <entry> + deltaTime + </entry> + <entry> + DELTATIME + </entry> + <entry> + Direct (get, set) + </entry> + <entry> +  + </entry> + </row> + <row> + <entry> + float + </entry> + <entry> + FLOAT + </entry> + <entry> + Direct (get, set) + </entry> + <entry> +  + </entry> + </row> + <row> + <entry> + double + </entry> + <entry> + DOUBLE + </entry> + <entry> + Direct (get, set) + </entry> + <entry> +  + </entry> + </row> + <row> + <entry> + uuid + </entry> + <entry> + UUID + </entry> + <entry> + Direct (get, set) + </entry> + <entry> +  + </entry> + </row> + <row> + <entry> + map + </entry> + <entry> + FTABLE + </entry> + <entry> + Direct (get, set) + </entry> + <entry> +  + </entry> + </row> + <row> + <entry> + hilo8,16,32,64 + </entry> + <entry> + U8,16,32,64 + </entry> + <entry> + Counter (inc, dec) + </entry> + <entry> + Generates value, valueMin, valueMax + </entry> + </row> + <row> + <entry> + count8,16,32,64 + </entry> + <entry> + U8,16,32,64 + </entry> + <entry> + Counter (inc, dec) + </entry> + <entry> +  + </entry> + </row> + <row> + <entry> + mma32,64 + </entry> + <entry> + U32,64 + </entry> + <entry> + Direct + </entry> + <entry> + Generates valueMin, valueMax, valueAverage, valueSamples + </entry> + </row> + <row> + <entry> + mmaTime + </entry> + <entry> + DELTATIME + </entry> + <entry> + Direct + </entry> + <entry> + Generates valueMin, valueMax, valueAverage, valueSamples + </entry> + </row> + </tbody> + </tgroup></table> + + <note><title>Important</title> + <para> + When writing a schema using the XML format, types used in + <property> or <arg> must be types that have + <emphasis>Direct</emphasis> accessor style. Any type may be used in + <statistic> tags. + </para> + </note> + +<!--h3--></section> +<!--h2--></section> + + <section role="h2" id="QpidManagementFramework-ClassKeysandClassVersioning"><title> + Class + Keys and Class Versioning + </title> + + <para/> + +<!--h2--></section> + + <!--h1--></section> + + <section role="h1" id="QpidManagementFramework-TheQMFProtocol"><title> + The QMF + Protocol + </title> + + <para> + The QMF protocol defines the message formats and communication + patterns used by the different QMF components to communicate with + one another. + </para><para> + A description of the current version of the QMF protocol can be + found at <xref linkend="qpid_QMF-20Protocol"/>. + </para><para> + A proposal for an updated protocol based on map-messages is in + progress and can be found at <xref linkend="qpid_QMF-20Map-20Message-20Protocol"/>. + </para> +<!--h1--></section> + + <section role="h1" id="QpidManagementFramework-HowtoWriteaQMFConsole"><title> + How + to Write a QMF Console + </title> + + <para> + Please see the <xref linkend="qpid_QMF-20Python-20Console-20Tutorial"/> for information about using the console API with + Python. + </para> +<!--h1--></section> + + <section role="h1" id="QpidManagementFramework-HowtoWriteaQMFAgent"><title> + How to + Write a QMF Agent + </title> + <para/> + <!--h1--></section> + + +</chapter> diff --git a/qpid/doc/book/src/RASC.xml b/qpid/doc/book/src/RASC.xml new file mode 100644 index 0000000000..fc021b8962 --- /dev/null +++ b/qpid/doc/book/src/RASC.xml @@ -0,0 +1,306 @@ +<?xml version="1.0" encoding="utf-8"?> +<chapter xmlns:html="http://www.w3.org/1999/xhtml"><title> + Apache Qpid : RASC + </title> + + <section role="h2" id="RASC-BuildingtheC-5CBrokerandClientLibraries"><title> + Building the + C++ Broker and Client Libraries + </title> + <para> + The root directory for the C++ distribution is named + qpidc-0.4. The README file in that directory gives + instructions for building the broker and client libraries. In + most cases you will do the following: + </para> + <programlisting> +[qpidc-0.4]$ ./configure}} +[qpidc-0.4]$ make +</programlisting> + <!--h2--></section> + <section role="h2" id="RASC-RunningtheC-5CBroker"><title> + Running the C++ Broker + </title> + <para> + Once you have built the broker and client libraries, you can + start the broker from the command line: + </para> + <programlisting> +[qpidc-0.4]$ src/qpidd +</programlisting> + <para> + Use the --daemon option to run the broker as a daemon + process: + </para> + <programlisting> +[qpidc-0.4]$ src/qpidd --daemon +</programlisting> + <para> + You can stop a running daemon with the --quit option: + </para> + <programlisting> +[qpidc-0.4]$ src/qpidd --quit +</programlisting> + <para> + You can see all available options with the --help option + </para> + <programlisting> +[qpidc-0.4]$ src/qpidd --help +</programlisting> + <!--h2--></section> + <section role="h2" id="RASC-Mostcommonquestionsgettingqpiddrunning"><title> + Most + common questions getting qpidd running + </title> + <section role="h3" id="RASC-Errorwhenstartingbroker-3A-22nodatadirectory-22"><title> + Error + when starting broker: "no data directory" + </title> + <para> + The qpidd broker requires you to set a data directory or specify + --no-data-dir (see help for more details). The data + directory is used for the journal, so it is important when + reliability counts. Make sure your process has write permission + to the data directory. + </para><para> + The default location is + </para> + <programlisting> +/lib/var/qpidd +</programlisting> + <para> + An alternate location can be set with --data-dir + </para> + <!--h3--></section> + <section role="h3" id="RASC-Errorwhenstartingbroker-3A-22thatprocessislocked-22"><title> + Error + when starting broker: "that process is locked" + </title> + <para> + Note that when qpidd starts it creates a lock file is data + directory are being used. If you have a un-controlled exit, + please mail + the trace from the core to the dev@qpid.apache.org mailing list. + To clear the lock run + </para> + <programlisting> +./qpidd -q +</programlisting> + <para> + It should also be noted that multiple brokers can be run on the + same host. To do so set alternate data directories for each qpidd + instance. + </para> + <!--h3--></section> + <section role="h3" id="RASC-Usingaconfigurationfile"><title> + Using a configuration + file + </title> + <para> + Each option that can be specified on the command line can also be + specified in a configuration file. To see available options, use + --help on the command line: + </para> + <programlisting> +./qpidd --help +</programlisting> + <para> + A configuration file uses name/value pairs, one on each line. To + convert a command line option to a configuration file entry: + </para><para> + a.) remove the '--' from the beginning of the option. + b.) place a '=' between the option and the value (use + <emphasis>yes</emphasis> or <emphasis>true</emphasis> to enable options that take no + value when specified on the command line). + c.) place one option per line. + </para><para> + For instance, the --daemon option takes no value, the + --log-to-syslog option takes the values yes or + no. The following configuration file sets these two + options: + </para> + <programlisting> +daemon=yes +log-to-syslog=yes +</programlisting> + <!--h3--></section> + <section role="h3" id="RASC-CanIuseanyLanguageclientwiththeC-5CBroker-3F"><title> + Can I use + any Language client with the C++ Broker? + </title> + <para> + Yes, all the clients work with the C++ broker; it is written in + C+<emphasis>, but uses the AMQP wire protocol. Any broker can be used + with any client that uses the same AMQP version. When running the + C</emphasis>+ broker, it is highly recommended to run AMQP 0-10. + </para><para> + Note that JMS also works with the C++ broker. For more details on + using the Java client refer to these pages: + </para><itemizedlist> + <listitem><para> + <xref linkend="qpid_How-20to-20Use-20JNDI"/> + </para></listitem> + <listitem><para> + <xref linkend="qpid_URL-20Formats"/> + </para></listitem> + <listitem><para> + <xref linkend="qpid_Example-20Classes"/> + </para></listitem> + </itemizedlist> + <!--h3--></section> + <!--h2--></section> + <section role="h2" id="RASC-Authentication"><title> + Authentication + </title> + <section role="h3" id="RASC-Linux"><title> + Linux + </title> + <para> + The PLAIN authentication is done on a username+password, which is + stored in the sasldb_path file. Usernames and passwords can be + added to the file using the command: + </para> + <programlisting> +saslpasswd2 -f /var/lib/qpidd/qpidd.sasldb -u <REALM> <USER> +</programlisting> + <para> + The REALM is important and should be the same as the + --auth-realm + option to the broker. This lets the broker properly find the user + in + the sasldb file. + </para><para> + Existing user accounts may be listed with: + </para> + <programlisting> +sasldblistusers2 -f /var/lib/qpidd/qpidd.sasldb +</programlisting> + <para> + NOTE: The sasldb file must be readable by the user running the + qpidd daemon, and should be readable only by that user. + </para> + <!--h3--></section> + <section role="h3" id="RASC-Windows"><title> + Windows + </title> + <para> + On Windows, the users are authenticated against the local + machine. You should add the appropriate users using the standard + Windows tools (Control Panel->User Accounts). To run many of + the examples, you will need to create a user "guest" with + password "guest". + </para><para> + If you cannot or do not want to create new users, you can run + without authentication by specifying the no-auth option to the + broker. + </para> + <!--h3--></section> + <!--h2--></section> + + <section role="h2" id="RASC-Slightlymorecomplexconfiguration"><title> + Slightly more + complex configuration + </title> + <para> + The easiest way to get a full listing of the broker's options are + to use the --help command, run it locally for the latest set of + options. These options can then be set in the conf file for + convenience (see above) + </para> + <programlisting> +./qpidd --help + +Usage: qpidd OPTIONS +Options: + -h [ --help ] Displays the help message + -v [ --version ] Displays version information + --config FILE (/etc/qpidd.conf) Reads configuration from FILE + +Module options: + --module-dir DIR (/usr/lib/qpidd) Load all .so modules in this directory + --load-module FILE Specifies additional module(s) to be loaded + --no-module-dir Don't load modules from module directory + +Broker Options: + --data-dir DIR (/var/lib/qpidd) Directory to contain persistent data generated by the broker + --no-data-dir Don't use a data directory. No persistent + configuration will be loaded or stored + -p [ --port ] PORT (5672) Tells the broker to listen on PORT + --worker-threads N (3) Sets the broker thread pool size + --max-connections N (500) Sets the maximum allowed connections + --connection-backlog N (10) Sets the connection backlog limit for the + server socket + --staging-threshold N (5000000) Stages messages over N bytes to disk + -m [ --mgmt-enable ] yes|no (1) Enable Management + --mgmt-pub-interval SECONDS (10) Management Publish Interval + --ack N (0) Send session.ack/solicit-ack at least every + N frames. 0 disables voluntary ack/solitict + -ack + +Daemon options: + -d [ --daemon ] Run as a daemon. + -w [ --wait ] SECONDS (10) Sets the maximum wait time to initialize the + daemon. If the daemon fails to initialize, prints + an error and returns 1 + -c [ --check ] Prints the daemon's process ID to stdout and + returns 0 if the daemon is running, otherwise + returns 1 + -q [ --quit ] Tells the daemon to shut down +Logging options: + --log-output FILE (stderr) Send log output to FILE. FILE can be a file name + or one of the special values: + stderr, stdout, syslog + -t [ --trace ] Enables all logging + --log-enable RULE (error+) Enables logging for selected levels and component + s. RULE is in the form 'LEVEL+:PATTERN' + Levels are one of: + trace debug info notice warning error critical + For example: + '--log-enable warning+' logs all warning, error + and critical messages. + '--log-enable debug:framing' logs debug messages + from the framing namespace. This option can be + used multiple times + --log-time yes|no (1) Include time in log messages + --log-level yes|no (1) Include severity level in log messages + --log-source yes|no (0) Include source file:line in log messages + --log-thread yes|no (0) Include thread ID in log messages + --log-function yes|no (0) Include function signature in log messages +</programlisting> + <!--h2--></section> + <section role="h2" id="RASC-Loadingextramodules"><title> + Loading extra modules + </title> + <para> + By default the broker will load all the modules in the module + directory, however it will NOT display options for modules that + are not loaded. So to see the options for extra modules loaded + you need to load the module and then add the help command like + this: + </para> + <programlisting> +./qpidd --load-module libbdbstore.so --help +Usage: qpidd OPTIONS +Options: + -h [ --help ] Displays the help message + -v [ --version ] Displays version information + --config FILE (/etc/qpidd.conf) Reads configuration from FILE + + + / .... non module options would be here ... / + + +Store Options: + --store-directory DIR Store directory location for persistence (overrides + --data-dir) + --store-async yes|no (1) Use async persistence storage - if store supports + it, enables AIO O_DIRECT. + --store-force yes|no (0) Force changing modes of store, will delete all + existing data if mode is changed. Be SURE you want + to do this! + --num-jfiles N (8) Number of files in persistence journal + --jfile-size-pgs N (24) Size of each journal file in multiples of read + pages (1 read page = 64kiB) +</programlisting> +<!--h2--></section> +</chapter> diff --git a/qpid/doc/book/src/SASL_Compatibility.xml b/qpid/doc/book/src/SASL_Compatibility.xml new file mode 100644 index 0000000000..96c523e519 --- /dev/null +++ b/qpid/doc/book/src/SASL_Compatibility.xml @@ -0,0 +1,50 @@ +h2. Qpid Interoperability Documentation + +This page documents the various interoperable features of the Qpid clients. + + +h3. SASL +{anchor:sasl} +h4. Standard Mechanisms +[SASL Mechanisms |http://en.wikipedia.org/wiki/Simple_Authentication_and_Security_Layer#SASL_Mechanisms] + +This table list the various SASL mechanisms that each component supports. The version listed shows when this +functionality was added to the product. + +|| Component || ANONYMOUS || CRAM-MD5 || DIGEST-MD5 || EXTERNAL || GSSAPI/Kerberos || PLAIN || +| C++ Broker |M3\[[#1]\] |M3\[[#1],[#2]\]| | |M3\[[#1],[#2]\] | M1 | +| C++ Client |M3\[[#1]\] | | | | | M1 | +| Java Broker | | M1 | | | | M1 | +| Java Client | | M1 | | | | M1 | +| .Net Client | M2 | M2 | M2 | M2 | | M2 | +| Python Client | | | | | | ? | +| Ruby Client | | | | | | ? | + +{anchor:1} +1: Support for these will be in M3 (currently available on trunk). +{anchor:2} +2: C++ Broker uses [Cyrus Sasl|http://freshmeat.net/projects/cyrussasl/] which supports CRAM-MD5 and GSSAPI but these have not been tested yet + +h4. Custom Mechanisms + +There have been some custom mechanisms added to our implementations. + +|| Component || AMQPLAIN || CRAM-MD5-HASHED || +| C++ Broker | | | +| C++ Client | | | +| Java Broker | M1 | M2 | +| Java Client | M1 | M2 | +| .Net Client | | | +| Python Client | M2 | | +| Ruby Client | M2 | | + + +h5. AMQPLAIN + +h5. CRAM-MD5-HASHED + +The Java SASL implementations require that you have the password of the user to validate the incoming request. This then means that the user's password must be stored on disk. For this to be secure either the broker must encrypt the password file or the need for the password being stored must be removed. + +The CRAM-MD5-HASHED SASL plugin removes the need for the plain text password to be stored on disk. The mechanism defers all functionality to the build in CRAM-MD5 module the only change is on the client side where it generates the hash of the password and uses that value as the password. This means that the Java Broker only need store the password hash on the file system. While a one way hash is not very secure compared to other forms of encryption in environments where the having the password in plain text is unacceptable this will provide and additional layer to protect the password. In particular this offers some protection where the same password may be shared amongst many systems. It offers no real extra protection against attacks on the broker (the secret is now the hash rather than the password). + + diff --git a/qpid/doc/book/src/SSL.xml b/qpid/doc/book/src/SSL.xml new file mode 100644 index 0000000000..7eb188e921 --- /dev/null +++ b/qpid/doc/book/src/SSL.xml @@ -0,0 +1,159 @@ +<?xml version="1.0" encoding="utf-8"?> +<chapter xmlns:html="http://www.w3.org/1999/xhtml"><title> + Apache Qpid : SSL + </title> + + <section role="h1" id="SSL-SSLHowto"><title> + SSL How to + </title> + + <section role="h2" id="SSL-C-5Cbroker-28M4andup-29"><title> + C++ broker (M4 and up) + </title> + <itemizedlist> + <listitem><para>You need to get a certificate signed by a CA, trusted by your + client. + </para></listitem> + </itemizedlist><itemizedlist> + <listitem><para>If you require client authentication, the clients certificate + needs to be signed by a CA trusted by the broker. + </para></listitem> + </itemizedlist><itemizedlist> + <listitem><para>Setting up the certificates for testing. + <itemizedlist> + <listitem><para>For testing purposes you could use the <xref linkend="qpid_gtstd"/> to setup your certificates. + </para></listitem> + <listitem><para>In summary you need to create a root CA and import it to + the brokers certificate data base. + </para></listitem> + <listitem><para>Create a certificate for the broker, sign it using the + root CA and then import it into the brokers certificate data + base. + </para></listitem> + </itemizedlist> + </para></listitem> + </itemizedlist><itemizedlist> + <listitem><para>Load the acl module using --load-module or if loading more + than one module, copy ssl.so to the location pointed by + --module-dir + + <programlisting> +Ex if running from source. ./qpidd --load-module /libs/ssl.so +</programlisting> + + </para></listitem> + </itemizedlist><itemizedlist> + <listitem><para>Specify the password file (a plain text file with the + password), certificate database and the brokers certificate name + using the following options + + <programlisting> +Ex ./qpidd ... --ssl-cert-password-file ~/pfile --ssl-cert-db ~/server_db/ --ssl-cert-name localhost.localdomain +</programlisting> + + </para></listitem> + </itemizedlist><itemizedlist> + <listitem><para>If you require client authentication you need to add + --ssl-require-client-authentication as a command line argument. + </para></listitem> + </itemizedlist><itemizedlist> + <listitem><para>Please note that the default port for SSL connections is + 5671, unless specified by --ssl-port + </para></listitem> + </itemizedlist><para> + Here is an example of a broker instance that requires SSL client + side authenticaiton + </para> + <programlisting> +./qpidd ./qpidd --load-module /libs/ssl.so --ssl-cert-password-file ~/pfile --ssl-cert-db ~/server_db/ --ssl-cert-name localhost.localdomain --ssl-require-client-authentication +</programlisting> + <!--h2--></section> + <section role="h2" id="SSL-JavaClient-28M4andup-29"><title> + Java Client (M4 and up) + </title> + <itemizedlist> + <listitem><para>This guide is for connecting with the Qpid c++ broker. + </para></listitem> + </itemizedlist><itemizedlist> + <listitem><para>Setting up the certificates for testing. In summary, + <itemizedlist> + <listitem><para>You need to import the trusted CA in your trust store and + keystore + </para></listitem> + <listitem><para>Generate keys for the certificate in your key store + </para></listitem> + <listitem><para>Create a certificate request using the generated keys + </para></listitem> + <listitem><para>Create a certficate using the request, signed by the + trusted CA. + </para></listitem> + <listitem><para>Import the signed certificate into your keystore. + </para></listitem> + </itemizedlist> + </para></listitem> + </itemizedlist><itemizedlist> + <listitem><para>Pass the following JVM arguments to your client. + + <programlisting> +-Djavax.net.ssl.keyStore=/home/bob/ssl_test/keystore.jks + -Djavax.net.ssl.keyStorePassword=password + -Djavax.net.ssl.trustStore=/home/bob/ssl_test/certstore.jks + -Djavax.net.ssl.trustStorePassword=password +</programlisting> + + </para></listitem> + </itemizedlist> + <!--h2--></section> + + <section role="h2" id="SSL-.NetClient-28M4andup-29"><title> + .Net Client (M4 and up) + </title> + <itemizedlist> + <listitem><para>If the Qpid broker requires client authentication then you + need to get a certificate signed by a CA, trusted by your client. + </para></listitem> + </itemizedlist><para> + Use the connectSSL instead of the standard connect method of the + client interface. + </para><para> + connectSSL signature is as follows: + </para> + <programlisting> +public void connectSSL(String host, int port, String virtualHost, String username, String password, String serverName, String certPath, bool rejectUntrusted) +</programlisting> + <para> + Where + </para><itemizedlist> + <listitem><para>host: Host name on which a Qpid broker is deployed + </para></listitem> + <listitem><para>port: Qpid broker port + </para></listitem> + <listitem><para>virtualHost: Qpid virtual host name + </para></listitem> + <listitem><para>username: User Name + </para></listitem> + <listitem><para>password: Password + </para></listitem> + <listitem><para>serverName: Name of the SSL server + </para></listitem> + </itemizedlist><itemizedlist> + <listitem><para>certPath: Path to the X509 certificate to be used when the + broker requires client authentication + </para></listitem> + <listitem><para>rejectUntrusted: If true connection will not be established + if the broker is not trusted (the server certificate must be + added in your truststore) + </para></listitem> + </itemizedlist> + <!--h2--></section> + + <section role="h2" id="SSL-Python-26RubyClient-28M4andup-29"><title> + Python & + Ruby Client (M4 and up) + </title> + <para> + Simply use amqps:// in the URL string as defined above + </para> + <!--h2--></section> + <!--h1--></section> +</chapter> diff --git a/qpid/doc/book/src/Starting a cluster.xml b/qpid/doc/book/src/Starting a cluster.xml new file mode 100644 index 0000000000..71d062a4cd --- /dev/null +++ b/qpid/doc/book/src/Starting a cluster.xml @@ -0,0 +1,204 @@ +<?xml version="1.0" encoding="utf-8"?> +<chapter xmlns:html="http://www.w3.org/1999/xhtml"><title> + Apache Qpid : Starting a cluster + </title><section role="h1" id="Startingacluster-RunningaQpiddcluster"><title> + Running a + Qpidd cluster + </title> + <para> + There are several pre-requisites to running a qpidd cluster: + </para><section role="h2" id="Startingacluster-Installandconfigureopenais-2Fcorosync"><title> + Install + and configure openais/corosync + </title> + <para> + Qpid clustering uses a multicast protocol provided by the + corosync (formerly called openais) library. Install whichever is + available on your OS. + E.g. in fedora10: yum install corosync. + </para><para> + The configuration file is /etc/ais/openais.conf on openais, + /etc/corosync.conf on early corosync versions and + /etc/corosync/corosync.conf on recent corosync versions. You will + need to edit the default file created when you installed + </para><para> + Here is an example, with places marked that you will + change. ( Below, I will describe how to change the file. ) + </para> + <programlisting> +# Please read the openais.conf.5 manual page + +totem { + version: 2 + secauth: off + threads: 0 + interface { + ringnumber: 0 + ## You must change this address ## + bindnetaddr: 20.0.100.0 + mcastaddr: 226.94.32.36 + mcastport: 5405 + } +} + +logging { + debug: off + timestamp: on + to_file: yes + logfile: /tmp/aisexec.log +} + +amf { + mode: disabled +} +</programlisting> + <para> + You must sent the bindnetaddr entry in the configuration file to + the network address of your network interface. This must be a + real network interface, not the loopback address 127.0.0.1 + </para><para> + You can find your network interface by running ifconfig. This + will list the address and the mask, e.g. + </para> + <programlisting> +inet addr:20.0.20.32 Bcast:20.0.20.255 Mask:255.255.255.0 +</programlisting> + <para> + The bindnetaddr is the logical AND of the inet addr and mask + values, in the example above 20.0.20.0 + </para> +<!--h2--></section> + + <section role="h2" id="Startingacluster-Openyourfirewall"><title> + Open your + firewall + </title> + <para> + In the above example file, I use mcastport 5405. + This implies that your firewall must allow UDP + protocol over port 5405, or that you disable the firewall + </para> + <!--h2--></section> + + <section role="h2" id="Startingacluster-Usetheproperidentity."><title> + Use the + proper identity. + </title> + <para> + The qpidd process must be started with the correct identity in + order to use the corosync/openais library. + </para><para> + For openais and early corosync versions the installation of + openAIS/corosync on your system will create a new + group called "ais". The user that starts the qpidd processes of + the cluster + must have "ais" as its effective group id. You can create a user + specifically for this purpose with ais as the primary group, + or + a user that has ais as a secondary group can use "newgrp" to set + the primary group to ais when running qpidd. + </para><para> + For recent corosync versions you no longer need to set your group + to "ais" but you do need to create a file in + /etc/corosync/uidgid.d/ to allow access for whatever user/group + ID you want to use. For example create + /etc/corosync/uidgid.d/qpid th the contents: + </para> + <programlisting> +uidgid { + uid: qpid + gid: qpid +} +</programlisting> +<!--h2--></section> + + + <section role="h2" id="Startingacluster-StartingaCluster"><title> + Starting a + Cluster + </title> + <para> + To be a member of a cluster you must pass the --cluster-name + argument to qpidd. This is the only required option to join + a cluster, other options can be set as for a normal qpidd. + </para><para> + For example to start a cluster of 3 brokers on the current + host + Here is an example of starting a cluster of 3 members, all on the + current host but with different ports and different log files: + </para> + <programlisting> +qpidd -p5672 --cluster-name=MY_CLUSTER --log-output=cluster0.log -d --no-data-dir --auth=no +qpidd -p5673 --cluster-name=MY_CLUSTER --log-output=cluster0.log -d --no-data-dir --auth=no +qpidd -p5674 --cluster-name=MY_CLUSTER --log-output=cluster0.log -d --no-data-dir --auth=no +</programlisting> + <para> + In a deployed system, cluster members will normally be on + different hosts but for development its useful to be able to + create a cluster on a single host. + </para> + <!--h2--></section> + + <section role="h2" id="Startingacluster-SELinuxconflicts"><title> + SELinux conflicts + </title> + <para> + Developers will often start openais/corosync as a service like + this: + </para><para> + service openais start + </para><para> + But will then will start a cluster-broker without using the + service script like this: + </para><para> + /usr/sbin/qpidd --cluster-name my_cluster ... + </para><para> + If SELinux is in enforcing mode this may cause qpidd to hang due + because of the different SELinux contexts. + There are 3 ways to resolve this: + </para><itemizedlist> + <listitem><para>run both qpidd and openais/corosync as services. + </para></listitem> + <listitem><para>run both qpidd and openais/corosync as user processes. + </para></listitem> + <listitem><para>make selinux permissive: + </para></listitem> + </itemizedlist><para> + To check what mode selinux is running: + </para> + <programlisting> +# getenforce +</programlisting> + <para> + To change the mode: + </para> + <programlisting> +# setenforce permissive +</programlisting> + <para> + Note that in a deployed system both openais/corosync and qpidd + should be started as services, in which case there is no problem + with SELinux running in enforcing mode. + </para> + <!--h2--></section> + + <section role="h2" id="Startingacluster-Troubleshootingchecklist."><title> + Troubleshooting + checklist. + </title> + <para> + If you have trouble starting your cluster, make sure that: + </para><para> + 1. You have edited the correct openais/corosync configuration + file and set bindnetaddr correctly + 1. Your firewall allows UDP on the openais/corosync + mcastport + 2. Your effective group is "ais" (openais/old corosync) or you + have created an appropriate ID file (new corosync) + 3. Your firewall allows TCP on the ports used by qpidd. + 4. If you're starting openais as a service but running qpidd + directly, ensure selinux is in permissive mode + </para> +<!--h2--></section> +<!--h1--></section> +</chapter> diff --git a/qpid/doc/book/src/Using Broker Federation.xml b/qpid/doc/book/src/Using Broker Federation.xml new file mode 100644 index 0000000000..1af7491a4d --- /dev/null +++ b/qpid/doc/book/src/Using Broker Federation.xml @@ -0,0 +1,681 @@ +<?xml version="1.0" encoding="utf-8"?> +<chapter xmlns:html="http://www.w3.org/1999/xhtml"><title> + Apache Qpid : Using Broker Federation + </title><section role="h1" id="UsingBrokerFederation-Introduction"><title> + Introduction + </title> + <para> + Please note: Whereas broker federation was introduced in the M3 + milestone release, the discussion in this document is based on + the richer capabilities of federation in the M4 release. + </para><para> + This document presents broker federation for the administrative + user. For design and developer information, please see <xref linkend="qpid_Federation-20Design-20Note"/>. + </para> + <!--h1--></section> + + <section role="h1" id="UsingBrokerFederation-WhatIsBrokerFederation-3F"><title> + What Is + Broker Federation? + </title> + <para> + The Qpid C++ messaging broker supports broker federation, a + mechanism by which large messaging networks can be built using + multiple brokers. Some scenarios in which federation is useful: + </para><itemizedlist> + <listitem><para> + <emphasis>Connecting disparate locations across a wide area + network.</emphasis> In this case full connectivity across the + enterprise can be achieved while keeping local message traffic + isolated to a single location. + </para></listitem> + <listitem><para> + <emphasis>Departmental brokers</emphasis> that have a policy which + controls the flow of inter-departmental message traffic. + </para></listitem> + <listitem><para> + <emphasis>Scaling of capacity</emphasis> for expensive broker operations. + High-function exchanges like the XML exchange can be replicated + to scale performance. + </para></listitem> + <listitem><para> + <emphasis>Co-Resident brokers</emphasis> Some applications benefit from + having a broker co-resident with the client. This is + particularly true if the client produces data that must be + delivered reliably but connectivity to the consumer(s) is + non-reliable. In this case, a co-resident broker provides + queueing and durablilty not available in the client alone. + </para></listitem> + <listitem><para> + <emphasis>Bridging disjoint IP networks.</emphasis> Message brokers can + be configured to allow message connectivity between networks + where there is no IP connectivity. For example, an isolated, + private IP network can have messaging connectivity to brokers + in other outside IP networks. + </para></listitem> + </itemizedlist> + <!--h1--></section> + + <section role="h1" id="UsingBrokerFederation-TheqpidrouteUtility"><title> + The + qpid-route Utility + </title> + <para> + The qpid-route command line utility is provided with the Qpid + broker. This utility is used to configure federated networks of + brokers and to view the status and topology of networks. + </para><para> + qpid-route accesses the managed brokers remotely. It does not + need to be invoked from the same host on which the broker is + running. If network connectivity permits, an entire enterprise + can be configured from a single location. + </para><para> + In the following sections, federation concepts will be introduced + and illustrated using qpid-route. + </para><section role="h2" id="UsingBrokerFederation-LinksandRoutes"><title> + Links and + Routes + </title> + <para> + Federation occurs when a <emphasis>link</emphasis> is established between two + brokers and one or more <emphasis>routes</emphasis> are created within that + link. A <emphasis>link</emphasis> is a transport level connection (tcp, rdma, + ssl, etc.) initiated by one broker and accepted by another. The + initiating broker assumes the role of <emphasis>client</emphasis> with regard + to the connection. The accepting broker annotates the connection + as being for federation but otherwise treats it as a normal + client connection. + </para><para> + A <emphasis>route</emphasis> is associated with an AMQP session established + over the link connection. There may be multiple routes sharing + the same link. A route controls the flow of messages across the + link between brokers. Routes always consist of a session and a + subscription for consuming messages. Depending on the + configuration, a route may have a private queue on the source + broker with a binding to an exchange on that broker. + </para><para> + Routes are unidirectional. A single route provides for the flow + of messages in one direction across a link. If bidirectional + connectivity is required (and it almost always is), then a pair + of routes must be created, one for each direction of message + flow. + </para><para> + The qpid-route utility allows the administrator to configure and + manage links and routes separately. However, when a route is + created and a link does not already exist, qpid-route will + automatically create the link. It is typically not necessary to + create a link by itself. It is, however, useful to get a list of + links and their connection status from a broker: + </para> + <programlisting> +$ qpid-route link list localhost:10001 + +Host Port Transport Durable State Last Error +============================================================================= +localhost 10002 tcp N Operational +localhost 10003 tcp N Operational +localhost 10009 tcp N Waiting Connection refused +</programlisting> + <para> + The example above shows a <emphasis>link list</emphasis> query to the + broker at "localhost:10001". In the example, this broker has + three links to other brokers. Two are operational and the third + is waiting to connect because there is not currently a broker + listening at that address. + </para><section role="h3" id="UsingBrokerFederation-TheLifeCycleofaLink"><title> + The Life + Cycle of a Link + </title> + <para> + When a link is created on a broker, that broker attempts to + establish a transport-level connection to the peer broker. If it + fails to connect, it retries the connection at an increasing time + interval. If the connection fails due to authentication failure, + it will not continue to retry as administrative intervention is + needed to fix the problem. + </para><para> + If an operational link is disconnected, the initiating broker + will attempt to re-establish the connection with the same + interval back-off. + </para><para> + The shortest retry-interval is 2 seconds and the longest is 64 + seconds. Once enough consecutive retries have occurred that the + interval has grown to 64 seconds, the interval will then stay at + 64 seconds. + </para> + <!--h3--></section> + + <section role="h3" id="UsingBrokerFederation-DurableLinksandRoutes"><title> + Durable + Links and Routes + </title> + <para> + If, when a link or a route is created using qpid-route, the + --durable option is used, it shall be durable. This + means that its life cycle shall span restarts of the broker. If + the broker is shut down, when it is restarted, the link will be + restored and will begin establishing connectivity. + </para><para> + A non-durable route can be created for a durable link but a + durable route cannot be created for a non-durable link. + </para> + <programlisting> +$ qpid-route dynamic add localhost:10003 localhost:10004 fed.topic +$ qpid-route dynamic add localhost:10003 localhost:10004 fed.topic2 --durable +Failed: Can't create a durable route on a non-durable link +</programlisting> + <para> + In the above example, a transient (non-durable) dynamic route was + created between localhost:10003 and localhost:10004. Because + there was no link in place, a new transient link was created. The + second command is attempting to create a durable route over the + same link and is rejected as illegal. + </para> + <!--h3--></section> + <!--h2--></section> + + <section role="h2" id="UsingBrokerFederation-DynamicRouting"><title> + Dynamic + Routing + </title> + <para> + Dynamic routing provides the simplest configuration for a network + of brokers. When configuring dynamic routing, the administrator + need only express the logical topology of the network (i.e. which + pairs of brokers are connected by a unidirectional route). Queue + configuration and bindings are handled automatically by the + brokers in the network. + </para><para> + Dynamic routing uses the <emphasis>Distributed Exchange</emphasis> concept. + From the client's point of view, all of the brokers in the + network collectively offer a single logical exchange that behaves + the same as a single exchange in a single broker. Each client + connects to its local broker and can bind its queues to the + distributed exchange and publish messages to the exchange. + </para><para> + When a consuming client binds a queue to the distributed + exchange, information about that binding is propagated to the + other brokers in the network to ensure that any messages matching + the binding will be forwarded to the client's local broker. + Messages published to the distributed exchange are forwarded to + other brokers only if there are remote consumers to receive the + messages. The dynamic binding protocol ensures that messages are + routed only to brokers with eligible consumers. This includes + topologies where messages must make multiple hops to reach the + consumer. + </para><para> + When creating a dynamic routing network, The type and name of the + exchange must be the same on each broker. It is <emphasis>strongly</emphasis> + recommended that dynamic routes <emphasis>NOT</emphasis> be created using the + standard exchanges (that is unless all messaging is intended to + be federated). + </para><para> + A simple, two-broker network can be configured by creating an + exchange on each broker then a pair of dynamic routes (one for + each direction of message flow): + </para><para> + Create exchanges: + </para> + <programlisting> +$ qpid-config -a localhost:10003 add exchange topic fed.topic +$ qpid-config -a localhost:10004 add exchange topic fed.topic +</programlisting> + <para> + Create dynamic routes: + </para> + <programlisting> +$ qpid-route dynamic add localhost:10003 localhost:10004 fed.topic +$ qpid-route dynamic add localhost:10004 localhost:10003 fed.topic +</programlisting> + <para> + Information about existing routes can be gotten by querying each + broker individually: + </para> + <programlisting> +$ qpid-route route list localhost:10003 +localhost:10003 localhost:10004 fed.topic <dynamic> +$ qpid-route route list localhost:10004 +localhost:10004 localhost:10003 fed.topic <dynamic> +</programlisting> + <para> + A nicer way to view the topology is to use <emphasis>qpid-route route + map</emphasis>. The argument to this command is a single broker that + serves as an entry point. <emphasis>qpid-route</emphasis> will attempt to + recursively find all of the brokers involved in federation + relationships with the starting broker and map all of the routes + it finds. + </para> + <programlisting> +$ qpid-route route map localhost:10003 + +Finding Linked Brokers: + localhost:10003... Ok + localhost:10004... Ok + +Dynamic Routes: + + Exchange fed.topic: + localhost:10004 <=> localhost:10003 + +Static Routes: + none found +</programlisting> + <para> + More extensive and realistic examples are supplied later in this + document. + </para> + <!--h2--></section> + + <section role="h2" id="UsingBrokerFederation-StaticRouting"><title> + Static Routing + </title> + <para> + Dynamic routing provides simple, efficient, and automatic + handling of the bindings that control routing as long as the + configuration keeps within a set of constraints (i.e. exchanges + of the same type and name, bidirectional traffic flow, etc.). + However, there are scenarios where it is useful for the + administrator to have a bit more control over the details. In + these cases, static routing is appropriate. + </para> + + <section role="h3" id="UsingBrokerFederation-ExchangeRoutes"><title> + Exchange + Routes + </title> + <para> + An exchange route is like a dynamic route except that the + exchange binding is statically set at creation time instead of + dynamically tracking changes in the network. + </para><para> + When an exchange route is created, a private queue (auto-delete, + exclusive) is declared on the source broker. The queue is bound + to the indicated exchange with the indicated key and the + destination broker subscribes to the queue with a destination of + the indicated exchange. Since only one exchange name is supplied, + this means that exchange routes require that the source and + destination exchanges have the same name. + </para><para> + Static exchange routes are added and deleted using <emphasis>qpid-route + route add</emphasis> and <emphasis>qpid-route route del</emphasis> respectively. The + following example creates a static exchange route with a binding + key of "global.#" on the default topic exchange: + </para> + <programlisting> +$ qpid-route route add localhost:10001 localhost:10002 amq.topic global.# +</programlisting> + <para> + The route can be viewed by querying the originating broker (the + destination in this case, see discussion of push and pull routes + for more on this): + </para> + <programlisting> +$ qpid-route route list localhost:10001 +localhost:10001 localhost:10002 amq.topic global.# +</programlisting> + <para> + Alternatively, the <emphasis>route map</emphasis> feature can be used to view + the topology: + </para> + <programlisting> +$ qpid-route route map localhost:10001 + +Finding Linked Brokers: + localhost:10001... Ok + localhost:10002... Ok + +Dynamic Routes: + none found + +Static Routes: + + localhost:10001(ex=amq.topic) <= localhost:10002(ex=amq.topic) key=global.# +</programlisting> + <para> + This example causes messages delivered to the <emphasis>amq.topic</emphasis> + exchange on broker <emphasis>localhost:10002</emphasis> that have a key that + matches <emphasis>global.#</emphasis> (i.e. starts with the string "global.") + to be delivered to the <emphasis>amq.topic</emphasis> exchange on broker + <emphasis>localhost:10001</emphasis>. This delivery will occur regardless of + whether there are any consumers on <emphasis>localhost:10001</emphasis> that + will receive the messages. + </para><para> + Note that this is a uni-directional route. No messages will be + forwarded in the opposite direction unless another static route + is created in the other direction. + </para><para> + The following diagram illustrates the result, in terms of AMQP + objects, of the example static exchange route. In this diagram, + the exchanges, both named "amq.topic" exist prior to the creation + of the route. The creation of the route causes the private queue, + the binding, and the subscription of the queue to the destination + to be created. + </para> + <programlisting> + -------------------------------------------------+ +------------------------ + localhost:10002 | | localhost:10001 + | | + +-------------+ | | +-------------+ + | | | | | | + | | global.# ---------------+ | | | | + | amq.topic |-----------> private queue |--------------->| amq.topic | + | | ---------------+ | | | | + | | | | | | + +-------------+ | | +-------------+ + | | + | | + -------------------------------------------------+ +------------------------ +</programlisting> +<!--h3--></section> + + <section role="h3" id="UsingBrokerFederation-QueueRoutes"><title> + Queue Routes + </title> + <para> + A queue route causes the destination broker to create a + subscription to a pre-existing, possibly shared, queue on the + source broker. There's no requirement that the queue be bound to + any particular exchange. Queue routes can be used to connect + exchanges of different names and/or types. They can also be used + to distribute or balance traffic across multiple destination + brokers. + </para><para> + Queue routes are created and deleted using the <emphasis>qpid-route + queue add</emphasis> and <emphasis>qpid-route queue del</emphasis> commands + respectively. The following example creates a static queue route + to a public queue called "public" that feeds the amq.fanout + exchange on the destination: + </para><para> + Create a queue on the source broker: + </para> + <programlisting> +$ qpid-config -a localhost:10002 add queue public +</programlisting> + <para> + Create a queue route to the new queue + </para> + <programlisting> +$ qpid-route queue add localhost:10001 localhost:10002 amq.fanout public +</programlisting> + <!--h3--></section> + <section role="h3" id="UsingBrokerFederation-Pullvs.PushRoutes"><title> + Pull vs. + Push Routes + </title> + <para> + When qpid-route creates or deletes a route, it establishes a + connection to one of the brokers involved in the route and + configures that broker. The configured broker then takes it upon + itself to contact the other broker and exchange whatever + information is needed to complete the setup of the route. + </para><para> + The notion of <emphasis>push</emphasis> vs. <emphasis>pull</emphasis> is concerned with + whether the configured broker is the source or the destination. + The normal case is the pull route, where qpid-route configures + the destination to pull messages from the source. A push route + occurs when qpid-route configures the source to push messages to + the destination. + </para><para> + Dynamic routes are always pull routes. Static routes are normally + pull routes but may be inverted by using the src-local + option when creating (or deleting) a route. If src-local + is specified, qpid-route will make its connection to the source + broker rather than the destination and configure the route to + push rather than pull. + </para><para> + Push routes are useful in applications where brokers are + co-resident with data sources and are configured to send data to + a central broker. Rather than configure the central broker for + each source, the sources can be configured to send to the + destination. + </para> + <!--h3--></section> + <!--h2--></section> + + <section role="h2" id="UsingBrokerFederation-qpidrouteSummaryandOptions"><title> + qpid-route + Summary and Options + </title> + + <programlisting> +$ qpid-route +Usage: qpid-route [OPTIONS] dynamic add <dest-broker> <src-broker> <exchange> [tag] [exclude-list] + qpid-route [OPTIONS] dynamic del <dest-broker> <src-broker> <exchange> + + qpid-route [OPTIONS] route add <dest-broker> <src-broker> <exchange> <routing-key> [tag] [exclude-list] + qpid-route [OPTIONS] route del <dest-broker> <src-broker> <exchange> <routing-key> + qpid-route [OPTIONS] queue add <dest-broker> <src-broker> <exchange> <queue> + qpid-route [OPTIONS] queue del <dest-broker> <src-broker> <exchange> <queue> + qpid-route [OPTIONS] route list [<dest-broker>] + qpid-route [OPTIONS] route flush [<dest-broker>] + qpid-route [OPTIONS] route map [<broker>] + + qpid-route [OPTIONS] link add <dest-broker> <src-broker> + qpid-route [OPTIONS] link del <dest-broker> <src-broker> + qpid-route [OPTIONS] link list [<dest-broker>] + +Options: + --timeout seconds (10) Maximum time to wait for broker connection + -v [ --verbose ] Verbose output + -q [ --quiet ] Quiet output, don't print duplicate warnings + -d [ --durable ] Added configuration shall be durable + -e [ --del-empty-link ] Delete link after deleting last route on the link + -s [ --src-local ] Make connection to source broker (push route) + --ack N Acknowledge transfers over the bridge in batches of N + -t <transport> [ --transport <transport>] + Specify transport to use for links, defaults to tcp + + dest-broker and src-broker are in the form: [username/password@] hostname | ip-address [:<port>] + ex: localhost, 10.1.1.7:10000, broker-host:10000, guest/guest@localhost +</programlisting> + <para> + There are several transport options available for the federation + link: + </para><table><title/><tgroup cols="2"> + <tbody> + <row> + <entry> + Transport + </entry> + <entry> + Description + </entry> + </row> + <row> + <entry> + tcp + </entry> + <entry> + (default) A cleartext TCP connection + </entry> + </row> + <row> + <entry> + ssl + </entry> + <entry> + A secure TLS/SSL over TCP connection + </entry> + </row> + <row> + <entry> + rdma + </entry> + <entry> + A Connection using the RDMA interface (typically for an + Infiniband network) + </entry> + </row> + </tbody> + </tgroup></table><para> + The <emphasis>tag</emphasis> and <emphasis>exclude-list</emphasis> arguments are not + needed. They have been left in place for backward compatibility + and for advanced users who might have very unusual requirements. + If you're not sure if you need them, you don't. Leave them alone. + If you must know, please refer to "Message Loop Prevention" in + the advanced topics section below. The prevention of message + looping is now automatic and requires no user action. + </para><para> + If the link between the two sites has network latency, this can + be compensated for by increasing the ack frequency with --ack N + to achieve better batching across the link between the two sites. + </para> + <!--h2--></section> + + <section role="h2" id="UsingBrokerFederation-Caveats-2CLimitations-2CandThingstoAvoid"><title> + Caveats, + Limitations, and Things to Avoid + </title> + + + <section role="h3" id="UsingBrokerFederation-RedundantPaths"><title> + Redundant + Paths + </title> + <para> + The current implementation of federation in the M4 broker imposes + constraints on redundancy in the topology. If there are parallel + paths from a producer to a consumer, multiple copies of messages + may be received. + </para><para> + A future release of Qpid will solve this problem by allowing + redundant paths with cost metrics. This will allow the deployment + of networks that are tolerant of connection or broker loss. + </para> + <!--h3--></section> + + <section role="h3" id="UsingBrokerFederation-LackofFlowControl"><title> + Lack of + Flow Control + </title> + <para> + M4 broker federation uses unlimited flow control on the + federation sessions. Flow control back-pressure will not be + applied on inter-broker subscriptions. + </para> + <!--h3--></section> + + + <section role="h3" id="UsingBrokerFederation-LackofClusterFailoverSupport"><title> + Lack of + Cluster Failover Support + </title> + <para> + The client functionality embedded in the broker for inter-broker + links does not currently support cluster fail-over. This will be + added in a subsequent release. + </para> + <!--h3--></section> + <!--h2--></section> + <!--h1--></section> + + <section role="h1" id="UsingBrokerFederation-ExampleScenarios"><title> + Example + Scenarios + </title> + + <section role="h2" id="UsingBrokerFederation-UsingQPIDtobridgedisjointIPnetworks"><title> + Using + QPID to bridge disjoint IP networks + </title> + <section role="h2" id="UsingBrokerFederation-Multitieredtopology"><title> + Multi-tiered + topology + </title> + + <programlisting> + +-----+ + | 5 | + +-----+ + / \ + +-----+ +-----+ + | 2 | | 6 | + +-----+ +-----+ + / | \ | \ + +-----+ +-----+ +-----+ +-----+ +-----+ + | 1 | | 3 | | 4 | | 7 | | 8 | + +-----+ +-----+ +-----+ +-----+ +-----+ +</programlisting> + <para> + This topology can be configured using the following script. + </para> + <programlisting> +## +## Define URLs for the brokers +## +broker1=localhost:10001 +broker2=localhost:10002 +broker3=localhost:10003 +broker4=localhost:10004 +broker5=localhost:10005 +broker6=localhost:10006 +broker7=localhost:10007 +broker8=localhost:10008 + +## +## Create Topic Exchanges +## +qpid-config -a $broker1 add exchange topic fed.topic +qpid-config -a $broker2 add exchange topic fed.topic +qpid-config -a $broker3 add exchange topic fed.topic +qpid-config -a $broker4 add exchange topic fed.topic +qpid-config -a $broker5 add exchange topic fed.topic +qpid-config -a $broker6 add exchange topic fed.topic +qpid-config -a $broker7 add exchange topic fed.topic +qpid-config -a $broker8 add exchange topic fed.topic + +## +## Create Topic Routes +## +qpid-route dynamic add $broker1 $broker2 fed.topic +qpid-route dynamic add $broker2 $broker1 fed.topic + +qpid-route dynamic add $broker3 $broker2 fed.topic +qpid-route dynamic add $broker2 $broker3 fed.topic + +qpid-route dynamic add $broker4 $broker2 fed.topic +qpid-route dynamic add $broker2 $broker4 fed.topic + +qpid-route dynamic add $broker2 $broker5 fed.topic +qpid-route dynamic add $broker5 $broker2 fed.topic + +qpid-route dynamic add $broker5 $broker6 fed.topic +qpid-route dynamic add $broker6 $broker5 fed.topic + +qpid-route dynamic add $broker6 $broker7 fed.topic +qpid-route dynamic add $broker7 $broker6 fed.topic + +qpid-route dynamic add $broker6 $broker8 fed.topic +qpid-route dynamic add $broker8 $broker6 fed.topic +</programlisting> + <!--h2--></section> + <section role="h2" id="UsingBrokerFederation-Loadsharingacrossbrokers"><title> + Load-sharing + across brokers + </title> + <para/> + <!--h2--></section> <!-- Level mixing in original - this is intentional --> + <!--h2--></section> + <!--h1--></section> + + <section role="h1" id="UsingBrokerFederation-AdvancedTopics"><title> + Advanced + Topics + </title> + <section role="h2" id="UsingBrokerFederation-FederationQueueNaming"><title> + Federation + Queue Naming + </title> + <para/> + <!--h2--></section> + <section role="h2" id="UsingBrokerFederation-MessageLoopPrevention"><title> + Message + Loop Prevention + </title> + <para/> + <!--h2--></section> +<!--h1--> + </section> + + +</chapter> diff --git a/qpid/doc/book/src/foo.xml b/qpid/doc/book/src/foo.xml new file mode 100644 index 0000000000..bdfd2c3c53 --- /dev/null +++ b/qpid/doc/book/src/foo.xml @@ -0,0 +1,42 @@ +<section> +<title>foo</title> +<table><title/><tgroup cols="2"> + + <tbody> + <row> + <entry> + Transport + </entry> + <entry> + Description + </entry> + </row> + + <row> + <entry> + tcp + </entry> + <entry> + (default) A cleartext TCP connection + </entry> + </row> + <row> + <entry> + ssl + </entry> + <entry> + A secure TLS/SSL over TCP connection + </entry> + </row> + <row> + <entry> + rdma + </entry> + <entry> + A Connection using the RDMA interface (typically for an + Infiniband network) + </entry> + </row> + </tbody> + </tgroup></table> +</section>
\ No newline at end of file diff --git a/qpid/doc/book/src/queue state replication.xml b/qpid/doc/book/src/queue state replication.xml new file mode 100644 index 0000000000..9196aa403b --- /dev/null +++ b/qpid/doc/book/src/queue state replication.xml @@ -0,0 +1,271 @@ +<?xml version="1.0" encoding="utf-8"?> +<chapter xmlns:html="http://www.w3.org/1999/xhtml"><title> + Apache Qpid : queue state replication + </title> + + <section role="h2" id="queuestatereplication-AsynchronousReplicationofQueueState"><title> + Asynchronous + Replication of Queue State + </title> + <section role="h3" id="queuestatereplication-Overview"><title> + Overview + </title> + <para> + There is support in qpidd for selective asynchronous replication + of queue state. This is achieved by: + </para><para> + (a) enabling event generation for the queues in question + </para><para> + (b) loading a plugin on the 'source' broker to encode those + events as messages on a replication queue (this plugin is + called + replicating_listener.so) + </para><para> + (c) loading a custom exchange plugin on the 'backup' broker (this + plugin is called replication_exchange.so) + </para><para> + (d) creating an instance of the replication exchange type on the + backup broker + </para><para> + (e) establishing a federation bridge between the replication + queue on the source broker and the replication exchange on the + backup broker + </para><para> + The bridge established between the source and backup brokers for + replication (step (e) above) should have acknowledgements turned + on (this may be done through the --ack N option to qpid-route). + This ensures that replication events are not lost if the bridge + fails. + </para><para> + The replication protocol will also eliminate duplicates to ensure + reliably replicated state. Note though that only one bridge per + replication exchange is supported. If clients try to publish to + the replication exchange or if more than a the single required + bridge from the replication queue on the source broker is + created, replication will be corrupted. (Access control may be + used to restrict access and help prevent this). + </para><para> + The replicating event listener plugin (step (b) above) has the + following options: + </para> + <programlisting> +Queue Replication Options: + --replication-queue QUEUE Queue on which events for + other queues are recorded + --replication-listener-name NAME (replicator) name by which to register the + replicating event listener + --create-replication-queue if set, the replication will + be created if it does not + exist +</programlisting> + <para> + The name of the queue is required. It can either point to a + durable queue whose definition has been previously recorded, or + the --create-replication-queue option can be specified in which + case the queue will be created a simple non-durable queue if it + does not already exist. + </para> + <!--h3--></section> + + <section role="h3" id="queuestatereplication-UsewithClustering"><title> + Use with + Clustering + </title> + <para> + The source and/or backup brokers may also be clustered brokers. + In this case the federated bridge will be re-established between + replicas should either of the originally connected nodes fail. + There are however the following limitations at present: + </para><itemizedlist> + <listitem><para>The backup site does not process membership updates after it + establishes the first connection. In order for newly added + members on a source cluster to be eligible as failover targets, + the bridge must be recreated after those members have been added + to the source cluster. + </para></listitem> + </itemizedlist><itemizedlist> + <listitem><para>New members added to a backup cluster will not receive + information about currently established bridges. Therefore in + order to allow the bridge to be re-established from these members + in the event of failure of older nodes, the bridge must be + recreated after the new members have joined. + </para></listitem> + </itemizedlist><itemizedlist> + <listitem><para>Only a single URL can be passed to create the initial link + from backup site to the primary site. this means that at the time + of creating the initial connection the initial node in the + primary site to which the connection is made needs to be running. + Once connected the backup site will receive a membership update + of all the nodes in the primary site, and if the initial + connection node in the primary fails, the link will be + re-established on the next node that was started (time) on the + primary site. + </para></listitem> + </itemizedlist><para> + Due to the acknowledged transfer of events over the bridge (see + note above) manual recreation of the bridge and automatic + re-establishment of te bridge after connection failure (including + failover where either or both ends are clustered brokers) will + not result in event loss. + </para> + <!--h3--></section> + + <section role="h3" id="queuestatereplication-OperationsonBackupQueues"><title> + Operations + on Backup Queues + </title> + <para> + When replicating the state of a queue to a backup broker it is + important to recognise that any other operations performed + directly on the backup queue may break the replication. + </para><para> + If the backup queue is to be an active (i.e. accessed by clients + while replication is on) only enqueues should be selected + for + replication. In this mode, any message enqueued on the source + brokers copy of the queue will also be enqueued on the backup + brokers copy. However not attempt will be made to remove messages + from the backup queue in response to removal of messages from the + source queue. + </para> + <!--h3--></section> + + <section role="h3" id="queuestatereplication-SelectingQueuesforReplication"><title> + Selecting + Queues for Replication + </title> + <para> + Queues are selected for replication by specifying the types of + events they should generate (it is from these events that the + replicating plugin constructs messages which are then pulled and + processed by the backup site). This is done through options + passed to the initial queue-declare command that creates the + queue and may be done either through qpid-config or similar + tools, or by the application. + </para><para> + With qpid-config, the --generate-queue-events options is used: + </para> + <programlisting> + --generate-queue-events N + If set to 1, every enqueue will generate an event that can be processed by + registered listeners (e.g. for replication). If set to 2, events will be + generated for enqueues and dequeues +</programlisting> + <para> + From an application, the arguments field of the queue-declare + AMQP command is used to convey this information. An entry should + be added to the map with key 'qpid.queue_event_generation' and an + integer value of 1 (to replicate only enqueue events) or 2 (to + replicate both enqueue and dequeue events). + </para><para> + Applications written using the c++ client API may fine the + qpid::client::QueueOptions class convenient. This has a + enableQueueEvents() method on it that can be used to set the + option (the instance of QueueOptions is then passed as the value + of the arguments field in the queue-declare command. The boolean + option to that method should be set to true if only enequeue + events should be replicated; by default it is false meaning that + both enqueues and dequeues will be replicated. E.g. + </para> + <programlisting> + QueueOptions options; + options.enableQueueEvents(false); + session.queueDeclare(arg::queue="my-queue", arg::arguments=options); +</programlisting> + <!--h3--></section> + + <section role="h3" id="queuestatereplication-Example"><title> + Example + </title> + <para> + Lets assume we will run the primary broker on host1 and the + backup on host2, have installed qpidd on both and have the + replicating_listener and replication_exchange plugins in qpidd's + module directory(*1). + </para><para> + On host1 we start the source broker and specifcy that a queue + called 'replication' should be used for storing the events until + consumed by the backup. We also request that this queue be + created (as transient) if not already specified: + </para> + <programlisting> + qpidd --replication-queue replication-queue --create-replication-queue true --log-enable info+ +</programlisting> + <para> + On host2 we start up the backup broker ensuring that the + replication exchange module is loaded: + </para> + <programlisting> + qpidd +</programlisting> + <para> + We can then create the instance of that replication exchange that + we will use to process the events: + </para> + <programlisting> + qpid-config -a host2 add exchange replication replication-exchange +</programlisting> + <para> + If this fails with the message "Exchange type not implemented: + replication", it means the replication exchange module was + not + loaded. Check that the module is installed on your system and if + necessary provide the full path to the library. + </para><para> + We then connect the replication queue on the source broker with + the replication exchange on the backup broker using the + qpid-route command: + </para> + <programlisting> + qpid-route --ack 50 queue add host2 host1 replication-exchange replication-queue +</programlisting> + <para> + The example above configures the bridge to acknowledge messages + in batches of 50. + </para><para> + Now create two queues (on both source and backup brokers), one + replicating both enqueues and dequeues (queue-a) and the + other + replicating only dequeues (queue-b): + </para> + <programlisting> + qpid-config -a host1 add queue queue-a --generate-queue-events 2 + qpid-config -a host1 add queue queue-b --generate-queue-events 1 + + qpid-config -a host2 add queue queue-a + qpid-config -a host2 add queue queue-b +</programlisting> + <para> + We are now ready to use the queues and see the replication. + </para><para> + Any message enqueued on queue-a will be replicated to the backup + broker. When the message is acknowledged by a client connected to + host1 (and thus dequeued), that message will be removed from the + copy of the queue on host2. The state of queue-a on host2 will + thus mirror that of the equivalent queue on host1, albeit with a + small lag. (Note + however that we must not have clients connected to host2 publish + to-or consume from- queue-a or the state will fail to replicate + correctly due to conflicts). + </para><para> + Any message enqueued on queue-b on host1 will also be enqueued on + the equivalent queue on host2. However the acknowledgement and + consequent dequeuing of messages from queue-b on host1 will have + no effect on the state of queue-b on host2. + </para><para> + (*1) If not the paths in the above may need to be modified. E.g. + if using modules built from a qpid svn checkout, the following + would be added to the command line used to start qpidd on host1: + </para> + <programlisting> + --load-module <path-to-qpid-dir>/src/.libs/replicating_listener.so +</programlisting> + <para> + and the following for the equivalent command line on host2: + </para> + <programlisting> + --load-module <path-to-qpid-dir>/src/.libs/replication_exchange.so +</programlisting> +<!--h3--></section> +<!--h2--></section> +</chapter> diff --git a/qpid/doc/book/src/queue%20state%20replication.xml b/qpid/doc/book/src/queue%20state%20replication.xml new file mode 100644 index 0000000000..70c232110d --- /dev/null +++ b/qpid/doc/book/src/queue%20state%20replication.xml @@ -0,0 +1,232 @@ +<?xml version="1.0"?> +<section> + <title> + Asynchronous + Replication of Queue State + </title> + <para> + There is support in qpidd for selective asynchronous replication + of queue state. This is achieved by: + </para> + <para> + (a) enabling event generation for the queues in question + </para> + <para> + (b) loading a plugin on the 'source' broker to encode those + events as messages on a replication queue (this plugin is + called + replicating_listener.so) + </para> + <para> + (c) loading a custom exchange plugin on the 'backup' broker (this + plugin is called replication_exchange.so) + </para> + <para> + (d) creating an instance of the replication exchange type on the + backup broker + </para> + <para> + (e) establishing a federation bridge between the replication + queue on the source broker and the replication exchange on the + backup broker + </para> + <para> + The bridge established between the source and backup brokers for + replication (step (e) above) should have acknowledgements turned + on (this may be done through the --ack N option to qpid-route). + This ensures that replication events are not lost if the bridge + fails. + </para> + <para> + The replication protocol will also eliminate duplicates to ensure + reliably replicated state. Note though that only one bridge per + replication exchange is supported. If clients try to publish to + the replication exchange or if more than a the single required + bridge from the replication queue on the source broker is + created, replication will be corrupted. (Access control may be + used to restrict access and help prevent this). + </para> + <para> + The replicating event listener plugin (step (b) above) has the + following options: + </para> + <para> + The name of the queue is required. It can either point to a + durable queue whose definition has been previously recorded, or + the --create-replication-queue option can be specified in which + case the queue will be created a simple non-durable queue if it + does not already exist. + </para> + <bridgehead id="queuestatereplication-UsewithClustering"><anchor id="queuestatereplication-UsewithClustering"/>Use with + Clustering + </bridgehead> + <para> + The source and/or backup brokers may also be clustered brokers. + In this case the federated bridge will be re-established between + replicas should either of the originally connected nodes fail. + There are however the following limitations at present: + </para> + <itemizedlist> + <listitem> + <para>The backup site does not process membership updates after it + establishes the first connection. In order for newly added + members on a source cluster to be eligible as failover targets, + the bridge must be recreated after those members have been added + to the source cluster. + </para> + </listitem> + </itemizedlist> + <itemizedlist> + <listitem> + <para>New members added to a backup cluster will not receive + information about currently established bridges. Therefore in + order to allow the bridge to be re-established from these members + in the event of failure of older nodes, the bridge must be + recreated after the new members have joined. + </para> + </listitem> + </itemizedlist> + <itemizedlist> + <listitem> + <para>Only a single URL can be passed to create the initial link + from backup site to the primary site. this means that at the time + of creating the initial connection the initial node in the + primary site to which the connection is made needs to be running. + Once connected the backup site will receive a membership update + of all the nodes in the primary site, and if the initial + connection node in the primary fails, the link will be + re-established on the next node that was started (time) on the + primary site. + </para> + </listitem> + </itemizedlist> + <para> + Due to the acknowledged transfer of events over the bridge (see + note above) manual recreation of the bridge and automatic + re-establishment of te bridge after connection failure (including + failover where either or both ends are clustered brokers) will + not result in event loss. + </para> + <bridgehead id="queuestatereplication-OperationsonBackupQueues"><anchor id="queuestatereplication-OperationsonBackupQueues"/>Operations + on Backup Queues + </bridgehead> + <para> + When replicating the state of a queue to a backup broker it is + important to recognise that any other operations performed + directly on the backup queue may break the replication. + </para> + <para> + If the backup queue is to be an active (i.e. accessed by clients + while replication is on) only enqueues should be selected + for + replication. In this mode, any message enqueued on the source + brokers copy of the queue will also be enqueued on the backup + brokers copy. However not attempt will be made to remove messages + from the backup queue in response to removal of messages from the + source queue. + </para> + <bridgehead id="queuestatereplication-SelectingQueuesforReplication"><anchor id="queuestatereplication-SelectingQueuesforReplication"/>Selecting + Queues for Replication + </bridgehead> + <para> + Queues are selected for replication by specifying the types of + events they should generate (it is from these events that the + replicating plugin constructs messages which are then pulled and + processed by the backup site). This is done through options + passed to the initial queue-declare command that creates the + queue and may be done either through qpid-config or similar + tools, or by the application. + </para> + <para> + With qpid-config, the --generate-queue-events options is used: + </para> + <para> + From an application, the arguments field of the queue-declare + AMQP command is used to convey this information. An entry should + be added to the map with key 'qpid.queue_event_generation' and an + integer value of 1 (to replicate only enqueue events) or 2 (to + replicate both enqueue and dequeue events). + </para> + <para> + Applications written using the c++ client API may fine the + qpid::client::QueueOptions class convenient. This has a + enableQueueEvents() method on it that can be used to set the + option (the instance of QueueOptions is then passed as the value + of the arguments field in the queue-declare command. The boolean + option to that method should be set to true if only enequeue + events should be replicated; by default it is false meaning that + both enqueues and dequeues will be replicated. E.g. + </para> + <bridgehead id="queuestatereplication-Example"><anchor id="queuestatereplication-Example"/>Example + </bridgehead> + <para> + Lets assume we will run the primary broker on host1 and the + backup on host2, have installed qpidd on both and have the + replicating_listener and replication_exchange plugins in qpidd's + module directory(*1). + </para> + <para> + On host1 we start the source broker and specifcy that a queue + called 'replication' should be used for storing the events until + consumed by the backup. We also request that this queue be + created (as transient) if not already specified: + </para> + <para> + On host2 we start up the backup broker ensuring that the + replication exchange module is loaded: + </para> + <para> + We can then create the instance of that replication exchange that + we will use to process the events: + </para> + <para> + If this fails with the message "Exchange type not implemented: + replication", it means the replication exchange module was + not + loaded. Check that the module is installed on your system and if + necessary provide the full path to the library. + </para> + <para> + We then connect the replication queue on the source broker with + the replication exchange on the backup broker using the + qpid-route command: + </para> + <para> + The example above configures the bridge to acknowledge messages + in batches of 50. + </para> + <para> + Now create two queues (on both source and backup brokers), one + replicating both enqueues and dequeues (queue-a) and the + other + replicating only dequeues (queue-b): + </para> + <para> + We are now ready to use the queues and see the replication. + </para> + <para> + Any message enqueued on queue-a will be replicated to the backup + broker. When the message is acknowledged by a client connected to + host1 (and thus dequeued), that message will be removed from the + copy of the queue on host2. The state of queue-a on host2 will + thus mirror that of the equivalent queue on host1, albeit with a + small lag. (Note + however that we must not have clients connected to host2 publish + to-or consume from- queue-a or the state will fail to replicate + correctly due to conflicts). + </para> + <para> + Any message enqueued on queue-b on host1 will also be enqueued on + the equivalent queue on host2. However the acknowledgement and + consequent dequeuing of messages from queue-b on host1 will have + no effect on the state of queue-b on host2. + </para> + <para> + (*1) If not the paths in the above may need to be modified. E.g. + if using modules built from a qpid svn checkout, the following + would be added to the command line used to start qpidd on host1: + </para> + <para> + and the following for the equivalent command line on host2: + </para> +</section> diff --git a/qpid/doc/book/src/schemas.xml b/qpid/doc/book/src/schemas.xml index 8e6e68ccd4..960fffaa2a 100644 --- a/qpid/doc/book/src/schemas.xml +++ b/qpid/doc/book/src/schemas.xml @@ -1,5 +1,21 @@ <?xml version="1.0"?> <locatingRules xmlns="http://thaiopensource.com/ns/locating-rules/1.0"> + <uri resource="SSL.xml" typeId="DocBook"/> + <uri resource="Using Broker Federation.xml" typeId="DocBook"/> + <uri resource="Cheat Sheet for configuring Exchange Options.xml" typeId="DocBook"/> + <uri resource="foo.xml" typeId="DocBook"/> + <uri resource="Download.xml" typeId="DocBook"/> + <uri resource="Download.xml" typeId="DocBook"/> + <uri resource="Starting a cluster.xml" typeId="DocBook"/> + <uri resource="queue state replication.xml" typeId="DocBook"/> + <uri resource="LVQ.xml" typeId="DocBook"/> + <uri resource="LVQ.xml" typeId="DocBook"/> + <uri resource="Cheat Sheet for configuring Queue Options.xml" typeId="DocBook"/> + <uri resource="Cheat Sheet for configuring Exchange Options.xml" typeId="DocBook"/> + <uri resource="Cheat Sheet for configuring Exchange Options.xml" typeId="DocBook"/> + <uri resource="AMQP-Messaging-Broker-CPP.xml" typeId="DocBook"/> + <uri resource="RASC.xml" typeId="DocBook"/> + <uri resource="Brokers.xml" typeId="DocBook"/> <uri resource="AMQP.xml" typeId="DocBook"/> <uri resource="qpid-book.xml" typeId="DocBook"/> </locatingRules> |