Using Qpid for Messaging

Using the Qpid messaging API The Qpid messaging API is quite simple, consisting of only a handful of core classes. A message consists of a standard set of fields (e.g. subject, reply-to), an application-defined set of properties, and message content (the main body of the message). A connection represents a network connection to a remote endpoint. A session provides a sequentially ordered context for sending and receiving messages. A session is obtained from a connection. A sender sends messages to a target using the sender.send method. A sender is obtained from a session for a given target address. A receiver receives messages from a source using the receiver.fetch method. A receiver is obtained from a session for a given source address. The following sections show how to use these classes in a simple messaging program.

A Simple Messaging Program in C++ The following C++ program shows how to create a connection, create a session, send messages using a sender, and receive messages using a receiver. "Hello world!" in C++ #include #include #include #include #include using namespace qpid::messaging; int main(int argc, char** argv) { std::string broker = argc > 1 ? argv[1] : "localhost:5672"; std::string address = argc > 2 ? argv[2] : "amq.topic"; Connection connection(broker); try { connection.open(); Session session = connection.createSession(); Receiver receiver = session.createReceiver(address); Sender sender = session.createSender(address); sender.send(Message("Hello world!")); Message message = receiver.fetch(Duration::SECOND * 1); std::cout << message.getContent() << std::endl; session.acknowledge(); connection.close(); return 0; } catch(const std::exception& error) { std::cerr << error.what() << std::endl; connection.close(); return 1; } }]]>

A Simple Messaging Program in Python The following Python program shows how to create a connection, create a session, send messages using a sender, and receive messages using a receiver. "Hello world!" in Python

Addresses An address is the name of a message target or message source. In the programs we have just seen, we used the address amq.topic (which is the name of an exchange on an AMQP 0-10 messaging broker). The methods that create senders and receivers require an address. The details of sending to a particular target or receiving from a particular source are then handled by the sender or receiver. A different target or source can be used simply by using a different address. An address resolves to a node. The Qpid messaging API recognises two kinds of nodes, queues and topics. The terms queue and topic here were chosen to align with their meaning in JMS. These two addressing 'patterns', queue and topic, are sometimes refered as point-to-point and publish-subscribe. AMQP 0-10 has an exchange type called a topic exchange. When the term topic occurs alone, it refers to a messaging API topic, not the topic exchange.. A queue stores each message until it has been received and acknowledged, and only one receiver can receive a given message There are exceptions to this rule; for instance, a receiver can use browse mode, which leaves messages on the queue for other receivers to read. A topic immediately delivers a message to all eligible receivers; if there are no eligible receivers, it discards the message. In the AMQP 0-10 implementation of the API, The AMQP 0-10 implementation is the only one that currently exists. queues map to AMQP queues, and topics map to AMQP exchanges. In AMQP 0-10, messages are sent to exchanges, and read from queues. The messaging API also allows a sender to send messages to a queue; internally, Qpid implements this by sending the message to the default exchange, with the name of the queue as the routing key. The messaging API also allows a receiver to receive messages from a topic; internally, Qpid implements this by setting up a private subscription queue for the receiver and binding the subscription queue to the exchange that corresponds to the topic. In the rest of this tutorial, we present many examples using two programs that take an address as a command line parameter. spout sends messages to the target address, drain receives messages from the source address. The source code is available in both C++ and Python, and can be found in the examples directory for each language. These programs can use any address string as a source or a destination, and have many command line options to configure behavior—use the -h option for documentation on these options. Currently, the Python and C++ implementations of drain and spout have slightly different options. This tutorial uses the C++ implementation. The options will be reconciled in the near future. The examples in this tutorial also use the qpid-config utility to configure AMQP 0-10 queues and exchanges on a Qpid broker. Queues Create a queue with qpid-config, send a message using spout, and read it using drain: $ qpid-config add queue hello-world $ ./spout -a hello-world $ ./drain -a hello-world Message(properties={spout-id:c877e622-d57b-4df2-bf3e-6014c68da0ea:0}, content='') The queue stored the message sent by spout and delivered it to drain when requested. Once the message has been delivered and and acknowledged by drain, it is no longer available on the queue. If we run drain one more time, no messages will be retrieved. $ ./drain -a hello-world $ Topics This example is similar to the previous example, but it uses a topic instead of a queue. First, use qpid-config to remove the queue and create an exchange with the same name: $ qpid-config del queue hello-world $ qpid-config add exchange topic hello-world Now run drain and spout the same way we did in the previous example: $ ./spout -a hello-world $ ./drain -a hello-world $ Topics deliver messages immediately to any interested receiver, and do not store messages. Because there were no receivers at the time spout sent the message, it was simply discarded. When we ran drain, there were no messages to receive. Now let's run drain first, using the -t option to specify a timeout in seconds. While drain is waiting for messages, run spout in another window. First Window: $ ./drain -a hello-word -t 30 Second Window: $ ./spout -a hello-word Once spout has sent a message, return to the first window to see the output from drain: Message(properties={spout-id:7da2d27d-93e6-4803-8a61-536d87b8d93f:0}, content='') You can run drain in several separate windows; each will create a subscription for the exchange, and each will receive all messages sent to the exchange.

Address Strings So far, our examples have used address strings that contain only the name of a node. An address string can also contain a subject and options. The syntax for an address string is: [ / ] [ ; ] options ::= { : , ... } ]]> Addresses, subjects, and keys are strings. Values can be numbers, strings (with optional single or double quotes), maps, or lists. A complete BNF for address strings appears in . So far, the address strings in this tutorial have used only addresses. The following sections show how to use subjects and options.

Subjects Every message has a property called subject, which is analogous to the subject on an email message. If no subject is specified, the message's subject is null. For convenience, address strings also allow a subject. If a sender's address contains a subject, it is used as the default subject for the messages it sends. If a receiver's address contains a subject, it is used to select only messages that match the subject—the matching algorithm depends on the message source. In AMQP 0-10, each exchange type has its own matching algorithm, and queues do not provide filtering. This is discussed in . Currently, a receiver bound to a queue ignores subjects, receiving messages from the queue without filtering. In the future, if a receiver is bound to a queue, and its address contains a subject, the subject will be used as a selector to filter messages. Using subjects In this example we will show how subjects affect message flow. First, let's use qpid-config to create a topic exchange. $ qpid-config add exchange topic news-service Now we will use drain to receive messages from news-service that match the subject sports. First Window: $ ./drain -a news-service/sports -t 30 In a second window, let's send messages to news-service using two different subjects: Second Window: $ ./spout -a news-service/sports $ ./spout -a news-service/news Now look at the first window, and you will see the message with the subject sports has been received, but not the message with the subject news: Message(properties={qpid.subject:sports, spout-id:9441674e-a157-4780-a78e-f7ccea998291:0}, content='') If you run drain in multiple windows using the same subject, all instances of drain receive the messages for that subject. The AMQP exchange type we are using here, amq.topic, can also do more sophisticated matching. A sender's subject can contain multiple words separated by a . delimiter. For instance, in a news application, the sender might use subjects like usa.news, usa.weather, europe.news, or europe.weather. The receiver's subject can include wildcard characters— # matches one or more words in the message's subject, * matches a single word. For instance, if the subject in the source address is *.news, it matches messages with the subject europe.news or usa.news; if it is europe.#, it matches messages with subjects like europe.news or europe.pseudo.news. Subjects with multi-word keys This example uses drain and spout to demonstrate the use of subjects with two-word keys. Let's use drain with the subject *.news to listen for messages in which the second word of the key is news. First Window: $ ./drain -a news-service/*.news -t 30 Now let's send messages using several different two-word keys: Second Window: $ ./spout -a news-service/usa.news $ ./spout -a news-service/usa.sports $ ./spout -a news-service/europe.sports $ ./spout -a news-service/europe.news In the first window, the messages with news in the second word of the key have been received: Message(properties={qpid.subject:usa.news, spout-id:73fc8058-5af6-407c-9166-b49a9076097a:0}, content='') Message(properties={qpid.subject:europe.news, spout-id:f72815aa-7be4-4944-99fd-c64c9747a876:0}, content='') Next, let's use drain with the subject #.news to match any sequence of words that ends with news. First Window: $ ./drain -a news-service/#.news -t 30 In the second window, let's send messages using a variety of different multi-word keys: Second Window: $ ./spout -a news-service/news $ ./spout -a news-service/sports $ ./spout -a news-service/usa.news $ ./spout -a news-service/usa.sports $ ./spout -a news-service/usa.faux.news $ ./spout -a news-service/usa.faux.sports In the first window, messages with news in the last word of the key have been received: Message(properties={qpid.subject:news, spout-id:cbd42b0f-c87b-4088-8206-26d7627c9640:0}, content='') Message(properties={qpid.subject:usa.news, spout-id:234a78d7-daeb-4826-90e1-1c6540781eac:0}, content='') Message(properties={qpid.subject:usa.faux.news, spout-id:6029430a-cfcb-4700-8e9b-cbe4a81fca5f:0}, content='')

Address String Options The options in an address string contain additional information for the senders or receivers created for it, including: Policies for assertions about the node to which an address refers. For instance, in the address string my-queue; {assert: always, node:{ type: queue }}, the node named my-queue must be a queue; if not, the address does not resolve to a node, and an exception is raised. Policies for automatically creating or deleting the node to which an address refers. For instance, in the address string xoxox ; {create: always}, the queue xoxox is created, if it does not exist, before the address is resolved. Extension points that can be used for sender/receiver configuration. For instance, if the address for a receiver is my-queue; {mode: browse}, the receiver works in browse mode, leaving messages on the queue so other receivers can receive them. Let's use some examples to show how these different kinds of address string options affect the behavior of senders and receives. First, let's use the assert option to ensure that the address resolves to a node of the required type. Assertions on Nodes Let's use qpid-config to create a queue and a topic. $ qpid-config add queue my-queue $ qpid-config add exchange topic my-topic We can now use the address specified to drain to assert that it is of a particular type: $ ./drain -a 'my-queue; {assert: always, node:{ type: queue }}' $ ./drain -a 'my-queue; {assert: always, node:{ type: topic }}' 2010-04-20 17:30:46 warning Exception received from broker: not-found: not-found: Exchange not found: my-queue (../../src/qpid/broker/ExchangeRegistry.cpp:92) [caused by 2 \x07:\x01] Exchange my-queue does not exist The first attempt passed without error as my-queue is indeed a queue. The second attempt however failed; my-queue is not a topic. We can do the same thing for my-topic: $ ./drain -a 'my-topic; {assert: always, node:{ type: topic }}' $ ./drain -a 'my-topic; {assert: always, node:{ type: queue }}' 2010-04-20 17:31:01 warning Exception received from broker: not-found: not-found: Queue not found: my-topic (../../src/qpid/broker/SessionAdapter.cpp:754) [caused by 1 \x08:\x01] Queue my-topic does not exist Now let's use the create option to create the queue xoxox if it does not already exist: Creating a Queue Automatically First Window: $ ./drain -a "xoxox ; {create: always}" -t 30 In previous examples, we created the queue before listening for messages on it. Using create: always, the queue is automatically created if it does not exist. Now we can send messages to this queue: Second Window: $ ./spout -a "xoxox ; {create: always}" Returning to the first window, we see that drain has received this message: Message(properties={spout-id:1a1a3842-1a8b-4f88-8940-b4096e615a7d:0}, content='') Other options specify message transfer semantics; for instance, they may state whether messages should be consumed or read in browsing mode, or specify reliability characteristics. The following example uses the browse option to receive messages without removing them from a queue. Browsing a Queue Let's use the browse mode to receive messages without removing them from the queue. First we send three messages to the queue: $ ./spout -a my-queue --content one $ ./spout -a my-queue --content two $ ./spout -a my-queue --content three Now we use drain to get those messages, using the browse option: $ ./drain -a 'my-queue; {mode: browse}' Message(properties={spout-id:fbb93f30-0e82-4b6d-8c1d-be60eb132530:0}, content='one') Message(properties={spout-id:ab9e7c31-19b0-4455-8976-34abe83edc5f:0}, content='two') Message(properties={spout-id:ea75d64d-ea37-47f9-96a9-d38e01c97925:0}, content='three') We can confirm the messages are still on the queue by repeating the drain: $ ./drain -a 'my-queue; {mode: browse}' Message(properties={spout-id:fbb93f30-0e82-4b6d-8c1d-be60eb132530:0}, content='one') Message(properties={spout-id:ab9e7c31-19b0-4455-8976-34abe83edc5f:0}, content='two') Message(properties={spout-id:ea75d64d-ea37-47f9-96a9-d38e01c97925:0}, content='three') Address Options option value semantics assert one of: always, never, sender or receiver Asserts that the properties specified in the node option match whatever the address resolves to. If they do not, resolution fails and an exception is raised. create one of: always, never, sender or receiver Creates the node to which an address refers if it does not exist. No error is raised if the node does exist. The details of the node may be specified in the node option. delete one of: always, never, sender or receiver Delete the node when the sender or receiver is closed. node A nested map containing the entries shown in . Specifies properties of the node to which the address refers. These are used in conjunction with the assert or create options. link A nested map containing the entries shown in . Used to control the establishment of a conceptual link from the client application to or from the target/source address. mode one of: browse, consume This option is only of relevance for source addresses that resolve to a queue. If browse is specified the messages delivered to the receiver are left on the queue rather than being removed. If consume is specified the normal behaviour applies; messages are removed from teh queue once the client acknoweldges their receipt.

Node Properties property value semantics type topic, queue durable True, False Indicates whether the node will survive a loss of volatile storage e.g. if the broker is restarted. x-declare A nested map whose values correspond to the valid fields on an AMQP 0-10 queue-declare or exchange-declare command. These values are used to fine tune the creation or assertion process. Note however that they are protocol specific. x-bindings A nested list each of whose entries is a map that may contain fields (queue, exchange, key and arguments) describing an AMQP 0-10 binding, using the following format: , queue: , key: , arguments: } ]]> In conjunction with the create option, each of these bindings will be established as the address is resolved. In conjunction with the assert option, the existence of each of these bindings will be verified during resolution. Again, these are protocol specific.

Link Properties option value semantics reliability one of: unreliable, at-least-once, at-most-once, exactly-once durable True, False Indicates whether the link will survive a loss of volatile storage e.g. if the broker is restarted. x-declare A nested map whose values correspond to the valid fields of an AMQP 0-10 queue-declare command. These values can be used to customise the subscription queue in the case of receiving from an exchange. Note however that they are protocol specific. x-subscribe A nested map whose values correspond to the valid fields of an AMQP 0-10 message-subscribe command. These values can be used to customise the subscription. x-bindings A nested list each of whose entries is a map that may contain fields (queue, exchange, key and arguments) describing an AMQP 0-10 binding. These bindings will be established during resolution independent of the create option. They are considered logically part of the linking process rather than of node creation.

Address String Grammar This section provides a formal grammar for address strings. Tokens The following regular expressions define the tokens used to parse address strings: Grammar The formal grammar for addresses is given below: The address string options map supports the following parameters: [ / ] ; { create: always | sender | receiver | never, delete: always | sender | receiver | never, assert: always | sender | receiver | never, mode: browse | consume, node: { type: queue | topic, durable: True | False, x-declare: { ... ... }, x-bindings: [, ... ] }, link: { name: , durable: True | False, reliability: unreliable | at-most-once | at-least-once | exactly-once, x-declare: { ... ... }, x-bindings: [, ... ], x-subscribe: { ... ... } } } ]]> Create, Delete, and Assert Policies The create, delete, and assert policies specify who should perfom the associated action: always: the action will be performed by any messaging client sender: the action will only be performed by a sender receiver: the action will only be performed by a receiver never: the action will never be performed (this is the default) Node-Type The node-type is one of: topic: in the AMQP 0-10 mapping, a topic node defaults to the topic exchange, x-declare may be used to specify other exchange types queue: this is the default node-type

Reconnect and Failover Connections in the Qpid messaging API support automatic reconnection. The following table lists some of the connection properties that control this. See the reference documentation for details on how to set these on connections fro each langauge. Connection properties property value semantics reconnect True, False Transparently reconnect if the connection is lost. reconnect_timeout N Total number of seconds to continue reconnection attempts before giving up and raising an exception. reconnect_limit N Maximum number of reconnection attempts before giving up and raising an exception. reconnect_interval_min N Minimum number of seconds between reconnection attempts. The first reconnection attempt is made immediately; if that fails, the first reconnection delay is set to the value of reconnect_interval_min; if that attempt fails, the reconnect interval increases exponentially until a reconnection attempt succeeds or reconnect_interval_max is reached. reconnect_interval_max N Maximum reconnect interval. reconnect_interval N Sets both reconnection_interval_min and reconnection_interval_max to the same value.

The AMQP 0-10 mapping This section describes the AMQP 0-10 mapping for the Qpid messaging API. The interaction with the broker triggered by creating a sender or receiver depends on what the specified address resolves to. Where the node type is not specified in the address, the client queries the broker to determine whether it refers to a queue or an exchange. When sending to a queue, the queue's name is set as the routing key and the message is transfered to the default (or nameless) exchange. When sending to an exchange, the message is transfered to that exchange and the routing key is set to the message subject if one is specified. A default subject may be specified in the target address. The subject may also be set on each message individually to override the default if required. In each case any specified subject is also added as a qpid.subject entry in the application-headers field of the message-properties. When receiving from a queue, any subject in the source address is currently ignored. The client sends a message-subscribe request for the queue in question. The accept-mode is determined by the reliability option in the link properties; for unreliable links the accept-mode is none, for reliable links it is explicit. The default for a queue is reliable. The acquire-mode is determined by the value of the mode option. If the mode is set to browse the acquire mode is not-acquired, otherwise it is set to pre-acquired. The exclusive and arguments fields in the message-subscribe command can be controlled using the x-subscribe map. When receiving from an exchange, the client creates a subscription queue and binds that to the exchange. The subscription queue's arguments can be specified using the x-declare map within the link properties. The reliability option determines most of the other parameters. If the reliability is set to unreliable then an auto-deleted, exclusive queue is used meaning that if the client or connection fails messages may be lost. For exactly-once the queue is not set to be auto-deleted. The durability of the subscription queue is determined by the durable option in the link properties. The binding process depends on the type of the exchange the source address resolves to. For a topic exchange, if no subject is specified and no x-bindings are defined for the link, the subscription queue will by be bound using a wildcard matching any routing key (thus satisfying the expectation that any message sent to that address will be received from it). If a subject is specified in the source address however, it will be used for the binding key (this means that the subject in the source address may be a binding pattern incuding wildcards). For a fanout exchange the binding key is irrelevant to matching. A receiver created from a source address that resolves to a fanout exchange will receive all messages sent to that exchange regardless of any subject the source address may contain. An x-bindings element in the link properties should be used if there is any need to set the arguments to the bind. A source address that resolves to a direct exchange must either contain a subject or must include a value for the x-bindings option in the link properties. This is because there is no way to receive all messages sent to an exchange of that type. The subject specified will be used as the binding key (this means it must match the message subject exactly). For a headers exchange, if no subject is specified the binding arguments will simply contain an x-match entry an no other entries, causing all messages to match. If a subject is specified then the binding arguments will contain an x-match entry set to all and an entry for qpid.subject whose value is the subject in the source address (this means the subject in the source address must match the message subject exactly). For more control the x-bindings element in the link properties must be used. For the XML exchange,Note that the XML exchange is not a standard AMQP exchange type. It is a Qpid extension and is currently only supported by the C++ broker. if a subject is specified it is used as the bidning key and an xquery is defined that will match any message with that value for qpid.subject. Again this means that only messages whose subject exactly match that specified in the source address will be received. For more control the x-bindings element in the link properties must be used. A source address that resolves to the XML exchange must contain either a subject or an x-bindings element in the link properties as there is no way at present to receive any message regardless of routing key. If an x-bindings list is present in the link options a binding is created for each element within that list. Each element is a nested map that may contain values named queue, exchange, key or arguments. If the queue value is absent the queue name the address resolves to is implied. If the exchange value is absent the exchange name the address resolves to is implied. The following table shows how Qpid Messaging API message properties are mapped to AMQP 0-10 message properties and delivery properties. In this table msg refers to the Message class defined in the Qpid Messaging API, mp refers to an AMQP 0-10 message-properties struct, and dp refers to an AMQP 0-10 delivery-properties struct. Mapping to AMQP 0-10 Message Properties Python API C++ API AMQP 0-10 PropertyIn these entries, mp> refers to an AMQP message property, and dp refers to an AMQP delivery property. msg.idmsg.{get,set}MessageId()mp.message_id msg.subjectmsg.{get,set}Subject()mp.application_headers["qpid.subject"] msg.user_idmsg.{get,set}UserId()mp.user_id msg.reply_tomsg.{get,set}ReplyTo()mp.reply_toThe reply_to is converted from the protocol representation into an address. msg.correlation_idmsg.{get,set}CorrelationId()mp.correlation_id msg.durablemsg.{get,set}Durable()dp.delivery_mode == delivery_mode.persistentNote that msg.durable is a boolean, not an enum. msg.prioritymsg.{get,set}Priority()dp.priority msg.ttlmsg.{get,set}Ttl()dp.ttl msg.redeliveredmsg.{get,set}Redelivered()dp.redelivered msg.propertiesmsg.{get,set}Properties()mp.application_headers msg.content_typemsg.{get,set}ContentType()mp.content_type