summaryrefslogtreecommitdiff
path: root/src/amqqueue.erl
Commit message (Collapse)AuthorAgeFilesLines
* amqqueue: Pass `Type` to amqqueue_v1:new/9Jean-Sébastien Pédron2020-10-051-1/+2
| | | | | | | | | | | | | | | | | | | | | | | amqqueue_v1:new/9 only accepts the classic queue type. Therefore, by passing the type, we ensure that the request to create a v1 record with a non-classic queue type will crash. Before this change, amqqueue:new/9 with a non-classic queue type would return "successfully" a classic queue record, effectively ignoring the queue type argument. This should help to detect missing dependency between feature flags. For instance, the `stream_queue` feature flag would depend on the `quorum_queue` feature flag because the latter introduced the v2 record (the one with the `type` record member). Without this change, it means that a user could request a new stream queue (after enabling the `stream_queue` feature flag), but a classic queue would be created instead. There would be no error. Now, with this change, the request will crash. The same user can report the bug: he enabled the `stream_queue` feature flag but the declare crashes.
* Stream Queuekjnilsson2020-09-301-2/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is an aggregated commit of all changes related to the initial implementation of queue types and on top of that the stream queue type. The varios commit messages have simply been included mostly un-edited below. Make rabbit_amqqueue:not_found_or_absent_dirty/1 visible For use in the stream plugin. Use bigger retention policy on max-age test Set coordinator timeout to 30s Handle coordinator unavailable error Handle operator policies as maps when checking if is applicable Add is_policy_applicable/2 to classic queues Ignore restart commands if the stream has been deleted It could happen that after termination some of the monitors are still up and trigger writer/replica restarts Policy support on stream queues Remove subscription events on stream coordinator Ensure old leaders are removed from monitors Introduce delay when retrying a failed phase Note that this ensures monitor is setup, there was a bug where no monitor was really started when re-trying the same phase Restart replicas after leader election instead of relying on old monitors Use timer for stream coordinator retries Fix stream stats for members/online Multiple fixes for replica monitoring and restart Ensure pending commands are appended at the end and re-run Ensure phase is reset with the state Remove duplicates from replica list Restart current phase on state_enter Remove unused import Ensure rabbit is running when checking for stream quorum Restart replicas Add a close/1 function to queue types So that we can get a chance of cleaning up resources if needed. Stream queues close their osiris logs at this point. fix compiler errors stream-queue: take retention into account When calculating ready messages metrics. Add osiris to the list of rabbit deps Retry restart of replicas Do not restart replicas or leaders after receiving a delete cluster command Add more logging to the stream coordinator Monitor subscribed processes on the stream coordinator Memory breakdown for stream queues Update quorum queue event formatter rabbit_msg_record fixes Refactor channel confirms Remove old unconfirmed_messages module that was designed to handle multiple queue fan in logic including all ha mirrors etc. Replaced with simpler rabbit_confirms module that handles the fan out and leaves any queue specific logic (such as confirms from mirrors) to the queue type implemention. Also this module has a dedicated test module. Which is nice. Backward compatibility with 3.8.x events Supports mixed version cluster upgrades Match specification when stream queue already exists Max age retention for stream queues Stop all replicas before starting leader election stream: disallow global qos remove IS_CLASSIC|QUORUM macros Ensure only classic queues are notified on channel down This also removes the delivering_queues map in the channel state as it should not be needed for this and just cause additional unecessary accounting. Polish AMQP 1.0/0.9.1 properties conversion Support byte in application properties, handle 1-bit representation for booleans. Use binary in header for long AMQP 1.0 ID Fix AMQP 1.0 to 0.9.1 conversion Fix test due to incorrect type Convert timestamp application properties to/from seconds AMQP 1.0 uses milliseconds for timestamp and AMQP 0.9.1 uses seconds, so conversion needed. Dialyzer fixes Handle all message-id types AMQP 1.0 is more liberal in it's allowed types of message-id and correlation-id - this adds headers to describe the type of the data in the message_id / correlation_id properties and also handles the case where the data cannot fit by again using headers. Resize stream coordinator cluster when broker configuration changes convert timestamp to and fro seconds user_id should be a binary message annotations keys need to be symbols stream-queue: default exchange and routing key As these won't be present for data written using the rabbitmq-stream plugin. Add exchange, routing key as message annotations To the AMQP 1.0 formatted data to enable roundtrip. Add osiris logging module config And update logging config test suite. Restart election when start of new leader fails The node might have just gone down so we need to try another one Only aux keeps track of phase now, as it might change if the leader election fails Stream coordinator refactor - all state is kept on the ra machine Ensure any ra cluster not a qq is not cleaned up Fixes to recovery and monitoring Add AMQP 1.0 common to dependencies Add rabbit_msg_record module To handle conversions into internal stream storage format. Use rabbitmq-common stream-queue branch Use SSH for osiris dependency Stream coordinator: delete replica Stream coordinator: add replica Stream coordinator: leader failover Stream coordinator: declare and delete Test consuming from a random offset Previous offsets should not be delivered to consumers Consume from stream replicas and multiple test fixes Use max-length-bytes and add new max-segment-size Use SSH for osiris dependency Basic cancel for stream queues Publish stream queues and settle/reject/requeue refactor Consume from stream queues Fix recovery Publish stream messages Add/delete stream replicas Use safe queue names Set retention policy for stream queues Required by the ctl command [#171207092] Stream queue delete queue fix missing callback impl Stream queue declare Queue type abstraction And use the implementing module as the value of the amqqueue record `type` field. This will allow for easy dispatch to the queue type implementation. Queue type abstraction Move queue declare into rabbit_queue_type Move queue delete into queue type implementation Queue type: dequeue/basic_get Move info inside queue type abstraction Move policy change into queue type interface Add purge to queue type Add recovery to the queue type interface Rename amqqueue quorum_nodes field To a more generic an extensible opaque queue type specific map. Fix tests and handle classic API response Fix HA queue confirm bug All mirrors need to be present as queue names. This introduces context linking allowing additional queue refs to be linked to a single "master" queue ref contining the actual queue context. Fix issue with events of deleted queues Also update queue type smoke test to use a cluster by default. correct default value of amqqueue getter Move classic queues further inside queue type interface why [TrackerId] Dialyzer fixes
* Switch to Mozilla Public License 2.0 (MPL 2.0)dcorbacho2020-07-101-12/+3
|
* Fix several type specsJean-Sébastien Pédron2020-05-191-4/+4
| | | | | | The most significant one is the type spec of `gm_pids` in the amqqueue records. I'm surprised Dialyzer didn't say anything about it. However it was reported by Dialyzer in `v3.7.x`.
* Update copyright (year 2020)Jean-Sébastien Pédron2020-03-101-1/+1
|
* (c) bumpMichael Klishin2019-12-291-1/+1
|
* Add marker rabbit_queue_type behaviourkjnilsson2019-08-071-25/+34
| | | | | | | | And use the implementing module as the value of the amqqueue record `type` field. This will allow for easy dispatch to the queue type implementation. Make amqqueue compatible with the classic queue tag
* amqqueue*: Uniformize APIJean-Sébastien Pédron2019-06-131-3/+80
| | | | | | Now, both modules export the same set of functions. This will help with the backport to `v3.7.x` and thus the life of plugin developers who will not have to have one copy of their plugin per RabbitMQ version.
* URL CleanupSpring Operator2019-03-201-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit updates URLs to prefer the https protocol. Redirects are not followed to avoid accidentally expanding intentionally shortened URLs (i.e. if using a URL shortener). # HTTP URLs that Could Not Be Fixed These URLs were unable to be fixed. Please review them to see if they can be manually resolved. * http://alvaro-videla.com/2013/09/rabbitmq-internals-credit-flow-for-erlang-processes.html (200) with 1 occurrences could not be migrated: ([https](https://alvaro-videla.com/2013/09/rabbitmq-internals-credit-flow-for-erlang-processes.html) result SSLHandshakeException). * http://blog.listincomprehension.com/search/label/procket (200) with 2 occurrences could not be migrated: ([https](https://blog.listincomprehension.com/search/label/procket) result ClosedChannelException). * http://dozzie.jarowit.net/trac/wiki/TOML (200) with 2 occurrences could not be migrated: ([https](https://dozzie.jarowit.net/trac/wiki/TOML) result SSLHandshakeException). * http://dozzie.jarowit.net/trac/wiki/subproc (200) with 2 occurrences could not be migrated: ([https](https://dozzie.jarowit.net/trac/wiki/subproc) result SSLHandshakeException). * http://e2project.org (200) with 2 occurrences could not be migrated: ([https](https://e2project.org) result AnnotatedConnectException). * http://erlang.org/doc/man/kernel_app.html (200) with 2 occurrences could not be migrated: ([https](https://erlang.org/doc/man/kernel_app.html) result ConnectTimeoutException). * http://erlang.org/pipermail/erlang-questions/2012-September/069320.html (200) with 1 occurrences could not be migrated: ([https](https://erlang.org/pipermail/erlang-questions/2012-September/069320.html) result ConnectTimeoutException). * http://nitrogenproject.com/ (200) with 4 occurrences could not be migrated: ([https](https://nitrogenproject.com/) result ConnectTimeoutException). * http://proper.softlab.ntua.gr (200) with 2 occurrences could not be migrated: ([https](https://proper.softlab.ntua.gr) result SSLHandshakeException). * http://proper.softlab.ntua.gr/ (200) with 1 occurrences could not be migrated: ([https](https://proper.softlab.ntua.gr/) result SSLHandshakeException). * http://rubybunny.info (200) with 1 occurrences could not be migrated: ([https](https://rubybunny.info) result AnnotatedConnectException). * http://yaws.hyber.org (200) with 2 occurrences could not be migrated: ([https](https://yaws.hyber.org) result AnnotatedConnectException). * http://choven.ca (503) with 2 occurrences could not be migrated: ([https](https://choven.ca) result ConnectTimeoutException). # Fixed URLs ## Fixed But Review Recommended These URLs were fixed, but the https status was not OK. However, the https status was the same as the http request or http redirected to an https URL, so they were migrated. Your review is recommended. * http://fixprotocol.org/ (301) with 1 occurrences migrated to: https://fixtrading.org ([https](https://fixprotocol.org/) result SSLHandshakeException). * http://erldb.org (UnknownHostException) with 1 occurrences migrated to: https://erldb.org ([https](https://erldb.org) result UnknownHostException). * http://www.erlang.org/cgi-bin/ezmlm-cgi?2:mss:1569 (404) with 1 occurrences migrated to: https://www.erlang.org/cgi-bin/ezmlm-cgi?2:mss:1569 ([https](https://www.erlang.org/cgi-bin/ezmlm-cgi?2:mss:1569) result 404). * http://www.rabbitmq.com/quorum-queues.html (404) with 1 occurrences migrated to: https://www.rabbitmq.com/quorum-queues.html ([https](https://www.rabbitmq.com/quorum-queues.html) result 404). ## Fixed Success These URLs were switched to an https URL with a 2xx status. While the status was successful, your review is still recommended. * http://cloudi.org/ with 27 occurrences migrated to: https://cloudi.org/ ([https](https://cloudi.org/) result 200). * http://erlware.org/ with 1 occurrences migrated to: https://erlware.org/ ([https](https://erlware.org/) result 200). * http://inaka.github.io/cowboy-trails/ with 1 occurrences migrated to: https://inaka.github.io/cowboy-trails/ ([https](https://inaka.github.io/cowboy-trails/) result 200). * http://ninenines.eu with 6 occurrences migrated to: https://ninenines.eu ([https](https://ninenines.eu) result 200). * http://pivotal.io with 1 occurrences migrated to: https://pivotal.io ([https](https://pivotal.io) result 200). * http://pubs.opengroup.org/onlinepubs/009695399/utilities/kill.html with 1 occurrences migrated to: https://pubs.opengroup.org/onlinepubs/009695399/utilities/kill.html ([https](https://pubs.opengroup.org/onlinepubs/009695399/utilities/kill.html) result 200). * http://www.actordb.com/ with 2 occurrences migrated to: https://www.actordb.com/ ([https](https://www.actordb.com/) result 200). * http://www.cs.kent.ac.uk/projects/wrangler/Home.html with 1 occurrences migrated to: https://www.cs.kent.ac.uk/projects/wrangler/Home.html ([https](https://www.cs.kent.ac.uk/projects/wrangler/Home.html) result 200). * http://www.erlang.org/ with 1 occurrences migrated to: https://www.erlang.org/ ([https](https://www.erlang.org/) result 200). * http://www.rabbitmq.com/access-control.html with 3 occurrences migrated to: https://www.rabbitmq.com/access-control.html ([https](https://www.rabbitmq.com/access-control.html) result 200). * http://www.rabbitmq.com/authentication.html with 2 occurrences migrated to: https://www.rabbitmq.com/authentication.html ([https](https://www.rabbitmq.com/authentication.html) result 200). * http://www.rabbitmq.com/clustering.html with 5 occurrences migrated to: https://www.rabbitmq.com/clustering.html ([https](https://www.rabbitmq.com/clustering.html) result 200). * http://www.rabbitmq.com/configure.html with 2 occurrences migrated to: https://www.rabbitmq.com/configure.html ([https](https://www.rabbitmq.com/configure.html) result 200). * http://www.rabbitmq.com/confirms.html with 1 occurrences migrated to: https://www.rabbitmq.com/confirms.html ([https](https://www.rabbitmq.com/confirms.html) result 200). * http://www.rabbitmq.com/dlx.html with 1 occurrences migrated to: https://www.rabbitmq.com/dlx.html ([https](https://www.rabbitmq.com/dlx.html) result 200). * http://www.rabbitmq.com/documentation.html with 1 occurrences migrated to: https://www.rabbitmq.com/documentation.html ([https](https://www.rabbitmq.com/documentation.html) result 200). * http://www.rabbitmq.com/download.html with 2 occurrences migrated to: https://www.rabbitmq.com/download.html ([https](https://www.rabbitmq.com/download.html) result 200). * http://www.rabbitmq.com/heartbeats.html with 1 occurrences migrated to: https://www.rabbitmq.com/heartbeats.html ([https](https://www.rabbitmq.com/heartbeats.html) result 200). * http://www.rabbitmq.com/lazy-queues.html with 1 occurrences migrated to: https://www.rabbitmq.com/lazy-queues.html ([https](https://www.rabbitmq.com/lazy-queues.html) result 200). * http://www.rabbitmq.com/ldap.html with 4 occurrences migrated to: https://www.rabbitmq.com/ldap.html ([https](https://www.rabbitmq.com/ldap.html) result 200). * http://www.rabbitmq.com/management.html with 6 occurrences migrated to: https://www.rabbitmq.com/management.html ([https](https://www.rabbitmq.com/management.html) result 200). * http://www.rabbitmq.com/memory-use.html with 3 occurrences migrated to: https://www.rabbitmq.com/memory-use.html ([https](https://www.rabbitmq.com/memory-use.html) result 200). * http://www.rabbitmq.com/memory.html with 2 occurrences migrated to: https://www.rabbitmq.com/memory.html ([https](https://www.rabbitmq.com/memory.html) result 200). * http://www.rabbitmq.com/monitoring.html with 1 occurrences migrated to: https://www.rabbitmq.com/monitoring.html ([https](https://www.rabbitmq.com/monitoring.html) result 200). * http://www.rabbitmq.com/nettick.html with 2 occurrences migrated to: https://www.rabbitmq.com/nettick.html ([https](https://www.rabbitmq.com/nettick.html) result 200). * http://www.rabbitmq.com/networking.html with 7 occurrences migrated to: https://www.rabbitmq.com/networking.html ([https](https://www.rabbitmq.com/networking.html) result 200). * http://www.rabbitmq.com/partitions.html with 2 occurrences migrated to: https://www.rabbitmq.com/partitions.html ([https](https://www.rabbitmq.com/partitions.html) result 200). * http://www.rabbitmq.com/persistence-conf.html with 3 occurrences migrated to: https://www.rabbitmq.com/persistence-conf.html ([https](https://www.rabbitmq.com/persistence-conf.html) result 200). * http://www.rabbitmq.com/plugins.html with 1 occurrences migrated to: https://www.rabbitmq.com/plugins.html ([https](https://www.rabbitmq.com/plugins.html) result 200). * http://www.rabbitmq.com/previous.html with 1 occurrences migrated to: https://www.rabbitmq.com/previous.html ([https](https://www.rabbitmq.com/previous.html) result 200). * http://www.rabbitmq.com/shovel.html with 2 occurrences migrated to: https://www.rabbitmq.com/shovel.html ([https](https://www.rabbitmq.com/shovel.html) result 200). * http://www.rabbitmq.com/ssl.html with 2 occurrences migrated to: https://www.rabbitmq.com/ssl.html ([https](https://www.rabbitmq.com/ssl.html) result 200). * http://www.rabbitmq.com/stomp.html with 1 occurrences migrated to: https://www.rabbitmq.com/stomp.html ([https](https://www.rabbitmq.com/stomp.html) result 200). * http://www.rebar3.org with 1 occurrences migrated to: https://www.rebar3.org ([https](https://www.rebar3.org) result 200). * http://contributor-covenant.org with 1 occurrences migrated to: https://contributor-covenant.org ([https](https://contributor-covenant.org) result 301). * http://contributor-covenant.org/version/1/3/0/ with 1 occurrences migrated to: https://contributor-covenant.org/version/1/3/0/ ([https](https://contributor-covenant.org/version/1/3/0/) result 301). * http://inaka.github.com/apns4erl with 1 occurrences migrated to: https://inaka.github.com/apns4erl ([https](https://inaka.github.com/apns4erl) result 301). * http://inaka.github.com/edis/ with 1 occurrences migrated to: https://inaka.github.com/edis/ ([https](https://inaka.github.com/edis/) result 301). * http://lasp-lang.org/ with 1 occurrences migrated to: https://lasp-lang.org/ ([https](https://lasp-lang.org/) result 301). * http://rabbitmq.com//cluster-formation.html with 1 occurrences migrated to: https://rabbitmq.com//cluster-formation.html ([https](https://rabbitmq.com//cluster-formation.html) result 301). * http://rabbitmq.com/access-control.html with 4 occurrences migrated to: https://rabbitmq.com/access-control.html ([https](https://rabbitmq.com/access-control.html) result 301). * http://rabbitmq.com/authentication.html with 2 occurrences migrated to: https://rabbitmq.com/authentication.html ([https](https://rabbitmq.com/authentication.html) result 301). * http://rabbitmq.com/clustering.html with 1 occurrences migrated to: https://rabbitmq.com/clustering.html ([https](https://rabbitmq.com/clustering.html) result 301). * http://rabbitmq.com/configure.html with 1 occurrences migrated to: https://rabbitmq.com/configure.html ([https](https://rabbitmq.com/configure.html) result 301). * http://rabbitmq.com/documentation.html with 2 occurrences migrated to: https://rabbitmq.com/documentation.html ([https](https://rabbitmq.com/documentation.html) result 301). * http://rabbitmq.com/heartbeats.html with 1 occurrences migrated to: https://rabbitmq.com/heartbeats.html ([https](https://rabbitmq.com/heartbeats.html) result 301). * http://rabbitmq.com/lazy-queues.html with 1 occurrences migrated to: https://rabbitmq.com/lazy-queues.html ([https](https://rabbitmq.com/lazy-queues.html) result 301). * http://rabbitmq.com/ldap.html with 3 occurrences migrated to: https://rabbitmq.com/ldap.html ([https](https://rabbitmq.com/ldap.html) result 301). * http://rabbitmq.com/logging.html with 1 occurrences migrated to: https://rabbitmq.com/logging.html ([https](https://rabbitmq.com/logging.html) result 301). * http://rabbitmq.com/management.html with 5 occurrences migrated to: https://rabbitmq.com/management.html ([https](https://rabbitmq.com/management.html) result 301). * http://rabbitmq.com/memory.html with 1 occurrences migrated to: https://rabbitmq.com/memory.html ([https](https://rabbitmq.com/memory.html) result 301). * http://rabbitmq.com/networking.html with 3 occurrences migrated to: https://rabbitmq.com/networking.html ([https](https://rabbitmq.com/networking.html) result 301). * http://rabbitmq.com/persistence-conf.html with 1 occurrences migrated to: https://rabbitmq.com/persistence-conf.html ([https](https://rabbitmq.com/persistence-conf.html) result 301). * http://rabbitmq.com/plugins.html with 1 occurrences migrated to: https://rabbitmq.com/plugins.html ([https](https://rabbitmq.com/plugins.html) result 301). * http://rabbitmq.com/shovel.html with 1 occurrences migrated to: https://rabbitmq.com/shovel.html ([https](https://rabbitmq.com/shovel.html) result 301). * http://rabbitmq.com/ssl.html with 1 occurrences migrated to: https://rabbitmq.com/ssl.html ([https](https://rabbitmq.com/ssl.html) result 301). * http://rabbitmq.com/stomp.html with 1 occurrences migrated to: https://rabbitmq.com/stomp.html ([https](https://rabbitmq.com/stomp.html) result 301). * http://saleyn.github.com/erlexec with 1 occurrences migrated to: https://saleyn.github.com/erlexec ([https](https://saleyn.github.com/erlexec) result 301). * http://www.erlang.org/doc/man/inet.html with 2 occurrences migrated to: https://www.erlang.org/doc/man/inet.html ([https](https://www.erlang.org/doc/man/inet.html) result 301). * http://www.erlang.org/doc/man/sys.html with 1 occurrences migrated to: https://www.erlang.org/doc/man/sys.html ([https](https://www.erlang.org/doc/man/sys.html) result 301). * http://www.mozilla.org/MPL/ with 228 occurrences migrated to: https://www.mozilla.org/MPL/ ([https](https://www.mozilla.org/MPL/) result 301). * http://zhongwencool.github.io/observer_cli with 1 occurrences migrated to: https://zhongwencool.github.io/observer_cli ([https](https://zhongwencool.github.io/observer_cli) result 301).
* Type spec: amqqueue:get_policy/1 returns a proplistMichael Klishin2019-03-151-1/+1
| | | | Spotted while dialyzing Federation plugin.
* Fix more Dialyzer warningsLoïc Hoguin2019-02-121-3/+3
|
* Make #amqqueue{} a private record + add a feature flagJean-Sébastien Pédron2019-02-011-0/+682
The #amqqueue{} record is now isolated in the `amqqueue` module: all accesses and modifications to the record are made through this helper. The type `rabbit_types:amqqueue()` is now `amqqueue:amqqueue()`. `amqqueue` knows about the new #amqqueue{} record introduced with quorum queues. If the actual record is the old one (as defined in RabbitMQ 3.7.x), it calls `amqqueue_v1` which knows about the old definition. When it needs to produce an #amqqueue{} record, it verifies if it is allowed to, using the new Feature flags subsystem and the `quorum_queue` feature flag specifically, and proceeds. If an old format #amqqueue{} is required, the creation is deferred to `amqqueue_v1`. The new modules come with a couple headers: they provide macros to replace pattern matching with `when` conditions. The `amqqueue_v*.hrl` headers are generated using the `macros()` functions in `amqqueue` and `amqqueue_v1` modules. Some operations now depend on the state of the `quorum_queue` feature flag. In particular, creating a quorum queue is denied if the feature flag is disabled. In the process, the following modules were moved from rabbitmq-common to rabbitmq-server: * rabbit_backing_queue * rabbit_queue_master_locator Likewise, the few functions were copied from `rabbit_misc` to `rabbit_amqqueue` because they depend on `amqqueue`. They don't really need to live in rabbitmq-common and they prevent further progress with the feature flags: feature flags need informations about the cluster. The `quorum_queue` feature flag comes with a migration function which is responsible of converting the #amqqueue{} records in Mnesia tables `rabbit_queue` and `rabbit_durable_queue`. Therefore, the upgrade function which did this is removed in this commit. [#159298729]