You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
ARTEMIS-5606 refactor doc chapter on duplicate detection
Improvements in this commit include:
- Fixing a broken code example for the Core client
- Adding sub-sections to make it easier to find relevant info quickly
- Adding more configuration samples
- Fixing grammatical issues to increase readability
- Adding new sub-section on performance
Apache ActiveMQ Artemis includes powerful automatic duplicate message detection, filtering out duplicate messages without you having to code your own fiddly duplicate detection logic at the application level.
7
-
This chapter will explain what duplicate detection is, how Apache ActiveMQ Artemis uses it and how and where to configure it.
7
+
This chapter will explain what duplicate detection is, how Apache ActiveMQ Artemis uses it, and how to configure it.
8
8
9
9
When sending messages from a client to a server, or indeed from a server to another server, if the target server or connection fails sometime after sending the message, but before the sender receives a response that the send (or commit) was processed successfully then the sender cannot know for sure if the message was sent successfully to the address.
10
10
11
11
If the target server or connection failed after the send was received and processed but before the response was sent back then the message will have been sent to the address successfully, but if the target server or connection failed before the send was received and finished processing then it will not have been sent to the address successfully.
12
12
From the senders point of view it's not possible to distinguish these two cases.
13
13
14
14
When the server recovers this leaves the client in a difficult situation.
15
-
It knows the target server failed, but it does not know if the last message reached its destination ok.
16
-
If it decides to resend the last message, then that could result in a duplicate message being sent to the address.
15
+
It knows the target server failed, but it does not know if the last message reached its destination successfully.
16
+
If it decides to resend the last message then that could result in a duplicate message being sent to the address.
17
17
If each message was an order or a trade then this could result in the order being fulfilled twice or the trade being double booked.
18
18
This is clearly not a desirable situation.
19
19
20
-
Sending the message(s) in a transaction does not help out either.
20
+
Sending the message(s) in a transaction does not help either.
21
21
If the server or connection fails while the transaction commit is being processed it is also indeterminate whether the transaction was successfully committed or not!
22
22
23
23
To solve these issues Apache ActiveMQ Artemis provides automatic duplicate messages detection for messages sent to addresses.
24
24
25
25
== Using Duplicate Detection for Message Sending
26
26
27
-
Enabling duplicate message detection for sent messages is simple: you just need to set a special property on the message to a unique value.
27
+
To enable duplicate message detection for sent messages you just need to set a special duplicate ID property on the message to a *unique* value.
28
28
You can create the value however you like, as long as it is unique.
29
-
When the target server receives the message it will check if that property is set, if it is, then it will check in its in memory cache if it has already received a message with that value of the header.
30
-
If it has received a message with the same value before then it will ignore the message.
31
29
32
30
[NOTE]
33
31
====
34
-
35
-
36
32
Using duplicate detection to move messages between nodes can give you the same _once and only once_ delivery guarantees as if you were using an XA transaction to consume messages from source and send them to the target, but with less overhead and much easier configuration than using XA.
37
33
====
38
34
39
-
If you're sending messages in a transaction then you don't have to set the property for _every_ message you send in that transaction, you only need to set it once in the transaction.
40
-
If the server detects a duplicate message for any message in the transaction, then it will ignore the entire transaction.
35
+
If you're sending messages in a transaction then you don't have to set the property for _every_ message you send in that transaction.
36
+
You only need to set it once in the transaction.
37
+
If the server detects a duplicate message for any message in the transaction then it will ignore the entire transaction.
41
38
42
-
The name of the property that you set is given by the value of `org.apache.activemq.artemis.api.core.Message.HDR_DUPLICATE_DETECTION_ID`, which is `_AMQ_DUPL_ID`
39
+
The name of the duplicate ID property is `_AMQ_DUPL_ID`. As a convenience for Java-based applications using the Core client `org.apache.activemq.artemis.api.core.Message.HDR_DUPLICATE_DETECTION_ID` can be used.
43
40
44
-
When using JMS the value of the property must be a `String`, and similarly a string type would be used in other client APIs or protocols used with the broker.
45
-
Its value should be unique.
46
-
An easy way of generating a unique id is by generating a UUID.
41
+
When using JMS the property's value must be a `String`, and similarly a string type would be used in other client APIs or protocols used with the broker.
47
42
48
43
Here's an example of setting the property using the JMS API:
49
44
50
45
[,java]
51
46
----
52
-
...
53
-
54
47
Message jmsMessage = session.createMessage();
55
48
56
49
String myUniqueID = "This is my unique id"; // Could use a UUID for this
The server maintains caches of received values of the `org.apache.activemq.artemis.core.message.impl.HDR_DUPLICATE_DETECTION_ID` property sent to each address.
81
-
Each address has its own distinct cache.
69
+
The server maintains a *circular*, fixed-size, per-address cache of duplicate IDs from messages it receives.
82
70
83
-
The cache is a circular fixed size cache.
84
-
If the cache has a maximum size of `n` elements, then the ``n + 1``th id stored will overwrite the ``0``th element in the cache.
71
+
When the server receives the message it will check if the duplicate ID property is set.
72
+
If it is then it will check to see if its cache for the correspond address already contains that duplicate ID.
73
+
If the cache already contains that duplicate ID then the message will not be routed to any queues, and the server will log a `WARN` message, e.g.:
74
+
[,console]
75
+
----
76
+
WARN [org.apache.activemq.artemis.core.server] AMQ222059: Duplicate message detected - message will not be routed. Message information:
If the cache does not contain that duplicate ID then it is added to the cache and the message is routed to any applicable queues.
85
80
86
-
The maximum size of the cache is configured by the parameter `id-cache-size` in `broker.xml`, the default value is `20000` elements.
81
+
Since the cache is circular then if it has a maximum size of `n` elements the ``n + 1``th id stored will overwrite the ``0``th element in the cache.
82
+
Duplicate IDs are _only_ removed from the cache when they are overwritten or cleared administratively (e.g. using the `clearDuplicateIdCache` operation on the corresponding xref:management.adoc#address-management[`AddressControl`] from the web console).
83
+
Even if a message is acknowledged or expires its duplicate ID is not removed from the cache because another message with that same duplicate ID may still be sent.
87
84
88
-
To implement an address-specific `id-cache-size`, you can add to the
89
-
corresponding address-settings section in `broker.xml`. Specify the
90
-
desired `id-cache-size` value for the particular address. When a message
91
-
is sent to an address with a specific `id-cache-size` configured, it
92
-
will take precedence over the global `id-cache-size` value, allowing
93
-
for greater flexibility and optimization of duplicate ID caching.
85
+
== Configuring the Duplicate ID Cache
94
86
95
-
The caches can also be configured to persist to disk or not.
96
-
This is configured by the parameter `persist-id-cache`, also in `broker.xml`.
97
-
If this is set to `true` then each id will be persisted to permanent storage as they are received.
98
-
The default value for this parameter is `true`.
87
+
The size of the duplicate ID cache can be configured globally for all addresses or on a per-address basis.
88
+
89
+
Whether the cache is persisted to storage is also configurable.
99
90
100
91
[NOTE]
101
92
====
102
-
103
-
104
93
When choosing a size of the duplicate id cache be sure to set it to a larger enough size so if you resend messages all the previously sent ones are in the cache not having been overwritten.
105
94
====
106
95
96
+
=== Global Configuration
97
+
98
+
The maximum size of the cache is configured by the parameter `id-cache-size` in `broker.xml`, e.g.:
99
+
100
+
[,xml]
101
+
----
102
+
<core>
103
+
...
104
+
<id-cache-size>5000</id-cache-size>
105
+
...
106
+
</core>
107
+
----
108
+
109
+
The default value for the global `id-cache-size` is `20000`. A value of `0` disables caching.
110
+
111
+
=== Address-Specific Configuration
112
+
113
+
To configure the cache size on a per-address basis use the `id-cache-size` `address-settings` section in `broker.xml`, e.g.:
114
+
115
+
[,xml]
116
+
----
117
+
<address-setting match="myAddress">
118
+
...
119
+
<id-cache-size>1000</id-cache-size>
120
+
...
121
+
</address-setting>
122
+
----
123
+
124
+
When a message is sent to an address with a specific `id-cache-size` configured it will take precedence over the global `id-cache-size` value.
125
+
This allows for greater flexibility and optimization of duplicate ID caches.
126
+
127
+
The default value for the per-address `id-cache-size` is `20000`. A value of `0` disables caching.
128
+
129
+
=== Persisting the Cache to Storage
130
+
131
+
Duplicate ID caches are persisted to storage by default.
132
+
The benefit to persisting the cache to storage is that if the broker is stopped for any reason then when it restarts the data will be read from storage back into the cache so duplicate messages can still be detected even if they were sent before the broker restarted.
133
+
However, there is a cost in terms of performance since it takes longer to persist the data.
134
+
135
+
Duplicate ID cache persistence is configured by the parameter `persist-id-cache` in `broker.xml`, e.g.:
136
+
137
+
[,xml]
138
+
----
139
+
<core>
140
+
...
141
+
<persist-id-cache>false</id-cache-size>
142
+
...
143
+
</core>
144
+
----
145
+
If `persist-id-cache` is set to `true` then each ID will be persisted to storage as it is received.
146
+
This is configured globally.
147
+
It can't be configured on a per-address basis.
148
+
149
+
The default value for `persist-id-cache` is `true`.
150
+
107
151
== Duplicate Detection and Bridges
108
152
109
153
Core bridges can be configured to automatically add a unique duplicate id value (if there isn't already one in the message) before forwarding the message to its target.
@@ -125,3 +169,18 @@ To configure a cluster connection to add the duplicate id header, simply set the
125
169
The default value for this parameter is `true`.
126
170
127
171
For more information on cluster connections and how to configure them, please see xref:clusters.adoc#clusters[Clusters].
172
+
173
+
== Performance Considerations
174
+
175
+
If you *do not need* duplicate detection at all or only for certain addresses it is best to set the global `id-cache-size` to `0` to prevent the server from pre-allocating internal cache-related objects, e.g.:
176
+
177
+
[,xml]
178
+
----
179
+
<core>
180
+
...
181
+
<id-cache-size>0</id-cache-size>
182
+
...
183
+
</core>
184
+
----
185
+
186
+
This will prevent needless consumption of heap memory so it is available to the broker for other uses.
0 commit comments