Where’s my message? Durability and you-程序员宅基地

技术标签: RabbitMQ  

There’s a dirty secret about creating queues and exchanges in Rabbit: by default they don’t survive reboot. That’s right; restart your RabbitMQ server and watch those queues and exchanges go poof (along with the messages inside). 

The reason is because of a property on every queue and exchange called  durable. It defaults to false, and tells RabbitMQ whether the queue (or exchange) should be re-created after a crash or restart of Rabbit. Set it to true and you won’t have to re-create those queues and exchanges when the power supply in your server dies. You might also think that setting durable to true on the exchanges and queues is all you need to do to make your messages survive a reboot, but you’d be wrong. Whereas queues and exchanges must be durable to allow messages to survive reboot, it isn’t enough on its own. 

 A message that can survive a crash of the AMQP broker is called persistent. You flag a message as persistent by setting the  delivery mode option of the message to 2 (your AMQP client may use a human-friendly constant instead) before publishing it. At this  point, the message is indicated as persistent, but it must be published to an exchange  that’s durable and arrive in a queue that’s durable to survive. If this weren’t the case,  the queue (or exchange) a persistent message was sitting in when Rabbit crashed  wouldn’t exist when Rabbit restarted, thereby orphaning the message. So, for a mes sage that’s in flight inside Rabbit to survive a crash, the message must 

Have its delivery mode option set to 2 (persistent) 
 Be published into a durable exchange 
 Arrive in a durable queue 

Do these three things and you won't have to play Where’s Waldo with your critical
messages. 
 The way that RabbitMQ ensures persistent messages survive a restart is by writing them to the disk inside of a persistency log file. When you publish a persistent message to a durable exchange, Rabbit won’t send the response until the message is committed to the log file. Keep in mind, though, that if it gets routed to a non durable queue after that, it’s automatically removed from the persistency log and won’t survive a restart. When you use persistent messages it’s crucial that you make sure  all three elements required for a message to persist are in place (we can’t stress this enough).
Once you consume a persistent message from a durable queue (and acknowledge it), RabbitMQ flags it in the persistency log for garbage collection. If Rabbit restarts any-time before you consume a persistent message, it’ll automatically re-create theexchanges and queues (and bindings) and replay any messages in the persistency log
into the appropriate queues or exchanges (depending on where in the routing process the messages were when Rabbit died). 


 You might be thinking that you should use persistent messaging for all of your messages. You could do that, but you’d pay a price for ensuring your messages survive Rabbit restarts: performance. The act of writing messages to disk is much slower than just storing them in RAM, and will significantly decrease the number of messages per second your RabbitMQ server can process. It’s not uncommon to see a 10x or more decrease in message throughput when using persistency.

There’s also the issue thatpersistent messages don’t play well with RabbitMQ’s built-in clustering. Though
RabbitMQ clustering allows you to talk to any queue present in the cluster from any
node, those queues are actually evenly distributed among the nodes  without redun-
dancy (there’s no backup copy of any queue on a second node in the cluster). If the
cluster node hosting your seed_bin queue crashes, the queue disappears from the
cluster until the node is restored … if the queue was durable. More important, while
the node is down its queues aren’t available and the durable ones can’t be re-created.
This can lead to black-holing of messages. We’ll cover the behavior in more detail and
show alternate clustering approaches to get around this in chapter 5. 
 Given the trade-offs, when should you use persistent/durable messaging? First, you
need to analyze (and test) your performance needs. Do you need to process 100,000

messages per second on a single Rabbit server? If so, you should probably look at
other ways of ensuring message delivery (or get a very fast storage system). For exam-
ple, your producer could listen to a reply queue on a separate channel. Every time it
publishes a message, it includes the name of the reply queue so that the consumer can
send a reply back to confirm receipt. If a message isn’t replied to within a reasonable
amount of time, the producer can republish the message. That said, the critical
nature of messages requiring guaranteed delivery generally means they’re lower in
volume than other types of messages (such as logging messages). So if persistent mes-
saging meets your performance needs, it’s an excellent way to help ensure delivery.
We use it a lot for critical messages. We’re just selective about what types of content
use persistent messaging. For example, we run two types of Rabbit clusters: traditional
RabbitMQ clustering for nonpersistent messaging, and pairs of active/hot-standby
nonclustered Rabbit servers for persistent messaging (using load balancers). This
ensures the processing load for persistent messaging doesn’t slow down nonpersistent
messages. It also means Rabbit’s built-in clustering won’t black-hole persistent mes-
sages when a node dies. Do keep in mind that while Rabbit can help ensure delivery, it
can never absolutely guarantee it. Hard drive corruption, buggy behavior by a con-
sumer, or other extreme events can trash/black-hole persistent messages. It’s ulti-
mately up to you to ensure your messages arrive where they need to go, and persistent
messaging is a great tool to help you get there. 
 A concept that’s related to the durability of a message is the AMQP transaction. So
far we’ve talked about marking messages, queues, and exchanges as durable. That’s all
well and good for keeping a message safe once RabbitMQ has it in its custody, but
since a publish operation returns no response to the producer, how do you know if
the broker has persisted the durable message to disk? Should the broker die before it
can write the message to disk, the message would be lost and you wouldn’t know.
That’s where transactions come in. When you absolutely need to be sure the broker
has the message in custody (and has routed the message to all matching subscribed
queues) before you move on to another t奋斗ask, you need to wrap it in a transaction. If 
you come from a database background, it’s important not to confuse AMQP transactions with what “transaction” means in most databases. In  AMQP, after you place a channel into transaction mode, you send it the publish you want to confirm, followed

by zero or more other AMQP commands that should be executed or ignored depend-
ing on whether the initial publish succeeded. Once you’ve sent all of the commands,
you commit the transaction. If the transaction’s initial publish succeeds, then the chan-
nel will complete the other AMQP commands in the transaction. If the publish fails,
none of the other  AMQP commands will be executed. Transactions close the “last
mile” gap between producers publishing messages and RabbitMQ committing them
to disk, but there’s a better way to close that gap. 
 Though transactions are a part of the formal AMQP 0-9-1 specification, they have
an Achilles heel in that they’re huge drains on Rabbit performance. Not only can
using transactions drop your message throughput by a factor of 2–10x, but they also
make your producer app synchronous, which is one of the things you’re trying to get 
rid of with messaging. Knowing all of this, the guys at RabbitMQ decided to come up with a better way to ensure message delivery: publisher confirms.

2
 Similar to transactions,
you have to tell Rabbit to place the channel into confirm mode, and you can’t turn it
off without re-creating the channel. Once a channel is in confirm mode, every mes-
sage published on the channel will be assigned a unique  ID number (starting at 1).
Once the message has been delivered to all queues that have bindings matching the
message’s routing key, the channel will issue a publisher confirm to the producer app
(containing the message’s unique  ID). This lets the producer know the message has
been safely queued at all of its destinations. If the message and the queues are dura-
ble, the confirm is issued only after the queues have written the message to disk. The
major benefit of publisher confirms is that they’re asynchronous. Once a message has
been published, the producer app can go on to the next message while waiting for the
confirm. When the confirm for that message is finally received, a callback function in
the producer app will be fired so it can wake up and handle the confirmation
. If an
internal error occurs inside Rabbit that causes a message to be lost, Rabbit will send a
message nack (not acknowledged) that’s like a publisher confirm (it has the message’s
unique ID) but indicates the message was lost. Also, since there’s no concept of mes-
sage rollback (as with transactions), publisher confirms are much lighter weight and
have an almost negligible performance hit on the Rabbit broker. 
 Now you have the individual parts of RabbitMQ down, from consumers and pro-
ducers to durable messaging, but how do they all fit together? What does the lifecycle
of an actual message look like from beginning to end? The best way to answer that is
to look at the life of a message in code. 

版权声明:本文为博主原创文章,遵循 CC 4.0 BY-SA 版权协议,转载请附上原文出处链接和本声明。
本文链接:https://blog.csdn.net/hanruikai/article/details/49024411

智能推荐

计算机毕业设计Java疫情防控医用品管理(系统+源码+mysql数据库+Lw文档)_疫情防护用品销售管理系统 论文-程序员宅基地

文章浏览阅读467次。计算机毕业设计Java疫情防控医用品管理(系统+源码+mysql数据库+Lw文档)springboot基于SpringBoot的婚庆策划系统的设计与实现。JSP健身俱乐部网站设计与实现sqlserver和mysql。JSP网上测试系统的研究与设计sqlserver。ssm基于SpringMvC的流浪狗领养系统。ssm基于Vue.js的音乐播放器设计与实现。ssm校园流浪猫图鉴管理系统的设计与实现。_疫情防护用品销售管理系统 论文

android插件化开发打包,Android项目开发如何设计整体架构-程序员宅基地

文章浏览阅读988次,点赞28次,收藏28次。最后小编想说:不论以后选择什么方向发展,目前重要的是把Android方面的技术学好,毕竟其实对于程序员来说,要学习的知识内容、技术有太多太多,要想不被环境淘汰就只有不断提升自己,从来都是我们去适应环境,而不是环境来适应我们!这里附上我整理的几十套腾讯、字节跳动,京东,小米,头条、阿里、美团等公司19年的Android面试题。把技术点整理成了视频和PDF(实际上比预期多花了不少精力),包含知识脉络 + 诸多细节。由于篇幅有限,这里以图片的形式给大家展示一小部分。

基于单片机数码管秒表控制系统设计-程序员宅基地

文章浏览阅读600次,点赞11次,收藏6次。*单片机设计介绍,基于单片机数码管秒表控制系统设计。

Python小程序之验证码图片生成_小程序图片验证码后端生成-程序员宅基地

文章浏览阅读235次。python小程序之验证码图片的生成定义随机字母的生成函数定义随机颜色生成函数,采用RGB格式,生成一个元组调用Image,生成画布,填充底色为白色调用画笔函数Draw,传入画布对象填充画布的每一个色块,作为背景在画布上控制间距,填上每一个字在最后的图上进行模糊操作代码# 生成一个随机的二维码小程序from PIL import Image,ImageDraw,ImageF..._小程序图片验证码后端生成

思科自防御网络安全方案典型配置_思科设备怎么ranga)服务器区域独立防护;-程序员宅基地

文章浏览阅读2.2k次。 1. 用户需求分析客户规模:客户有一个总部,具有一定规模的园区网络; 一个分支机构,约有20-50名员工; 用户有很多移动办公用户 客户需求:组建安全可靠的总部和分支LAN和WAN; 总部和分支的终端需要提供安全防护,并实现网络准入控制,未来实现对VPN用户的网络准入检查; 需要提供IPSEC/SSLVPN接入; 在内部各主要部门间,及内外网络间进_思科设备怎么ranga)服务器区域独立防护;

苹果账号迁移流程_apple 账号迁移-程序员宅基地

文章浏览阅读445次。4、转移账号生成的 p8 文件(证书文件)1、转移苹果账号的 teamID。2、接受苹果账号的 teamID。5、接受账号生成的 p8 文件。3、转移应用的 AppID。_apple 账号迁移

随便推点

深度学习中优化方法之动量——momentum、Nesterov Momentum、AdaGrad、Adadelta、RMSprop、Adam_momentum seg-程序员宅基地

文章浏览阅读1k次。https://blog.csdn.net/u012328159/article/details/80311892_momentum seg

动态数据生成静态html页_监听数据变更自动生成静态html-程序员宅基地

文章浏览阅读816次。主要的原理就是替换模板里的特殊字符。 1、静态模板页面 template.html,主要是定义了一些特殊字符,用来被替换。 HTML code DOCTYPE HT_监听数据变更自动生成静态html

预防按钮的多次点击 恶意刷新-程序员宅基地

文章浏览阅读494次。 今日在做一个新闻系统的评论时. 想到了预防"提交"按钮的多次点击的问提 (prevent multiple clicks of a submit button in ASP.NET). 以前碰到此类问提总是用重定位页面来解决. 这次我想找到一个一劳永逸的办法. 通过查讯Google,找到了一些代码,挑选一些较好的修改了一下。public void pa

sokcs5软件dante配置指南_dante 代理 配置pam用户名密码 模式-程序员宅基地

文章浏览阅读4.7k次。近来公司业务有需要做socks5代理的需求,研究了一下,主要的开源实现有2个:dante http://www.inet.no/dante/ss5 http://ss5.sourceforge.net/比较了一下,还是比较倾向于dante,因为看到有人这样评价ss5:Project has an incredibly poor source code quality. Th_dante 代理 配置pam用户名密码 模式

Excel vba 求助。_vba countifs 源码-程序员宅基地

文章浏览阅读809次。在excel vba 中用到countifs 函数,但用来统计带有特殊符号* 时总是统计chu_vba countifs 源码

web前端基础——实现动画效果_web前端实现图片动画效果-程序员宅基地

文章浏览阅读2.6k次。当两个效果之间变换时,可以使用transition过渡属性,但是有多个效果来回变换时,就需要使用动画效果,且动画过程可控(重复播放,画面暂停,最终画面等)文章目录1、简介2、实现步骤3、复合属性animation4、动画属性1、简介动画的本质是快速切换大量图片在人脑中形成的具有连续性的画面构成动画的最小单元:帧或者动画帧2、实现步骤定义动画@keyframes 动画名称{ from{} to{}}@keyframes 动画名称{ 0%{} 10%{} 20%{} 50._web前端实现图片动画效果

推荐文章

热门文章

相关标签