问题描述
Tl;dr:我如何通过一堆异步、无序的微服务推送消息，并知道该消息何时通过每个微服务?"
我正在努力为特定的微服务架构寻找合适的消息传递系统/协议.这不是一个哪个最好"的问题，而是一个关于我的设计模式/协议的选择的问题.
- 我在开始队列中有一条消息.假设一个带有序列化 JSON 的 RabbitMQ 消息
- 我需要该消息通过任意数量的微服务
- 这些微服务中的每一个都需要长时间运行，必须是独立的，并且可以用多种语言实现
- 消息通过的服务顺序无关紧要.事实上，它不应该是同步的.
- 每个服务都可以附加数据到原始消息中，但其他服务会忽略该数据.应该没有合并冲突(每个服务写一个唯一的键).任何服务都不会更改或破坏数据.
- 一旦轮到所有服务，消息应该发布到带有原始数据和新数据的第二个 RabbitMQ 队列.
- 微服务不会有其他副作用.如果这一切都在一个单体应用程序中(并且使用同一种语言)，那么函数式编程将是完美的.
那么，问题是，通过各种服务管理该消息的合适方法是什么?我不想一次要做一个，而且顺序并不重要.但是，如果是这种情况，系统如何知道所有服务何时都已完成并且最终消息可以写入结束队列(让下一批服务开始运行).
我能想出的唯一的、半优雅的解决方案是
- 让第一个遇到消息的服务将该消息写入公共存储(例如 mongodb)
- 让每个服务做它自己的事情，标记它已完成该消息，然后检查是否所有服务都轮到了
- 如果是这样，最后一个服务将发布消息
但这仍然需要每个服务都知道所有其他服务并且需要每个服务留下自己的标记.这两个都不想要.
我愿意接受某种牧羊人"服务.
我会感谢我错过的任何选项，并愿意承认它们可能是更好的基本设计.
谢谢.
管理长时间运行的进程(或涉及多个微服务的处理)有两种方法:编排和编排.有很多文章描述它们.
长话短说:在编排中，您有一个微服务来跟踪流程状态，而在编排中，所有微服务都知道去哪里发送下一条消息和/或在进程完成时发送.
这篇
编排优势
- 当有同步处理时，为控制应用程序的流程提供了一个很好的方法.例如，如果服务 A 需要在调用服务 B 之前成功完成.
编排权衡
将服务耦合在一起创建依赖项.如果服务 A 关闭，则永远不会调用服务 B 和 C.
如果所有请求都有一个协调器的中央共享实例，那么协调器就是单点故障.如果它出现故障，则所有处理都会停止.
利用阻止请求的同步处理.在本例中，端到端的总处理时间是调用服务 A + 服务 B + 服务 C 所需的时间总和.
编舞
编舞的好处
支持更快的端到端处理，因为服务可以并行/异步执行.
更容易添加/更新服务，因为它们可以轻松地插入/拔出事件流.
非常适合敏捷交付模型，因为团队可以专注于特定服务而不是整个应用程序.
控制是分布式的，因此不再有一个单一的协调器作为故障的中心点.
多种模式可以与反应式架构一起使用，以提供额外的好处.例如，事件溯源是指事件流存储所有事件并启用事件重播.这样，如果一个服务在事件仍在产生时出现故障，当它重新上线时，它可以重播这些事件以追赶上.此外，可以应用命令查询职责分离 (CQRS) 来分离读取和写入活动.这使得这些中的每一个都可以独立缩放.如果您的应用程序读重读轻写，反之亦然.
编舞权衡
异步编程对于开发人员来说通常是一个重要的思想转变.我倾向于认为它类似于递归，您无法通过查看代码来确定代码将如何执行，您必须考虑在特定时间点可能为真的所有可能性.
复杂性发生了变化.流控制现在不再集中在编排器中，而是分解并分布在各个服务中.每个服务都有自己的流逻辑，该逻辑将根据事件流中的特定数据确定何时以及如何做出反应.
Tl;dr: "How can I push a message through a bunch of asynchronous, unordered microservices and know when that message has made it through each of them?"
I'm struggling to find the right messaging system/protocol for a specific microservices architecture. This isn't a "which is best" question, but a question about what my options are for a design pattern/protocol.
- I have a message on the beginning queue. Let's say a RabbitMQ message with serialized JSON
- I need that message to go through an arbitrary number of microservices
- Each of those microservices are long running, must be independent, and may be implemented in a variety of languages
- The order of services the message goes through does not matter. In fact, it should not be synchronous.
- Each service can append data to the original message, but that data is ignored by the other services. There should be no merge conflicts (each service writes a unique key). No service will change or destroy data.
- Once all the services have had their turn, the message should be published to a second RabbitMQ queue with the original data and the new data.
- The microservices will have no other side-effects. If this were all in one monolithic application (and in the same language), functional programming would be perfect.
So, the question is, what is an appropriate way to manage that message through the various services? I don't want to have to do one at a time, and the order isn't important. But, if that's the case, how can the system know when all the services have had their whack and the final message can be written onto the ending queue (to have the next batch of services have their go).
The only, semi-elegant solution I could come up with was
- to have the first service that encounters a message write that message to common storage (say mongodb)
- Have each service do its thing, mark that it has completed for that message, and then check to see if all the services have had their turn
- If so, that last service would publish the message
But that still requires each service to be aware of all the other services and requires each service to leave its mark. Neither of those is desired.
I am open to a "Shepherd" service of some kind.
I would appreciate any options that I have missed, and am willing to concede that their may be a better, fundamental design.
Thank you.
There are two methods of managing a long running process (or a processing involving multiple microservices): Orchestration and choreography. There are a lot of articles describing them.
Long story short: In Orchestration you have a microservice that keeps track of the process status and in Choreography all the microservices know where to send next the message and/or when the process is done.
This article explains the benefits and tradeofs of the two styles.
Orchestration
Orchestration Benefits
- Provides a good way for controlling the flow of the application when there is synchronous processing. For example, if Service A needs to complete successfully before Service B is invoked.
Orchestration Tradeoffs
Couples the services together creating dependencies. If service A is down, service B and C will never be called.
If there is a central shared instance of the orchestrator for all requests, then the orchestrator is a single point of failure. If it goes down, all processing stops.
Leverages synchronous processing that blocks requests. In this example, the total end-to-end processing time is the sum of time it takes for Service A + Service B + Service C to be called.
Choreography
Choreography Benefits
Enables faster end-to-end processing as services can be executed in parallel/asynchronously.
Easier to add/update services as they can be plugged in/out of the event stream easily.
Aligns well with an agile delivery model as teams can focus on particular services instead of the entire application.
Control is distributed, so there is no longer a single orchestrator serving as a central point of failure.
Several patterns can be used with a reactive architecture to provide additional benefits. For example, Event Sourcing is when the Event Stream stores all of the events and enables event replay. This way, if a service went down while events were still being produced, when it came back online it could replay those events to catch back up. Also, Command Query Responsibility Segregation (CQRS) can be applied to separate out the read and write activities. This enables each of these to be scaled independently. This comes in handy if you have an application that is read-heavy and light on writes or vice versa.
Choreography Tradeoffs
Async programming is often a significant mindshift for developers. I tend to think of it as similar to recursion, where you can’t figure out how code will execute by just looking at it, you have to think through all of the possibilities that could be true at a particular point in time.
Complexity is shifted. Instead of having the flow control centralized in the orchestrator, the flow control is now broken up and distributed across the individual services. Each service would have its own flow logic, and this logic would identify when and how it should react based on specific data in the event stream.
这篇关于微服务架构 - 当顺序无关紧要时通过服务传递消息的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！