A Brief History of Microservices and Message Queues
Ever since Dr. Peter Rodgers first coined the term "Micro-Web-Services" during a presentation at the Web Services Edge conference in 2005, the IT industry slowly shifted from monolithic design to microservices.
Netflix made the decision to break their monolith into microservices back in 2009. Best Buy started to transform their monolithic architecture to a microservice architecture in 2010. eBay introduced microservices in 2011.
If you go back to 2001, Amazon’s retail website was a large architectural monolith. At Amazon’s re:Invent 2015 conference, Rob Brigham explained the change. By separating their backend into loosely-coupled services, isolating the services irrelevant in terms of their tech stack, and having smaller dev teams focus on a well-defined service, they could deliver software more efficiently and at scale.
Today, backends almost always follow the microservices architectural design pattern. While message queue-based inter-process or inter-thread communication dates back to the early 1980s with UNIX System V API or perhaps other real-time OS kernels, the birth of message brokers operating in a network of different computer systems as we know it primarily started at 2000s.
RabbitMQ v1.0.0 was released on 1 July 2007. Kafka was originally developed at LinkedIn, and was subsequently open sourced in early 2011. Amazon SQS went into beta in late 2004 and became generally available in mid 2006.
Synchronous vs. Asynchronous
Once a microservice architecture is chosen as the path for the development of the backend, a decision must be made for communication between the services.
- Synchronous protocol: like HTTP (REST, SOAP or any other RPC)
- Asynchronous protocol: message queue protocols like AMQP(RabbitMQ), Apache Kafka, STOMP, MQTT, etc.
A synchronous approach is more error-prone, harder to debug, and harder to recover. Many industry professionals pass this option over.
On the other hand, the asynchronous approach gives the architecture a single, solid message bus. This makes it easier to debug, less error-prone, and more secure. Data is delivered between services in a reliable fashion, making asynchronous the obvious choice.
That is, unless you have legacy services in an older programming language, or have an older infrastructure that can’t be easily modernized. Moving into the cloud-native era however, usually requires businesses to choose an asynchronous design.
Once the asynchronous approach is chosen, the architects need to decide what message broker software and the message queue protocol to use.
The World of Message Queues
My journey of understanding the similarities and differences began with the objective of adding support of several message queue protocols like AMQP, Apache Kafka, Redis etc. to Mizu. Mizu is a Kubernetes-aware deep inspection tool that lets you inspect the traffic between your microservices.
Implementing the protocol dissectors for such message queue protocols gave me in-depth knowledge for both the protocols and the broker software.
Here is a list of prominent message broker software and cloud services in the year 2021:
- Apache Qpid
- Apache Kafka
- Apache ActiveMQ
- Eclipse OpenMQ
- Eclipse Mosquitto
- Solace PubSub+
- Google Cloud Pub/Sub
- Amazon SQS
- Amazon MQ
- IBM MQ
- Azure Event Hubs
- Azure Service Bus
Protocol Comparison Matrix
Here is a message queue protocol comparison matrix based on the client libraries' perspective:
Note: headers indicates a dictionary/map with arbitrary amount of keys while attrs indicates a limited set of key-value pairs.
Note: We didn't include XMPP into this list of comparison because it's not so popular in the area of microservices.
- Every protocol has some sort of a queue concept.
- All protocols are TCP based.
- Every protocol has some sort of a producer/consumer concept.
- Every protocol has payload/body.
- AMQP has different message delivery models.
- Cloud-based message queues have dead-letter queues.
- The message retrieval method differs.
- Routing key has a limited support.
- Headers and/or meta attributes have limited support.
- Redis, STOMP, MQTT are the least feature-rich while AMQP is the most feature-rich.
- Cloud based message queues have some unique configuration related API.
Protocols In Detail
The choice of the message queue protocol is actually more important than the choice of the broker software. This is because if you choose a more generic protocol, it's easy to replace the broker with something else.
Advanced Message Queue Protocol (AMQP) is a TCP-based binary protocol, which has become an ISO and OASIS standard. It's primarily used by RabbitMQ. There are different versions of the AMQP specification and they are, in some cases, wildly different from the previous one. RabbitMQ uses AMQP version 0-9-1 making that version the most commonly-used. The other versions are 0-8, 0-9, 0-10 and 1.0. AMQP version 1.0 is the most common among the cloud service providers and RabbitMQ also supports 1.0 via a plugin. 1.0 is the OASIS standard.
- The different message delivery models reduces the architectural complexity by solving different use cases at protocol level.
- It's an ISO and OASIS standard and it's widely adopted.
- It's fast, secure and maybe the most mature among the message queue protocols.
- Effortlessly switching between cloud providers like Azure Service Bus or Solace PubSub+ to your own RabbitMQ server or vice versa is possible.
- Queue mirroring both using classical or Quorum queues makes it easily scalable.
- Message size limit in RabbitMQ was 2GB up to version 3.7.0, with 3.8.0 it's reduced to 128MB.
- It's not backward compatible with old versions. Client libraries implement only a single version of the protocol and migrating between the versions might be time consuming.
- Relying on many RabbitMQ plugins may introduce maintenance challenges.
- Debugging and monitoring can be problematic.
- While AMQP feels like the standardized message queue protocol, it's not really supported by the majority of message broker software alternatives.
Apache Kafka is both the name of the message broker software and the protocol itself. As of 2021, there are 12 versions of the protocol and client libraries implement all of them at once.
It's a project developed and maintained by the Apache Software Foundation, written in Scala and Java.
The Apache Kafka team chooses to define their own protocol, instead of adopting AMQP or STOMP for example. This is because they hold that the protocol defines the implementation, therefore it reduces their freedom for creating a truly distributed messaging system and performing certain optimizations.
- Apache Kafka with partitioning and replication makes it easy to scale.
- The protocol provides a way (and also suggests) to send small messages in batches. This makes the protocol highly efficient compared to others.
- The Admin API makes it easy to debug.
- Kafka ZooKeeper and Kafka Brokers requires you to deploy at least two containers. Configuration of this deployment might be challenging for beginners.
- Client libraries might have a hard-time of keeping up with ever-changing protocol specification. Therefore upgrades might also be challenging.
- Message size limit is 1MB.
Much like AMQP, Streaming Text Oriented Messaging Protocol (STOMP) is a text-based protocol not specific to a certain message broker software. But it lacks lot of functionality and optimizations that other protocols have. On the other hand, the protocol's simplicity makes it easier to adopt. Therefore, there are many client libraries for this protocol and RabbitMQ supports STOMP via a plugin as well. It's logical to use STOMP for simpler use cases or for fast prototyping.
- Simplicity makes it easier to adopt.
- RabbitMQ support via a plugin.
- Less functionality and optimizations than other protocols.
Message Queuing Telemetry Transport (MQTT) is the message queue protocol for the Internet of Things (IoT). It's an ISO and OASIS standard. The most notable brokers that support MQTT are Eclipse Mosquitto and HiveMQ.
- Not suitable for microservices that are not IoT related.
- Not feature-rich.
Redis Serialization Protocol (Redis/RESP) is the protocol of Redis, an in-memory key–value database. Redis is not technically a message queue software, but through some client libraries, it can be used for that purpose. It's mainly the choice for the people who are so in love with Redis in their tech stack that they also use Redis as the message bus.
- Quite fast because of its in-memory nature.
- Shallow learning curve for people who use Redis in their tech stack already.
- Not feature-rich.
- Not suitable for every use case that other protocols solve.
The message queue services of cloud providers are the way to go if you don't want to maintain the infrastructure, you want to automatically scale, and you already use cloud services directly as your backend such as Firebase or serverless functions. They are all HTTP-based, so don't expect to have the efficiency of a dedicated message queue protocol.
So Which Protocol Should I Choose?
In conclusion, go with AMQP (RabbitMQ) or Apache Kafka in general. They work and scale well.
Don't go with STOMP; it's a wasted potential. It hasn’t been adopted well and it’s not really optimized in any way.
If you're an IoT company then go with MQTT. It's cut out for IoT.
If you're in love with Redis and can't handle adding a new technology to your stack then go with Redis.
If you're not maintaining your own infrastructure and you’re heavily dependent on cloud services already then go with a cloud service for the message queue solution as well.
UP9 recently released Version 0.14.0 of its open source API Traffic Viewer, Mizu. Read more about the changes here.
The most notable update was increased protocol support. This has brought support for Kafka, Redis, and AMQP (aka RabbitMQ). These protocols are in addition to REST and gRPC.
Additionally, you can now use Mizu to view API traffic between microservices that are using numerous protocols to communicate. For a quick guide on how to see multiple-protocol support in action, go here.
UP9 enables your APIs perform reliably and securely, and helps you ensure they're production-ready.
Through unpacking synchronous and message-based protocols and continuously monitoring your API payload, UP9 detects API regressions and prevents vulnerabilities from drifting into production.
UP9 can be installed in under 5 minutes in pre and post-production environments (e.g. CI, staging, canary, or production) without coding or SDKs.
You can sign up now for free!