In the past few years, we've seen the asynchronous communication paradigm explode in popularity. This has led to the massive adoption of event stream processing applications, like Apache Kafka. This app is used by many but fully understood by few.
While these days it may be tempting to go with Kafka as soon as development is underway, there are more considerations for DevOps and engineering leaders. No two architectures are exactly the same. There are nearly infinite uses for applications out there. Instead of going with what is most popular, leaders should consider finding the right fit.
With that in mind, there are a few important questions to ask yourself when selecting an event streaming application.
Note: For the purposes of this discussion, apps like Kafka will be referred to as a "message bus" or "event streaming processor", although Kafka (and similar services) may have more/other use cases.
Breaking the Kafka Monolith
Some reading this may be wondering, "well, why not go with Kafka?"
That's a fantastic question. After all, Confluent, built atop Apache Kafka, is used by many of the top enterprises in the world. It has gained popularity for its extreme usefulness and adaptability. Though difficult to master, it's not difficult to understand Kafka's utility for cloud-native applications, especially with loosely coupled services that benefit from asynchronous communication patterns.
This is why some product and engineering leads refer to Kafka as the new "IBM monolith" that Apple Computers famously broke in 1984 when they introduced the world to the Mackintosh. Sometimes leaders go with Kafka because "nobody ever got fired for buying IBM." In other words, Kafka is a safe bet due to its popularity.
But, since 1984, whenever one product becomes dominant in the world of computing, competitors rise to break its hold. The same thing is happening today in the world of event streaming.
There is no one-size-fits-all solution to event streaming and messaging. Different apps facilitate different needs. Throughout the past couple of years, competitors to Kafka have arisen, some of whom use a similar frontend but a more lightweight and efficient backend. Others have come up to serve more bespoke needs or converted to function within much larger systems like Amazon and their Kinesis project.
For an engineering leader at a company of any size, the risk of selecting the wrong service is clear. If one ends up with the wrong service, or at least something that doesn't fit their needs, then migrating to something else can be costly in terms of time, money, and energy.
There is also the risk of committing (financially and architecturally) to a service that has too many irrelevant functions. A smaller architecture could be served by something else. After all, this is why some engineering leaders start out with a monolithic architecture instead of immediately jumping to microservices.
For example, there doesn't seem to be a singular direction that the developers of Apache Kafka itself are taking. They are building a turnkey solution. But that key might not fit your lock. For example, Kafka can be used for event streaming, analytics, data storage, monitoring, event processing, data management, and more. That's why its users vary so much, from Uber to Netflix, to Intuit.
With all that said, it is the author's opinion that Kafka should not necessarily be the first choice, but it should be considered along with other alternatives such as Pulsar, Redis for event streams, and RabbitMQ and ActiveMQ for message queues.
Here are questions to ask yourself before going with Kafka, or shopping somewhere else.
1- How much of your app will be an event stream?
Some applications use a message bus as a little reference tool or database within the larger framework. Compared to the entire architecture, the message bus is tiny. Some organizations need less event processing power and can get by with a simpler message queue like RabbitMQ or ActiveMQ
On the other end of the spectrum, Cloud Native apps can be comprised mostly of an application like Kafka. It depends on the level of complex data processing the application requires.
Before selecting your message bus, ask yourself how much volume you expect it to utilize in your architecture. The more space you expect, the more functionality you can expect as well.
2- What volume of data do you expect?
Since every application is different, you could see exponentially more or less data passing through your system. You don't need a 12 core processor to run Candy Crush, after all.
Some engineering leaders will prefer to focus on data caching without sacrificing throughput. In this instance, they may explore using Redis Streams (or its native Windows port Memurai) instead of Kafka. They have similar durability and fault tolerance with advantages in terms of crash resistances. And they cost far less to maintain, making them a favored choice for new and/or smaller teams. Especially because Redis has viable scaling strategies as the project and team grow.
3- How comfortable would you be migrating to another service down the road?
This is an incredibly important question for startups. We've discussed in the past whether startups should use a monolithic or microservice architecture from the very beginning. Similarly, a message bus may be something you want your architecture to grow into, rather than going with something limited and then migrating later.
This is why some startups choose Apache Kafka because they know that it will always match their scale. Unless they get as large as Amazon, I suppose. But even then, there are other options that can also scale with time.
4- Who on your team has the expertise to optimize/tune something like this?
This is a pivotal question, especially for smaller teams. Not every team has access to a Kafka guru or someone who's experienced working with Redis, Memurai, or Pulsar.
Generally speaking, because of Apache Kafka's popularity, it's easier to find a Kafka expert and have them make sure you get the most out of it. But few people claim to have full mastery of all of Kafka's use cases, and if they are, they may be padding their resume.
So engineering leaders have to know who on their team has the skills to tune, optimize, and maintain an event stream processor, and which apps they have the most experience working with.
5- What is your budget?
Most event streams operate with a freemium model, making it completely free to get basic features and then more complex features as the user rises in tiers. As you shop for the right event stream, you need to make sure you aren't paying for features you don't use.
Confluent offers Apache Kafka in a model where costs scale well with use, but they're not the only ones. Many services built atop Redis and Pulsar have custom pricing as well. You should consider going with a service that scales costs with function, but at the very least, one that doesn't break your budget.
Though Kafka has considerable advantages, you may decide to go with something else, or at least start with something else. During the next few years, competitors and niche event stream services and message queues will continue to gain traction to serve developers in new and better ways. You may decide to get ahead of that curve and ditch Kafka, depending on your answers to those questions. Or, you might decide that what works for Uber, Netflix, Intuit, and Citigroup works for you.
UP9 is a microservice testing platform for Cloud Native systems that can scale across any number of services. UP9 helps developers prevent software regressions and increase engineering productivity through effortless testing and instant virtualization.
- Understand how services interact with each other with automatic contract discovery.
- Ensure system-wide contract adherence with tests that write themselves.
- Start testing early with test environments as code, automatically mocking all service dependencies.
UP9 offloads the microservice testing workload from developers, giving them precious time back.
You can sign up now for free!