The Message Bus Mocking Problem
Today we have a new inter-service communication approach gaining more and more popularity: the asynchronous message bus. It allows developers to make services less coupled, which leads to higher resiliency and better performance. Modern applications are often built as event-driven microservices, have larger scale and experience high load. The technologies that implement message bus include Kafka, RabbitMQ, Redis, and many others.
Mocking for HTTP and REST is well-known and many developers leverage its benefits; namely the ability to have a lightweight and predictable replacement for real APIs. The most famous use case for mocks is “dependency isolation” where the desired service (or group of services) gets isolated from third-party or even own microservices. This is done by replacing the dependency with a mock, because the dependency is consuming too much resources or is too hard to control (e.g. to create desired database state).
For asynchronous communication, there is no well-adopted practice today that would offer an experience similar to classic mocks. But developers need the same “dependency isolation” capability to properly organize development and testing.
In this article, we’ll review the “Mock Actors” approach, using Kafka as our message bus implementation (though the very same concept is applicable to any other asynchronous messaging technology. Closer to the end of the article, we’ll learn about an Open Source tool that implements the concept of Mock Actors, the Mockintosh.
The Concept of “Mock Actors”
When aiming to achieve “dependency isolation” with asynchronous communication, one of the first ideas is to consider Kafka as yet another service and attempt to mock it. But that is actually not necessary at all, because the message bus plays no active role in application functioning.
Let’s look at the following diagram:
What’s important is that the actual data processing happens inside the HTTP service for the synchronous case. But in the asynchronous case, the data processing is done by some event-driven microservice, consuming the message from Kafka and doing actual work. The result of asynchronous processing may be delivered back to the originator via the same message bus. The message bus in the async case plays a very passive role by just providing the transport function.
This reveals the place in the architecture that holds the actual business logic... and thus is a subject for mocking. It is not Kafka, rather it is microservices that produce or consume messages in topics. And if we replace these services with some simple and predictable actors, we will get the desired dependency isolation. The “Mock Actor” approach for message bus architectures does exactly that:
The Mock Actor reproduces the functionality of services working with the message bus, be it producers, consumers, or both. And just like a regular mock, it requires much less resources to deploy and configure, while preserving the asynchronous communication patterns of the real service. Kafka keeps existing in the architecture, providing the passive transport for messages between mock actors and real services.
One important question arises when adopting the Mock Actor approach in your application setup: which Kafka instance to use? One approach is to keep using the same Kafka instance as before, and connect Mock Actors to that instance. This might be simpler in setup, but it has a chance to create undesired noise in topics for shared Kafka installation. Alternatively, you can deploy a dedicated Kafka instance specifically for Mock Actors, and reconfigure real services to use that instance. The latter approach provides a truly hermetic and isolated environment.
Let’s look at some of the most frequent Mock Actor behavior patterns in detail below.
Four Actor Behavior Patterns
In HTTP, the interaction between client and server always happens in “request + response” pairs. Unlike in HTTP, the interaction with a message bus happens with one of two actions: producing a message or consuming a message. From these primitives, the following four Mock Actor behaviors are suggested:
When the isolated service has a message on the bus as its input, we need the ability to trigger such a message to appear at a predictable moment in time. This usually happens as part of an automated test or when a developer is experimenting with the service.
The simplest way to trigger anything in today’s technology is by means of HTTP API call. This is exactly how the “on-demand producer” accepts the commands:
The actual message that will be published into the Kafka topic is defined by a configuration of the Mock Actor, usually a piece of YAML or JSON, which does not require any programming knowledge.
An HTTP API is simpler to interact with compared to Kafka protocol. Clients for HTTP exist in any environment; one can even use a Web browser for that. Finally, the developer or QA person might be not familiar with programming Kafka interactions in a certain language. Using a Mock Actor with HTTP interface hides the complexity of underlying Kafka communication.
If we modify the “on-demand producer” pattern to trigger message sending by some periodic timer, we get the “scheduled producer” pattern. This comes in handy in many situations, when you research application behavior under regular flow of input messages.
Scheduled producers also allow one to simulate “noisy environments” in isolation, with many Kafka messages produced into many topics.
Developers don’t need to do much with “outgoing messages” that are not consumed within an isolated environment. But there are a couple of important cases where we still need to access those messages. One is the assertion phase of automated testing, when we need to validate that the expected Kafka message was actually sent into a topic. Another is the ability to quickly access and view the published messages for exploratory testing and troubleshooting.
For these cases, the “validating consumer” pattern is used:
At the moment the message is expected to appear on the bus, it gets consumed by a specially-configured Mock Actor. The Actor makes sure the message conforms to expectations, like JSONSchema or a set of regular expressions. After that, the message is considered “captured” by the Mock Actor and that fact can be checked via a simple HTTP API. We have already discussed the benefits of using HTTP API in conjunction with Kafka couple of sections above.
If we merge the “validating consumer” together with “on-demand producer,” we get the “reactive producer” pattern. In this pattern, the fact of the consumer getting the message is a “trigger” that tells the producer to publish the “reaction” message into a different topic:
The Mock Actor may use pieces of consumed “trigger message” to compose the “reaction message”. Some delay might be configured between the consume and produce events, to simulate the “processing time” of dependency service.
The usage of “reactive producer” allows us to isolate the services from heavy dependencies, while maintaining the functionality loops via simulated data processing.
Implementation for Mock Actors
There are other usage situations and patterns for Mock Actors in addition to the four described above. What is important is the ability to replace real dependency services with lightweight and obedient Mock Actors.
To implement the Mock Actors approach we have an Open Source tool called Mockintosh. It offers an opinionated approach for configuring classic HTTP Mocks as well as asynchronous Mock Actors. The holistic approach simplifies the learning process for Kafka-related mocking with analogue to synchronous mocks.
Some features make Mockintosh especially useful for those who practice service isolation and mocking:
- A dashboard to trigger on-demand producers and view captured messages
- The interactive UI viewer of requests, responses and messages from validating consumers
- The ability to alter the configuration without restarting the mock container
- Sophisticated language for configuring message properties
With Mockintosh, developers don’t need to write their own implementations of Mock Actors for each situation. QA people can experiment with async services with no programming knowledge required. Teams that work on different services get the common ground for learning and using the mocks.