Kafka Mocks with Mockintosh

UP9 and Mockintosh offer mocking services for asynchronous communication such as Kafka and RabbitMQ. Service mocks provide predictable behavior from service dependencies to speed up both the development and testing stages. UP9 builds mocks automatically from observed service traffic, which can then be executed with Mockintosh.

This is configured similarly to HTTP mocks, and mocks the four most common patterns of consumers and producers:

  • On-demand producer: when we want something to happen, it produces a message on the bus.
  • Validating consumer: validate that a message appears on the bus by consuming the message, or if we require the message to be consumed. Remember the message, access its fields, test with an assertion.
  • Scheduled producer: same as on-demand, but scheduled (for example, every five seconds, push a message into my message queue)
  • Reactive consumer/producer: Merges validating consumer and on-demand producer, so after a message on one queue is consumed, another message on another queue is produced. This simulates a whole service acting on the bus.

We'll use the Asynchronous Microservices Demo repository and Docker Compose to demonstrate using Mockintosh with asynchronous communication.

Install and Run

Clone the Asynchronous Microservices Demo repository to your machine.

This will install an asynchronous microservices demo program that looks like this:

Async_Demo

We have asynchronous message buses between frontend, job manager, and grayscaler that we will mock with Mockintosh.

To run the application normally, run it with Docker Compose in the top root of the repository: docker-compose up

To run the isolated front-end, navigate to the frontend subfolder and run docker-compose up.

To run the job manager, navigate to the manager subfolder and run docker-compose up.

To run the grayscaler, navigate to the grayscaler subfolder and run docker-compose up.

Open http://localhost:8000/ to see the mocked Kafka traffic log in the Mockintosh command UI.

To stop the application at any time, press Ctrl+C in your terminal.

Reactive Consumer/Producer Mock

When the application starts for the first time, we will receive warnings about topics not existing. This is normal because our Kafka is empty.

Open http://localhost/ in the browser to see the demo application. Upload a colorful image via the web form and wait for the resulting image to appear.

Demo_app

Notice that after the first result, the Kafka topics have been created and there are no more log warnings. In the log, we can see messages about different services working with Kafka topics.

Grayscaler

We are going to focus on the grayscaler microservice. Grayscaler receives images from the grayscaler-job queue, grayscales them, then passes the finished product to the grayscaler-result queue. We need Mockintosh to mock an on-demand producer sending images to the grayscaler-job queue, and a validating consumer receiving images from the grayscaler-result queue.

grayscaler

We can see the mock code in the mockintosh.yaml configuration file. First, it sets the Mockintosh command UI at port 8000, and defines the mocked services as type kafka. It opens port 9092 by default, but this can be changed.

This mock acts as the on-demand producer. It sends a message to the grayscaler-job queue with a key and a static file image (@color_panel.png).

      - name: Job-Trigger
        produce:
          queue: grayscaler-job
          key: job-{{date.timestamp}}
          value: "@color_panel.png"

We also have a result validator mock to validate the consumer. This consumes messages from the grayscaler-result queue and captures the last 5 messages for investigation.

      - name: Result-Validator
        consume:
          queue: grayscaler-result
          capture: 5

Change directory to the grayscaler subfolder and run docker-compose up to run the isolated grayscaler service. Open http://localhost:8000/ to see the Mockintosh control UI.

Mockintosh_local

We can see the mocked Kafka traffic log in the Async Actors tab. We could click the Trigger button and View Captured Message buttons to test manually, but we are not receiving any traffic from the job queue yet. We need to mock this traffic.

Pytest

In another console, open tests.py in the repository's grayscaler folder. UP9 builds tests, mocks, and contracts like tests.py automatically from observed service traffic.

This Python test first clears our message cache, then mocks traffic on the grayscaler-jobs queue. The code below is equivalent to clicking the Trigger button in the Mockintosh UI.

requests.post("http://localhost:8000/async/producers/Job-Trigger").raise_for_status()

The test then waits for 10 seconds for the resulting message from the grayscaler-result queue, and prints the message it receives. If it receives nothing, it returns Didn't get the result.

Run the test. We should receive a message like below. The string output is a Base64 representation of the color_panel.png image.

Pytest_result

We can then check the mock traffic in the Mockintosh control UI. We can see that a message was put into the bus with the grayscaler-job input, and the result was output with the same message key to the grayscaler-result bus.

Mockintosh_result

We have now confirmed that our grayscaler asynchronous queues are operational using Mockintosh, with mocks produced automatically from UP9 observing traffic.

Support

For support, feel free to use any one of the three: