Mockintosh: Asynchronous Kafka Actors

Welcome, in this part we will talk about the asynchronous Kafka actors in Mockintosh. ​ Besides the HTTP mocking capabilities, Mockintosh is able to mock the asynchronous tasks through communicating with the message queues. ​

Setting Up Apache Kafka

To be able to work with asynchronous Kafka actors you first need an Apache Kafka instance up and running. ​ So you can head over to kafka.apache.org/quickstart and follow the instructions in Step 1 and Step 2 or you can use the Apache Kafka Docker image that's maintained by the Mockintosh team. ​ docker run -it --net=host up9inc/mockintosh:self-contained-kafka ​ At this point, you should have an Apache Kafka instance listening to port 9092. ​

services:
    - name: Kafka Mocks
      type: kafka
      address: "{{ env 'KAFKA_ADDR' 'localhost:9092' }}"
      actors: []

Kafka Service

Now, let's define a Kafka service.
​ A Mockintosh service can have the optional field type to define what type of a service it is. The type defaults to http. To have a service that will contain Kafka mocks, you should set the type field to kafka and the required field address to the hostname and port of the target Apache Kafka instance. ​ The actors field is a list Kafka actors to mock the asynchronous tasks. An actor can have only one of these roles: ​

  • Validating Consumers
  • On-Demand Producers
  • Scheduled Producers
  • and Reactive Producers
    ​ The role is determined automatically by how you defined the actor. ​
management:
    port: 8080
services:
    - name: Kafka Mocks
      type: kafka
      address: "{{ env 'KAFKA_ADDR' 'localhost:9092' }}"
      actors:
          - name: Simplest Validating Consumer
            consume:
                queue: topic1

Validating Consumers

A validating consumer is an asynchronous actor that has only the consume field and not the produce, delay, or limit fields.
​ So a validating consumer in its simplest form is a consumer with only the queue field present. This consumer will consume every possible message in the topic topic1. ​ Let's create the topic topic1 and produce an example message using the Kafka's shell scripts.

management:
  port: 8080
services:
- name: Kafka Mocks
  type: kafka
  address: "{{ env 'KAFKA_ADDR' 'localhost:9092' }}"
  actors:
    - name: Complex Validating Consumer
      consume:
        queue: topic1
        key: "{{ regEX 'prefix-(.*)' }}"
        value: "{{ regEx '(.*)-suffix' }}"
        headers:
          client-id: "{{ regEx '[0-9]+' }}

​ Visit the management UI to validate if the message is consumed. ​ A more complex validating consumer example would be having key, value and headers fields defined using regEx helper to describe the criteria for matching to incoming messages. This actor expects key to have a prefix, value to have a suffix, and a header named client-id to be a number. ​

On-Demand Producers

An on-demand producer is an asynchronous actor that has only the produce field and not the consume, delay, or limit fields. These kinds of actors can be triggered using either the management API (through the actor index or the name) or the management UI, and they produce the exact message according to the actor description. ​ Notice how templating is possible in key, value, and headers below.

management:
    port: 8080
services:
    - name: Kafka Mocks
      type: kafka
      address: "{{ env 'KAFKA_ADDR' 'localhost:9092' }}"
      actors:
          - name: templated-producer
            consume:
                queue: templated-producer
                key: "prefix-{{ random.uuid4 }}"
                value: "{{ fake.name }}"
                headers:
                    name: templated
                    constant: constant-value
                    timestamp: "{{ date.timestamp }}"
                    counter: "{{ counter 'kafkaCounter' }}"
                    fromFile: "@templates/example.txt"

​ So let's create listen to the templated-producer topic using the shell scripts. ​ Do a POST request to localhost:8080/async/producers/0 and another POST request to localhost:8080/async/producers/templated-producer. ​ Now head over to the management UI and trigger this producer for the third time. ​ You can see that how it's logged into the consumer that we have started through the shell scripts. ​ Using the same logic a validating consumer can consume a message produced by an on-demand producer or any other producer. ​

services:
    - name: Kafka Mocks
      type: kafka
      address: "{{ env 'KAFKA_ADDR' 'localhost:9092' }}"
      actors:
          - produce:
                create: true
                queue: topic2
                key: key2
                value: value2
                headers:
                    hdr2: val2
            delay: 2
            limit: 100

Scheduled Producers

A scheduled producer is an asynchronous actor that has the required produce and delay fields, an optional limit field, but not the consume field. ​ In this example the producer triggered for each 2 seconds up to 100 times. It triggers as soon as Mockintosh is initialized. ​

services:
    - name: Kafka Mocks
      type: kafka
      address: "{{ env 'KAFKA_ADDR' 'localhost:9092' }}"
      actors:
          - produce:
                create: true
                queue: topic2
                key: key2
                value: value2
                headers:
                    hdr2: val2
            delay: 2
            limit: 100

​ It's logical to set the create field to true for creating the topic if the topic is missing. By default it's set to false. create field is avaiable for all kind of producers. ​ If the limit is not specified, then the scheduled producer runs indefinitely. ​

Reactive Producers

A reactive producer is kind of a wired up version of a validating consumer and an on-demand producer. If a validating consumer has also the produce field, then it automatically becomes a reactive producer.

services:
- name: Kafka Mocks
  type: kafka
  address: "{{ env 'KAFKA_ADDR' 'localhost:9092' }}"
  actors:
    - consume:
        queue: topic3
      delay: 3
      produce:
        queue: topic4
        key: key4
        value: "value4 and {{ consumed.key }} {{ consumed.value }} {{{{ consumed.headers.hdr3 }}
        headers:
          hdr4: val4

​ In this example, whenever a message is consumed from topic3, a message is produced into the topic4 after a three second delay. ​ Notice that the consumed object is avaiable to use in the templating of a reactive producer with the purpose of passing the information from the consumer to the producer. Just like how the request object works in the HTTP mocks. ​

Support

You can view these examples in action in this video, and learn more at Mockintosh and UP9. Join our community Slack at up9.slack.com.