Mockintosh: Asynchronous Kafka Actors
Welcome, in this part we will talk about the asynchronous Kafka actors in Mockintosh. Besides the HTTP mocking capabilities, Mockintosh is able to mock the asynchronous tasks through communicating with the message queues.
Setting Up Apache Kafka
To be able to work with asynchronous Kafka actors you first need an Apache Kafka instance up and running.
So you can head over to kafka.apache.org/quickstart and follow the instructions in Step 1 and Step 2
or you can use the Apache Kafka Docker image that's maintained by the Mockintosh team.
docker run -it --net=host up9inc/mockintosh:self-contained-kafka
At this point, you should have an Apache Kafka instance listening to port 9092.
services:
- name: Kafka Mocks
type: kafka
address: "{{ env 'KAFKA_ADDR' 'localhost:9092' }}"
actors: []
Kafka Service
Now, let's define a Kafka service.
A Mockintosh service can have the optional field type to define what type of a service it is.
The type defaults to http. To have a service that will contain Kafka mocks, you should set the type field to kafka and the required field address to the hostname
and port of the target Apache Kafka instance.
The actors field is a list Kafka actors to mock the asynchronous tasks. An actor can have only one of these roles:
- Validating Consumers
- On-Demand Producers
- Scheduled Producers
- and Reactive Producers
The role is determined automatically by how you defined the actor.
management:
port: 8080
services:
- name: Kafka Mocks
type: kafka
address: "{{ env 'KAFKA_ADDR' 'localhost:9092' }}"
actors:
- name: Simplest Validating Consumer
consume:
queue: topic1
Validating Consumers
A validating consumer is an asynchronous actor that has only the consume field and not the produce, delay, or limit fields.
So a validating consumer in its simplest form is a consumer with only the queue field present. This consumer will consume every possible message in the topic topic1.
Let's create the topic topic1 and produce an example message using the Kafka's shell scripts.
management:
port: 8080
services:
- name: Kafka Mocks
type: kafka
address: "{{ env 'KAFKA_ADDR' 'localhost:9092' }}"
actors:
- name: Complex Validating Consumer
consume:
queue: topic1
key: "{{ regEX 'prefix-(.*)' }}"
value: "{{ regEx '(.*)-suffix' }}"
headers:
client-id: "{{ regEx '[0-9]+' }}
Visit the management UI to validate if the message is consumed.
A more complex validating consumer example would be having key, value and headers fields defined using regEx helper to describe the criteria for matching to incoming messages. This actor expects key to have a prefix, value to have a suffix,
and a header named client-id to be a number.
On-Demand Producers
An on-demand producer is an asynchronous actor that has only the produce field and not the consume, delay, or limit fields. These kinds of actors can be triggered using either
the management API (through the actor index or the name)
or the management UI, and they produce the exact message according to the actor description.
Notice how templating is possible in key, value, and headers below.
management:
port: 8080
services:
- name: Kafka Mocks
type: kafka
address: "{{ env 'KAFKA_ADDR' 'localhost:9092' }}"
actors:
- name: templated-producer
consume:
queue: templated-producer
key: "prefix-{{ random.uuid4 }}"
value: "{{ fake.name }}"
headers:
name: templated
constant: constant-value
timestamp: "{{ date.timestamp }}"
counter: "{{ counter 'kafkaCounter' }}"
fromFile: "@templates/example.txt"
So let's create listen to the templated-producer topic using the shell scripts.
Do a POST request to localhost:8080/async/producers/0
and another POST request to localhost:8080/async/producers/templated-producer.
Now head over to the management UI and trigger this producer for the third time. You can see that how it's logged into the consumer that we have started through the shell scripts. Using the same logic a validating consumer can consume a message produced by an on-demand producer or any other producer.
services:
- name: Kafka Mocks
type: kafka
address: "{{ env 'KAFKA_ADDR' 'localhost:9092' }}"
actors:
- produce:
create: true
queue: topic2
key: key2
value: value2
headers:
hdr2: val2
delay: 2
limit: 100
Scheduled Producers
A scheduled producer is an asynchronous actor that has the required produce and delay fields, an optional limit field, but not the consume field.
In this example the producer triggered for each 2 seconds up to 100 times. It triggers as soon as Mockintosh is initialized.
services:
- name: Kafka Mocks
type: kafka
address: "{{ env 'KAFKA_ADDR' 'localhost:9092' }}"
actors:
- produce:
create: true
queue: topic2
key: key2
value: value2
headers:
hdr2: val2
delay: 2
limit: 100
It's logical to set the create field to true for creating the
topic if the topic is missing. By default it's set to false.
create field is avaiable for all kind of producers.
If the limit is not specified, then the scheduled producer runs indefinitely.
Reactive Producers
A reactive producer is kind of a wired up version of a validating consumer and an on-demand producer.
If a validating consumer has also the produce field,
then it automatically becomes a reactive producer.
services:
- name: Kafka Mocks
type: kafka
address: "{{ env 'KAFKA_ADDR' 'localhost:9092' }}"
actors:
- consume:
queue: topic3
delay: 3
produce:
queue: topic4
key: key4
value: "value4 and {{ consumed.key }} {{ consumed.value }} {{{{ consumed.headers.hdr3 }}
headers:
hdr4: val4
In this example, whenever a message is consumed from topic3, a message is produced into the topic4 after a three second delay.
Notice that the consumed object is avaiable to use in the templating of a reactive producer with the
purpose of passing the information from the consumer to the producer. Just like how the request object
works in the HTTP mocks.
Support
You can view these examples in action in this video, and learn more at Mockintosh and UP9. Join our community Slack at up9.slack.com.