The second part of this series was all about theory - we had a closer look at some helpful resilience design patterns. Now it's time to take some action 💪. Let's finally bring all threads together and see how we can realize this with Quarkus and Kotlin.
Do you still remember the shocking scenario for all of this? We are doomed cause we live in an area with poisonous snakes and must rely on our own service to be warned about the dangerous animals in time.
Let me introduce you to the Snake Alarm Detection Fulfilment Scenario where we are going to handle this in a resilient way.
Imagine small snake detector boxes, which can magically detect snakes passing by. Sometimes the detectors have false positives with cute and harmless lizards. To handle these false positives the snake detector boxes can also send a photo of the detected animal via REST endpoint so it can be evaluated, for example, by means of machine learning. When we are sure that it's a snake, we want to send a push notifications to all mobile devices in that area. Sadly, we have to use an external service for the push notifications and this is pretty expensive. We have two options here: one service, which is very expensive and 100% reliable but a little bit slow and another services, which is much cheaper and faster, but not so reliable at all.
The plan is to build a robust roundtrip from a) detecting a "snake or lizard" with the box to b) sending a warning push notification to everyone in the area. Ideally the push notification goes out fast and we do not spend that much money on sending those push notifications.
Let's try: the snake detector box could make a POST request to a SnakeAlarmEndpoint. This could trigger a SnakeAlarmRepository to emit an event into an AMQP messaging channel named snake-alarm-out. Then we subscribe to those events in another snake-alarm-in channel to be able to scale independently. Given the illustration below, you can see our SnakeAlarmProcessor, which can process new SnakeAlarms coming in from the AMQP channel snake-alarms-in.
The SnakeAlarmProcessor itself calls the LizardCheck to verify that we have a snake and not a lizard coming in.
The LizardCheck has a runtime between 0,2s and 1,06s. Having the runtime measured, we can configure our SnakeAlarmProcessor call to the LizardCheck with a timeout of 1s and a maximum of 3 retries to make most calls succeed in an acceptable time.
Finally we call the push notification service. The primary service of choice will be tried first, but fails with a probability of 50% and has an average response time of 1s. We give that a maximum of 2 retries. After that we call the secondary service of choice as a fallback, which has a probability of ~100% and a response time of 3s.
Sample output of the running result
Here you find the output of the final result. Try to see where we had a false positive and where not. Where did we use the expensive push notification service? It's all written in the logs.
2022-07-24 16:03:53,936 INFO [de.fay.sna.SnakeAlarmProcessor] (vert.x-eventloop-thread-0) Received an alarm for snake alarm fulfillment: [SnakeItems(id=1, imageASCIIArtString=e--==--==--)], False positive - Successfully verified a lizard
2022-07-24 16:03:53,994 DEBUG [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-2) SRMSG16211: Sending AMQP message to address `snake-alarms`
2022-07-24 16:03:54,000 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Disposition{role=RECEIVER, first=1, last=1, settled=true, state=Accepted{}, batchable=false}
2022-07-24 16:03:54,001 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Transfer{handle=0, deliveryId=1, deliveryTag=\x00, messageFormat=0, settled=false, more=false, rcvSettleMode=null, state=null, resume=false, aborted=false, batchable=false}[\x00Ss\xd0\x00\x00\x00)\x00\x00\x00\x07@@\xa1\x0csnake-alarms@@@\xa3\x10application/json\x00Su\xa0\x9c{"snakes":[{"id":"1","imageASCIIArtString":"e--==--==--"}],"snakeVerification":"Snake could also be a lizard :-)","estimatedPushNotificationTimestamp":null}]
2022-07-24 16:03:54,460 DEBUG [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-2) SRMSG16211: Sending AMQP message to address `snake-alarms`
2022-07-24 16:03:54,463 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Transfer{handle=0, deliveryId=2, deliveryTag=\x01, messageFormat=0, settled=false, more=false, rcvSettleMode=null, state=null, resume=false, aborted=false, batchable=false}[\x00Ss\xd0\x00\x00\x00)\x00\x00\x00\x07@@\xa1\x0csnake-alarms@@@\xa3\x10application/json\x00Su\xa0\x9c{"snakes":[{"id":"1","imageASCIIArtString":"e--==--==--"}],"snakeVerification":"Snake could also be a lizard :-)","estimatedPushNotificationTimestamp":null}]
2022-07-24 16:03:54,464 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Disposition{role=RECEIVER, first=2, last=2, settled=true, state=Accepted{}, batchable=false}
2022-07-24 16:03:54,579 INFO [de.fay.sna.SnakeAlarmProcessor] (vert.x-eventloop-thread-0) Received an alarm for snake alarm fulfillment: [SnakeItems(id=1, imageASCIIArtString=e--==--==--)], False positive - Successfully verified a lizard
2022-07-24 16:03:54,937 INFO [de.fay.sna.SnakeAlarmProcessor] (vert.x-eventloop-thread-0) Received an alarm for snake alarm fulfillment: [SnakeItems(id=1, imageASCIIArtString=e--==--==--)], False positive - Successfully verified a lizard
2022-07-24 16:03:56,180 DEBUG [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-2) SRMSG16211: Sending AMQP message to address `snake-alarms`
2022-07-24 16:03:56,185 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Disposition{role=RECEIVER, first=3, last=3, settled=true, state=Accepted{}, batchable=false}
2022-07-24 16:03:56,186 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Transfer{handle=0, deliveryId=3, deliveryTag=\x00, messageFormat=0, settled=false, more=false, rcvSettleMode=null, state=null, resume=false, aborted=false, batchable=false}[\x00Ss\xd0\x00\x00\x00)\x00\x00\x00\x07@@\xa1\x0csnake-alarms@@@\xa3\x10application/json\x00Su\xa0\x9c{"snakes":[{"id":"1","imageASCIIArtString":"e--==--==--"}],"snakeVerification":"Snake could also be a lizard :-)","estimatedPushNotificationTimestamp":null}]
2022-07-24 16:03:56,618 DEBUG [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-2) SRMSG16211: Sending AMQP message to address `snake-alarms`
2022-07-24 16:03:56,622 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Transfer{handle=0, deliveryId=4, deliveryTag=\x01, messageFormat=0, settled=false, more=false, rcvSettleMode=null, state=null, resume=false, aborted=false, batchable=false}[\x00Ss\xd0\x00\x00\x00)\x00\x00\x00\x07@@\xa1\x0csnake-alarms@@@\xa3\x10application/json\x00Su\xa0\x9c{"snakes":[{"id":"1","imageASCIIArtString":"e--==--==--"}],"snakeVerification":"Snake could also be a lizard :-)","estimatedPushNotificationTimestamp":null}]
2022-07-24 16:03:56,625 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Disposition{role=RECEIVER, first=4, last=4, settled=true, state=Accepted{}, batchable=false}
2022-07-24 16:03:57,105 DEBUG [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-2) SRMSG16211: Sending AMQP message to address `snake-alarms`
2022-07-24 16:03:57,110 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Disposition{role=RECEIVER, first=5, last=5, settled=true, state=Accepted{}, batchable=false}
2022-07-24 16:03:57,111 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Transfer{handle=0, deliveryId=5, deliveryTag=\x02, messageFormat=0, settled=false, more=false, rcvSettleMode=null, state=null, resume=false, aborted=false, batchable=false}[\x00Ss\xd0\x00\x00\x00)\x00\x00\x00\x07@@\xa1\x0csnake-alarms@@@\xa3\x10application/json\x00Su\xa0\x9c{"snakes":[{"id":"1","imageASCIIArtString":"e--==--==--"}],"snakeVerification":"Snake could also be a lizard :-)","estimatedPushNotificationTimestamp":null}]
2022-07-24 16:03:57,219 INFO [de.fay.sna.SnakeAlarmProcessor] (vert.x-eventloop-thread-0) Received an alarm for snake alarm fulfillment: [SnakeItems(id=1, imageASCIIArtString=e--==--==--)], False positive - Successfully verified a lizard
2022-07-24 16:03:57,534 DEBUG [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-2) SRMSG16211: Sending AMQP message to address `snake-alarms`
2022-07-24 16:03:57,538 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Disposition{role=RECEIVER, first=6, last=6, settled=true, state=Accepted{}, batchable=false}
2022-07-24 16:03:57,539 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Transfer{handle=0, deliveryId=6, deliveryTag=\x00, messageFormat=0, settled=false, more=false, rcvSettleMode=null, state=null, resume=false, aborted=false, batchable=false}[\x00Ss\xd0\x00\x00\x00)\x00\x00\x00\x07@@\xa1\x0csnake-alarms@@@\xa3\x10application/json\x00Su\xa0\x9c{"snakes":[{"id":"1","imageASCIIArtString":"e--==--==--"}],"snakeVerification":"Snake could also be a lizard :-)","estimatedPushNotificationTimestamp":null}]
2022-07-24 16:03:58,039 INFO [de.fay.sna.SnakeAlarmProcessor] (vert.x-eventloop-thread-0) Received an alarm for snake alarm fulfillment: [SnakeItems(id=1, imageASCIIArtString=e--==--==--)], False positive - Successfully verified a lizard
2022-07-24 16:03:58,041 INFO [de.fay.sna.SnakeAlarmProcessor] (vert.x-eventloop-thread-0) Received an alarm for snake alarm fulfillment: [SnakeItems(id=1, imageASCIIArtString=e--==--==--)], Danger - Successfully verified a snake
2022-07-24 16:03:58,052 INFO [de.fay.sna.PushNotificationService] (vert.x-eventloop-thread-0) Push notification estimated in 3 sec
2022-07-24 16:03:58,420 INFO [de.fay.sna.SnakeAlarmProcessor] (vert.x-eventloop-thread-0) Received an alarm for snake alarm fulfillment: [SnakeItems(id=1, imageASCIIArtString=e--==--==--)], False positive - Successfully verified a lizard
Postman
This is an exemplary POST request we are going to use to fire up our services. If we make the request right, we should receive a 202 Accepted response. You will find this as a Postman collection in the Github sources.
Snake alarm endpoint
Our endpoint calls the SnakeAlarmRepository to take care of everything.
@Path("snake-alarm")
class SnakeAlarmEndpoint(
private val snakeAlarmRepository: SnakeAlarmRepository
) {
@POST
@Consumes(MediaType.APPLICATION_JSON)
@Produces(MediaType.TEXT_PLAIN)
suspend fun snakeAlarm(request: SnakeAlarmRequest): Response {
return when (snakeAlarmRepository.alarm(request.snakes)) {
true -> Response.accepted().build()
false -> Response.serverError().build()
}
}
}
The SnakeAlarmRepository emits the alarm without verification using an AMQP messaging channel named snake-alarms-out.
@ApplicationScoped
class SnakeAlarmRepository(
private val pushNotificationService: PushNotificationService,
@Channel("snake-alarms-out") private val snakeAlarmEmitter: Emitter<SnakeAlarm>,
private val logger: Logger
) {
suspend fun alarm(items: List<SnakeItems>): Boolean {
val snakeAlarm = SnakeAlarm(
snakes = items,
snakeVerification = "Snake could also be a lizard :-)",
estimatedPushNotificationTimestamp = null
)
return try {
snakeAlarmEmitter.send(snakeAlarm).await()
true
} catch (e: Exception) {
logger.log(Level.SEVERE, "Error while sending snake alarm", e)
false
}
}
}
Application Properties
That's where we need to configure our loosely coupled AMPQ messaging channels, which will run using a Docker image (a sample Docker file is provided in the Github sources).
quarkus.log.level=DEBUG
quarkus.log.category."org.jboss.resteasy".min-level=DEBUG
# The AMQP broker location and credentials
amqp-host=localhost
amqp-port=5672
amqp-username=quarkus
amqp-password=quarkus
# Configure snake alarm messaging
mp.messaging.incoming.snake-alarms-in.connector=smallrye-amqp
mp.messaging.incoming.snake-alarms-in.address=snake-alarms
mp.messaging.outgoing.snake-alarms-out.connector=smallrye-amqp
mp.messaging.outgoing.snake-alarms-out.address=snake-alarms
When we receive a message from this channel, the SnakeAlarmProcessor checks if it really is a snake by using the LizardCheck.
@ApplicationScoped
class SnakeAlarmProcessor(
private val pushNotificationService: PushNotificationService,
private val lizardCheck: LizardCheck,
private val logger: Logger
) {
@Incoming("snake-alarms-in")
suspend fun processSnakeAlarm(snakeAlarm: SnakeAlarm) {
val isVerifiedSnake = lizardCheck.checkSnakeAlarmValid(snakeAlarm)
val verificationResultText = if (isVerifiedSnake) "Danger - Successfully verified a snake" else "False positive - Successfully verified a lizard"
logger.info { "Received an alarm for snake alarm fulfillment: ${snakeAlarm.snakes}, $verificationResultText" }
if(isVerifiedSnake) pushNotificationService.announcePrimaryPushNotificationService()
}
}
The LizardCheck has often timeouts. Therefor we have configured timeout and retry options by using annotations.
@ApplicationScoped
class LizardCheck {
@Timeout(1, unit = ChronoUnit.SECONDS)
@Retry(maxRetries = 3)
suspend fun checkSnakeAlarmValid(): Boolean {
if (Random.nextBoolean()) {
delay(Random.nextLong(200, 1060))
return false
}
return true
}
}
The PushNotificationServices has a method to announce each notification services. The first method has the second method has a fallback. If the fallback is used, we have a max retry of two attempts for that.
@ApplicationScoped
class PushNotificationService(
private val logger: Logger
) {
@Fallback(fallbackMethod = "announceSecondaryPushNotificationService")
fun announcePrimaryPushNotificationService(): Date {
val fail = Random.nextBoolean()
if (fail) {
error("Can't reach primary push notification service")
}
logger.info { "Push notification estimated in 1 sec" }
return Calendar.getInstance().apply {
add(Calendar.SECOND, 1)
}.time
}
@Retry(maxRetries = 2)
fun announceSecondaryPushNotificationService(): Date {
logger.info { "Push notification estimated in 3 sec" }
return Calendar.getInstance().apply {
add(Calendar.SECOND, 3)
}.time
}
}
Let's re-check the logs:
2022-07-24 16:03:53,936 INFO [de.fay.sna.SnakeAlarmProcessor] (vert.x-eventloop-thread-0) Received an alarm for snake alarm fulfillment: [SnakeItems(id=1, imageASCIIArtString=e--==--==--)], False positive - Successfully verified a lizard
2022-07-24 16:03:53,994 DEBUG [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-2) SRMSG16211: Sending AMQP message to address `snake-alarms`
2022-07-24 16:03:54,000 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Disposition{role=RECEIVER, first=1, last=1, settled=true, state=Accepted{}, batchable=false}
2022-07-24 16:03:54,001 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Transfer{handle=0, deliveryId=1, deliveryTag=\x00, messageFormat=0, settled=false, more=false, rcvSettleMode=null, state=null, resume=false, aborted=false, batchable=false}[\x00Ss\xd0\x00\x00\x00)\x00\x00\x00\x07@@\xa1\x0csnake-alarms@@@\xa3\x10application/json\x00Su\xa0\x9c{"snakes":[{"id":"1","imageASCIIArtString":"e--==--==--"}],"snakeVerification":"Snake could also be a lizard :-)","estimatedPushNotificationTimestamp":null}]
2022-07-24 16:03:54,460 DEBUG [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-2) SRMSG16211: Sending AMQP message to address `snake-alarms`
2022-07-24 16:03:54,463 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Transfer{handle=0, deliveryId=2, deliveryTag=\x01, messageFormat=0, settled=false, more=false, rcvSettleMode=null, state=null, resume=false, aborted=false, batchable=false}[\x00Ss\xd0\x00\x00\x00)\x00\x00\x00\x07@@\xa1\x0csnake-alarms@@@\xa3\x10application/json\x00Su\xa0\x9c{"snakes":[{"id":"1","imageASCIIArtString":"e--==--==--"}],"snakeVerification":"Snake could also be a lizard :-)","estimatedPushNotificationTimestamp":null}]
2022-07-24 16:03:54,464 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Disposition{role=RECEIVER, first=2, last=2, settled=true, state=Accepted{}, batchable=false}
2022-07-24 16:03:54,579 INFO [de.fay.sna.SnakeAlarmProcessor] (vert.x-eventloop-thread-0) Received an alarm for snake alarm fulfillment: [SnakeItems(id=1, imageASCIIArtString=e--==--==--)], False positive - Successfully verified a lizard
2022-07-24 16:03:54,937 INFO [de.fay.sna.SnakeAlarmProcessor] (vert.x-eventloop-thread-0) Received an alarm for snake alarm fulfillment: [SnakeItems(id=1, imageASCIIArtString=e--==--==--)], False positive - Successfully verified a lizard
2022-07-24 16:03:56,180 DEBUG [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-2) SRMSG16211: Sending AMQP message to address `snake-alarms`
2022-07-24 16:03:56,185 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Disposition{role=RECEIVER, first=3, last=3, settled=true, state=Accepted{}, batchable=false}
2022-07-24 16:03:56,186 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Transfer{handle=0, deliveryId=3, deliveryTag=\x00, messageFormat=0, settled=false, more=false, rcvSettleMode=null, state=null, resume=false, aborted=false, batchable=false}[\x00Ss\xd0\x00\x00\x00)\x00\x00\x00\x07@@\xa1\x0csnake-alarms@@@\xa3\x10application/json\x00Su\xa0\x9c{"snakes":[{"id":"1","imageASCIIArtString":"e--==--==--"}],"snakeVerification":"Snake could also be a lizard :-)","estimatedPushNotificationTimestamp":null}]
2022-07-24 16:03:56,618 DEBUG [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-2) SRMSG16211: Sending AMQP message to address `snake-alarms`
2022-07-24 16:03:56,622 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Transfer{handle=0, deliveryId=4, deliveryTag=\x01, messageFormat=0, settled=false, more=false, rcvSettleMode=null, state=null, resume=false, aborted=false, batchable=false}[\x00Ss\xd0\x00\x00\x00)\x00\x00\x00\x07@@\xa1\x0csnake-alarms@@@\xa3\x10application/json\x00Su\xa0\x9c{"snakes":[{"id":"1","imageASCIIArtString":"e--==--==--"}],"snakeVerification":"Snake could also be a lizard :-)","estimatedPushNotificationTimestamp":null}]
2022-07-24 16:03:56,625 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Disposition{role=RECEIVER, first=4, last=4, settled=true, state=Accepted{}, batchable=false}
2022-07-24 16:03:57,105 DEBUG [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-2) SRMSG16211: Sending AMQP message to address `snake-alarms`
2022-07-24 16:03:57,110 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Disposition{role=RECEIVER, first=5, last=5, settled=true, state=Accepted{}, batchable=false}
2022-07-24 16:03:57,111 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Transfer{handle=0, deliveryId=5, deliveryTag=\x02, messageFormat=0, settled=false, more=false, rcvSettleMode=null, state=null, resume=false, aborted=false, batchable=false}[\x00Ss\xd0\x00\x00\x00)\x00\x00\x00\x07@@\xa1\x0csnake-alarms@@@\xa3\x10application/json\x00Su\xa0\x9c{"snakes":[{"id":"1","imageASCIIArtString":"e--==--==--"}],"snakeVerification":"Snake could also be a lizard :-)","estimatedPushNotificationTimestamp":null}]
2022-07-24 16:03:57,219 INFO [de.fay.sna.SnakeAlarmProcessor] (vert.x-eventloop-thread-0) Received an alarm for snake alarm fulfillment: [SnakeItems(id=1, imageASCIIArtString=e--==--==--)], False positive - Successfully verified a lizard
2022-07-24 16:03:57,534 DEBUG [io.sma.rea.mes.amqp] (vert.x-eventloop-thread-2) SRMSG16211: Sending AMQP message to address `snake-alarms`
2022-07-24 16:03:57,538 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Disposition{role=RECEIVER, first=6, last=6, settled=true, state=Accepted{}, batchable=false}
2022-07-24 16:03:57,539 FINE [pro.trace] (vert.x-eventloop-thread-0) IN: CH[0] : Transfer{handle=0, deliveryId=6, deliveryTag=\x00, messageFormat=0, settled=false, more=false, rcvSettleMode=null, state=null, resume=false, aborted=false, batchable=false}[\x00Ss\xd0\x00\x00\x00)\x00\x00\x00\x07@@\xa1\x0csnake-alarms@@@\xa3\x10application/json\x00Su\xa0\x9c{"snakes":[{"id":"1","imageASCIIArtString":"e--==--==--"}],"snakeVerification":"Snake could also be a lizard :-)","estimatedPushNotificationTimestamp":null}]
2022-07-24 16:03:58,039 INFO [de.fay.sna.SnakeAlarmProcessor] (vert.x-eventloop-thread-0) Received an alarm for snake alarm fulfillment: [SnakeItems(id=1, imageASCIIArtString=e--==--==--)], False positive - Successfully verified a lizard
2022-07-24 16:03:58,041 INFO [de.fay.sna.SnakeAlarmProcessor] (vert.x-eventloop-thread-0) Received an alarm for snake alarm fulfillment: [SnakeItems(id=1, imageASCIIArtString=e--==--==--)], Danger - Successfully verified a snake
2022-07-24 16:03:58,052 INFO [de.fay.sna.PushNotificationService] (vert.x-eventloop-thread-0) Push notification estimated in 3 sec
2022-07-24 16:03:58,420 INFO [de.fay.sna.SnakeAlarmProcessor] (vert.x-eventloop-thread-0) Received an alarm for snake alarm fulfillment: [SnakeItems(id=1, imageASCIIArtString=e--==--==--)], False positive - Successfully verified a lizard
If it is "Danger - Successfully verified a snake" or "False positive - Successfully verified a lizard" inside the log: we always know if we verified a snake. The push notifications are estimated with 1 second for the faster and less reliable primary version and with 3 seconds in case the secondary fallback version succeeds within the two retries.
Adding resilience to Quarkus using Kotlin is quite easy, cause most of it can be done by using annotations and loose coupling.
If you want to try the resilience example from above, you can find it on Github. Have phun and stay curious.
Epilog
Snakes are cool animals. The snake detector is just a fun fiction and does not have the intent to support fears when it comes to wild animals. Next time I'll choose a scenario helping animals :) - cause we should protect all of them. Buy the way: Tardigrades and ants are good examples for very resilient animals!
Top comments (0)