So we are a little way into this mini series on Akka now.
We have covered a few of the fundamental topics of Akka such as
- Supervision
- Mailboxes
- Logging
When we talked about mailboxes we discussed the fact that there were queues involves that are used as mailboxes.
Generally speaking where there are queues concerned there is also the possibility of dead letters.
In this post we will talk about what ‘dead letters’ are within Akka, and also look at how you can monitor for dead letters
What Are Dead Letters?
In Akka messages that can’t be delivered are routed to a synthetic actor which has the path “/deadLetters”. This is for NON transport lost messages.
Akka makes no guarantees for lost messages at the transport layer.
How Can You Monitor For Them?
Within Akka there is a concept of a event bus. That is a bus that Akka uses internally to send messages. You can also use this event bus for other purposes, such as publishing/subscribing to certain types of messages.
It is kind of like a topic based subscription system.
We can use this event bus to monitor for dead letters.
It is also worth noting that the event bus can be used a general pub/sub mechanism, you can read more about it here : Akka Event Bus
But for now let’s get back to the problem in hand which is how do we monitor for dead letters?
So let’s start out with this actor code
import akka.actor.Actor
import akka.event.Logging
class LifeCycleActor extends Actor {
val log = Logging(context.system, this)
log.info("LifeCycleActor: constructor")
override def preStart { log.info("LifeCycleActor: preStart") }
override def postStop { log.info("LifeCycleActor: postStop") }
override def preRestart(reason: Throwable, message: Option[Any]) {
log.info("LifeCycleActor: preRestart")
log.info(s"LifeCycleActor reason: ${reason.getMessage}")
log.info(s"LifeCycleActor message: ${message.getOrElse("")}")
super.preRestart(reason, message)
}
override def postRestart(reason: Throwable) {
log.info("LifeCycleActor: postRestart")
log.info(s"LifeCycleActor reason: ${reason.getMessage}")
super.postRestart(reason)
}
def receive = {
case RestartMessage => throw new Exception("RestartMessage seen")
case _ => log.info("LifeCycleActor : got a message")
}
}
We will then use this actor which is setup to receive dead letters
import akka.actor.{DeadLetter, Actor}
class DeadLetterMonitorActor
extends Actor
with akka.actor.ActorLogging {
log.info("DeadLetterMonitorActor: constructor")
def receive = {
case d: DeadLetter => {
log.error(s"DeadLetterMonitorActor : saw dead letter $d")
}
case _ => log.info("DeadLetterMonitorActor : got a message")
}
}
We then use this demo code
import akka.actor._
import scala.language.postfixOps
import scala.io.StdIn
import akka.event.Logging
class Runner {
def Run(): Unit = {
//create the actor system
val system = ActorSystem("LifeCycleSystem")
// default Actor constructor
val lifeCycleActor =
system.actorOf(Props[LifeCycleActor],
name = "lifecycleactor")
val deadLetterMonitorActor =
system.actorOf(Props[DeadLetterMonitorActor],
name = "deadlettermonitoractor")
//subscribe to system wide event bus 'DeadLetter'
system.eventStream.subscribe(
deadLetterMonitorActor, classOf[DeadLetter])
val log = Logging(system, classOf[Runner])
log.debug("Runner IS UP BABY")
log.debug("sending lifeCycleActor a few numbers")
lifeCycleActor ! 100
lifeCycleActor ! 200
Thread.sleep(1000)
log.debug("sending lifeCycleActor a poison pill (kill it)")
lifeCycleActor ! PoisonPill
Thread.sleep(1000)
log.debug("sending lifeCycleActor a few numbers")
lifeCycleActor ! 100
lifeCycleActor ! 200
log.debug("stop lifeCycleActor/deadLetterMonitorActor")
system.stop(lifeCycleActor)
system.stop(deadLetterMonitorActor)
//shutdown the actor system
log.debug("stop actor system")
system.terminate()
StdIn.readLine()
}
}
To to do the following
- Create the 2 actors above
- Subscribe the DeadLetterMonitorActor (above) to the “DeadLetter” topic of the inbuilt akka EventStream (internal pub/sub bus)
- We then send a few messages to the LifeCycleActor
- Then send the LifeCylceActor a PoisonPill (Which terminates it)
- We then send a few more messages to the LifeCycleActor (which it doesn’t get cos its dead like)
- We expect the DeadLetterMonitorActor to receive the DeadLetter messages and log these DeadLetter messages
If we run these bits of code this is the result, where it can plainly be seen that the DeadLetterMonitorActor does receive the DeadLetter messages
INFO [LifeCycleSystem-akka.actor.default-dispatcher-2] Slf4jLogger – Slf4jLogger started
16:55:49.546UTC DEBUG[LifeCycleSystem-akka.actor.default-dispatcher-2] EventStream – logger log1-Slf4jLogger started
16:55:49.547UTC DEBUG[LifeCycleSystem-akka.actor.default-dispatcher-2] EventStream – Default Loggers started
16:55:49.559UTC INFO [LifeCycleSystem-akka.actor.default-dispatcher-2] LifeCycleActor – LifeCycleActor: constructor
16:55:49.560UTC INFO [LifeCycleSystem-akka.actor.default-dispatcher-2] LifeCycleActor – LifeCycleActor: preStart
16:55:49.560UTC DEBUG[LifeCycleSystem-akka.actor.default-dispatcher-2] Runner – Runner IS UP BABY
16:55:49.561UTC DEBUG[LifeCycleSystem-akka.actor.default-dispatcher-2] Runner – sending lifeCycleActor a few numbers
16:55:49.561UTC INFO [LifeCycleSystem-akka.actor.default-dispatcher-2] DeadLetterMonitorActor – DeadLetterMonitorActor: constructor
16:55:49.563UTC INFO [LifeCycleSystem-akka.actor.default-dispatcher-2] LifeCycleActor – LifeCycleActor : got a message
16:55:49.563UTC INFO [LifeCycleSystem-akka.actor.default-dispatcher-2] LifeCycleActor – LifeCycleActor : got a message
16:55:50.561UTC DEBUG[LifeCycleSystem-akka.actor.default-dispatcher-2] Runner – sending lifeCycleActor a poison pill (kill it)
16:55:50.576UTC INFO [LifeCycleSystem-akka.actor.default-dispatcher-9] LifeCycleActor – LifeCycleActor: postStop
16:55:51.565UTC DEBUG[LifeCycleSystem-akka.actor.default-dispatcher-9] Runner – sending lifeCycleActor a few numbers
16:55:51.570UTC DEBUG[LifeCycleSystem-akka.actor.default-dispatcher-2] Runner – stop lifeCycleActor/deadLetterMonitorActor
16:55:51.573UTC DEBUG[LifeCycleSystem-akka.actor.default-dispatcher-7] Runner – stop actor system
16:55:51.581UTC INFO [LifeCycleSystem-akka.actor.default-dispatcher-9] RepointableActorRef – Message [java.lang.Integer]
from Actor[akka://LifeCycleSystem/deadLetters] to Actor[akka://LifeCycleSystem/user/lifecycleactor#-912643257]
was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration
settings ‘akka.log-dead-letters’ and ‘akka.log-dead-letters-during-shutdown’.
16:55:51.581UTC INFO [LifeCycleSystem-akka.actor.default-dispatcher-9] RepointableActorRef – Message [java.lang.Integer]
from Actor[akka://LifeCycleSystem/deadLetters] to Actor[akka://LifeCycleSystem/user/lifecycleactor#-912643257]
was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration
settings ‘akka.log-dead-letters’ and ‘akka.log-dead-letters-during-shutdown’.
16:55:51.588UTC ERROR[LifeCycleSystem-akka.actor.default-dispatcher-8] DeadLetterMonitorActor – DeadLetterMonitorActor :
saw dead letter DeadLetter(100,Actor[akka://LifeCycleSystem/deadLetters],Actor[akka://LifeCycleSystem/user/lifecycleactor#-912643257])
16:55:51.590UTC ERROR[LifeCycleSystem-akka.actor.default-dispatcher-8] DeadLetterMonitorActor – DeadLetterMonitorActor :
saw dead letter DeadLetter(200,Actor[akka://LifeCycleSystem/deadLetters],Actor[akka://LifeCycleSystem/user/lifecycleactor#-912643257])
16:55:51.609UTC INFO [LifeCycleSystem-akka.actor.default-dispatcher-6] LocalActorRef – Message [akka.actor.StopChild]
from Actor[akka://LifeCycleSystem/deadLetters] to Actor[akka://LifeCycleSystem/user] was not delivered. [3] dead letters encountered.
This logging can be turned off or adjusted with configuration settings ‘akka.log-dead-letters’ and ‘akka.log-dead-letters-during-shutdown’.
16:55:51.616UTC DEBUG[LifeCycleSystem-akka.actor.default-dispatcher-6] EventStream – shutting down: StandardOutLogger started
NOTE : According to the official Akka docs “Dead letters are not propagated over the network, if you want to collect them in one place you will have to subscribe one actor per network node and forward them manually.”
Are All Messages In DeadLetters A Problem
No, not all of the messages that end up in the deadLetters synthetic actor are problems. For example suppose an actor is sent several Terminate messages. Only one of these will take effect, the others will likely end up in deadLetters, but this is not a concern.
The deadLetters actor and the monitoring of it is more of a debugging aid than anything else. You will have to use some modicum of common sense when examining the results of the deadLetters logging, which we discussed above
Where Is The Code?
As previously stated all the code for this series will end up in this GitHub repo:
Sacha's Blog 