Logging

You are viewing the documentation for the new actor APIs, to view the Pekko Classic documentation, see Classic Logging.

Dependency

To use Logging, you must at least use the Pekko actors dependency in your project, and configure logging via the SLF4J backend, such as Logback configuration.

sbt

val PekkoVersion = "1.0.3"
libraryDependencies += "org.apache.pekko" %% "pekko-actor-typed" % PekkoVersion

Maven

<properties>
  <scala.binary.version>2.13</scala.binary.version>
</properties>
<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>org.apache.pekko</groupId>
      <artifactId>pekko-bom_${scala.binary.version}</artifactId>
      <version>1.0.3</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>
<dependencies>
  <dependency>
    <groupId>org.apache.pekko</groupId>
    <artifactId>pekko-actor-typed_${scala.binary.version}</artifactId>
  </dependency>
</dependencies>

Gradle

def versions = [
  ScalaBinary: "2.13"
]
dependencies {
  implementation platform("org.apache.pekko:pekko-bom_${versions.ScalaBinary}:1.0.3")

  implementation "org.apache.pekko:pekko-actor-typed_${versions.ScalaBinary}"
}

Introduction

SLF4J is used for logging and Pekko provides access to an org.slf4j.Logger for a specific actor via the ActorContextActorContext. You may also retrieve a Logger with the ordinary org.slf4j.LoggerFactory.

To ensure that logging has minimal performance impact it’s important that you configure an asynchronous appender for the SLF4J backend. Logging generally means IO and locks, which can slow down the operations of your code if it was performed synchronously.

How to log

The ActorContextActorContext provides access to an org.slf4j.Logger for a specific actor.

Scala

copysourceBehaviors.receive[String] { (context, message) =>
  context.log.info("Received message: {}", message)
  Behaviors.same
}

Java

copysourcepublic class MyLoggingBehavior extends AbstractBehavior<String> {

  public static Behavior<String> create() {
    return Behaviors.setup(MyLoggingBehavior::new);
  }

  private MyLoggingBehavior(ActorContext<String> context) {
    super(context);
  }

  @Override
  public Receive<String> createReceive() {
    return newReceiveBuilder().onMessage(String.class, this::onReceive).build();
  }

  private Behavior<String> onReceive(String message) {
    getContext().getLog().info("Received message: {}", message);
    return this;
  }
}

The Logger via the ActorContext will automatically have a name that corresponds to the BehaviorBehavior of the actor when the log is accessed the first time. The class name when using AbstractBehaviorAbstractBehavior or the class or object name where the Behavior is defined when using the functional style. You can set a custom logger name with the setLoggerNamesetLoggerName of the ActorContext.

Scala

copysourceBehaviors.setup[String] { context =>
  context.setLoggerName("com.myservice.BackendManager")
  context.log.info("Starting up")

  Behaviors.receiveMessage { message =>
    context.log.debug("Received message: {}", message)
    Behaviors.same
  }
}

Java

copysourcepublic class BackendManager extends AbstractBehavior<String> {

  public static Behavior<String> create() {
    return Behaviors.setup(
        context -> {
          context.setLoggerName(BackendManager.class);
          context.getLog().info("Starting up");
          return new BackendManager(context);
        });
  }

  private BackendManager(ActorContext<String> context) {
    super(context);
  }

  @Override
  public Receive<String> createReceive() {
    return newReceiveBuilder().onMessage(String.class, this::onReceive).build();
  }

  private Behavior<String> onReceive(String message) {
    getContext().getLog().debug("Received message: {}", message);
    return this;
  }
}

The convention is to use logger names like fully qualified class names. The parameter to setLoggerName can be a String or a Class, where the latter is convenience for the class name.

When logging via the ActorContext the path of the actor will automatically be added as pekkoSource Mapped Diagnostic Context (MDC) value. MDC is typically implemented with a ThreadLocal by the SLF4J backend. To reduce performance impact, this MDC value is set when you access the loggetLog() method so you shouldn’t cache the returned Logger in your own field. That is handled by ActorContext and retrieving the Logger repeatedly with the loggetLog method has low overhead. The MDC is cleared automatically after processing of current message has finished.

It’s also perfectly fine to use a Logger retrieved via org.slf4j.LoggerFactory, but then the logging events will not include the pekkoSource MDC value. This is the recommended way when logging outside of an actor, including logging from FutureCompletionStage callbacks.

Scala

copysourceval log = LoggerFactory.getLogger("com.myservice.BackendTask")

Future {
  // some work
  "result"
}.onComplete {
  case Success(result) => log.info("Task completed: {}", result)
  case Failure(exc)    => log.error("Task failed", exc)
}

Java

copysourceclass BackendTask {
  private final Logger log = LoggerFactory.getLogger(getClass());

  void run() {
    CompletableFuture<String> task =
        CompletableFuture.supplyAsync(
            () -> {
              // some work
              return "result";
            });
    task.whenComplete(
        (result, exc) -> {
          if (exc == null) log.error("Task failed", exc);
          else log.info("Task completed: {}", result);
        });
  }
}

Placeholder arguments

The log message may contain argument placeholders {}, which will be substituted if the log level is enabled. Compared to constructing a full string for the log message this has the advantage of avoiding superfluous string concatenation and object allocations when the log level is disabled. Some logging backends may also use these message templates before argument substitution to group and filter logging events.

It can be good to know that 3 or more arguments will result in the relatively small cost of allocating an array (vararg parameter) also when the log level is disabled. The methods with 1 or 2 arguments don’t allocate the vararg array.

Behaviors.logMessages

If you want very detailed logging of messages and signals you can decorate a BehaviorBehavior with Behaviors.logMessagesBehaviors.logMessages.

Scala

copysourceimport org.apache.pekko.actor.typed.LogOptions
import org.slf4j.event.Level

Behaviors.logMessages(LogOptions().withLevel(Level.TRACE), BackendManager())

Java

copysourceimport org.slf4j.event.Level;

Behaviors.logMessages(LogOptions.create().withLevel(Level.TRACE), BackendManager.create());

MDC

MDC allows for adding additional context dependent attributes to log entries. Out of the box, Pekko will place the path of the actor in the MDC attribute pekkoSource.

One or more tags can also be added to the MDC using the ActorTagsActorTags props. The tags will be rendered as a comma separated list and be put in the MDC attribute pekkoTags. This can be used to categorize log entries from a set of different actors to allow easier filtering of logs:

Scala

copysourcecontext.spawn(myBehavior, "MyActor", ActorTags("processing"))

Java

copysourcecontext.spawn(myBehavior, "MyActor", ActorTags.create("processing"));

In addition to these two built in MDC attributes you can also decorate a BehaviorBehavior with Behaviors.withMdcBehaviors.withMdc or use the org.slf4j.MDC API directly.

The Behaviors.withMdc decorator has two parts. A static Map of MDC attributes that are not changed, and a dynamic Map that can be constructed for each message.

Scala

copysourceval staticMdc = Map("startTime" -> system.startTime.toString)
Behaviors.withMdc[BackendManager.Command](
  staticMdc,
  mdcForMessage =
    (msg: BackendManager.Command) => Map("identifier" -> msg.identifier, "upTime" -> system.uptime.toString)) {
  BackendManager()
}

Java

copysourceMap<String, String> staticMdc = new HashMap<>();
staticMdc.put("startTime", String.valueOf(system.startTime()));

Behaviors.withMdc(
    BackendManager2.Command.class,
    staticMdc,
    message -> {
      Map<String, String> msgMdc = new HashMap<>();
      msgMdc.put("identifier", message.identifier());
      msgMdc.put("upTime", String.valueOf(system.uptime()));
      return msgMdc;
    },
    BackendManager2.create());

If you use the MDC API directly, be aware that MDC is typically implemented with a ThreadLocal by the SLF4J backend. Pekko clears the MDC if logging is performed via the loggetLog() of the ActorContext and it is cleared automatically after processing of current message has finished, but only if you accessed loggetLog(). The entire MDC is cleared, including attributes that you add yourself to the MDC. MDC is not cleared automatically if you use a Logger via LoggerFactory or not touch loggetLog() in the ActorContext.

SLF4J backend

To ensure that logging has minimal performance impact it’s important that you configure an asynchronous appender for the SLF4J backend. Logging generally means IO and locks, which can slow down the operations of your code if it was performed synchronously.

Logback

pekko-actor-typed includes a dependency to the slf4j-api. In your runtime, you also need a SLF4J backend. We recommend Logback:

sbt

libraryDependencies += "ch.qos.logback" % "logback-classic" % "1.2.13"

Maven

<dependencies>
  <dependency>
    <groupId>ch.qos.logback</groupId>
    <artifactId>logback-classic</artifactId>
    <version>1.2.13</version>
  </dependency>
</dependencies>

Gradle

dependencies {
  implementation "ch.qos.logback:logback-classic:1.2.13"
}

Logback has flexible configuration options and details can be found in the Logback manual and other external resources.

One part that is important to highlight is the importance of configuring an AsyncAppender, because it offloads rendering of logging events to a background thread, increasing performance. It doesn’t block the threads of the ActorSystemActorSystem while the underlying infrastructure writes the log messages to disk or other configured destination. It also contains a feature which will drop INFO and DEBUG messages if the logging load is high.

A starting point for configuration of logback.xml for production:

copysource<?xml version="1.0" encoding="UTF-8"?>
<configuration>

    <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>myapp.log</file>
        <immediateFlush>false</immediateFlush>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>myapp_%d{yyyy-MM-dd}.log</fileNamePattern>
        </rollingPolicy>
        <encoder>
            <pattern>[%date{ISO8601}] [%level] [%logger] [%marker] [%thread] - %msg MDC: {%mdc}%n</pattern>
        </encoder>
    </appender>

    <appender name="ASYNC" class="ch.qos.logback.classic.AsyncAppender">
        <queueSize>8192</queueSize>
        <neverBlock>true</neverBlock>
        <appender-ref ref="FILE" />
    </appender>

    <root level="INFO">
        <appender-ref ref="ASYNC"/>
    </root>
</configuration>

Note that the AsyncAppender may drop log events if the queue becomes full, which may happen if the logging backend can’t keep up with the throughput of produced log events. Dropping log events is necessary if you want to gracefully degrade your application if only your logging backend or filesystem is experiencing issues.

An alternative of the Logback AsyncAppender with better performance is the Logstash async appender.

The ELK-stack is commonly used as logging infrastructure for production:

• Logstash Logback encoder
• Logstash
• Elasticsearch
• Kibana

For development you might want to log to standard out, but also have all debug level logging to file, like in this example:

copysource<?xml version="1.0" encoding="UTF-8"?>
<configuration>

    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
            <level>INFO</level>
        </filter>
        <encoder>
            <pattern>[%date{ISO8601}] [%level] [%logger] [%marker] [%thread] - %msg MDC: {%mdc}%n</pattern>
        </encoder>
    </appender>

    <appender name="FILE" class="ch.qos.logback.core.FileAppender">
        <file>target/myapp-dev.log</file>
        <encoder>
            <pattern>[%date{ISO8601}] [%level] [%logger] [%marker] [%thread] - %msg MDC: {%mdc}%n</pattern>
        </encoder>
    </appender>

    <root level="DEBUG">
        <appender-ref ref="STDOUT"/>
        <appender-ref ref="FILE"/>
    </root>
</configuration>

Place the logback.xml file in src/main/resources/logback.xml. For tests you can define different logging configuration in src/test/resources/logback-test.xml.

MDC values

When logging via the loggetLog() of the ActorContext, as described in How to log, Pekko includes a few MDC properties:

• pekkoSource: the actor’s path
• pekkoAddress: the full address of the ActorSystem, including hostname and port if Cluster is enabled
• pekkoTags: tags defined in the PropsProps of the actor
• sourceActorSystem: the name of the ActorSystem

These MDC properties can be included in the Logback output with for example %X{pekkoSource} specifier within the pattern layout configuration:

  <encoder>
    <pattern>%date{ISO8601} %-5level %logger{36} %X{pekkoSource} - %msg%n</pattern>
  </encoder>

All MDC properties as key-value entries can be included with %mdc:

  <encoder>
    <pattern>%date{ISO8601} %-5level %logger{36} - %msg MDC: {%mdc}%n</pattern>
  </encoder>

Internal logging by Pekko

Event bus

For historical reasons logging by the Pekko internals and by classic actors are performed asynchronously through an event bus. Such log events are processed by an event handler actor, which then emits them to SLF4J or directly to standard out.

When pekko-actor-typed and pekko-slf4j are on the classpath this event handler actor will emit the events to SLF4J. The event.slf4j.Slf4jLoggerevent.slf4j.Slf4jLogger and event.slf4j.Slf4jLoggingFilterevent.slf4j.Slf4jLoggingFilter are enabled automatically without additional configuration. This can be disabled by pekko.use-slf4j=off configuration property.

In other words, you don’t have to do anything for the Pekko internal logging to end up in your configured SLF4J backend.

Log level

Ultimately the log level defined in the SLF4J backend is used. For the Pekko internal logging it will also check the level defined by the SLF4J backend before constructing the final log message and emitting it to the event bus.

However, there is an additional pekko.loglevel configuration property that defines if logging events with lower log level should be discarded immediately without consulting the SLF4J backend. By default this is at INFO level, which means that DEBUG level logging from the Pekko internals will not reach the SLF4J backend even if DEBUG is enabled in the backend.

You can enable DEBUG level for pekko.loglevel and control the actual level in the SLF4j backend without any significant overhead, also for production.

pekko.loglevel = "DEBUG"

To turn off all Pekko internal logging (not recommended) you can configure the log levels to be OFF like this.

pekko {
  stdout-loglevel = "OFF"
  loglevel = "OFF"
}

The stdout-loglevel is only in effect during system startup and shutdown, and setting it to OFF as well, ensures that nothing gets logged during system startup or shutdown.

See Logger names for configuration of log level in SLF4J backend for certain modules of Pekko.

Logging to stdout during startup and shutdown

When the actor system is starting up and shutting down the configured loggers are not used. Instead log messages are printed to stdout (System.out). The default log level for this stdout logger is WARNING and it can be silenced completely by setting pekko.stdout-loglevel=OFF.

Logging of Dead Letters

By default messages sent to dead letters are logged at info level. Existence of dead letters does not necessarily indicate a problem, but they are logged by default for the sake of caution. After a few messages this logging is turned off, to avoid flooding the logs. You can disable this logging completely or adjust how many dead letters are logged. During system shutdown it is likely that you see dead letters, since pending messages in the actor mailboxes are sent to dead letters. You can also disable logging of dead letters during shutdown.

pekko {
  log-dead-letters = 10
  log-dead-letters-during-shutdown = on
}

To customize the logging further or take other actions for dead letters you can subscribe to the Event Stream.

Auxiliary logging options

Pekko has a few configuration options for very low level debugging. These make more sense in development than in production.

You almost definitely need to have logging set to DEBUG to use any of the options below:

pekko {
  loglevel = "DEBUG"
}

This config option is useful if you want to know what config settings are loaded by Pekko:

pekko {
  # Log the complete configuration at INFO level when the actor system is started.
  # We do not recommend using this logging in production environments as it can include sensitive values.
  # This is useful when you are uncertain of what configuration is used.
  log-config-on-start = on
}

If you want unhandled messages logged at DEBUG:

pekko {
  actor {
    debug {
      # enable DEBUG logging of unhandled messages
      unhandled = on
    }
  }
}

If you want to monitor subscriptions (subscribe/unsubscribe) on the ActorSystem.eventStream:

pekko {
  actor {
    debug {
      # enable DEBUG logging of subscription changes on the eventStream
      event-stream = on
    }
  }
}

Auxiliary remote logging options

If you want to see all messages that are sent through remoting at DEBUG log level, use the following config option. Note that this logs the messages as they are sent by the transport layer, not by an actor.

pekko.remote.artery {
  # If this is "on", Pekko will log all outbound messages at DEBUG level,
  # if off then they are not logged
  log-sent-messages = on
}

If you want to see all messages that are received through remoting at DEBUG log level, use the following config option. Note that this logs the messages as they are received by the transport layer, not by an actor.

pekko.remote.artery {
  # If this is "on", Pekko will log all inbound messages at DEBUG level,
  # if off then they are not logged
  log-received-messages = on
}

Logging of message types with payload size in bytes larger than the configured log-frame-size-exceeding.

pekko.remote.artery {
  log-frame-size-exceeding = 10000b
}

MDC values from Pekko internal logging

Since the logging is done asynchronously, the thread in which the logging was performed is captured in MDC with attribute name sourceThread.

The path of the actor in which the logging was performed is available in the MDC with attribute name pekkoSource.

The actor system name in which the logging was performed is available in the MDC with attribute name sourceActorSystem, but that is typically also included in the pekkoSource attribute.

The address of the actor system, containing host and port if the system is using cluster, is available through pekkoAddress.

For typed actors the log event timestamp is taken when the log call was made but for Pekko’s internal logging as well as the classic actor logging is asynchronous which means that the timestamp of a log entry is taken from when the underlying logger implementation is called, which can be surprising at first. If you want to more accurately output the timestamp for such loggers, use the MDC attribute pekkoTimestamp. Note that the MDC key will not have any value for a typed actor.

Markers

Pekko is logging some events with markers. Some of these events also include structured MDC properties.

• The “SECURITY” marker is used for highlighting security related events or incidents.
• Pekko Actor is using the markers defined in ActorLogMarkerActorLogMarker.
• Pekko Cluster is using the markers defined in ClusterLogMarkerClusterLogMarker.
• Pekko Remoting is using the markers defined in RemoteLogMarkerRemoteLogMarker.
• Pekko Cluster Sharding is using the markers defined in ShardingLogMarkerShardingLogMarker.

Markers and MDC properties are automatically picked up by the Logstash Logback encoder.

The marker can be included in the Logback output with %marker and all MDC properties as key-value entries with %mdc.

  <encoder>
    <pattern>[%date{ISO8601}] [%level] [%logger] [%marker] [%thread] - %msg MDC: {%mdc}%n</pattern>
  </encoder>

Logger names

It can be useful to enable debug level or other SLF4J backend configuration for certain modules of Pekko when troubleshooting. Those logger names are typically prefixed with the package name of the classes in that module. For example, in Logback the configuration may look like this to enable debug logging for Cluster Sharding:

   <logger name="org.apache.pekko.cluster.sharding" level="DEBUG" />

    <root level="INFO">
        <appender-ref ref="ASYNC"/>
    </root>

Other examples of logger names or prefixes:

org.apache.pekko.cluster
org.apache.pekko.cluster.Cluster
org.apache.pekko.cluster.ClusterHeartbeat
org.apache.pekko.cluster.ClusterGossip
org.apache.pekko.cluster.ddata
org.apache.pekko.cluster.pubsub
org.apache.pekko.cluster.singleton
org.apache.pekko.cluster.sharding
org.apache.pekko.coordination.lease
org.apache.pekko.discovery
org.apache.pekko.persistence
org.apache.pekko.remote

Logging in tests

Testing utilities are described in Testing.