Coordination

Pekko Coordination is a set of tools for distributed coordination.

Module info

sbt
val PekkoVersion = "1.1.2+24-bcd44ee3-SNAPSHOT"
libraryDependencies += "org.apache.pekko" %% "pekko-coordination" % PekkoVersion
Gradle
def versions = [
  ScalaBinary: "2.13"
]
dependencies {
  implementation platform("org.apache.pekko:pekko-bom_${versions.ScalaBinary}:1.1.2+24-bcd44ee3-SNAPSHOT")

  implementation "org.apache.pekko:pekko-coordination_${versions.ScalaBinary}"
}
Maven
<properties>
  <scala.binary.version>2.13</scala.binary.version>
</properties>
<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>org.apache.pekko</groupId>
      <artifactId>pekko-bom_${scala.binary.version}</artifactId>
      <version>1.1.2+24-bcd44ee3-SNAPSHOT</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>
<dependencies>
  <dependency>
    <groupId>org.apache.pekko</groupId>
    <artifactId>pekko-coordination_${scala.binary.version}</artifactId>
  </dependency>
</dependencies>
Project Info: Pekko Coordination
Artifact
org.apache.pekko
pekko-coordination
1.1.2+24-bcd44ee3-SNAPSHOT
JDK versions
OpenJDK 8
OpenJDK 11
OpenJDK 17
OpenJDK 21
Scala versions2.13.14, 2.12.20, 3.3.4
JPMS module namepekko.coordination
License
Home pagehttps://pekko.apache.org/
API documentation
Forums
Release notesRelease Notes
IssuesGithub issues
Sourceshttps://github.com/apache/pekko

Lease

The lease is a pluggable API for a distributed lock.

Using a lease

Leases are loaded with:

  • Lease name
  • Config location to indicate which implementation should be loaded
  • Owner name

Any lease implementation should provide the following guarantees:

  • A lease with the same name loaded multiple times, even on different nodes, is the same lease
  • Only one owner can acquire the lease at a time

To acquire a lease:

Scala
sourceval lease = LeaseProvider(system).getLease("<name of the lease>", "docs-lease", "owner")
val acquired: Future[Boolean] = lease.acquire()
val stillAcquired: Boolean = lease.checkLease()
val released: Future[Boolean] = lease.release()
Java
sourceLease lease =
    LeaseProvider.get(system).getLease("<name of the lease>", "jdocs-lease", "<owner name>");
CompletionStage<Boolean> acquired = lease.acquire();
boolean stillAcquired = lease.checkLease();
CompletionStage<Boolean> released = lease.release();

Acquiring a lease returns a FutureCompletionStage as lease implementations typically are implemented via a third party system such as the Kubernetes API server or Zookeeper.

Once a lease is acquired, checkLease can be called to ensure that the lease is still acquired. As lease implementations are based on other distributed systems, a lease can be lost due to a timeout with the third party system. This operation is not asynchronous, so it can be called before performing any action for which having the lease is important.

A lease has an owner. If the same owner tries to acquire the lease multiple times, it will succeed i.e. leases are reentrant.

It is important to pick a lease name that will be unique for your use case. If a lease needs to be unique for each node in a Cluster the cluster host port can be used:

Scala
sourceval owner = Cluster(system).selfAddress.hostPort
Java
source// String owner = Cluster.get(system).selfAddress().hostPort();

For use cases where multiple different leases on the same node then something unique must be added to the name. For example a lease can be used with Cluster Sharding and in this case the shard Id is included in the lease name for each shard.

Setting a lease heartbeat

If a node with a lease crashes or is unresponsive the heartbeat-timeout is how long before other nodes can acquire the lease. Without this timeout operator intervention would be needed to release a lease in the case of a node crash. This is the safest option but not practical in all cases.

The value should be greater than the max expected JVM pause e.g. garbage collection, otherwise a lease can be acquired by another node and then when the original node becomes responsive again there will be a short time before the original lease owner can take action e.g. shutdown shards or singletons.

Usages in other Pekko modules

Leases can be used for Split Brain Resolver, Cluster Singleton, and Cluster Sharding.

Lease implementations

Implementing a lease

Implementations should extend the org.apache.pekko.coordination.lease.scaladsl.Leaseorg.apache.pekko.coordination.lease.javadsl.Lease

Scala
sourceclass SampleLease(settings: LeaseSettings) extends Lease(settings) {

  override def acquire(): Future[Boolean] = {
    Future.successful(true)
  }

  override def acquire(leaseLostCallback: Option[Throwable] => Unit): Future[Boolean] = {
    Future.successful(true)
  }

  override def release(): Future[Boolean] = {
    Future.successful(true)
  }

  override def checkLease(): Boolean = {
    true
  }
}
Java
sourcestatic class SampleLease extends Lease {

  private LeaseSettings settings;

  public SampleLease(LeaseSettings settings) {
    this.settings = settings;
  }

  @Override
  public LeaseSettings getSettings() {
    return settings;
  }

  @Override
  public CompletionStage<Boolean> acquire() {
    return CompletableFuture.completedFuture(true);
  }

  @Override
  public CompletionStage<Boolean> acquire(Consumer<Optional<Throwable>> leaseLostCallback) {
    return CompletableFuture.completedFuture(true);
  }

  @Override
  public CompletionStage<Boolean> release() {
    return CompletableFuture.completedFuture(true);
  }

  @Override
  public boolean checkLease() {
    return true;
  }
}

The methods should provide the following guarantees:

  • acquire should complete with: true if the lease has been acquired, false if the lease is taken by another owner, or fail if it can’t communicate with the third party system implementing the lease.
  • release should complete with: true if the lease has definitely been released, false if the lease has definitely not been released, or fail if it is unknown if the lease has been released.
  • checkLease should return true if the lease has been acquired, should return false until an acquire FutureCompletionStage has completed, and should return false if the lease is lost due to an error communicating with the third party. Check lease should not block.
  • The acquire lease lost callback should only be called after an acquire FutureCompletionStage has completed and should be called if the lease is lost e.g. due to losing communication with the third party system.

In addition, it is expected that a lease implementation will include a time to live mechanism meaning that a lease won’t be held for ever in case the node crashes. If a user prefers to have outside intervention in this case for maximum safety then the time to live can be set to infinite.

The configuration must define the lease-class property for the FQCN of the lease implementation.

The lease implementation should have support for the following properties where the defaults come from pekko.coordination.lease:

source# if the node that acquired the leases crashes, how long should the lease be held before another owner can get it
heartbeat-timeout = 120s

# interval for communicating with the third party to confirm the lease is still held
heartbeat-interval = 12s

# lease implementations are expected to time out acquire and release calls or document
# that they do not implement an operation timeout
lease-operation-timeout = 5s

This configuration location is passed into getLease.

Scala
sourcepekko.actor.provider = cluster
docs-lease {
  lease-class = "docs.coordination.SampleLease"
  heartbeat-timeout = 100s
  heartbeat-interval = 1s
  lease-operation-timeout = 1s
  # Any lease specific configuration
}
Java
sourcepekko.actor.provider = cluster
docs-lease {
  lease-class = "docs.coordination.SampleLease"
  heartbeat-timeout = 100s
  heartbeat-interval = 1s
  lease-operation-timeout = 1s
  # Any lease specific configuration
}
1.1.2+24-bcd44ee3*