Kubernetes Lease
This module is marked as May Change The API, configuration and behavior may change based on feedback from initial usage.
This module is an implementation of a Pekko Coordination Lease in Kubernetes backed by two implementations:
- a Custom Resource Definition (CRD). Resources in Kubernetes offer concurrency control and consistency that have been used to build a distributed lease/lock.
- a native Kubernetes Lease Object.
A lease can be used for:
- Split Brain Resolver. An additional safety measure so that only one SBR instance can make the decision to remain up.
- Cluster Singleton. A singleton manager can be configured to acquire a lease before creating the singleton.
- Cluster Sharding. Each
Shard
can be configured to acquire a lease before creating entity actors.
In all cases the use of the lease increases the consistency of the feature. However, as the Kubernetes API server and its backing etcd
cluster can also be subject to failure and network issues any use of this lease can reduce availability.
Lease Instances
- With Split Brain Resolver there will be one lease per Pekko Cluster
- With multiple Pekko Clusters using SBRs in the same namespace you must ensure different
ActorSystem
names because they all need a separate lease. - With Cluster Sharding and Cluster Singleton there will be more leases
- For Cluster Singleton there will be one per singleton.
- For Cluster Sharding, there will be one per shard per type.
Configuring
Dependency
- sbt
val PekkoManagementVersion = "1.1.0" libraryDependencies += "org.apache.pekko" %% "pekko-lease-kubernetes" % PekkoManagementVersion
- Maven
<properties> <pekko.management.version>1.1.0</pekko.management.version> <scala.binary.version>2.13</scala.binary.version> </properties> <dependencies> <dependency> <groupId>org.apache.pekko</groupId> <artifactId>pekko-lease-kubernetes_${scala.binary.version}</artifactId> <version>${pekko.management.version}</version> </dependency> </dependencies>
- Gradle
def versions = [ PekkoManagementVersion: "1.1.0", ScalaBinary: "2.13" ] dependencies { implementation "org.apache.pekko:pekko-lease-kubernetes_${versions.ScalaBinary}:${versions.PekkoManagementVersion}" }
Creating the Custom Resource Definition for the lease
For the CRD Implementation, a Custom Resource must be created. This requires admin privileges to your Kubernetes / Open Shift cluster but only needs doing once.
Kubernetes:
kubectl apply -f lease.yml
Where lease.yml contains:
sourceapiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
# name must match the spec fields below, and be in the form: <plural>.<group>
name: leases.pekko.apache.org
spec:
group: pekko.apache.org
versions:
- name: v1
storage: true
served: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
owner:
type: string
version:
type: string
time:
type: integer
scope: Namespaced
names:
# kind is normally the CamelCased singular type. Your resource manifests use this.
kind: Lease
listKind: LeaseList
# singular name to be used as an alias on the CLI and for display
singular: lease
# plural name to be used in the URL: /apis/<group>/<version>/<plural>
plural: leases
Enable the native implementation
To enable the native implementation, the lease class must be changed in the configuration to the following value:
pekko.coordination.lease.kubernetes {
lease-class = "org.apache.pekko.coordination.lease.kubernetes.NativeKubernetesLease"
}
Role based access control
Each pod needs permission to read/create and update lease resources. They only need access for the namespace they are in.
An example RBAC that can be used:
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: lease-access
rules:
# Using CRD implementation
- apiGroups: ["pekko.apache.org"]
resources: ["leases"]
verbs: ["get", "create", "update", "list"]
# Using native implementation
- apiGroups: ["coordination.k8s.io"]
resources: ["leases"]
verbs: ["get", "create", "update", "list"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: lease-access
subjects:
- kind: User
name: system:serviceaccount:<YOUR NAMESPACE>:default
roleRef:
kind: Role
name: lease-access
apiGroup: rbac.authorization.k8s.io
This defines a Role
that is allowed to get
, create
and update
lease objects and a RoleBinding
that gives the default service user this role in <YOUR NAMESPACE>
.
Future versions may also require delete
access for cleaning up old resources. Current uses within Pekko only create a single lease so cleanup is not an issue.
To avoid giving an application the access to create new leases an empty lease can be created in the same namespace as the application with:
Kubernetes:
kubelctl create -f sbr-lease.yml -n <YOUR_NAMESPACE>
Where sbr-lease.yml
contains:
apiVersion: "pekko.apache.org/v1"
kind: Lease
metadata:
name: <YOUR_ACTORSYSTEM_NAME>-pekko-sbr
spec:
owner: ""
time: 0
The lease gets created only during an actual Split Brain.
Enable in SBR
To enable the lease for use within SBR:
pekko {
cluster {
downing-provider-class = "org.apache.pekko.cluster.sbr.SplitBrainResolverProvider"
split-brain-resolver {
active-strategy = "lease-majority"
lease-majority {
lease-implementation = "pekko.coordination.lease.kubernetes"
}
}
}
}
Full configuration options
source# SPDX-License-Identifier: Apache-2.0
pekko.coordination.lease.kubernetes {
lease-class = "org.apache.pekko.coordination.lease.kubernetes.KubernetesLease"
api-ca-path = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
api-token-path = "/var/run/secrets/kubernetes.io/serviceaccount/token"
# Host for the Kubernetes API server. Typically this will be set via an environment
# variable that is set when running inside Kubernetes
api-service-host = "localhost"
api-service-host = ${?KUBERNETES_SERVICE_HOST}
# Port for the Kubernetes API server. Typically this will be set via an environment
# variable that is set when running inside Kubernetes
api-service-port = 8080
api-service-port = ${?KUBERNETES_SERVICE_PORT}
# Namespace file path. The namespace is to create the lock in. Can be overridden by "namespace"
#
# If this path doesn't exist, the namespace will default to "default".
namespace-path = "/var/run/secrets/kubernetes.io/serviceaccount/namespace"
# Namespace to create the lock in. If set to something other than "<namespace>" then overrides any value
# in "namespace-path"
namespace = "<namespace>"
# How often to write time into CRD so that if the holder crashes
# another node can take the lease after a given timeout. If left blank then the default is
# max(5s, heartbeat-timeout / 10) which will be 12s with the default heartbeat-timeout
heartbeat-interval = ""
# How long a lease must not be updated before another node can assume
# the holder has crashed.
# If the lease holder hasn't crashed its next heart beat will fail due to the version
# having been updated
heartbeat-timeout = 120s
# The individual timeout for each HTTP request. Defaults to 2/5 of the lease-operation-timeout
# Can't be greater than then lease-operation-timeout
api-server-request-timeout = ""
# Use TLS & auth token for communication with the API server
# set to false for plain text with no auth
secure-api-server = true
# The amount of time to wait for a lease to be acquired or released. This includes all requests to the API
# server that are required. If this timeout is hit then the lease *may* be taken due to the response being lost
# on the way back from the API server but will be reported as not taken and can be safely retried.
lease-operation-timeout = 5s
}
F.A.Q
Q. What happens if the node that holds the lease crashes?
A. Each lease has a Time To Live (TTL) that is set pekko.coordination.lease.kubernetes.heartbeat-timeout
which defaults to 120s. A lease holder updates the lease every 1/10
of the timeout to keep the lease. If the TTL passes without the lease being updated another node is allowed to take the lease. For ultimate safety this timeout can be set very high but then an operator would need to come and clear the lease if a lease owner crashes with the lease taken.