Kubernetes API

If you want to use Kubernetes for Cluster Bootstrap, please follow the Cluster Bootstrap Kubernetes API documentation that is tailored for that use case.

The typical way to consume a service in Kubernetes is to discover it through DNS: this will take into account liveness/readiness checks, and depending on the configuration take care of load balancing (removing the need for client-side load balancing). For this reason, for general usage the pekko-dns implementation is usually a better fit for discovering services in Kubernetes. However, in some cases, such as for Cluster Bootstrap, it is desirable to connect to the pods directly, bypassing any liveness/readiness checks or load balancing. For such situations we provide a discovery implementation that uses the Kubernetes API.

Project Info

Dependencies and usage

First, add the dependency on the component:

sbt

val PekkoManagementVersion = "1.1.0"
libraryDependencies += "org.apache.pekko" %% "pekko-discovery-kubernetes-api" % PekkoManagementVersion

Gradle

def versions = [
  PekkoManagementVersion: "1.1.0",
  ScalaBinary: "2.13"
]
dependencies {
  implementation "org.apache.pekko:pekko-discovery-kubernetes-api_${versions.ScalaBinary}:${versions.PekkoManagementVersion}"
}

Maven

<properties>
  <pekko.management.version>1.1.0</pekko.management.version>
  <scala.binary.version>2.13</scala.binary.version>
</properties>
<dependencies>
  <dependency>
    <groupId>org.apache.pekko</groupId>
    <artifactId>pekko-discovery-kubernetes-api_${scala.binary.version}</artifactId>
    <version>${pekko.management.version}</version>
  </dependency>
</dependencies>

pekko-discovery-kubernetes-api can be used with Pekko 1.1.2 or later. You have to override the following Pekko dependencies by defining them explicitly in your build and define the Pekko version to the one that you are using. Latest patch version of Pekko is recommended and a later version than 1.1.2 can be used.

sbt

val PekkoVersion = "1.1.2"
libraryDependencies ++= Seq(
  "org.apache.pekko" %% "pekko-cluster" % PekkoVersion,
  "org.apache.pekko" %% "pekko-discovery" % PekkoVersion
)

Gradle

def versions = [
  PekkoVersion: "1.1.2",
  ScalaBinary: "2.13"
]
dependencies {
  implementation "org.apache.pekko:pekko-cluster_${versions.ScalaBinary}:${versions.PekkoVersion}"
  implementation "org.apache.pekko:pekko-discovery_${versions.ScalaBinary}:${versions.PekkoVersion}"
}

Maven

<properties>
  <pekko.version>1.1.2</pekko.version>
  <scala.binary.version>2.13</scala.binary.version>
</properties>
<dependencies>
  <dependency>
    <groupId>org.apache.pekko</groupId>
    <artifactId>pekko-cluster_${scala.binary.version}</artifactId>
    <version>${pekko.version}</version>
  </dependency>
  <dependency>
    <groupId>org.apache.pekko</groupId>
    <artifactId>pekko-discovery_${scala.binary.version}</artifactId>
    <version>${pekko.version}</version>
  </dependency>
</dependencies>

As described above, it is uncommon to use the Kubernetes API discovery mechanism as your default discovery mechanism. When using it with Pekko Cluster Bootstrap, it is sufficient to configure it as described here. Otherwise, to load it manually, use loadServiceDiscovery on the Discovery extension:

Scala

copysourceval discovery = Discovery(system).loadServiceDiscovery("kubernetes-api")

Java

copysourceServiceDiscovery discovery = Discovery.get(system).loadServiceDiscovery("kubernetes-api");

To find other pods, this method needs to know how they are labeled, what the name of the target port is, and what namespace they reside in. Below, you’ll find the default configuration. It can be customized by changing these values in your application.conf.

pekko.discovery {
  kubernetes-api {
    # Namespace discovery path
    #
    # If this path doesn't exist, the namespace will default to "default".
    pod-namespace-path = "/var/run/secrets/kubernetes.io/serviceaccount/namespace"

    # Namespace to query for pods.
    #
    # Set this value to a specific string to override discovering the namespace using pod-namespace-path.
    pod-namespace = "<pod-namespace>"

    # Selector value to query pod API with.
    # `%s` will be replaced with the configured effective name, which defaults to the actor system name
    pod-label-selector = "app=%s"
  }
}

This configuration complements the following Deployment specification:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: example
  name: example
spec:
  replicas: 4
  selector:
    matchLabels:
      app: example
  template:
    metadata:
      labels:
        app: example
    spec:
      containers:
      - name: example
        image: example/image:1.0.0
        imagePullPolicy: IfNotPresent
        ports:
        # pekko remoting
        - name: remoting
          containerPort: 7355
          protocol: TCP
        # When
        # pekko.management.cluster.bootstrap.contact-point-discovery.port-name
        # is defined, it must correspond to this name:
        - name: management
          containerPort: 7626
          protocol: TCP

Role-Based Access Control

If your Kubernetes cluster has Role-Based Access Control (RBAC) enabled, you’ll also have to grant the Service Account that your pods run under access to list pods. The following configuration can be used as a starting point. It creates a Role, pod-reader, which grants access to query pod information. It then binds the default Service Account to the Role by creating a RoleBinding. Adjust as necessary.

Using Google Kubernetes Engine? Your user will need permission to grant roles. See Google’s Documentation for more information.

copysource#
# Create a role, `pod-reader`, that can list pods and
# bind the default service account in the namespace
# that the binding is deployed to to that role.
#

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: pod-reader
rules:
- apiGroups: [""] # "" indicates the core API group
  resources: ["pods"]
  verbs: ["get", "watch", "list"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: read-pods
subjects:
  # Uses the default service account.
  # Consider creating a dedicated service account to run your
  # Apache Pekko Cluster services and binding the role to that one.
- kind: ServiceAccount
  name: default
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io