Kubernetes API

The Kubernetes API can be used to discover peers and form a Pekko Cluster. The kubernetes-api mechanism queries the Kubernetes API server to find pods with a given label. A Kubernetes service isn’t required for the cluster bootstrap but may be used for external access to the application.

The following Kubernetes resources are created:

• Deployment: For creating the Pekko pods
• Role and RoleBinding to give the pods access to the API server

An example deployment (used for integration testing):

copysourceapiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: pekko-bootstrap-demo
  name: pekko-bootstrap-demo
spec:
  replicas: 3
  selector:
    matchLabels:
     app: pekko-bootstrap-demo
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate

  template:
    metadata:
      labels:
        app: pekko-bootstrap-demo
        actorSystemName: pekko-bootstrap-demo
    spec:
      containers:
      - name: pekko-bootstrap-demo
        image: integration-test-kubernetes-api:1.3.3.7
        # Remove for a real project, the image is picked up locally for the integratio test
        imagePullPolicy: Never
        livenessProbe:
          httpGet:
            path: /alive
            port: management
        readinessProbe:
          httpGet:
            path: /ready
            port: management
        ports:
        # pekko-management bootstrap
        - containerPort: 7626
          protocol: TCP
          # when contact-point-discovery.port-name is set for cluster bootstrap,
          # the management port must be named accordingly:
          # name: management
        env:
        # The Kubernetes API discovery will use this service name to look for
        # nodes with this value in the 'app' label.
        # This can be customized with the 'pod-label-selector' setting.
        - name: PEKKO_CLUSTER_BOOTSTRAP_SERVICE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: "metadata.labels['app']"

An example Role and Rolebinding to allow the nodes to query the Kubernetes API server:

copysource#
# Create a role, `pod-reader`, that can list pods and
# bind the default service account in the namespace
# that the binding is deployed to to that role.
#

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: pod-reader
rules:
- apiGroups: [""] # "" indicates the core API group
  resources: ["pods"]
  verbs: ["get", "watch", "list"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: read-pods
subjects:
  # Uses the default service account.
  # Consider creating a dedicated service account to run your
  # Apache Pekko Cluster services and binding the role to that one.
- kind: ServiceAccount
  name: default
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

The User name includes the namespace, this will need updated for your namespace.

The following configuration is required:

• Set pekko.management.cluster.bootstrap.contact-point-discovery.discovery-method to kubernetes-api
• Set pekko.discovery.kubernetes-api.pod-label-selector to a label selector that will match the Pekko pods e.g. app=%s

copysourcepekko.discovery {
  kubernetes-api {
    # in fact, this is already the default:
    pod-label-selector = "app=%s"
  }
}

pekko.management {
  cluster.bootstrap {
    contact-point-discovery {
      discovery-method = kubernetes-api
    }
  }
}

The lookup needs to know which namespace to look in. By default, this will be detected by reading the namespace from the service account secret, in /var/run/secrets/kubernetes.io/serviceaccount/namespace, but can be overridden by setting pekko.discovery.kubernetes-api.pod-namespace.

For more details on how to configure the Kubernetes deployment see recipes.