Production considerations

Apache Pekko Connectors Kafka API

1. Do not use Consumer.atMostOnceSource in production as it internally commits the offset after every element.
2. If you create Producer sinks in “inner flows”, be sure to share the Producer instance. This avoids the expensive creation of KafkaProducer instances.

Security setup

The different security setups offered by Kafka brokers are described in the Apache Kafka documentation.

SSL

The properties described in Kafka’s Configuring Kafka Clients for SSL go in the pekko.kafka.consumer.kafka-clients and pekko.kafka.producer.kafka-clients sections of the configuration, or can be added programmatically via ProducerSettings.withProperties and ConsumerSettings.withProperties. The necessary property name constants are available in SslConfigs.

pekko.kafka.producer { # and pekko.kafka.consumer respectively
  kafka-clients {
    security.protocol=SSL
    ssl.truststore.location=/var/private/ssl/kafka.client.truststore.jks
    ssl.truststore.password=test1234
    ssl.keystore.location=/var/private/ssl/kafka.client.keystore.jks
    ssl.keystore.password=test1234
    ssl.key.password=test1234
  }
}

The truststore and keystore locations may specify URLs, absolute paths or relative paths (starting with ./).

You have the option to pass the passwords as command line parameters or environment values via the support in Config.

Kerberos

The properties described in Kafka’s Configuring Kafka Clients for Kerberos go in the pekko.kafka.consumer.kafka-clients and pekko.kafka.producer.kafka-clients sections of the configuration, or can be added programmatically via ProducerSettings.withProperties and ConsumerSettings.withProperties.

pekko.kafka.producer { # and pekko.kafka.consumer respectively
  kafka-clients {
    security.protocol=SASL_PLAINTEXT # (or SASL_SSL)
    sasl.mechanism=GSSAPI
    sasl.kerberos.service.name=kafka
  }
}