KAFKA on Pigsty v5.0

Module: KAFKA

Mon, 01 Jan 0001 00:00:00 +0000

Kafka is a distributed event-streaming platform. Pigsty’s KAFKA module deploys Apache Kafka 4.1+ dynamic KRaft clusters on managed nodes from RPM/DEB packages, with unified management of security, resources, lifecycle, and observability.

Current status: Beta module

The Kafka module is currently in Beta. Test it thoroughly and confirm it meets your requirements before using it in serious production. That includes dynamic KRaft, strict rolling restart, TLS/SCRAM/ACL, declarative topics/users, credential and certificate rotation, and the full monitoring pipeline.

Quick Start

Mon, 01 Jan 0001 00:00:00 +0000

This tutorial starts from a minimal single-node cluster, walking through topic creation and message read/write; it then deploys a separate, secured three-node cluster with application users, ACLs, quotas, and production topics; finally, it demonstrates core parameter changes, client access, monitoring verification, and pre-launch checks.

Tutorial Scope

Here, “from scratch” means starting before any Kafka is deployed. You will need a working Pigsty admin node with the base INFRA services already deployed; if you don’t have one yet, complete the Pigsty quick install first. The target nodes require SSH/sudo access and must be manageable by the NODE module.

Configuration

Mon, 01 Jan 0001 00:00:00 +0000

The KAFKA module expresses cluster intent through 15 persistent public parameters; everything else — topology, listeners, storage subdirectories, replication safety, authorization, and Exporter placement — is derived by the role in a single, consistent way. For a first deployment, start with the Quickstart; for the full field reference, see Parameters.

Plan First, Then Format

kafka_seq becomes the KRaft node.id. For a new cluster, the randomly generated Cluster ID, the initial controller identities, the security profile, and the initial replication policy are all written into the bootstrap manifest. Once storage has been formatted, do not casually change identities, the security profile, or the controller set. The role validates the live state against the manifest and fails closed on any conflict; it never silently overwrites or reformats data.

Parameters

Mon, 01 Jan 0001 00:00:00 +0000

The KAFKA role deliberately exposes only 15 persistent parameters. Details such as topology, listeners, security implementation, storage subdirectories, replication safety, and Exporter placement are derived by the role in a single, consistent way, and cannot be overridden as additional persistent variables.

Parameter Overview

Parameter	Level	Default	Description
`kafka_cluster`	Cluster	required	Kafka cluster identity
`kafka_seq`	Instance	required	Cluster-unique KRaft `node.id`
`kafka_role`	Instance	`combined`	`combined`, `broker`, or `controller`
`kafka_cluster_id`	Cluster	unset	Takeover/recovery assertion; randomly generated for a new cluster
`kafka_data`	Instance	`/data/kafka`	Role-owned data root directory
`kafka_heap_opts`	Instance	`-Xms1G -Xmx1G`	Kafka JVM heap
`kafka_port`	Instance	`9092`	Broker/client port
`kafka_controller_port`	Instance	`9093`	KRaft controller port
`kafka_rack`	Instance	unset	Broker fault-domain label
`kafka_parameters`	Cluster/Instance	`{}`	Non-role-owned broker parameters
`kafka_jmx_exporter_port`	Instance	`9404`	JMX Exporter HTTP port
`kafka_exporter_port`	Instance	`9308`	Protocol Exporter HTTP port
`kafka_security`	Cluster	`plaintext`	`plaintext` or the production `scram` profile
`kafka_users`	Cluster	`[]`	User credentials, ACLs, and quotas
`kafka_topics`	Cluster	`[]`	Declarative topics

kafka_cluster and kafka_seq must be defined; kafka_role has a real default. The cluster’s roles are either all omitted or all declared explicitly.

Administration

Mon, 01 Jan 0001 00:00:00 +0000

The KAFKA module installs Kafka under /opt/kafka, manages the service with Systemd, and keeps its persistent intent in pigsty.yml. The files generated on the nodes are not meant to be edited by hand.

All of the Kafka CLI examples below use the role-generated /etc/kafka/admin.properties. Even when the current profile is plaintext, keep --command-config on every command: that way the command structure stays the same when you switch the admin channel to scram. Replace <broker>:9092 with a reachable inventory_hostname and port.

Playbook

Mon, 01 Jan 0001 00:00:00 +0000

The KAFKA module ships two playbooks: kafka.yml deploys an Apache Kafka 4.1+ dynamic KRaft cluster and converges its security, resource, and monitoring state; kafka-rm.yml tears down a cluster or removes a member.

Cluster Completeness Constraint

Every selected kafka_cluster must include all of its members: a partial selection fails before anything is written, while one cluster, several complete clusters, or a bare run over the whole inventory are all allowed. Run --check first against the exact same target; before the real run, still verify the backup/rebuild intent, capacity, business window, rollback plan, and change approval by hand.

Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Pigsty gives the KAFKA module a unified observability stack that combines metrics, logs, dashboards, and alerts. Monitoring covers both the Kafka JVM internals and the Kafka protocol view, so you never end up seeing only that the process is alive without visibility into partitions, ISR, and consumer lag, nor seeing only cluster metadata without visibility into the JVM, request queues, and KRaft controller health.

Scrape Architecture

The KAFKA module uses two complementary exporters:

Metrics

Mon, 01 Jan 0001 00:00:00 +0000

The KAFKA module uses two kinds of metric sources, both registered under the same job=kafka scrape job: the JMX target (with a role label) collects the internal state of each JVM, while the protocol exporter target (without a role label) collects the state of the logical cluster, topics, partitions, and consumer groups over the Kafka protocol. The protocol exporter is placed only on the one or two broker-capable nodes with the smallest kafka_seq, and a single-broker cluster runs just one.

FAQ

Mon, 01 Jan 0001 00:00:00 +0000

How mature is the current KAFKA module?

The current role implements a production-grade v1 baseline: dynamic KRaft, full cluster guardrails, cold-start/repair, serial broker admission and dynamic controller join, member retirement (including dead nodes), three-command failed-node replacement, strict rolling restart, TLS/SCRAM/ACL, declarative convergence of topics/users, internal credential and certificate rotation, and the full monitoring pipeline.

It is not a managed Kafka product. Production still requires kafka_security: scram, an odd number of controllers, sufficient brokers/RF/minISR, plus your own capacity planning, reassignment/data balancing, upgrade, backup, restore, and failure drills. The default plaintext is only suitable for development or a trusted, isolated network.