Skip to content

This is the multi-page printable view of this section. .

Return to the regular view of this page.

Pigsty v5.0 Documentation

From core concepts and quick start to production deployment, module references, and day-to-day operations.

The Pigsty v5.0 documentation focuses on Pigsty itself: architecture, installation, deployment, configuration, operations, and the complete manual for every first-party module.

v5.0 Docs Preview OINK 0.6.0 Local First

Press ⌘K on macOS, or CtrlK, to open offline search and the command palette anywhere on the site.

Get Started

Core Modules

PGSQLCORE

Highly available PostgreSQL clusters, services, backup, monitoring, security, and daily administration.

INFRACORE

VictoriaMetrics, VictoriaLogs, Grafana, Nginx, and infrastructure services.

NODECORE

Host management, software baselines, log collection, VIP, and HAProxy load balancing.

ETCDCORE

Reliable distributed configuration storage for PostgreSQL high availability.

Optional Modules

MINIO / SiloSTORAGE

S3-compatible object storage and a PostgreSQL backup repository.

REDIS / ValkeyCACHE

Primary-replica, Sentinel, and native cluster modes.

JUICEFILESYSTEM

JuiceFS backed by PostgreSQL metadata and object storage.

KAFKAMESSAGING

Dynamic KRaft, TLS, ACL, and complete observability.

MYSQLDATABASE

MySQL 8.4 LTS and InnoDB Cluster.

DOCKERRUNTIME

A managed Docker service and container runtime.

VIBEDEVELOPMENT

Code-Server, Jupyter, and an AI coding sandbox.

Content Scope

This site does not duplicate the standalone manuals for Pig, Patroni, pg_exporter, pgBackRest, PgBouncer, the software repository, application templates, or pilot projects. When these components must be referenced, the documentation links to the existing site; Pigsty’s own integration, configuration, and operational guidance remains in the relevant module manual.

1 - Get Started

Deploy Pigsty single-node version on your laptop/cloud server, access DB and Web UI

Pigsty uses a scalable architecture design, suitable for both large-scale production environments and single-node development/demo environments. This guide focuses on the latter.

If you intend to learn about Pigsty, you can start with the Quick Start single-node deployment. A Linux virtual machine with 1C/2G is sufficient to run Pigsty.

You can use a Linux MiniPC, free/discounted virtual machines provided by cloud providers, Windows WSL, or create a virtual machine on your own laptop for Pigsty deployment. Pigsty provides out-of-the-box Vagrant templates and Terraform templates to help you provision Linux VMs with one click locally or in the cloud.

pigsty-arch

The single-node version of Pigsty includes all core features: 575 PG extensions, self-contained Grafana/Victoria monitoring, IaC provisioning capabilities, and local PITR point-in-time recovery. If you have external object storage (for PostgreSQL PITR backup), then for scenarios like demos, personal websites, and small services, even a single-node environment can provide a certain degree of data persistence guarantee. However, single-node cannot achieve High Availability—automatic failover requires at least 3 nodes.

If you want to install Pigsty in an environment without internet connection, please refer to the Offline Install mode. If you only need the PostgreSQL database itself, please refer to the Slim Install mode. If you are ready to start serious multi-node production deployment, please refer to the Deployment Guide.

Quick Start

Prepare a node with compatible Linux system, and execute as an admin user with passwordless ssh and sudo privileges:

curl -fsSL https://repo.pigsty.io/get | bash  # Install Pigsty and dependencies
cd ~/pigsty; ./configure -g                   # Generate config (with 1-node template, -g generates random passwords)
./deploy.yml                                  # Execute deployment playbook

Yes, it’s that simple. You can use pre-configured templates to bring up Pigsty with one click without understanding any details.

Next, you can explore the Graphical User Interface, access PostgreSQL database services; or perform configuration customization and execute playbooks to deploy more clusters.

1.1 - Single-Node Installation

Get started with Pigsty—complete single-node install on a fresh Linux host!

This is the Pigsty single-node install guide Single Node. For multi-node HA production deployment, refer to the Deployment docs.

Pigsty single-node installation consists of three steps: Install, Configure, and Deploy.

Summary

Prepare a node with compatible OS, and run as an admin user with nopass ssh and sudo:

Choose a Pigsty download mirror:

pigsty.io (Global)
curl -fsSL https://repo.pigsty.io/get | bash
pigsty.cc (China)
curl -fsSL https://repo.pigsty.cc/get | bash

This command runs the install script, downloads and extracts Pigsty source to your home directory and installs dependencies. Then complete Configure and Deploy:

Enter the Source Directory

Terminal 
cd ~/pigsty

Generate the Inventory

Terminal 
./configure -g

Skip this step if you already have a prepared pigsty.yml.

Run the Deployment Playbook

Terminal 
./deploy.yml

After installation, access the Web UI via IP/domain + port 80/443 through Nginx, and access the default PostgreSQL service via port 5432.

The complete process takes 3–10 minutes depending on server specs/network. Offline installation speeds this up significantly; for monitoring-free setups, use Slim Install for even faster deployment.

Video Example: Online Single-Node Installation (Debian 13, x86_64)

demo/install-hero.cast

Prepare

Installing Pigsty involves some preparation work. Here’s a checklist.

For single-node installations, many constraints can be relaxed—typically you only need to know your IP address. If you don’t have a static IP, use 127.0.0.1.

Item Requirement Item Requirement
Node 1-node, at least 1C2G, no upper limit Disk /data mount point, xfs recommended
OS Linux x86_64 / aarch64, EL/Debian/Ubuntu Network Static IPv4; single-node without fixed IP can use 127.0.0.1
SSH nopass SSH login via public key SUDO sudo privilege, preferably with nopass option

Typically, you only need to focus on your local IP address—as an exception, for single-node deployment, use 127.0.0.1 if no static IP available.

Install

Use the following commands to auto-install Pigsty source to ~/pigsty (recommended). Deployment dependencies (Ansible) are installed automatically.

Choose a Pigsty download mirror:

pigsty.io (Global)
curl -fsSL https://repo.pigsty.io/get | bash            # Install current default version
curl -fsSL https://repo.pigsty.io/get | bash -s v4.5.0  # Pin current public stable release
pigsty.cc (China)
curl -fsSL https://repo.pigsty.cc/get | bash            # Install current default version
curl -fsSL https://repo.pigsty.cc/get | bash -s v4.5.0  # Pin current public stable release

If you prefer not to run a remote script, you can manually download or clone the source. When using git, always checkout a specific version before use.

Terminal 
git clone https://github.com/pgsty/pigsty; cd pigsty;
git checkout v4.5.0;  # Always checkout a released tag when using git

For manual download/clone installations, run the bootstrap script to install Ansible and other dependencies. You can also install them yourself.

Terminal 
./bootstrap           # Install ansible for subsequent deployment

Configure

In Pigsty, deployment blueprints are defined by the inventory, the pigsty.yml configuration file. You can customize through declarative configuration.

Pigsty provides the configure script as an optional configuration wizard, which generates an inventory with good defaults based on your environment and input:

Terminal 
./configure -g                # Use config wizard to generate config with random passwords

The generated config file is at ~/pigsty/pigsty.yml by default. Review and customize as needed before installation.

Many configuration templates are available for reference. You can skip the wizard and directly edit pigsty.yml:

Terminal 
./configure                  # Default template, install PG 18 with essential extensions
./configure -v 16            # Use PG 16 instead of default PG 18
./configure -c rich          # Create local repo, download all extensions, install major ones
./configure -c slim          # Minimal install template, use with ./slim.yml playbook
./configure -c app/supa      # Use app/supa self-hosted Supabase template
./configure -c ivory         # Use IvorySQL kernel instead of native PG
./configure -i 10.11.12.13   # Explicitly specify primary IP address
./configure -r china         # Use China mirrors instead of default repos
./configure -c ha/full -s    # Use 4-node sandbox template, skip IP replacement/detection

The output below is from the current main branch (v5.0.0-preview). If you install another version, the first line reports that version.

Current main branch configure output
Example 1 Example configure output from the current main branch
configure output 
vagrant@meta:~/pigsty$ ./configure

configure pigsty v4.5.0 begin
[ OK ] region  = default
[ OK ] kernel  = Linux
[ OK ] machine = x86_64
[ OK ] package = rpm,dnf
[ OK ] vendor  = rocky (Rocky Linux)
[ OK ] version = 9 (9.6)
[ OK ] sudo = vagrant ok
[ OK ] ssh = [email protected] ok
[WARN] Multiple IP address candidates found:
    (1) 192.168.121.24	inet 192.168.121.24/24 brd 192.168.121.255 scope global dynamic noprefixroute eth0
    (2) 10.10.10.12	    inet 10.10.10.12/24 brd 10.10.10.255 scope global noprefixroute eth1
[ IN ] INPUT primary_ip address (of current meta node, e.g 10.10.10.10):
=> 10.10.10.12    # <------- INPUT YOUR PRIMARY IPV4 ADDRESS HERE!
[ OK ] primary_ip = 10.10.10.12 (from input)
[ OK ] admin = [email protected] ok
[ OK ] mode = meta (el9)
[ OK ] locale  = C.UTF-8
[ OK ] configure pigsty done
proceed with ./deploy.yml

Common configure Arguments

-i | --ip IPv4

The primary private IP of the current host, used to replace the 10.10.10.10 placeholder in the inventory.

-c | --conf string

A configuration template name relative to conf/, without the .yml suffix.

-v | --version integer

PostgreSQL major version 14 through 19; PG19 is Beta, so use the dedicated pg19 template.

-r | --region enum defaultdefault

Upstream repository region for faster downloads: default, china, or europe.

-n | --non-interactive boolean defaultfalse

Use command-line arguments for the primary IP and skip the interactive wizard.

-x | --proxy boolean defaultfalse

Use current environment variables to configure proxy_env.

If your machine has multiple IPs bound, use -i|--ip <ipaddr> to explicitly specify the primary IP, or provide it in the interactive prompt. The script replaces the placeholder 10.10.10.10 with your node’s primary IPv4 address. Choose a static IP; do not use public IPs.

Change default passwords!

We strongly recommend modifying default passwords and credentials in the config file before installation. See Security Recommendations for details.

Deploy

Pigsty’s deploy.yml playbook applies the blueprint from Configure to target nodes.

Terminal 
./deploy.yml     # Deploy the defined modules in the core path at once
Example deployment output
deploy output 
......

TASK [pgsql : pgsql init done] *************************************************
ok: [10.10.10.11] => {
    "msg": "postgres://10.10.10.11/postgres | meta  | dbuser_meta dbuser_view "
}
......

TASK [pg_monitor : load grafana datasource meta] *******************************
changed: [10.10.10.11]

PLAY RECAP *********************************************************************
10.10.10.11                : ok=302  changed=232  unreachable=0    failed=0    skipped=65   rescued=0    ignored=1
localhost                  : ok=6    changed=3    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0

When you see pgsql init done, PLAY RECAP and similar output at the end, installation is complete!

Upstream repo changes may cause online installation failures!

Upstream repos used by Pigsty (like Linux/PGDG repos) can sometimes enter a broken state due to improper updates, causing deployment failures (this has happened multiple times)! You can wait for upstream fixes or use pre-made offline packages to solve this.

Avoid re-running the deployment playbook!

Warning: Running deploy.yml again on an existing deployment may restart services and overwrite configurations!

Interface

After single-node installation, you typically have four modules installed on the current node: PGSQL, INFRA, NODE, and ETCD.

ID NODE PGSQL INFRA ETCD
1 10.10.10.10 pg-meta-1 infra-1 etcd-1

The INFRA module provides a graphical management interface, accessible via Nginx on ports 80/443.

The PGSQL module provides a PostgreSQL database server, listening on 5432, also accessible via Pgbouncer/HAProxy proxies.

Pigsty online demo homepage

More

Use the current node as a base to deploy and monitor more clusters: add cluster definitions to the inventory and run:

bin/node-add   pg-test      # Add the 3 nodes of cluster pg-test to Pigsty management
bin/pgsql-add  pg-test      # Initialize a 3-node pg-test HA PG cluster
bin/redis-add  redis-ms     # Initialize Redis cluster: redis-ms

Most modules require the NODE module installed first. See available modules for details:

PGSQL, INFRA, NODE, ETCD, MINIO, REDIS, DOCKER……

1.2 - Docker Deployment

Spin up Pigsty in Docker containers for quick testing on macOS/Windows

Pigsty is designed for native Linux, but can also run in Linux containers with systemd. If you don’t have native Linux (e.g., macOS or Windows), use Docker to spin up a local single-node Pigsty for testing.

Quick Start

Enter the docker/ dir in Pigsty source and launch with one command:

cd ~/pigsty/docker
make launch          # Start container + generate config + deploy

After deployment, access services:

Service URL / Command Credentials
SSH ssh root@localhost -p 2222 Password: pigsty
Web Portal http://localhost:8080 -
Grafana http://localhost:8080/ui admin / grafana_admin_password
PostgreSQL psql 'postgres://dbuser_dba:<pg_admin_password>@localhost:5432/postgres' pg_admin_password

make launch runs ./configure -g internally to generate random passwords. You can check them with:

cd ~/pigsty/docker
make pass | grep -E 'grafana_admin_password|pg_admin_password'
Web Portal & PostgreSQL

Web Portal and PostgreSQL are only available after Deployment (./deploy.yml) completes.

Prepare

Docker deployment requires:

Item Requirement Item Requirement
Docker Docker 20.10+ (Desktop or CE) CPU At least 1 core
RAM At least 2GB Disk At least 20GB free

Ensure default host ports (2222/8080/8443/5432) are available, or edit .env first.

Good Use Cases
  • Quick Pigsty experience on macOS/Windows without native Linux
  • Learning and testing Pigsty features, dev and debug
  • Quick local PostgreSQL dev environment
Not Recommended For
  • Production: Container perf and stability inferior to native Linux
  • HA Clusters: Docker single-node mode can’t achieve multi-node HA
  • Large Scale: Use native Linux VMs or physical machines

Image

Pigsty provides an out-of-the-box Docker image on Docker Hub.

Image Pull Size Contents
pgsty/pigsty ~500MB 1.3GB Debian 13 + systemd + SSH + pig + Ansible
  • Supports both amd64 (x86_64) and arm64 (Apple Silicon, AWS Graviton)
  • Image tags follow Pigsty versions. The Docker configuration on the current main branch and the site baseline both use v5.0.0-preview; verify that the matching remote image exists before pulling or deploying.
  • Pre-configured with docker template, ready to run ./deploy.yml

Built on Debian 13 (Trixie), pre-installed with pig CLI and Ansible, Pigsty source already initialized.

Launch

Pigsty provides out-of-the-box Docker support in the docker/ source directory.

Simplest way is make launch, which auto-completes: start container, generate config, and deploy:

cd ~/pigsty/docker
make launch          # One-liner: up + config + deploy

Or step by step for inspection at each stage:

cd ~/pigsty/docker
make up              # Start container
make exec            # Enter container
./configure -c docker -g --ip 127.0.0.1  # Generate config (optional, pre-configured)
./deploy.yml         # Execute deployment

To build locally instead of pulling from Docker Hub:

cd ~/pigsty/docker
make build           # Build image locally
make launch          # Start container + generate config + deploy

Config

Customize image version and port mappings via .env:

PIGSTY_VERSION=v4.5.0         # Current main source default; verify the remote tag before pulling
PIGSTY_SSH_PORT=2222          # SSH port
PIGSTY_HTTP_PORT=8080         # Nginx HTTP port
PIGSTY_HTTPS_PORT=8443        # Nginx HTTPS port
PIGSTY_PG_PORT=5432           # PostgreSQL port

Port Mapping:

Env Var Default Container Description
PIGSTY_VERSION v4.5.0 - Current main source default; verify the remote tag separately
PIGSTY_SSH_PORT 2222 22 SSH access port
PIGSTY_HTTP_PORT 8080 80 Nginx HTTP port
PIGSTY_HTTPS_PORT 8443 443 Nginx HTTPS port
PIGSTY_PG_PORT 5432 5432 PostgreSQL port

Override via env vars if defaults are occupied:

PIGSTY_HTTP_PORT=8888 docker compose up -d

Commands

Pigsty Docker provides Makefile commands for container and image management.

Docker Compose

Recommended way to run:

make up           # Start container
make down         # Stop and remove container
make start        # Start stopped container
make stop         # Stop container
make restart      # Restart container
make pull         # Pull latest image
make config       # Run ./configure in container
make deploy       # Run ./deploy.yml in container
make launch       # One-liner: up + config + deploy

Container Access 

make exec         # Enter container bash
make ssh          # SSH into container
make log          # View container logs
make status       # View systemd status
make ps           # View process list
make conf         # View config file
make pass         # View passwords in config

Image Build 

make build        # Build image locally
make buildnc      # Build without cache
make push         # Build and push multi-arch image

Image Management 

make save         # Export image to pigsty-<version>-<arch>.tgz
make load         # Import image from tgz file
make rmi          # Remove current version's pigsty image

Cleanup 

make clean        # Stop and remove container
make purge        # Stop and remove the container, then directly delete ./data in the current directory
Use make purge with care

The current Makefile no longer provides a countdown prompt. After removing the container, make purge runs rm -rf -- ./data directly. Verify the current directory and target data first, and back it up when necessary.

Manual Run

If you prefer docker run over Docker Compose:

mkdir -p ./data
docker run -d --privileged --name pigsty \
  -p 2222:22 -p 8080:80 -p 5432:5432 \
  -v ./data:/data \
  pgsty/pigsty:<version>

docker exec -it pigsty ./configure -c docker -g --ip 127.0.0.1
docker exec -it pigsty ./deploy.yml

Or use Makefile’s make run:

make run          # Start with docker run
make exec         # Enter container
make clean        # Stop and remove container
make purge        # Remove container and directly delete ./data in the current directory

How It Works

Pigsty Docker image is based on Debian 13 (Trixie) with systemd as init. Service management inside container stays consistent with native Linux via systemctl.

Key features:

  • systemd support: Full systemd for proper service management
  • SSH access: Pre-configured SSH, root password is pigsty
  • Privileged mode: Requires --privileged for systemd
  • Data persistence: Via /data volume mount
  • Pre-installed: pig CLI + Ansible, Pigsty source initialized

Image build executes these init steps:

# Install pig CLI
RUN echo "deb [trusted=yes] https://repo.pigsty.io/apt/infra/ generic main" \
    > /etc/apt/sources.list.d/pigsty.list \
    && apt-get update && apt-get install -y pig

# Initialize Pigsty source and install Ansible
RUN pig sty init -v ${PIGSTY_VERSION} \
    && pig sty boot \
    && pig sty conf -c docker --ip 127.0.0.1

Running ./configure with -c docker applies the Docker-optimized config template:

  • Uses 127.0.0.1 as default IP
  • Tuned for container environment

FAQ

Container won’t start

Ensure Docker is properly installed with sufficient resources. On Docker Desktop, allocate at least 2GB RAM. Check for port conflicts on 2222, 8080, 8443, 5432.

Can’t access services

Web Portal and PostgreSQL only available after deployment. Ensure ./deploy.yml finished successfully. Use make status to check service status.

Port conflicts

Override via .env or env vars:

PIGSTY_HTTP_PORT=8888 PIGSTY_PG_PORT=5433 docker compose up -d

Data persistence

Container data mounted to ./data. To wipe and start fresh:

make purge        # Remove container and directly delete ./data in the current directory (no countdown)

macOS performance

On macOS with Docker Desktop, performance is worse than native Linux due to virtualization overhead. Expected—Docker deployment is for dev/testing. For production, use native Linux installation.

More

1.3 - Web Interface

Explore Pigsty’s Web graphical management interface, Grafana dashboards, and how to access them via domain names and HTTPS.

After single-node installation, you’ll have the INFRA module installed on the current node, which includes an out-of-the-box Nginx web server.

The default server configuration provides a WebUI graphical interface for displaying monitoring dashboards and unified proxy access to other component web interfaces.

Access

You can access this graphical interface by entering the deployment node’s IP address in your browser. By default, Nginx serves on standard ports 80/443.

Direct IP Access Domain (HTTP) Domain (HTTPS) Demo
http://10.10.10.10 http://i.pigsty https://i.pigsty https://demo.pigsty.io

Pigsty online demo homepage

Monitoring

To access Pigsty’s monitoring system dashboards (Grafana), visit the /ui endpoint on the server.

Direct IP Access Domain (HTTP) Domain (HTTPS) Demo
http://10.10.10.10/ui http://i.pigsty/ui https://i.pigsty/ui https://demo.pigsty.io/ui

If your service is exposed to Internet or office network, we recommend accessing via domain names and enabling HTTPS encryption—only minimal configuration is needed.

Endpoints

By default, Nginx exposes the following endpoints via different paths on the default server at ports 80/443:

Endpoint Component Native Port Description Public Demo
/ Nginx 80/443 Homepage, local repo, file service demo.pigsty.io
/ui/ Grafana 3000 Grafana dashboard portal demo.pigsty.io/ui/
/vmetrics/ VictoriaMetrics 8428 Time series database Web UI demo.pigsty.io/vmetrics/
/vlogs/ VictoriaLogs 9428 Log database Web UI demo.pigsty.io/vlogs/
/vtraces/ VictoriaTraces 10428 Distributed tracing Web UI demo.pigsty.io/vtraces/
/vmalert/ VMAlert 8880 Alert rule management demo.pigsty.io/vmalert/
/alertmgr/ AlertManager 9059 Alert management Web UI demo.pigsty.io/alertmgr/
/blackbox/ Blackbox 9115 Blackbox exporter
/haproxy/* HAProxy 9101 Load balancer admin Web UI
/pev PEV2 80 PostgreSQL execution plan visualizer demo.pigsty.io/pev
/nginx Nginx 80 Nginx status page (for metrics)

Domain Access

If you have your own domain name, you can point it to Pigsty server’s IP address to access various services via domain.

If you want to enable HTTPS, you should modify the home server configuration in the infra_portal parameter:

all:
  vars:
    infra_portal:
      home : { domain: i.pigsty } # Replace i.pigsty with your domain
all:
  vars:
    infra_portal:  # domain specifies the domain name  # certbot parameter specifies certificate name
      home : { domain: demo.pigsty.io ,certbot: mycert }

You can run make cert command after deployment to apply for a free Let’s Encrypt certificate for the domain. If you don’t define the certbot field, Pigsty will use the local CA to issue a self-signed HTTPS certificate by default. In this case, you must first trust Pigsty’s self-signed CA to access normally in your browser.

You can also mount local directories and other upstream services to Nginx. For more management details, refer to INFRA Management - Nginx.

1.4 - Getting Started with PostgreSQL

Get started with PostgreSQL—connect using CLI and graphical clients

PostgreSQL (abbreviated as PG) is the world’s most advanced and popular open-source relational database. Use it to store and retrieve multi-modal data.

This guide is for developers with basic Linux CLI experience but not very familiar with PostgreSQL, helping you quickly get started with PG in Pigsty.

We assume you’re a personal user deploying in the default single-node mode. For prod multi-node HA cluster access, refer to Prod Service Access.

Basics

In the default single-node installation template, you’ll create a PostgreSQL database cluster named pg-meta on the current node, with only one primary instance.

PostgreSQL listens on port 5432, and the cluster has a preset database meta available for use.

After installation, exit the current admin user ssh session and re-login to refresh environment variables. Then simply type pp and press Enter to access the database cluster via the psql CLI tool (p is the shortcut for the pig CLI):

vagrant@pg-meta-1:~$ pp
psql (18.6 (Ubuntu 18.6-1.pgdg24.04+1))
Type "help" for help.

postgres=#

You can also switch to the postgres OS user and execute psql directly to connect to the default postgres admin database.

Connecting to Database

To access a PostgreSQL database, use a CLI tool or graphical client and fill in the PostgreSQL connection string:

postgres://username:password@host:port/dbname

Some drivers and tools may require you to fill in these parameters separately. The following five are typically required:

Parameter Description Example Value Notes
host Database server address 10.10.10.10 Replace with your node IP or domain; can omit for localhost
port Port number 5432 PG default port, can be omitted
username Username dbuser_dba Pigsty default database admin
password Password DBUser.DBA Pigsty default admin password (change this!)
dbname Database name meta Default template database name

For personal use, you can directly use the Pigsty default database superuser dbuser_dba for connection and management. The dbuser_dba has full database privileges. By default, if you specified the configure -g parameter when configuring Pigsty, the password will be randomly generated and saved in ~/pigsty/pigsty.yml:

cat ~/pigsty/pigsty.yml | grep pg_admin_password

Default Accounts

Pigsty’s default single-node template presets the following database users, ready to use out of the box:

Username Password Role Purpose
dbuser_dba DBUser.DBA Superuser Database admin (change this!)
dbuser_meta DBUser.Meta Business admin App R/W (change this!)
dbuser_view DBUser.Viewer Read-only user Data viewing (change this!)

For example, you can connect to the meta database in the pg-meta cluster using three different connection strings with three different users:

postgres://dbuser_dba:[email protected]:5432/meta
postgres://dbuser_meta:[email protected]:5432/meta
postgres://dbuser_view:[email protected]:5432/meta

Note: These default passwords are automatically replaced with random strong passwords when using configure -g. Remember to replace the IP address and password with actual values.

Using CLI Tools

psql is the official PostgreSQL CLI client tool, powerful and the first choice for DBAs and developers.

On a server with Pigsty deployed, you can directly use psql to connect to the local database:

# Simplest way: use postgres system user for local connection (no password needed)
sudo -u postgres psql

# Use connection string (recommended, most universal)
psql 'postgres://dbuser_dba:[email protected]:5432/meta'

# Use parameter form
psql -h 10.10.10.10 -p 5432 -U dbuser_dba -d meta

# Use env vars to avoid password appearing in command line
export PGPASSWORD='DBUser.DBA'
psql -h 10.10.10.10 -p 5432 -U dbuser_dba -d meta

After successful connection, you’ll see a prompt like this:

psql (18.6)
Type "help" for help.

meta=#

Common psql Commands

After entering psql, you can execute SQL statements or use meta-commands starting with \:

Command Description Command Description
Ctrl+C Interrupt query Ctrl+D Exit psql
\? Show all meta commands \h Show SQL command help
\l List all databases \c dbname Switch to database
\d table View table structure \d+ table View table details
\du List all users/roles \dx List installed extensions
\dn List all schemas \dt List all tables

Executing SQL

In psql, directly enter SQL statements ending with semicolon ;:

-- Check PostgreSQL version
SELECT version();

-- Check current time
SELECT now();

-- Create a test table
CREATE TABLE test (id SERIAL PRIMARY KEY, name TEXT, created_at TIMESTAMPTZ DEFAULT now());

-- Insert data
INSERT INTO test (name) VALUES ('hello'), ('world');

-- Query data
SELECT * FROM test;

-- Drop test table
DROP TABLE test;

Using Graphical Clients

If you prefer graphical interfaces, here are some popular PostgreSQL clients:

Grafana

Pigsty’s INFRA module includes Grafana with a pre-configured PostgreSQL data source (Meta). You can directly query the database using SQL from the Grafana Explore panel through the browser graphical interface, no additional client tools needed.

Grafana’s default username is admin, and the password can be found in the grafana_admin_password field in the inventory (default pigsty).

DataGrip

DataGrip is a professional database IDE from JetBrains, with powerful features. IntelliJ IDEA’s built-in Database Console can also connect to PostgreSQL in a similar way.

DBeaver

DBeaver is a free open-source universal database tool supporting almost all major databases. It’s a cross-platform desktop client.

pgAdmin

pgAdmin is the official PostgreSQL-specific GUI tool from PGDG, available through browser or as a desktop client.

Pigsty provides a configuration template for one-click pgAdmin service deployment using Docker in Software Template: pgAdmin.

Viewing Monitoring Dashboards

Pigsty provides many PostgreSQL monitoring dashboards, covering everything from cluster overview to single-table analysis.

We recommend starting with PGSQL Overview. Many elements in the dashboards are clickable, allowing you to drill down layer by layer to view details of each cluster, instance, database, and even internal database objects like tables, indexes, and functions.

Trying Extensions

One of PostgreSQL’s most powerful features is its extension ecosystem. Extensions can add new data types, functions, index methods, and more to the database.

Pigsty provides 575 extensions covering 16 major categories including time-series, geographic, vector, and full-text search, installable with one click. Start with three commonly used extensions, then install more extensions such as timescaledb as needed.

  • postgis: Geographic information system for processing maps and location data (installed by default)
  • pgvector: Vector database supporting AI embedding vector similarity search (installed by default)
  • timescaledb: Time-series database for efficient storage and querying of time-series data (optional install)
\dx                            -- psql meta command, list installed extensions
TABLE pg_available_extensions; -- Query installed, available extensions
CREATE EXTENSION postgis;      -- Enable postgis extension

Next Steps

Congratulations on completing the PostgreSQL basics! Next, you can start configuring and customizing your database.

1.5 - Customize Pigsty with Configuration

Express your infra and clusters with declarative config files

Besides using the configuration wizard to auto-generate configs, you can write Pigsty config files from scratch. This tutorial guides you through building a complex inventory step by step.

If you define NODE, INFRA, ETCD, MINIO, and PGSQL in the inventory upfront, deploy.yml can deploy this core path in one run—but it hides the details. Optional modules such as Docker, Redis, Kafka, native MySQL, JUICE, and VIBE require their own playbooks.

This doc breaks down all modules and playbooks, showing how to incrementally build from a simple config to a complete deployment.

Minimal Configuration

The simplest valid config only defines the admin_ip variable—the IP address of the node where Pigsty is installed (admin node):

Minimal
all: { vars: { admin_ip: 10.10.10.10 } }
Mirror
# Set region: china to use mirrors
all: { vars: { admin_ip: 10.10.10.10, region: china } }

This config deploys nothing, but running ./deploy.yml generates a self-signed CA in files/pki/ca for issuing certificates.

For convenience, you can also set region to specify which region’s software mirrors to use (default, china, europe).

Add Nodes

Pigsty’s NODE module manages cluster nodes. Any IP address in the inventory will be managed by Pigsty with the NODE module installed.

Minimal
all:  # Remember to replace 10.10.10.10 with your actual IP
  children: { nodes: { hosts: { 10.10.10.10: {} } } }
  vars:
    admin_ip: 10.10.10.10                   # Current node IP
    region: default                         # Default repos
    node_repo_modules: node,pgsql,infra     # Add node, pgsql, infra repos
Mirror
all:  # Remember to replace 10.10.10.10 with your actual IP
  children: { nodes: { hosts: { 10.10.10.10: {} } } }
  vars:
    admin_ip: 10.10.10.10                 # Current node IP
    region: china                         # Use mirrors
    node_repo_modules: node,pgsql,infra   # Add node, pgsql, infra repos

We added two global parameters: node_repo_modules specifies repos to add; region specifies which region’s mirrors to use.

These parameters enable the node to use correct repositories and install required packages. The NODE module offers many customization options: node names, DNS, repos, packages, NTP, kernel params, tuning templates, monitoring, log collection, etc. Even without changes, the defaults are sufficient.

Run deploy.yml or more precisely node.yml to bring the defined node under Pigsty management.

ID NODE INFRA ETCD PGSQL Description
1 10.10.10.10 - - - Add node

Add Infrastructure

A full-featured RDS cloud database service needs infrastructure support: monitoring (metrics/log collection, alerting, visualization), NTP, DNS, and other foundational services.

Define a special group infra to deploy the INFRA module:

Minimal
all:  # Simply changed group name from nodes -> infra and added infra_seq
  children: { infra: { hosts: { 10.10.10.10: { infra_seq: 1 } } } }
  vars:
    admin_ip: 10.10.10.10
    region: default
    node_repo_modules: node,pgsql,infra
Mirror
all:  # Simply changed group name from nodes -> infra and added infra_seq
  children: { infra: { hosts: { 10.10.10.10: { infra_seq: 1 } } } }
  vars:
    admin_ip: 10.10.10.10
    region: china
    node_repo_modules: node,pgsql,infra

We also assigned an identity parameter: infra_seq to distinguish nodes in multi-node HA INFRA deployments.

Run infra.yml to install INFRA **](/docs/infra/) and [**NODE modules on 10.10.10.10:

./infra.yml   # Install INFRA module on infra group (includes NODE module)
demo/infra.cast

NODE module is implicitly defined as long as an IP exists. NODE is idempotent—re-running has no side effects.

After completion, you’ll have complete observability infrastructure and node monitoring, but PostgreSQL database service is not yet deployed.

If your goal is just to set up this monitoring system (Grafana + Victoria), you’re done! The infra template is designed for this. Everything in Pigsty is modular: you can deploy only monitoring infra without databases; or vice versa—run HA PostgreSQL clusters without infra—Slim Install.

ID NODE INFRA ETCD PGSQL Description
1 10.10.10.10 infra-1 - - Add infrastructure

Deploy Database Cluster

To provide PostgreSQL service, install the PGSQL` module and its dependency ETCD—just two lines of config:

Minimal
all:
  children:
    infra:   { hosts: { 10.10.10.10: { infra_seq: 1 } } }
    etcd:    { hosts: { 10.10.10.10: { etcd_seq:  1 } } } # Add etcd cluster
    pg-meta: { hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }, vars: { pg_cluster: pg-meta } } # Add pg cluster
  vars: { admin_ip: 10.10.10.10, region: default, node_repo_modules: node,pgsql,infra }
Mirror
all:
  children:
    infra:   { hosts: { 10.10.10.10: { infra_seq: 1 } } }
    etcd:    { hosts: { 10.10.10.10: { etcd_seq:  1 } } } # Add etcd cluster
    pg-meta: { hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }, vars: { pg_cluster: pg-meta } } # Add pg cluster
  vars: { admin_ip: 10.10.10.10, region: china, node_repo_modules: node,pgsql,infra }

We added two new groups: etcd and pg-meta, defining a single-node etcd cluster and a single-node PostgreSQL cluster.

Use ./deploy.yml to converge the defined modules in the core path again, or deploy incrementally:

./etcd.yml  -l etcd      # Install ETCD module on etcd group
./pgsql.yml -l pg-meta   # Install PGSQL module on pg-meta group

PGSQL depends on ETCD for HA consensus, so install ETCD first. After completion, you have a working PostgreSQL service!

ID NODE INFRA ETCD PGSQL Description
1 10.10.10.10 infra-1 etcd-1 pg-meta-1 Add etcd and PostgreSQL cluster

We used node.yml, infra.yml, etcd.yml, and pgsql.yml to deploy all four core modules on a single machine.

Define Databases and Users

In Pigsty, you can customize PostgreSQL cluster internals like databases and users through the inventory:

all:
  children:
    # Other groups and variables hidden for brevity
    pg-meta:
      hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
      vars:
        pg_cluster: pg-meta
        pg_users:       # Define database users
          - { name: dbuser_meta ,password: DBUser.Meta ,pgbouncer: true ,roles: [dbrole_admin] ,comment: admin user  }
        pg_databases:   # Define business databases
          - { name: meta ,baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty] ,extensions: [vector] }
  • pg_users: Defines a new user dbuser_meta with password DBUser.Meta
  • pg_databases: Defines a new database meta with Pigsty CMDB schema (optional) and vector extension

Pigsty offers rich customization parameters covering all aspects of databases and users. If you define these parameters upfront, they’re automatically created during ./pgsql.yml execution. For existing clusters, you can incrementally create or modify users and databases:

bin/pgsql-user pg-meta dbuser_meta      # Ensure user dbuser_meta exists in pg-meta
bin/pgsql-db   pg-meta meta             # Ensure database meta exists in pg-meta

Configure PG Version and Extensions

You can install different major versions of PostgreSQL, and up to 575 extensions. Let’s remove the current default PG 18 and install PG 16:

./pgsql-rm.yml -l pg-meta --check # Preflight old pg-meta removal; execute only after backup and target confirmation

We can customize parameters to install and enable common extensions by default: timescaledb, postgis, and pgvector:

all:
  children:
    infra:   { hosts: { 10.10.10.10: { infra_seq: 1 } } }
    etcd:    { hosts: { 10.10.10.10: { etcd_seq:  1 } } } # Add etcd cluster
    pg-meta:
      hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
      vars:
        pg_cluster: pg-meta
        pg_version: 16   # Specify PG version as 16
        pg_extensions: [ timescaledb, postgis, pgvector ]      # Install these extensions
        pg_libs: 'timescaledb,pg_stat_statements,auto_explain'  # Preload these extension libraries
        pg_databases: { { name: meta ,baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty] ,extensions: [vector, postgis, timescaledb ] } }
        pg_users: { { name: dbuser_meta ,password: DBUser.Meta ,pgbouncer: true ,roles: [dbrole_admin] ,comment: admin user } }

  vars:
    admin_ip: 10.10.10.10
    region: default
    node_repo_modules: node,pgsql,infra
./pgsql.yml -l pg-meta   # Install PG16 and extensions, recreate pg-meta cluster

Add More Nodes

Add more nodes to the deployment, bring them under Pigsty management, deploy monitoring, configure repos, install software…

# Add entire cluster at once, or add nodes individually
bin/node-add pg-test

bin/node-add 10.10.10.11
bin/node-add 10.10.10.12
bin/node-add 10.10.10.13
demo/node.cast

Deploy HA PostgreSQL Cluster

Now deploy a new database cluster pg-test on the three newly added nodes, using a three-node HA architecture:

all:
  children:
    infra:   { hosts: { 10.10.10.10: { infra_seq: 1 } } }
    etcd:    { hosts: { 10.10.10.10: { etcd_seq: 1 } } }, vars: { etcd_cluster: etcd } }
    pg-meta: { hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }, vars: { pg_cluster: pg-meta } }
    pg-test:
      hosts:
        10.10.10.11: { pg_seq: 1, pg_role: primary }
        10.10.10.12: { pg_seq: 2, pg_role: replica  }
        10.10.10.13: { pg_seq: 3, pg_role: replica  }
      vars: { pg_cluster: pg-test }
demo/pgsql.cast

Deploy Redis Cluster

Pigsty provides optional Redis support as a caching service in front of PostgreSQL:

bin/redis-add redis-ms
bin/redis-add redis-meta
bin/redis-add redis-test

Redis HA requires cluster mode or sentinel mode. See Redis Configuration.

Deploy Silo Object Storage

Pigsty’s MINIO module currently deploys Silo S3-compatible object storage, which can serve as a PostgreSQL backup repository. The module, inventory group, and playbooks retain the compatible minio name.

./minio.yml -l minio

Serious production Silo deployments typically require at least 4 nodes with 4 disks each (4N/16D).

Deploy Docker Module

If you want to use containers to run tools for managing PG or software using PostgreSQL, install the DOCKER module:

./docker.yml -l infra

Use pre-made application templates to launch common software tools with one click, such as the GUI tool for PG management: Pgadmin:

./app.yml    -l infra -e app=pgadmin

You can even self-host enterprise-grade Supabase with Pigsty, using external HA PostgreSQL clusters as the foundation and running stateless components in containers.

1.6 - Run Playbooks with Ansible

Use Ansible playbooks to deploy and manage Pigsty clusters

Pigsty uses Ansible to manage clusters, a very popular large-scale/batch/automation ops tool in the SRE community.

Ansible can use declarative approach for server configuration management. All module deployments are implemented through a series of idempotent Ansible playbooks.

For example, in single-node deployment, you’ll use the deploy.yml playbook. Pigsty has more built-in playbooks, you can choose to use as needed.

Understanding Ansible basics helps with better use of Pigsty, but this is not required, especially for single-node deployment.

Deploy Playbook

Pigsty provides a “one-stop” deploy playbook deploy.yml for the core path: CA/software repository, NODE, INFRA, ETCD, PGSQL, and MINIO when enabled in the inventory. Optional modules such as Redis, Kafka, and native MySQL require their own module playbooks even when defined in the inventory.

Playbook Command Group infra [nodes] etcd minio [pgsql]
infra.yml ./infra.yml -l infra ✓ ✓
node.yml ./node.yml ✓ ✓ ✓ ✓
etcd.yml ./etcd.yml -l etcd ✓
minio.yml ./minio.yml -l minio ✓
pgsql.yml ./pgsql.yml ✓

This is the simplest deployment method. You can also follow instructions in Customization Guide to incrementally complete deployment of all modules and nodes step by step.

Install Ansible

When using the Pigsty installation script or the bootstrap phase of offline installation, Pigsty will automatically install ansible and its dependencies for you.

If you want to manually install Ansible, refer to the following instructions. The minimum supported Ansible version is 2.9.

Debian / Ubuntu
sudo apt install -y ansible python3-jmespath
EL
sudo dnf install -y ansible python3.12-jmespath python3-cryptography  # EL 8
sudo dnf install -y ansible python3-jmespath                           # EL 9
sudo dnf install -y ansible                                            # EL 10
MacOS
brew install ansible
pip3 install jmespath
Change default passwords!

Please note that EL10 EPEL repo doesn’t yet provide a complete Ansible package. Pigsty PGSQL EL10 repo supplements this.

Ansible is also available on macOS. You can use Homebrew to install Ansible on Mac, and use it as an admin node to manage remote cloud servers. This is convenient for single-node Pigsty deployment on cloud VPS, but not recommended in prod envs.

Execute Playbook

Ansible playbooks are executable YAML files containing a series of task definitions to execute. Running playbooks requires the ansible-playbook executable in your environment variable PATH. Running ./node.yml playbook is essentially executing the ansible-playbook node.yml command.

You can use some parameters to fine-tune playbook execution. The following 4 parameters are essential for effective Ansible use:

Purpose Parameter Description
Target -l|--limit <pattern> Limit execution to specific groups/hosts/patterns
Tasks -t|--tags <tags> Only run tasks with specific tags
Params -e|--extra-vars <vars> Extra command-line parameters
Config -i|--inventory <path> Use a specific inventory file 
./node.yml                         # Run node playbook on all hosts
./pgsql.yml -l pg-test             # Run pgsql playbook on pg-test cluster
./infra.yml -t repo_build          # Run infra.yml subtask repo_build
./pgsql-rm.yml -l pg-test -e pg_rm_pkg=false --check # Preflight removal while keeping packages
./infra.yml -i conf/mynginx.yml    # Use another location's config file

Limit Hosts

Playbook execution targets can be limited with -l|--limit <selector>. This is convenient when running playbooks on specific hosts/nodes or groups/clusters. Here are some host limit examples:

./pgsql.yml                              # Run on all hosts (dangerous!)
./pgsql.yml -l pg-test                   # Run on pg-test cluster
./pgsql.yml -l 10.10.10.10               # Run on single host 10.10.10.10
./pgsql.yml -l pg-*                      # Run on hosts/groups matching glob `pg-*`
./pgsql.yml -l '10.10.10.11,&pg-test'    # Run on 10.10.10.11 in pg-test group
./pgsql-rm.yml -l 'pg-test,!10.10.10.11' --check # Preflight removal; verify target and backups before execution

See all details in Ansible documentation: Patterns: targeting hosts and groups

Use caution when running playbooks without host limits!

Missing this value can be dangerous—most playbooks execute on all hosts. Use with caution.

Limit Tasks

Execution tasks can be controlled with -t|--tags <tags>. If specified, only tasks with the given tags will execute instead of the entire playbook.

./infra.yml -t repo          # Create repo
./node.yml  -t node_pkg      # Install node packages
./pgsql.yml -t pg_install    # Install PG packages and extensions
./etcd.yml  -t etcd_config   # Render ETCD configuration again
./minio.yml -t minio_alias   # Write the mcli client alias

To run multiple tasks, specify multiple tags separated by commas -t tag1,tag2:

./node.yml  -t node_repo,node_pkg   # Add repos, then install packages
./pgsql.yml -t pg_hba,pg_reload     # Configure, then reload pg hba rules

Extra Vars

You can override config parameters at runtime using CLI arguments, which have highest priority.

Extra command-line parameters are passed via -e|--extra-vars KEY=VALUE, usable multiple times:

# Create admin using another admin user
./node.yml -e ansible_user=admin -k -K -t node_admin

# Initialize a specific Redis instance: 10.10.10.11:6379
./redis.yml -l 10.10.10.10 -e redis_port=6379 -t redis

# Remove PostgreSQL but keep packages and data
./pgsql-rm.yml -l pg-test -e pg_rm_pkg=false -e pg_rm_data=false --check

For complex parameters, use JSON strings to pass multiple complex parameters at once:

# Add repo and install packages
./node.yml -t node_install -e '{"node_repo_modules":"infra","node_packages":["duckdb"]}'

Specify Inventory

The default config file is pigsty.yml in the Pigsty home directory.

You can use -i <path> to specify a different inventory file path.

./pgsql.yml -i conf/rich.yml            # Initialize single node with all extensions per rich config
./pgsql.yml -i conf/ha/full.yml         # Initialize 4-node cluster per full config
./pgsql.yml -i conf/app/supa.yml        # Initialize 1-node Supabase deployment per supa.yml
Changing the default inventory file

To permanently change the default config file, modify the inventory parameter in ansible.cfg.

Convenience Scripts

Pigsty provides a series of convenience scripts to simplify common operations. These scripts are in the bin/ directory:

bin/node-add   <cls>            # Add nodes to Pigsty management: ./node.yml -l <cls>
bin/node-rm    <cls>            # Remove nodes from Pigsty: ./node-rm.yml -l <cls>
bin/pgsql-add  <cls>            # Initialize PG cluster: ./pgsql.yml -l <cls>
bin/pgsql-rm   <cls>            # Remove PG cluster: ./pgsql-rm.yml -l <cls>
bin/pgsql-user <cls> <username> # Add business user: ./pgsql-user.yml -l <cls> -e username=<user>
bin/pgsql-db   <cls> <dbname>   # Add business database: ./pgsql-db.yml -l <cls> -e dbname=<db>
bin/redis-add  <cls>            # Initialize Redis cluster: ./redis.yml -l <cls>
bin/redis-rm   <cls>            # Remove Redis cluster: ./redis-rm.yml -l <cls>

These scripts are simple wrappers around Ansible playbooks, making common operations more convenient.

Playbook List

Below are the built-in playbooks in Pigsty. You can also easily add your own playbooks, or customize and modify playbook implementation logic as needed.

Module Playbook Function
INFRA deploy.yml One-click deploy Pigsty on current node
INFRA infra.yml Initialize Pigsty infrastructure on infra nodes
INFRA infra-rm.yml Remove infrastructure components from infra nodes
INFRA cache.yml Create offline packages from target node
INFRA cert.yml Issue certificates using Pigsty self-signed CA
NODE node.yml Initialize node, adjust to desired state
NODE node-rm.yml Remove node from Pigsty
PGSQL pgsql.yml Initialize HA PostgreSQL cluster or add replica
PGSQL pgsql-rm.yml Remove PostgreSQL cluster or replica
PGSQL pgsql-db.yml Add new business database to existing cluster
PGSQL pgsql-user.yml Add new business user to existing cluster
PGSQL pgsql-pitr.yml Perform point-in-time recovery on cluster
PGSQL pgsql-monitor.yml Monitor remote PostgreSQL with local exporter
PGSQL pgsql-migration.yml Generate migration manual and scripts
PGSQL slim.yml Install Pigsty with minimal components
REDIS redis.yml Initialize Redis cluster/node/instance
REDIS redis-rm.yml Remove Redis cluster/node/instance
ETCD etcd.yml Initialize ETCD cluster or add new member
ETCD etcd-rm.yml Remove ETCD cluster/data or shrink member
MINIO minio.yml Initialize a Silo object-storage cluster
MINIO minio-rm.yml Remove Silo, its configuration, and optional data
DOCKER docker.yml Install Docker on nodes
DOCKER app.yml Install applications using Docker Compose
JUICE juice.yml Install and configure JuiceFS
VIBE vibe.yml Install the Vibe coding environment
KAFKA kafka.yml Create or converge a Kafka dynamic KRaft cluster
KAFKA kafka-rm.yml Remove a Kafka cluster or member
MYSQL (Pilot) mysql.yml Deploy native MySQL 8.4 standalone or three-node clusters
MYSQL (Pilot) mysql-rm.yml Stop and retire native MySQL while retaining local state

1.7 - Offline Installation

Install Pigsty in air-gapped env using offline packages

Pigsty installs from Internet upstream by default, but some envs are isolated from the Internet. To address this, Pigsty supports offline installation using offline packages. Think of them as Linux-native Docker images.

Overview

Offline packages bundle all required RPM/DEB packages and dependencies; they are snapshots of the local APT/YUM repo after a normal installation.

In serious prod deployments, we strongly recommend using offline packages. They ensure all future nodes have consistent software versions with the existing env, and avoid online installation failures caused by upstream changes (quite common!), guaranteeing you can run it independently forever.

Advantages of offline packages
  • Easy delivery in Internet-isolated envs.
  • Pre-download all packages in one pass to speed up installation.
  • No need to worry about upstream dependency breakage causing install failures.
  • If you have multiple nodes, all packages only need to be downloaded once, saving bandwidth.
  • Use local repo to ensure all nodes have consistent software versions for unified version management.
Disadvantages of offline packages
  • Offline packages are made for specific OS minor versions, typically cannot be used across versions.
  • It’s a snapshot at the time of creation, may not include the latest updates and OS security patches.
  • Offline packages are typically about 1GB, while online installation downloads on-demand, saving space.

Offline Packages

The following table records the historical v4.4.0 offline artifacts and the OS minor versions used to build them; these are not the currently recommended operating systems.

Linux Distribution System Code Minor Version Package
RockyLinux 9 x86_64 el9.x86_64 9.7 pigsty-pkg-v4.4.0.el9.x86_64.tgz
RockyLinux 9 aarch64 el9.aarch64 9.7 pigsty-pkg-v4.4.0.el9.aarch64.tgz
RockyLinux 10 x86_64 el10.x86_64 10.1 pigsty-pkg-v4.4.0.el10.x86_64.tgz
RockyLinux 10 aarch64 el10.aarch64 10.1 pigsty-pkg-v4.4.0.el10.aarch64.tgz
Debian 12 x86_64 d12.x86_64 12.14 pigsty-pkg-v4.4.0.d12.x86_64.tgz
Debian 12 aarch64 d12.aarch64 12.14 pigsty-pkg-v4.4.0.d12.aarch64.tgz
Debian 13 x86_64 d13.x86_64 13.6 pigsty-pkg-v4.4.0.d13.x86_64.tgz
Debian 13 aarch64 d13.aarch64 13.6 pigsty-pkg-v4.4.0.d13.aarch64.tgz
Ubuntu 26.04 x86_64 u26.x86_64 26.04.0 pigsty-pkg-v4.4.0.u26.x86_64.tgz
Ubuntu 26.04 aarch64 u26.aarch64 26.04.0 pigsty-pkg-v4.4.0.u26.aarch64.tgz
Ubuntu 24.04 x86_64 u24.x86_64 24.04.4 pigsty-pkg-v4.4.0.u24.x86_64.tgz
Ubuntu 24.04 aarch64 u24.aarch64 24.04.4 pigsty-pkg-v4.4.0.u24.aarch64.tgz
Ubuntu 22.04 x86_64 u22.x86_64 22.04.5 pigsty-pkg-v4.4.0.u22.x86_64.tgz
Ubuntu 22.04 aarch64 u22.aarch64 22.04.5 pigsty-pkg-v4.4.0.u22.aarch64.tgz

If your OS exactly matches one of these historical artifact baselines, you can use the corresponding v4.4.0 offline package. The v4.4.0 Community Edition publishes six dual-architecture artifacts for Debian 13, EL 10, and Ubuntu 24.04 on GitHub. Artifact names and checksums for Debian 12, EL 9, Ubuntu 22.04, and Ubuntu 26.04 remain listed here; those offline packages are available with the Professional Edition.

Download Community Edition artifacts from the GitHub release page. The MD5 checksums for all v4.4.0 offline packages are:

7de8b932412f1863fd9c033a7be355d7  pigsty-pkg-v4.4.0.d12.aarch64.tgz
2e5006a8d35eb1c087dc0ed11cf14d14  pigsty-pkg-v4.4.0.d12.x86_64.tgz
955308c00d3890f6e82a6a83bc624760  pigsty-pkg-v4.4.0.d13.aarch64.tgz
350f31c66de0aafff3bd91c2c9d740a0  pigsty-pkg-v4.4.0.d13.x86_64.tgz
0b4817a8edbab0bdf37ecee730fb0412  pigsty-pkg-v4.4.0.el10.aarch64.tgz
4584a61e4456749e68d86e4817cfe526  pigsty-pkg-v4.4.0.el10.x86_64.tgz
21621daf510a532829c36464d48f9198  pigsty-pkg-v4.4.0.el9.aarch64.tgz
504afd5030e2738a25e1b4c570d0e654  pigsty-pkg-v4.4.0.el9.x86_64.tgz
461c999424dee587ca33fe1a63df40d7  pigsty-pkg-v4.4.0.u22.aarch64.tgz
20ccc5ab8f9f4648b05bcd304f9fb5fc  pigsty-pkg-v4.4.0.u22.x86_64.tgz
d092c48ee55116ed5e2c99a3d909ccdd  pigsty-pkg-v4.4.0.u24.aarch64.tgz
24fa5399d8421305961fcaf91325b382  pigsty-pkg-v4.4.0.u24.x86_64.tgz
36f69b699d8b3041d35384970e157631  pigsty-pkg-v4.4.0.u26.aarch64.tgz
330047d117b20f04317dce506edd5d9a  pigsty-pkg-v4.4.0.u26.x86_64.tgz
Offline packages are made for specific Linux OS minor versions

When OS minor versions don’t match, it may work or may fail—we don’t recommend taking the risk.

Please note that the historical v4.4.0 artifacts above were built on EL 9.7/10.1, Debian 12.14/13.6, and Ubuntu 22.04.5/24.04.4/26.04.0. Cross-minor installation may fail due to OpenSSL/system library differences. Use online installation on matching OS versions to build your own offline package, or contact us for custom packages.

Using Offline Packages

Offline installation steps:

  1. Download Pigsty offline package, place it at /tmp/pkg.tgz
  2. Download Pigsty source package, extract and enter directory (assume extracted to home: cd ~/pigsty)
  3. ./bootstrap, it will extract the package and configure using local repo (and install ansible from it offline)
  4. ./configure -g -c rich, you can directly use the rich template configured for offline installation, or configure yourself
  5. Run ./deploy.yml as usual to install the core path from the local repository; other optional modules still require their own playbooks
demo/install-offline.cast
Warning

If you encounter “No package nginx available” errors during offline installation, it usually means a previous installation attempt failed. Delete the /www/pigsty directory and re-run the deployment.

If you want to use the already extracted and configured offline package in your own config, modify and ensure these settings:

  • repo_enabled: Set to true, will build local software repo (explicitly disabled in most templates)
  • node_repo_modules: Set to local, then all nodes in the env will install from the local software repo
    • In most templates, this is explicitly set to: node,infra,pgsql, i.e., install directly from these upstream repos.
    • Setting it to local will use the local software repo to install all packages, fastest, no interference from other repos.
    • If you want to use both local and upstream repos, you can add other repo module names too, e.g., local,node,infra,pgsql

The first parameter, if enabled, Pigsty will create a local software repo. The second parameter, if contains local, then all nodes in the env will use this local software repo. If it only contains local, then it becomes the sole repo for all nodes. If you still want to install other packages from other upstream repos, you can add other repo module names too, e.g., local,node,infra,pgsql.

Hybrid Installation Mode

If your environment has Internet access, there’s a hybrid approach that combines the advantages of offline and online installation. You can use the offline package as a base, and supplement missing packages online.

Using the historical v4.4.0 artifacts as an example, suppose you run RockyLinux 9.6 while the package was built for RockyLinux 9.7. You can use the el9 offline package (though made for 9.7), then execute make repo-build before formal installation to re-download missing packages for 9.6. Pigsty will download the required increments from upstream repos.

Making Offline Packages

If your OS isn’t in the default list, you can make your own offline package with the built-in cache.yml playbook:

  1. Find a node running the exact same OS version with Internet access
  2. Use the rich template for an online installation (./configure -c rich), and confirm that the target INFRA node has generated its local repository at /www/pigsty; if not, run ./infra.yml -t repo against that node first
  3. Run cd ~/pigsty; ./cache.yml -l <infra-host> to select one INFRA node that already has a local repository, build the package there, and fetch it
  4. By default, the artifact is ~/pigsty/dist/${version}/pigsty-pkg-${version}.${os}.${arch}.tgz; copy it to the offline environment (ftp, scp, USB, etc.), then unpack it with bootstrap

Current cache.yml defaults can be overridden with extra variables:

cache_pkg_name defaultpigsty-pkg-${version}.${os}.${arch}.tgz
Offline package filename template
cache_pkg_dir defaultdist/${version}
Output directory on the admin node
cache_repo defaultpigsty
Local repository to package on the target node; separate multiple repositories with commas

We offer paid services providing tested, pre-made offline packages for specific Linux major.minor versions (¥200).

Bootstrap

Pigsty relies on ansible to execute playbooks; this script is responsible for ensuring ansible is correctly installed in various ways.

./bootstrap       # Ensure ansible is correctly installed (if offline package exists, use offline installation and extract first)

Usually, you need to run this script in two cases:

  • You didn’t install Pigsty via the installation script, but by downloading or git clone of the source package, so ansible isn’t installed.
  • You’re preparing to install Pigsty via offline packages and need to use this script to install ansible from the offline package.

The bootstrap script will automatically detect if the offline package exists (-p to specify, default is /tmp/pkg.tgz). If it exists, it will extract and use it, then install ansible from it. If the offline package doesn’t exist, it will try to install ansible from the Internet. If that still fails, you’re on your own!

Where are my yum/apt repo files?

The bootloader will by default move away existing repo configurations to ensure only required repos are enabled. You can find them in /etc/yum.repos.d/backup (EL) or /etc/apt/backup (Debian / Ubuntu).

If you want to keep existing repo configurations during bootstrap, use the -k|--keep parameter.

./bootstrap -k # or --keep

1.8 - Slim Installation

Install only HA PostgreSQL clusters with minimal dependencies

If you only want HA PostgreSQL database cluster itself without monitoring, infra, etc., consider Slim Installation.

Slim installation has no INFRA module, no monitoring, no local repo—just ETCD and PGSQL and partial NODE functionality.

Slim installation is suitable for:
  • Only needing PostgreSQL database itself, no observability infra required.
  • Extremely resource-constrained envs unwilling to bear infra overhead (~0.2 vCPU / 500MB on single node).
  • Already having external monitoring system, wanting to use your own unified monitoring framework.
  • Not needing the Grafana visualization dashboard component.
Limitations of slim installation:
  • No INFRA module, cannot use WebUI and local software repo features.
  • Offline Install is limited to single-node mode; multi-node slim install can only be done online.

Overview

To use slim installation, you need to:

  1. Use the slim.yml slim install config template (configure -c slim)
  2. Run the slim.yml playbook instead of the default deploy.yml
curl https://repo.pigsty.io/get | bash
./configure -g -c slim
./slim.yml
demo/install-slim.cast

Description

Slim installation only installs/configures these components:

Component Required Description
patroni ⚠️ Required Bootstrap HA PostgreSQL cluster
etcd ⚠️ Required Meta database dependency (DCS) for Patroni
pgbouncer ✔️ Optional PostgreSQL connection pooler
vip-manager ✔️ Optional L2 VIP binding to PostgreSQL cluster primary
haproxy ✔️ Optional Auto-routing services via Patroni health checks
chronyd ✔️ Optional Time synchronization with NTP server
tuned ✔️ Optional Node tuning template and kernel parameter management

You can disable all optional components via configuration, keeping only the required patroni and etcd.

Because there’s no INFRA module’s Nginx providing local repo service, offline installation only works in single-node mode.

Configuration

Slim installation config file example: conf/slim.yml:

ID NODE PGSQL INFRA ETCD
1 10.10.10.10 pg-meta-1 No INFRA module etcd-1 
---
#==============================================================#
# File      :   slim.yml
# Desc      :   Pigsty slim installation config template
# Ctime     :   2020-05-22
# Mtime     :   2025-12-28
# Docs      :   https://pigsty.io/docs/conf/slim
# License   :   Apache-2.0 @ https://pigsty.io/docs/about/license/
# Copyright :   2018-2026  Ruohang Feng / Vonng ([email protected])
#==============================================================#

# This is the config template for slim / minimal installation
# No monitoring & infra will be installed, just raw postgresql
#
# Usage:
#   curl https://repo.pigsty.io/get | bash
#   ./configure -c slim
#   ./slim.yml

all:
  children:

    etcd: # dcs service for postgres/patroni ha consensus
      hosts: # 1 node for testing, 3 or 5 for production
        10.10.10.10: { etcd_seq: 1 }  # etcd_seq required
        #10.10.10.11: { etcd_seq: 2 }  # assign from 1 ~ n
        #10.10.10.12: { etcd_seq: 3 }  # three-member cluster keeps an odd voter count
      vars: # cluster level parameter override roles/etcd
        etcd_cluster: etcd  # mark etcd cluster name etcd

    #----------------------------------------------#
    # PostgreSQL Cluster
    #----------------------------------------------#
    pg-meta:
      hosts:
        10.10.10.10: { pg_seq: 1, pg_role: primary }
        #10.10.10.11: { pg_seq: 2, pg_role: replica } # you can add more!
        #10.10.10.12: { pg_seq: 3, pg_role: replica, pg_offline_query: true }
      vars:
        pg_cluster: pg-meta
        pg_users:
          - { name: dbuser_meta ,password: DBUser.Meta   ,pgbouncer: true ,roles: [dbrole_admin   ] ,comment: pigsty admin user }
          - { name: dbuser_view ,password: DBUser.Viewer ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer  }
        pg_databases:
          - { name: meta, baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty] ,extensions: [ vector ]}
        pg_hba_rules:   # https://pigsty.io/docs/pgsql/config/hba
          - { user: all ,db: all ,addr: intra ,auth: pwd ,title: 'everyone intranet access with password' ,order: 800 }
        pg_crontab:     # https://pigsty.io/docs/pgsql/admin/crontab
          - '00 01 * * * /pg/bin/pg-backup full'

  vars:
    version: v4.5.0                   # pigsty version string
    admin_ip: 10.10.10.10             # admin node ip address
    region: default                   # upstream mirror region: default,china,europe
    nodename_overwrite: false           # do not overwrite node hostname on single node mode
    node_repo_modules: node,infra,pgsql # add these repos directly to the singleton node
    node_tune: oltp                     # node tuning specs: oltp,olap,tiny,crit
    pg_conf: oltp.yml                   # pgsql tuning specs: {oltp,olap,tiny,crit}.yml
    pg_version: 18                      # Default PostgreSQL Major Version is 18
    pg_packages: [ pgsql-main, pgsql-common ]   # pg kernel and common utils
    #pg_extensions: [ pg18-time ,pg18-gis ,pg18-rag ,pg18-fts ,pg18-olap ,pg18-feat ,pg18-lang ,pg18-type ,pg18-util ,pg18-func ,pg18-admin ,pg18-stat ,pg18-sec ,pg18-fdw ,pg18-sim ,pg18-etl]

    #----------------------------------------------#
    # PASSWORD : https://pigsty.io/docs/setup/security/
    #----------------------------------------------#
    grafana_admin_password: pigsty
    grafana_view_password: DBUser.Viewer
    pg_admin_password: DBUser.DBA
    pg_monitor_password: DBUser.Monitor
    pg_replication_password: DBUser.Replicator
    patroni_password: Patroni.API
    haproxy_admin_password: pigsty
    minio_secret_key: S3User.MinIO
    etcd_root_password: Etcd.Root
...

Deployment

Slim installation uses the slim.yml playbook instead of deploy.yml:

./slim.yml

HA Cluster

Slim installation can also deploy HA clusters—just add more nodes to the etcd and pg-meta groups. A three-node deployment example:

ID NODE PGSQL INFRA ETCD
1 10.10.10.10 pg-meta-1 No INFRA module etcd-1
2 10.10.10.11 pg-meta-2 No INFRA module etcd-2
3 10.10.10.12 pg-meta-3 No INFRA module etcd-3 
all:
  children:
    etcd:
      hosts:
        10.10.10.10: { etcd_seq: 1 }
        10.10.10.11: { etcd_seq: 2 }  # <-- New
        10.10.10.12: { etcd_seq: 3 }  # <-- New

    pg-meta:
      hosts:
        10.10.10.10: { pg_seq: 1, pg_role: primary }
        10.10.10.11: { pg_seq: 2, pg_role: replica } # <-- New
        10.10.10.12: { pg_seq: 3, pg_role: replica } # <-- New
      vars:
        pg_cluster: pg-meta
        pg_users:
          - { name: dbuser_meta ,password: DBUser.Meta   ,pgbouncer: true ,roles: [dbrole_admin   ] ,comment: pigsty admin user }
          - { name: dbuser_view ,password: DBUser.Viewer ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer  }
        pg_databases:
          - { name: meta, baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty] ,extensions: [ vector ]}
        pg_crontab: [ '00 01 * * * /pg/bin/pg-backup full' ] # full backup daily at 1am
  vars:
    # omitted ……

1.9 - Security Recommendations

Basic security checks for quick-start and single-node deployments.

The default configuration targets local demonstrations and development or testing on a trusted intranet. If other hosts can reach the deployment, complete at least three checks: credentials, network boundaries, and critical files.

Production environments should also review the Security Model, Compliance, and Security Considerations.

Passwords

Pigsty default credentials are public in the source code and documentation and must not be used directly in production.

The configuration wizard can randomize built-in parameters and example credentials that it recognizes:

./configure -g

configure -g does not replace:

  • the pgBackRest cipher_pass;
  • Silo users and selected example passwords in ha/safe;
  • database, object-storage, or application credentials added by the user.

After generation, inspect pigsty.yml and replace every uncovered credential. The wizard prints generated passwords to the terminal, so protect terminal history and automation logs as sensitive data.

See the Default Credentials Checklist for the complete scope.

Firewall

node_firewall_mode defaults to zone. It trusts the intranet defined by node_firewall_intranet and restricts ports exposed to public networks.

Port Service Public by Default
22 SSH Yes
80 Nginx HTTP Yes
443 Nginx HTTPS Yes
5432 PostgreSQL Not in the base default; exposed additionally by the demo pigsty.yml

Production deployments should normally remove 5432 from the demo configuration. If applications need direct database access, restrict source addresses in the cloud security group, host firewall, and HBA.

Also verify that the intranet definition matches the actual trust boundary. The default RFC 1918 ranges may be too broad; office networks, container networks, and other tenant networks should not become trusted automatically.

Files

The following files and directories contain highly sensitive information:

  • pigsty.yml: system and application credentials, node definitions, and service configuration;
  • files/pki/ca/ca.key: local CA private key;
  • the administration user’s SSH private key, used to access managed nodes;
  • files/pki/misc/*.key: client-certificate private keys;
  • /pg/tmp/pg-user-*.sql: SQL containing plaintext passwords generated during user creation.

Restrict access to the admin node and configuration repository. Do not commit complete inventories or private keys to public repositories. Maintain controlled backups of the CA private key and required configuration.

Related Documentation

2 - Deployment

Multi-node, high-availability Pigsty deployment for production environments.

Unlike Getting Started, production Pigsty deployments require more Architecture Planning and Preparation.

This chapter helps you understand the complete deployment process and provides best practices for production environments.

Before deploying to production, we recommend testing in Pigsty’s Sandbox to fully understand the workflow. Use Vagrant to create a local 4-node sandbox, or leverage Terraform to provision larger simulation environments in the cloud.

pigsty-sandbox

For production, you typically need at least three nodes for high availability. You should understand Pigsty’s core Concepts and common administration procedures, including Configuration, Ansible Playbooks, and Security Hardening for enterprise compliance.

2.1 - Install Pigsty for Production

How to install Pigsty on Linux hosts for production?

This is the Pigsty production multi-node deployment guide. For single-node Demo/Dev setups, see Getting Started.

Summary

Prepare nodes with SSH access following your architecture plan, install a compatible Linux OS, then execute with an admin user having passwordless ssh and sudo:

curl -fsSL https://repo.pigsty.io/get | bash;         # International
curl -fsSL https://repo.pigsty.cc/get | bash;         # Backup Mirror

This runs the install script, downloading and extracting Pigsty source to your home directory with dependencies installed. Complete configuration and deployment to finish.

Before running deploy.yml for deployment, review and edit the configuration inventory: pigsty.yml.

cd ~/pigsty      # Enter Pigsty directory
./configure -g   # Generate config file (optional, skip if you know how to configure)
./deploy.yml     # Execute deployment playbook based on generated config

After installation, access the WebUI via IP/domain + ports 80/443, and PostgreSQL service via port 5432.

Full installation takes 3-10 minutes depending on specs/network. Offline installation significantly speeds this up; slim installation further accelerates when monitoring isn’t needed.

Video Example: 20-node Production Simulation (Ubuntu 24.04 x86_64)

demo/install-simu.cast

Prepare

Production Pigsty deployment involves preparation work. Here’s the complete checklist:

Item Requirement Item Requirement
Node At least 1C2G, no upper limit Plan Multiple homogeneous nodes: 2/3/4 or more
Disk /data as default mount point FS xfs recommended; ext4/zfs as needed
VIP L2 VIP, optional (unavailable in cloud) Network Static IPv4, single-node can use 127.0.0.1
CA Self-signed CA or specify existing certs Domain Local/public domain, optional, default i.pigsty
Kernel Linux x86_64 / aarch64 Linux el8, el9, el10, d12, d13, u22, u24, u26
Locale C.UTF-8 or C Firewall Ports: 80/443/22/5432 (optional)
User Avoid root and postgres Sudo sudo privilege, preferably with nopass
SSH Passwordless SSH via public key Accessible ssh <ip|alias> sudo ls no error

Install

Use the following to automatically install the Pigsty source package to ~/pigsty (recommended). Deployment dependencies (Ansible) are auto-installed.

pigsty.io (Global)
curl -fsSL https://repo.pigsty.io/get | bash            # Install current default version
curl -fsSL https://repo.pigsty.io/get | bash -s v4.5.0  # Explicitly install the current public stable release
pigsty.cc (China)
curl -fsSL https://repo.pigsty.cc/get | bash            # Install current default version
curl -fsSL https://repo.pigsty.cc/get | bash -s v4.5.0  # Explicitly install the current public stable release

If you prefer not to run remote scripts, manually download or clone the source. When using git, always checkout a specific version before use:

git clone https://github.com/pgsty/pigsty; cd pigsty;
git checkout v4.5.0;  # Always checkout a released tag when using git

For manual download/clone, additionally run bootstrap to manually install Ansible and other dependencies, or install them yourself:

./bootstrap           # Install ansible for subsequent deployment

Configure

In Pigsty, deployment details are defined by the configuration inventory—the pigsty.yml config file. Customize through declarative configuration.

Pigsty provides configure as an optional configuration wizard, generating a configuration inventory with good defaults based on your environment:

./configure -g                # Use wizard to generate config with random passwords

The generated config defaults to ~/pigsty/pigsty.yml. Review and customize before installation.

Many configuration templates are available for reference. You can skip the wizard and directly edit pigsty.yml:

./configure -c ha/full -g       # Use 4-node sandbox template
./configure -c ha/trio -g       # Use 3-node minimal HA template
./configure -c ha/dual -g -v 18 # Use 2-node semi-HA template with PG 18
./configure -c ha/simu -s       # Use 20-node production simulation, skip IP check, no random passwords
Example configure output
vagrant@meta:~/pigsty$ ./configure
configure pigsty v4.5.0 begin
[ OK ] region = china
[ OK ] kernel  = Linux
[ OK ] machine = x86_64
[ OK ] package = deb,apt
[ OK ] vendor  = ubuntu (Ubuntu)
[ OK ] version = 22 (22.04)
[ OK ] sudo = vagrant ok
[ OK ] ssh = [email protected] ok
[WARN] Multiple IP address candidates found:
    (1) 192.168.121.38	    inet 192.168.121.38/24 metric 100 brd 192.168.121.255 scope global dynamic eth0
    (2) 10.10.10.10	    inet 10.10.10.10/24 brd 10.10.10.255 scope global eth1
[ OK ] primary_ip = 10.10.10.10 (from demo)
[ OK ] admin = [email protected] ok
[ OK ] mode = meta (ubuntu22.04)
[ OK ] locale  = C.UTF-8
[ OK ] ansible = ready
[ OK ] pigsty configured
[WARN] don't forget to check it and change passwords!
proceed with ./deploy.yml

The wizard only replaces the current node’s IP (use -s to skip replacement). For multi-node deployments, replace other node IPs manually. Also customize the config as needed—modify default passwords, add nodes, etc.

Common configure parameters:

Parameter Description
-c|--conf Specify config template relative to conf/, without .yml suffix
-v|--version PostgreSQL major version 14 through 19; PG19 is currently Beta
-r|--region Upstream repo region for faster downloads: default|china|europe
-n|--non-interactive Use CLI params for primary IP, skip interactive wizard
-x|--proxy Configure proxy_env from current environment variables

If your machine has multiple IPs, explicitly specify one with -i|--ip <ipaddr> or provide it interactively. The script replaces IP placeholder 10.10.10.10 with the current node’s primary IPv4. Use a static IP; never use public IPs.

Generated config is at ~/pigsty/pigsty.yml. Review and modify before installation.

Change default passwords!

Change default passwords and credentials before installation. See Security Recommendations.

Deploy

Pigsty’s deploy.yml playbook applies the configuration blueprint to all target nodes.

./deploy.yml     # Deploy core modules on all target nodes at once
Example deployment output
......

TASK [pgsql : pgsql init done] *************************************************
ok: [10.10.10.11] => {
    "msg": "postgres://10.10.10.11/postgres | meta  | dbuser_meta dbuser_view "
}
......

TASK [pg_monitor : load grafana datasource meta] *******************************
changed: [10.10.10.11]

PLAY RECAP *********************************************************************
10.10.10.11                : ok=302  changed=232  unreachable=0    failed=0    skipped=65   rescued=0    ignored=1
localhost                  : ok=6    changed=3    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0

When output ends with pgsql init done, PLAY RECAP, etc., installation is complete!

Upstream repo changes may cause online installation failures!

Upstream repos (Linux/PGDG) may break due to improper updates, causing deployment failures (quite common)! For serious production deployments, we strongly recommend using verified offline packages for offline installation.

Avoid running deploy playbook repeatedly!

Warning: Running deploy.yml again on an initialized environment may restart services and overwrite configs. Be careful!

Interface

Assuming the 4-node deployment template, your Pigsty environment should have a structure like:

ID NODE PGSQL INFRA ETCD
1 10.10.10.10 pg-meta-1 infra-1 etcd-1
2 10.10.10.11 pg-test-1 - -
3 10.10.10.12 pg-test-2 - -
4 10.10.10.13 pg-test-3 - -

The INFRA module provides a graphical management interface via browser, accessible through Nginx’s 80/443 ports.

The PGSQL module provides a PostgreSQL database server on port 5432, also accessible via Pgbouncer/HAProxy proxies.

For production multi-node HA PostgreSQL clusters, use service access for automatic traffic routing.

Pigsty online demo homepage

More

After installation, explore the WebUI and access PostgreSQL service via port 5432.

Deploy and monitor more clusters—add definitions to the configuration inventory and run:

bin/node-add   pg-test      # Add pg-test cluster's 3 nodes to Pigsty management
bin/pgsql-add  pg-test      # Initialize a 3-node pg-test HA PG cluster
bin/redis-add  redis-ms     # Initialize Redis cluster: redis-ms

Most modules require the NODE module first. See available modules:

PGSQL, INFRA, NODE, ETCD, MINIO, REDIS, DOCKER…

2.2 - Prepare Resources for Serious Deployment

Production deployment preparation including hardware, nodes, disks, network, VIP, domain, software, and filesystem requirements.

Pigsty runs on nodes (physical machines or VMs). This document covers the planning and preparation required for deployment.

Node

Pigsty currently runs on Linux kernel with x86_64 / aarch64 architecture. A “node” refers to an SSH accessible resource that provides a bare Linux OS environment. It can be a physical machine, virtual machine, or a systemd-enabled container equipped with systemd, sudo, and sshd.

Deploying Pigsty requires at least 1 node. You can prepare more and deploy everything in one pass via playbooks, or add nodes later. The minimum spec requirement is 1C1G, but at least 1C2G is recommended. Higher is better—no upper limit. Parameters are auto-tuned based on available resources.

The number of nodes you need depends on your requirements. See Architecture Planning for details. Although a single-node deployment with external backup provides reasonable recovery guarantees, we recommend multiple nodes for production. A functioning HA setup requires at least 3 nodes; 2 nodes provide Semi-HA.

Disk

Pigsty uses /data as the default data directory. If you have a dedicated data disk, mount it there. Use /data1, /data2, /dataN for additional disk drives.

To use a different data directory, configure these parameters:

Name Description Default
node_data Node main data directory /data
pg_fs_main PG main data directory /data/postgres
pg_fs_backup PG backup directory /data/backups
etcd_data ETCD data directory /data/etcd
infra_data Infra data directory /data/infra
nginx_data Nginx data directory /data/nginx
minio_data Silo data directory /data/minio
redis_fs_main Redis data directory /data/redis
kafka_data Kafka data directory /data/kafka

The native MySQL 8.4 pilot module does not currently expose a data-directory parameter and always uses /var/lib/mysql.

Filesystem

You can use any supported Linux filesystem for data disks. For production, we recommend xfs.

xfs is a Linux standard with excellent performance and CoW capabilities for instant large database cluster cloning. Multi-drive Silo deployments require xfs. ext4 is another viable option with a richer data recovery tool ecosystem, but lacks CoW. zfs provides RAID and snapshot features but with significant performance overhead and requires separate installation.

Choose among these three based on your needs. Avoid NFS for database services.

Pigsty assumes /data is owned by root:root with 755 permissions. Admins can assign ownership for first-level directories; each application runs with a dedicated user in its subdirectory. See FHS for the directory structure reference.

Network

Pigsty defaults to online installation mode, requiring outbound Internet access. Offline installation eliminates the Internet requirement.

Internally, Pigsty requires a static network. Assign a fixed IPv4 address to each node.

The IP address serves as the node’s unique identifier—the primary IP bound to the main network interface for internal communications.

For single-node deployment without a fixed IP, use the loopback address 127.0.0.1 as a workaround.

Never use Public IP as identifier

Using public IP addresses as node identifiers can cause security and connectivity issues. Always use internal IP addresses.

VIP

Pigsty supports optional L2 VIP for NODE clusters (keepalived) and PGSQL clusters (vip-manager).

To use L2 VIP, you must explicitly assign an L2 VIP address for each node/database cluster. This is straightforward on your own hardware but may be challenging in public cloud environments.

L2 VIP requires L2 Networking

To use optional Node VIP and PG VIP features, ensure all nodes are on the same L2 network.

CA

Pigsty generates a self-signed CA infrastructure for each deployment, issuing all encryption certificates.

If you have an existing enterprise CA or self-signed CA, you can use it to issue the certificates Pigsty requires.

Domain

Pigsty uses a local static domain i.pigsty by default for WebUI access. This is optional—IP addresses work too.

For production, domain names are recommended to enable HTTPS and encrypted data transmission. Domains also allow multiple services on the same port, differentiated by domain name.

For Internet-facing deployments, use public DNS providers (Cloudflare, AWS Route53, etc.) to manage resolution. Point your domain to the Pigsty node’s public IP address. For LAN/office network deployments, use internal DNS servers with the node’s internal IP address.

For local-only access, add the following to /etc/hosts on machines accessing the Pigsty WebUI:

10.10.10.10 i.pigsty    # Replace with your domain and Pigsty node IP

Linux

Pigsty runs on Linux. It currently targets 16 platform combinations: eight distribution major versions across two architectures. See the Compatible OS List.

We recommend Rocky Linux 9.8 / 10.2, Debian 12.15 / 13.6, or Ubuntu 22.04.5 / 24.04.4 / 26.04.0 as default options.

On macOS and Windows, use VM software or Docker systemd images to run Pigsty.

We strongly recommend a fresh OS installation. If your server already runs Nginx, PostgreSQL, or similar services, consider deploying on new nodes.

Use the same OS version on all nodes

For multi-node deployments, ensure all nodes use the same Linux distribution, architecture, and version. Heterogeneous deployments may work but are unsupported and may cause unpredictable issues.

Locale

We recommend setting en_US as the primary OS language, or at minimum ensuring this locale is available, so PostgreSQL logs are in English.

Some distributions (e.g., Debian) may not provide the en_US locale by default. Enable it with:

localedef -i en_US -f UTF-8 en_US.UTF-8
localectl set-locale LANG=en_US.UTF-8

For PostgreSQL, we strongly recommend using the built-in C.UTF-8 collation (PG 17+) as the default.

The configuration wizard automatically sets C.UTF-8 as the collation when PG version and OS support are detected.

Ansible

Pigsty uses Ansible to control all managed nodes from the admin node. See Installing Ansible for details.

Pigsty installs Ansible on Infra nodes by default, making them usable as admin nodes (or backup admin nodes). For single-node deployment, the installation node serves as both the admin node running Ansible and the INFRA node hosting infrastructure.

Pigsty

You can install the current default Pigsty source with:

pigsty.io (Global)
curl -fsSL https://repo.pigsty.io/get | bash;
pigsty.cc (China)
curl -fsSL https://repo.pigsty.cc/get | bash;

To install a specific version, use the -s <version> parameter:

pigsty.io (Global)
curl -fsSL https://repo.pigsty.io/get | bash -s <version>  # Install a specific version (current stable: v4.5.0)
pigsty.cc (China)
curl -fsSL https://repo.pigsty.cc/get | bash -s <version>  # Install a specific version (current stable: v4.5.0)

To install the latest beta version:

pigsty.io (Global)
curl -fsSL https://repo.pigsty.io/beta | bash;
pigsty.cc (China)
curl -fsSL https://repo.pigsty.cc/beta | bash;

For developers or the latest development version, clone the repository directly:

git clone https://github.com/pgsty/pigsty.git;
cd pigsty; git checkout <tag>  # Use a released version (current stable tag: v4.5.0)

If your environment lacks Internet access, download the source tarball from GitHub Releases or the Pigsty repository:

wget https://repo.pigsty.io/src/pigsty-v<version>.tgz
wget https://repo.pigsty.cc/src/pigsty-v<version>.tgz

2.3 - Planning Architecture and Nodes

How many nodes? Which modules need HA? How to plan based on available resources and requirements?

Pigsty uses a modular architecture. You can combine modules like building blocks and express your intent through declarative configuration.

Common Patterns

Here are common deployment patterns for reference. Customize based on your requirements:

Pattern INFRA ETCD PGSQL MINIO Description
Single-node (meta) 1 1 1 Single-node deployment default
Slim deploy (slim) 1 1 Database only, no monitoring infra
Infra-only (infra) 1 Monitoring infrastructure only
Rich deploy (rich) 1 1 1 1 Single-node + object storage + local repo with all extensions
Multi-node Pattern INFRA ETCD PGSQL MINIO Description
Two-node (dual) 1 1 2 Semi-HA, tolerates specific node failure
Three-node (trio) 3 3 3 Standard HA, tolerates any one failure
Four-node (full) 1 1 1+3 Demo setup, single INFRA/ETCD
Production (simu) 2 3 n n 2 INFRA, 3 ETCD
Large-scale (custom) 3 5 n n 3 INFRA, 5 ETCD

Your architecture choice depends on reliability requirements and available resources. Serious production deployments require at least 3 nodes for HA configuration. With only 2 nodes, use Semi-HA configuration.

Expert Consulting: Architecture Planning

We offer Architecture Consulting Services to help plan your Pigsty configuration.

Trade-offs

  • Pigsty monitoring requires at least 1 INFRA node. Production typically uses 2; large-scale deployments use 3.
  • PostgreSQL HA requires at least 1 ETCD node. Production typically uses 3; large-scale uses 5. Even-member clusters work, but do not tolerate more failures than an odd cluster with one fewer member, so prefer odd sizes.
  • Silo object storage through the MINIO module requires at least 1 MINIO node. Production typically uses 4+ nodes in MNMD clusters.
  • Production PG clusters typically use at least two-node primary-replica configuration; serious deployments use 3 nodes; high read loads can have dozens of replicas.
  • For PostgreSQL, you can also use advanced configurations: offline instances, sync instances, standby clusters, delayed clusters, etc.

Single-Node Setup

The simplest configuration with everything on a single node. Installs four essential modules by default. Typically used for demos, devbox, or testing.

ID NODE PGSQL INFRA ETCD
1 node-1 pg-meta-1 infra-1 etcd-1

With an external S3/MinIO backup repository providing RTO/RPO guarantees, this configuration works for standard production environments.

Single-node variants:

Two-Node Setup

Two-node configuration enables database replication and Semi-HA capability with better data redundancy and limited failover support:

ID NODE PGSQL INFRA ETCD
1 node-1 pg-meta-1 (replica) infra-1 etcd-1
2 node-2 pg-meta-2 (primary)

Two-node HA auto-failover has limitations. This “Semi-HA” setup only auto-recovers from specific node failures:

  • If node-1 fails: No automatic failover—requires manual promotion of node-2
  • If node-2 fails: Automatic failover works—node-1 auto-promoted

Three-Node Setup

Three-node template provides true baseline HA configuration, tolerating any single node failure with automatic recovery.

ID NODE PGSQL INFRA ETCD
1 node-1 pg-meta-1 infra-1 etcd-1
2 node-2 pg-meta-2 infra-2 etcd-2
3 node-3 pg-meta-3 infra-3 etcd-3

Four-Node Setup

Pigsty Sandbox uses the standard four-node configuration.

ID NODE PGSQL INFRA ETCD
1 node-1 pg-meta-1 infra-1 etcd-1
2 node-2 pg-test-1
3 node-3 pg-test-2
4 node-4 pg-test-3

For demo purposes, INFRA / ETCD modules aren’t configured for HA. You can adjust further:

ID NODE PGSQL INFRA ETCD MINIO
1 node-1 pg-meta-1 infra-1 etcd-1 minio-1
2 node-2 pg-test-1 infra-2 etcd-2
3 node-3 pg-test-2 etcd-3
4 node-4 pg-test-3

More Nodes

With proper virtualization infrastructure or abundant resources, you can use more nodes for dedicated deployment of each module, achieving optimal reliability, observability, and performance.

ID NODE INFRA ETCD MINIO PGSQL
1 10.10.10.10 infra-1 pg-meta-1
2 10.10.10.11 infra-2 pg-meta-2
3 10.10.10.21 etcd-1
4 10.10.10.22 etcd-2
5 10.10.10.23 etcd-3
6 10.10.10.31 minio-1
7 10.10.10.32 minio-2
8 10.10.10.33 minio-3
9 10.10.10.34 minio-4
10 10.10.10.40 pg-src-1
11 10.10.10.41 pg-src-2
12 10.10.10.42 pg-src-3
13 10.10.10.50 pg-test-1
14 10.10.10.51 pg-test-2
15 10.10.10.52 pg-test-3
16 ……

2.4 - Setup Admin User and Privileges

Admin user, sudo, SSH, accessibility verification, and firewall configuration

Pigsty requires an OS admin user with passwordless SSH and Sudo privileges on all managed nodes.

This user must be able to SSH to all managed nodes and execute sudo commands on them.

User

Typically use names like dba or admin, avoiding root and postgres:

  • Using root for deployment is possible but not a production best practice.
  • Using postgres (pg_dbsu) as admin user is strictly prohibited.

Passwordless

The passwordless requirement is optional if you can accept entering a password for every ssh and sudo command.

Use -k|--ask-pass when running playbooks to prompt for SSH password, and -K|--ask-become-pass to prompt for sudo password.

./deploy.yml -k -K

Some enterprise security policies may prohibit passwordless ssh or sudo. In such cases, use the options above, or consider configuring a sudoers rule with a longer password cache time to reduce password prompts.

Create Admin User

Typically, your server/VM provider creates an initial admin user.

If unsatisfied with that user, Pigsty’s deployment playbook can create a new admin user for you.

Assuming you have root access or an existing admin user on the node, create an admin user with Pigsty itself:

./node.yml -k -K -t node_admin \
  -e ansible_user=[current_login_admin] \
  -e node_admin_username=[new_admin_to_create]

This leverages the existing admin to create a new one—a dedicated dba (uid=88) user described by these parameters, with sudo/ssh properly configured:

Name Description Default
node_admin_enabled Enable node admin user true
node_admin_uid Node admin user UID 88
node_admin_username Node admin username dba

Sudo

All admin users should have sudo privileges on all managed nodes, preferably with passwordless execution.

To configure an admin user with passwordless sudo from scratch, edit/create a sudoers file (assuming username vagrant):

echo '%vagrant ALL=(ALL) NOPASSWD: ALL' | sudo tee /etc/sudoers.d/vagrant

For admin user dba, the /etc/sudoers.d/dba content should be:

%dba ALL=(ALL) NOPASSWD: ALL

If your security policy prohibits passwordless sudo, remove the NOPASSWD: part:

%dba ALL=(ALL) ALL

Ansible relies on sudo to execute commands with root privileges on managed nodes. In environments where sudo is unavailable (e.g., inside Docker containers), install sudo first.

SSH

Your current user should have passwordless SSH access to all managed nodes as the corresponding admin user.

Your current user can be the admin user itself, but this isn’t required—as long as you can SSH as the admin user.

SSH configuration is Linux 101, but here are the basics:

Generate SSH Key

If you don’t have an SSH key pair, generate one:

ssh-keygen -t rsa -b 2048 -N '' -f ~/.ssh/id_rsa -q

Pigsty will do this for you during the bootstrap stage if you lack a key pair.

Copy SSH Key

Distribute your generated public key to remote (and local) servers, placing it in the admin user’s ~/.ssh/authorized_keys file on all nodes. Use the ssh-copy-id utility:

ssh-copy-id <ip>                        # Interactive password entry
sshpass -p <password> ssh-copy-id <ip>  # Non-interactive (use with caution)

Using Alias

When direct SSH access is unavailable (jumpserver, non-standard port, different credentials), configure SSH aliases in ~/.ssh/config:

Host meta
    HostName 10.10.10.10
    User dba                      # Different user on remote
    IdentityFile /etc/dba/id_rsa  # Non-standard key
    Port 24                       # Non-standard port

Reference the alias in the inventory using ansible_host for the real SSH alias:

nodes:
  hosts:          # If node `10.10.10.10` requires SSH alias `meta`
    10.10.10.10: { ansible_host: meta }  # Access via `ssh meta`

SSH parameters work directly in Ansible. See Ansible Inventory Guide for details. This technique enables accessing nodes in private networks via jumpservers, or using different ports and credentials, or using your local laptop as an admin node.

Check Accessibility

You should be able to passwordlessly ssh from the admin node to all managed nodes as your current user. The remote user (admin user) should have privileges to run passwordless sudo commands.

To verify passwordless ssh/sudo works, run this command on the admin node for all managed nodes:

ssh <ip|alias> 'sudo ls'

If there’s no password prompt or error, passwordless ssh/sudo is working as expected.

Firewall

Production deployments typically require firewall configuration to block unauthorized port access.

By default, block inbound access from office/Internet networks except:

  • SSH port 22 for node access
  • HTTP (80) / HTTPS (443) for WebUI services
  • PostgreSQL port 5432 for database access

If accessing PostgreSQL via other ports, allow them accordingly. See used ports for the complete port list.

  • 5432: PostgreSQL database
  • 6432: Pgbouncer connection pooler
  • 5433: PG primary service
  • 5434: PG replica service
  • 5436: PG default service
  • 5438: PG offline service

2.5 - Sandbox

4-node sandbox environment for learning, testing, and demonstration

Pigsty provides a standard 4-node sandbox environment for learning, testing, and feature demonstration.

The sandbox uses fixed IP addresses and predefined identity identifiers, making it easy to reproduce various demo use cases.

Description

The default sandbox environment consists of 4 nodes, using the ha/full.yml configuration template.

ID IP Address Node PostgreSQL INFRA ETCD MINIO
1 10.10.10.10 meta pg-meta-1 infra-1 etcd-1 minio-1
2 10.10.10.11 node-1 pg-test-1
3 10.10.10.12 node-2 pg-test-2
4 10.10.10.13 node-3 pg-test-3

The sandbox configuration can be summarized as the following config:

all:
  children:
    infra: { hosts: { 10.10.10.10: { infra_seq: 1 } } }
    etcd:  { hosts: { 10.10.10.10: { etcd_seq:  1 } }, vars: { etcd_cluster: etcd } }
    minio: { hosts: { 10.10.10.10: { minio_seq: 1 } }, vars: { minio_cluster: minio } }

    pg-meta:
      hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
      vars:  { pg_cluster: pg-meta }

    pg-test:
      hosts:
        10.10.10.11: { pg_seq: 1, pg_role: primary }
        10.10.10.12: { pg_seq: 2, pg_role: replica }
        10.10.10.13: { pg_seq: 3, pg_role: replica }
      vars: { pg_cluster: pg-test }

  vars:
    version: v4.5.0
    admin_ip: 10.10.10.10
    region: default
    pg_version: 18
pigsty-sandbox

PostgreSQL Clusters

The sandbox comes with a single-instance PostgreSQL cluster pg-meta on the meta node:

10.10.10.10 meta pg-meta-1
10.10.10.2  pg-meta          # Optional L2 VIP

There’s also a 3-instance PostgreSQL HA cluster pg-test deployed on the other three nodes:

10.10.10.11 node-1 pg-test-1
10.10.10.12 node-2 pg-test-2
10.10.10.13 node-3 pg-test-3
10.10.10.3  pg-test          # Optional L2 VIP

Two optional L2 VIPs are bound to the primary instances of pg-meta and pg-test clusters respectively.

Infrastructure

The meta node also hosts:

  • ETCD cluster: Single-node etcd cluster providing DCS service for PostgreSQL HA
  • Silo cluster: A single-node minio cluster managed by the MINIO module, providing S3-compatible object storage
10.10.10.10 etcd-1
10.10.10.10 minio-1

ha/full.yml also declares three Redis example topologies and enables Docker installation on the INFRA node. The standard deploy.yml does not deploy these two optional modules; run ./redis.yml and ./docker.yml separately when needed.

Creating Sandbox

Pigsty provides out-of-the-box templates. You can use Vagrant to create a local sandbox, or use Terraform to create a cloud sandbox.

Local Sandbox (Vagrant)

Local sandbox uses VirtualBox/libvirt to create local virtual machines, running free on your Mac / PC.

To run the full 4-node sandbox, your machine should have at least 4 CPU cores and 8GB memory.

cd ~/pigsty/vagrant
make full       # Create 4-node sandbox with default Ubuntu 24.04 image
make full9      # Create 4-node sandbox with RockyLinux 9
make full12     # Create 4-node sandbox with Debian 12
make full24     # Create 4-node sandbox with Ubuntu 24.04
make full26     # Create 4-node sandbox with Ubuntu 26.04

The current Vagrant configuration uses the cloud-image/* boxes from Vagrant Cloud. See Vagrant: Supported Images for available images, source-pinned versions, and architecture details. Boxes without a version pinned in source are resolved by Vagrant to their currently available version.

Cloud Sandbox (Terraform)

Cloud sandbox uses public cloud API to create virtual machines. Easy to create and destroy, pay-as-you-go, ideal for quick testing.

Use the spec/aliyun-full.tf template to create a 4-node sandbox on Alibaba Cloud:

cd ~/pigsty/terraform
cp spec/aliyun-full.tf terraform.tf
terraform init
terraform apply

For more details, please refer to Terraform documentation.

Other Specs

Besides the standard 4-node sandbox, Pigsty also provides other environment specs:

Run the following Makefile shortcuts from ~/pigsty/vagrant:

cd ~/pigsty/vagrant

Single Node Devbox (meta)

The simplest 1-node environment for quick start, development, and testing:

make meta       # Create single-node devbox

Two Node Environment (dual)

2-node environment for testing primary-replica replication:

make dual       # Create 2-node environment

Three Node Environment (trio)

3-node environment for testing basic high availability:

make trio       # Create 3-node environment

Production Simulation (simu)

20-node large simulation environment for full production environment testing:

make simu       # Create 20-node production simulation environment

This environment includes:

  • 3 infrastructure nodes (meta1, meta2, meta3)
  • 2 HAProxy proxy nodes
  • 4 MINIO (Silo) nodes
  • 5 ETCD nodes
  • 6 PostgreSQL nodes (2 clusters, 3 nodes each)

2.6 - Vagrant

Create local virtual machine environment with Vagrant

Vagrant is a popular local virtualization tool that creates local virtual machines in a declarative manner.

Pigsty requires a Linux environment to run. You can use Vagrant to easily create Linux virtual machines locally for testing.

The currently recommended and validated baselines are Rocky Linux 9.8 / 10.2, Debian 12.15 / 13.6, and Ubuntu 22.04.5 / 24.04.4 / 26.04.0. Major-version Vagrant aliases map to pinned box versions.

Quick Start

Install Dependencies

First, ensure you have Vagrant and a virtual machine provider (such as VirtualBox or libvirt) installed on your system.

On macOS, you can use Homebrew for one-click installation:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
brew install vagrant virtualbox ansible
VirtualBox requires reboot after installation

After installing VirtualBox, you need to restart your system and allow its kernel extensions in System Preferences.

On Linux, you can use VirtualBox or vagrant-libvirt as the VM provider.

Create Virtual Machines

Use the Pigsty-provided make shortcuts to create virtual machines:

cd ~/pigsty/vagrant

make meta       # 1 node devbox for quick start, development, and testing
make full       # 4 node sandbox for HA testing and feature demonstration
make simu       # 20 node simubox for production environment simulation

# Other less common specs
make dual       # 2 node environment
make trio       # 3 node environment
make deci       # 10 node environment

You can use variant aliases to specify different operating system images:

make meta9      # Create single node with Rocky Linux 9.8
make full12     # Create 4-node sandbox with Debian 12.15
make simu24     # Create 20-node simubox with Ubuntu 24.04.4
make full26     # Create 4-node sandbox with Ubuntu 26.04.0

Available OS suffixes: 8 (EL8), 9 (EL9), 10 (EL10), 12 (Debian 12.15), 13 (Debian 13.6), 22 (Ubuntu 22.04.5), 24 (Ubuntu 24.04.4), 26 (Ubuntu 26.04.0)

Build Environment

You can also use the following aliases to create Pigsty build environments. These templates won’t replace the base image:

make oss        # 7 node OSS build environment
make pro        # 7 node PRO build environment
make rpm        # 2 node EL9/10 build environment
make deb        # 5 node Debian12/13 Ubuntu22/24/26 build environment
make all        # 7 node full build environment

Spec Templates

Pigsty provides multiple predefined VM specs in the vagrant/spec/ directory:

Template Nodes Spec Description Alias
meta.rb 1 node 2c4g x 1 Single-node devbox Devbox
dual.rb 2 nodes 1c2g x 2 Two-node environment
trio.rb 3 nodes 1c2g x 3 Three-node environment
full.rb 4 nodes 2c4g + 1c2g x 3 4-node full sandbox Sandbox
deci.rb 10 nodes Mixed 10-node environment
simu.rb 20 nodes Mixed 20-node production simubox Simubox
minio.rb 4 nodes 1c2g x 4 + disk MinIO test environment
citus.rb 13 nodes Mixed Citus coordinator and six two-replica worker groups
oss.rb 7 nodes 2c2g x 7 7-platform OSS build environment
pro.rb 7 nodes 2c2g x 7 7-platform PRO build environment
rpm.rb 2 nodes 1c2g x 2 2-node EL build environment
deb.rb 5 nodes 1c2g x 5 5-node Deb build environment
all.rb 7 nodes 1c2g x 7 7-node full build environment

Each spec file contains a Specs variable describing the VM nodes. For example, full.rb contains the 4-node sandbox definition:

Current Vagrant templates explicitly provision a 32 GB primary system disk for every VM. Regular nodes also receive one data disk whose size comes from the spec’s disk value, defaulting to 128 GB when omitted. Object-storage nodes whose names begin with minio instead receive four 32 GB data disks mounted at /data1 through /data4. These disks depend on Vagrant’s experimental disks feature. The repository Makefile exports VAGRANT_EXPERIMENTAL=disks automatically; set it yourself when invoking vagrant directly.

# full: pigsty full-featured 4-node sandbox for HA-testing & tutorial & practices

Specs = [
  { "name" => "meta"   , "ip" => "10.10.10.10" ,  "cpu" => "2" ,  "mem" => "4096" ,  "image" => "cloud-image/ubuntu-24.04" },
  { "name" => "node-1" , "ip" => "10.10.10.11" ,  "cpu" => "1" ,  "mem" => "2048" ,  "image" => "cloud-image/ubuntu-24.04" },
  { "name" => "node-2" , "ip" => "10.10.10.12" ,  "cpu" => "1" ,  "mem" => "2048" ,  "image" => "cloud-image/ubuntu-24.04" },
  { "name" => "node-3" , "ip" => "10.10.10.13" ,  "cpu" => "1" ,  "mem" => "2048" ,  "image" => "cloud-image/ubuntu-24.04" },
]

simu Spec Details

simu.rb provides a 20-node production environment simulation configuration:

  • 3 x infra nodes (meta1-3): 4c16g
  • 2 x haproxy nodes (proxy1-2): 1c2g
  • 4 x minio nodes (minio1-4): 1c2g
  • 5 x etcd nodes (etcd1-5): 1c2g
  • 6 x pgsql nodes (pg-src-1-3, pg-dst-1-3): 2c4g

Config Script

Use the vagrant/config script to generate the final Vagrantfile based on spec and options:

cd ~/pigsty/vagrant
vagrant/config [spec] [image] [scale] [provider]

# Examples
vagrant/config meta u24            # Use 1-node spec with Ubuntu 24.04.4 image
vagrant/config dual el9            # Use 2-node spec with RockyLinux 9.7 image
vagrant/config trio d12 2          # Use 3-node spec with Debian 12.14, double resources
vagrant/config full u22 4          # Use 4-node spec with Ubuntu 22.04.5, 4x resources
vagrant/config simu u26 1 libvirt  # Use 20-node spec with Ubuntu 26.04.0, libvirt provider

Image Aliases

The config script supports various image aliases:

Distro Alias Vagrant Box
Rocky 8 el8, rocky8, r8 cloud-image/rocky-8
Rocky 9 el9, rocky9, el, r9 cloud-image/rocky-9
Rocky 10 el10, rocky10, r10 cloud-image/rocky-10
Debian 12 d12, debian12, deb12 cloud-image/debian-12
Debian 13 d13, debian13, deb13 cloud-image/debian-13
Ubuntu 22.04.5 u22, ubuntu22, ubuntu2204 cloud-image/ubuntu-22.04
Ubuntu 24.04.4 u24, ubuntu24, ubuntu2404, ubuntu cloud-image/ubuntu-24.04
Ubuntu 26.04.0 u26, ubuntu26, ubuntu2604 cloud-image/ubuntu-26.04
AlmaLinux 8 alma8 cloud-image/almalinux-8
AlmaLinux 9 alma9 cloud-image/almalinux-9
AlmaLinux 10 alma10 cloud-image/almalinux-10
RHEL 8 / 9 rhel8, rhel9 generic/rhel8, generic/rhel9
Oracle Linux 8 / 9 oracle8, oracle9 generic/oracle8, generic/oracle9

The historical d11/debian11/deb11 and u20/ubuntu20/ubuntu2004 aliases remain visible in the script mapping, but the current script explicitly rejects them; they are not supported images.

Resource Scaling

You can use the VM_SCALE environment variable to adjust the resource multiplier (default is 1):

VM_SCALE=2 vagrant/config meta     # Double the CPU/memory resources for meta spec

For example, using VM_SCALE=4 with the meta spec will adjust the default 2c4g to 8c16g:

Specs = [
  { "name" => "meta" , "ip" => "10.10.10.10", "cpu" => "8" , "mem" => "16384" , "image" => "cloud-image/ubuntu-24.04" },
]
simu and deci specs don’t support scaling

The simu and deci specs don’t support resource scaling. The scale parameter is automatically reset to 1 because their resource configurations are already optimized for simulation scenarios.

VM Management

The vagrant/Makefile provides shortcuts for managing virtual machines. Run the following commands from that directory:

cd ~/pigsty/vagrant
make           # Equivalent to make start
make new       # Destroy existing VMs and create new ones
make ssh       # Write VM SSH config to ~/.ssh/ (must run after creation)
make dns       # Write VM DNS records to /etc/hosts (optional)
make start     # Start VMs and configure SSH (up + ssh)
make up        # Start VMs with vagrant up
make halt      # Shutdown VMs (alias: down, dw)
make clean     # Destroy VMs (alias: del, destroy)
make status    # Show VM status (alias: st)
make pause     # Pause VMs (alias: suspend)
make resume    # Resume VMs
make nuke      # Destroy all VMs and volumes with virsh (libvirt only)
make info      # Show libvirt info (VMs, networks, storage volumes)

SSH Keys

Pigsty Vagrant templates use your ~/.ssh/id_rsa[.pub] as the SSH key for VMs by default.

Before starting, ensure you have a valid SSH key pair. If not, generate one with:

ssh-keygen -t rsa -b 2048 -N '' -f ~/.ssh/id_rsa -q

Supported Images

The standard EL, Debian, Ubuntu, and AlmaLinux matrix uses cloud-image/* boxes from Vagrant Cloud. Explicit RHEL and Oracle Linux aliases use generic/* boxes. The current config script applies the same cloud-image/* mapping to VirtualBox, libvirt, amd64, and arm64; actual payload availability is still resolved by Vagrant Cloud at runtime.

VirtualBox and libvirt use the same mapping. vagrant/config writes the validated versions below for every supported cloud-image/* image, making amd64 and arm64 environments reproducible:

OS Vagrant Box Source Version Policy
Rocky 8 cloud-image/rocky-8 8.10.20240528.0
Rocky 9 cloud-image/rocky-9 9.8.20260525.0
Rocky 10 cloud-image/rocky-10 10.2.20260525.0
Debian 12 cloud-image/debian-12 20260806.2562.0
Debian 13 cloud-image/debian-13 20260810.2566.0
Ubuntu 22.04 cloud-image/ubuntu-22.04 20260810.0.0
Ubuntu 24.04 cloud-image/ubuntu-24.04 20260801.0.0
Ubuntu 26.04 cloud-image/ubuntu-26.04 20260731.0.0
AlmaLinux 8 cloud-image/almalinux-8 8.10.20260803
AlmaLinux 9 cloud-image/almalinux-9 9.8.20260810
AlmaLinux 10 cloud-image/almalinux-10 10.2.20260526.0

The retained but unsupported Debian 11 and Ubuntu 20.04 aliases are pinned to 20260618.2513.0 and 20250624.0.0; experimental generic/* RHEL, Oracle Linux, and CentOS 7 images are pinned to their final 4.3.12 release. These legacy images are outside the current support matrix.

Environment Variables

You can use the following environment variables to control Vagrant behavior:

export VM_SPEC='meta'              # Spec name
export VM_IMAGE='cloud-image/rocky-9' # Image name
export VM_SCALE='1'                # Resource scaling multiplier
export VM_PROVIDER='virtualbox'    # Virtualization provider
export VAGRANT_EXPERIMENTAL=disks  # Enable disks for direct vagrant use; Makefile sets this automatically

Notes

VirtualBox Network Configuration

When using older versions of VirtualBox as Vagrant provider, additional configuration is required to use 10.x.x.x CIDR as Host-Only network:

echo "* 10.0.0.0/8" | sudo tee -a /etc/vbox/networks.conf
First-time image download is slow

The first time you use Vagrant to start a specific operating system, it will download the corresponding Box image file (typically 1-2 GB). After download, the image is cached and reused for subsequent VM creation.

libvirt Provider

If you’re using libvirt as the provider, you can use make info to view VMs, networks, and storage volume information, and make nuke to forcefully destroy all related resources.

2.7 - Terraform

Create virtual machine environment on public cloud with Terraform

Terraform is a popular “Infrastructure as Code” tool that you can use to create virtual machines on public clouds with one click.

Pigsty currently provides example Terraform templates for Alibaba Cloud, AWS (global and China), Azure, GCP, Tencent Cloud, Hetzner, Vultr, DigitalOcean, and Linode. The aliyun-s3.tf template also creates a private OSS bucket and dedicated RAM read/write credentials for S3/pgBackRest scenarios.

Quick Start

Install Terraform

On macOS, you can use Homebrew to install Terraform:

brew install terraform

For other platforms, refer to the Terraform Official Installation Guide.

Initialize and Apply

Enter the Terraform directory, select a template, initialize provider plugins, and apply the configuration:

cd ~/pigsty/terraform
cp spec/aliyun.tf terraform.tf         # Select template
terraform init                         # Install cloud provider plugins (first use)
terraform apply                        # Generate execution plan and create resources

After running the apply command, type yes to confirm when prompted. Terraform will create VMs and related cloud resources for you.

Get IP Address

After creation, print the public IP address of the admin node:

terraform output -raw meta_ip

Configure SSH Access

Global-cloud templates usually also provide an executable ssh_command output:

terraform output -raw ssh_command

The repository’s ./ssh script is a compatibility tool for legacy templates whose outputs are all IP addresses and whose root password is PigstyDemo4. It iterates over every Terraform output, treats it as an IP address, writes it to ~/.ssh/pigsty_config, and distributes keys with sshpass. It is suitable for compatibility templates such as aliyun.tf, aliyun-full.tf, aliyun-oss.tf, and aliyun-pro.tf. Do not run it against modern templates that output ssh_command, private IPs, or access keys.

When using a compatible template:

./ssh       # Write SSH config and distribute keys
ssh meta    # Login using hostname instead of IP
Using SSH Config File

If you want to use the configuration in ~/.ssh/pigsty_config, ensure your ~/.ssh/config includes:

Include ~/.ssh/pigsty_config

Destroy Resources

After testing, you can destroy all created cloud resources with one click:

terraform destroy

Template Specs

Pigsty provides multiple predefined cloud resource templates in the terraform/spec/ directory:

Template File Cloud Provider Description
aliyun.tf Alibaba Cloud Single-node meta template, supports all distributions and AMD/ARM (default)
aliyun-s3.tf Alibaba Cloud Single node + private OSS bucket and RAM read/write credentials for S3/pgBackRest
aliyun-full.tf Alibaba Cloud Four-node sandbox, supports all distributions and AMD/ARM
aliyun-oss.tf Alibaba Cloud Six-node build template, supports all distributions and AMD/ARM
aliyun-pro.tf Alibaba Cloud Seven-node multi-distribution test template
aws.tf AWS Global AWS single node, Debian 12/13, AMD/ARM
aws-cn.tf AWS Legacy single-node environment for AWS China
azure.tf Azure Single node, Debian 12/13, AMD/ARM
gcp.tf GCP Single node, Debian 12/13, AMD/ARM
qcloud.tf Tencent Cloud Tencent Cloud single-node environment
hetzner.tf Hetzner Single node, Debian 12/13, AMD/ARM
vultr.tf Vultr Single node, Debian 12/13, currently AMD only
digitalocean.tf DigitalOcean Single node, Debian 12/13, currently AMD only
linode.tf Linode Single node, Debian 12/13, currently AMD only

When using a template, copy the template file to terraform.tf:

cd ~/pigsty/terraform
cp spec/aliyun-full.tf terraform.tf   # Use Alibaba Cloud 4-node sandbox template
terraform init && terraform apply

Variable Configuration

Variables differ between templates. Alibaba Cloud templates support the full multi-distribution matrix and default to u26. Global AWS, Azure, GCP, Tencent Cloud, and Hetzner support Debian 12/13 with AMD/ARM selection and generally default to d12/amd64. Vultr, DigitalOcean, and Linode currently expose AMD instance choices only.

Architecture and Distribution 

variable "architecture" {
  description = "Architecture type (amd64 or arm64)"
  type        = string
  default     = "amd64"    # Comment this line to use arm64
  #default     = "arm64"   # Uncomment to use arm64
}

variable "distro" {
  description = "Distribution code (the exact set depends on the template)"
  type        = string
  default     = "d12"       # Global-cloud templates usually default to Debian 12; Alibaba Cloud defaults to u26
}

Resource Configuration

Alibaba Cloud templates expose the following resource parameters in a locals block. Other cloud templates use provider-specific instance, disk, and network variables or local values; consult the selected .tf file.

locals {
  bandwidth        = 100                    # Public bandwidth (Mbps)
  disk_size        = 40                     # System disk size (GB)
  spot_policy      = "SpotWithPriceLimit"   # Spot policy: NoSpot, SpotWithPriceLimit, SpotAsPriceGo
  spot_price_limit = 5                      # Max spot price (only effective with SpotWithPriceLimit)
}

Alibaba Cloud Configuration

Credential Setup

Add your Alibaba Cloud credentials to environment variables, for example in ~/.bash_profile or ~/.zshrc:

export ALICLOUD_ACCESS_KEY="<your_access_key>"
export ALICLOUD_SECRET_KEY="<your_secret_key>"
export ALICLOUD_REGION="cn-shanghai"

Supported Images

The following are commonly used ECS Public OS Image prefixes in Alibaba Cloud:

The currently recommended and validated baselines are Rocky Linux 9.8 / 10.2, Debian 12.15 / 13.6, and Ubuntu 22.04.5 / 24.04.4 / 26.04.0.

Distro Code x86_64 Image Prefix aarch64 Image Prefix
CentOS 7.9 el7 centos_7_9_x64 -
Rocky 8.10 el8 rockylinux_8_10_x64 rockylinux_8_10_arm64
Rocky 9.8 el9 rockylinux_9_8_x64 rockylinux_9_8_arm64
Rocky 10.2 el10 rockylinux_10_2_x64 rockylinux_10_2_arm64
Debian 11.11 d11 debian_11_11_x64 -
Debian 12.15 d12 debian_12_15_x64 debian_12_15_arm64
Debian 13.6 d13 debian_13_6_x64 debian_13_6_arm64
Ubuntu 22.04.5 LTS u22 ubuntu_22_04_x64_20G ubuntu_22_04_arm64_20G
Ubuntu 24.04.4 LTS u24 ubuntu_24_04_x64_20G ubuntu_24_04_arm64_20G
Ubuntu 26.04.0 LTS u26 ubuntu_26_04_x64_20G ubuntu_26_04_arm64_20G
Anolis 8.10 an8 anolisos_8_10_x64 anolisos_8_10_arm64
Alibaba Cloud Linux 3 al3 aliyun_3_x64_20G_alibase_[0-9]+ aliyun_3_arm64_20G_alibase_[0-9]+

OSS Storage Configuration

The aliyun-s3.tf template additionally creates an OSS bucket and related permissions for PostgreSQL PITR backup:

  • OSS Bucket: Creates a private bucket named pigsty-oss
  • RAM User: Creates a dedicated pigsty-oss-user user
  • Access Key: Generates AccessKey and saves to ~/pigsty.sk
  • RAM Policy: Grants the user oss:* permissions on the bucket and its objects for read/write use

AWS Configuration

Credential Setup

Both global and China-region templates can read standard AWS environment variables or credential files:

export AWS_ACCESS_KEY_ID="<your_access_key>"
export AWS_SECRET_ACCESS_KEY="<your_secret_key>"
export AWS_REGION="us-west-2"

# ~/.aws/config
[default]
region = us-west-2

# ~/.aws/credentials
[default]
aws_access_key_id = <YOUR_AWS_ACCESS_KEY>
aws_secret_access_key = <AWS_ACCESS_SECRET>

aws.tf reads ~/.ssh/id_rsa.pub by default. The legacy China-region aws-cn.tf instead reads this dedicated public key:

~/.aws/pigsty-key.pub
AWS templates may need adjustments

aws.tf uses a rolling lookup for official Debian AMIs. aws-cn.tf uses a hard-coded China-region AMI and ~/.aws/pigsty-key.pub; verify the target region, AMI, and key before deployment.

Tencent Cloud Configuration

Credential Setup

Add Tencent Cloud credentials to environment variables:

export TENCENTCLOUD_SECRET_ID="<your_secret_id>"
export TENCENTCLOUD_SECRET_KEY="<your_secret_key>"
export TENCENTCLOUD_REGION="ap-beijing"
Tencent Cloud templates may need adjustments

Tencent Cloud templates are community-contributed examples and may need adjustments based on your specific requirements.

Other Cloud Credentials 

# Azure: az login is recommended; for a service principal, use all four
export ARM_CLIENT_ID="<client_id>"
export ARM_CLIENT_SECRET="<client_secret>"
export ARM_SUBSCRIPTION_ID="<subscription_id>"
export ARM_TENANT_ID="<tenant_id>"

# GCP: gcloud auth application-default login is also supported
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account-key.json"

# Hetzner / Vultr / DigitalOcean / Linode
export HCLOUD_TOKEN="<api_token>"
export VULTR_API_KEY="<api_key>"
export DIGITALOCEAN_TOKEN="<api_token>"
export LINODE_TOKEN="<api_token>"

The GCP template also requires a project variable, for example terraform apply -var="project=my-project". Except for AWS China, current key-based templates read ~/.ssh/id_rsa.pub by default; edit the selected template to use another public-key path.

Shortcut Commands

Pigsty provides some Makefile shortcuts for Terraform operations:

cd ~/pigsty/terraform

make u          # terraform apply -auto-approve + run legacy ./ssh (compatible templates only)
make d          # terraform destroy -auto-approve
make apply      # terraform apply (interactive confirmation)
make destroy    # terraform destroy (interactive confirmation)
make out        # terraform output
make ssh        # Run ssh script to configure SSH access
make r          # Reset terraform.tf to repository state

For modern templates with ssh_command, private-IP, or other non-IP outputs, run terraform apply directly; do not use make u, which invokes the legacy ./ssh script afterward.

Notes

Cloud Resource Costs

Cloud resources created with Terraform incur costs. After testing, promptly use terraform destroy to destroy resources to avoid unnecessary expenses.

It’s recommended to use pay-as-you-go instance types for testing. Templates default to using Spot Instances to reduce costs.

Default Password

Alibaba Cloud and Tencent Cloud templates set the default root password to PigstyDemo4; Linode uses PigstyDemo4! to satisfy its password-complexity rules. Current AWS, Azure, GCP, Hetzner, Vultr, and DigitalOcean templates primarily use SSH public-key authentication and do not share a default root password. Example passwords are for temporary tests only; change them or disable password login in production.

Security Group Configuration

These templates target demonstration and development. Their current security groups or cloud firewalls allow all or nearly all inbound traffic from 0.0.0.0/0 (some also include ::/0), not just the ports Pigsty requires. Restrict source networks and ports before deployment; do not use these defaults unchanged in production.

SSH Access

After creation, SSH login to the admin node using:

ssh root@<public_ip>

Alibaba Cloud templates that retain the legacy output and password conventions can also use ./ssh or make ssh to write SSH aliases. For other templates, use their ssh_command output.

2.8 - Security Considerations

Credential, network, authentication, encryption, data protection, and audit checks for production Pigsty deployments.

Pigsty defaults target development, testing, and demonstrations on a trusted intranet. A production deployment must configure credentials, network boundaries, authentication, certificates, backup, and audit according to its threat model.

See Security and Compliance for mechanisms and boundaries, and the Launch Hardening Checklist for executable checks. ha/safe is a hardening example, not a substitute for reviewing each control.

Confidentiality

Critical Files

Protect these assets:

  • pigsty.yml and other inventories, which normally contain system and application credentials;
  • files/pki/ca/ca.key, which can issue certificates trusted by the deployment;
  • the administration user’s SSH private key, which can use sudo on managed nodes by default;
  • client-certificate private keys and backup-encryption keys;
  • generated /pg/tmp/pg-user-*.sql files.

Restrict access to the admin node and configuration repository. Do not commit complete inventories or private keys to public repositories. Back up the CA private key and recovery configuration through controlled channels.

Passwords

Replace every public default credential before production. Start with:

./configure -g

This option does not replace the pgBackRest cipher_pass, every Silo example credential in ha/safe, or user-defined values. Review the result against the Default Credentials Checklist.

PostgreSQL stores newly set or updated passwords with SCRAM-SHA-256 by default. To enforce complexity, preload passwordcheck through pg_libs, or configure credcheck. Declare account lifetime with expire_in or expire_at.

Credential rotation must also update database users, the PgBouncer user list, component configuration, and client connection information. Prepare a rollback plan before rotating.

Network Boundaries

IP Addresses

PostgreSQL listens on 0.0.0.0 by default. To constrain listen addresses, set:

pg_listen: '${ip},${vip},${lo}'

A listen address is not the only boundary. Production reviews should also cover:

The demo pigsty.yml inventory also exposes 5432 publicly. Remove that exception in production. If direct database access is required, limit it to explicit application CIDRs.

Network Traffic

  • PostgreSQL enables server-side TLS by default, but default intranet HBA rules do not require it.
  • PgBouncer TLS is disabled by default and controlled by pgbouncer_sslmode.
  • HTTPS for the Patroni REST API is disabled by default and controlled by patroni_ssl_enabled.
  • Nginx and the object-storage backend selected by the MINIO module enable HTTPS by default; etcd uses TLS for client and peer traffic.

HBA auth: ssl requires an encrypted connection only. Clients should also use sslmode=verify-full with a trusted CA to verify the database server; see Encrypted Communication.

Grafana, VictoriaMetrics, and other components may listen on node ports, but the default firewall does not expose them directly to public networks. Prefer Nginx for external access, and restrict management pages by source address and identity.

Authentication and Access Control

  • Use HBA to define the user, database, source address, and authentication method. Avoid broad world rules.
  • Use auth: cert for privileged remote users, with a process for delivering and revoking client certificates.
  • Assign application privileges through built-in roles; do not grant superuser to ordinary application accounts.
  • Set revokeconn: true for multi-tenant shared clusters, and inspect effective database ACLs.
  • Create objects through the declared database owner or a controlled administration role so default privileges apply.
  • To isolate offline queries, set role: offline explicitly on the HBA rule for dbrole_offline.

After changing HBA, users, or roles, compare both the inventory and the effective database state.

Integrity

Pigsty enables page checksums by default to detect page damage after write. Checksums do not detect every memory error, logical error, or incorrect application write.

The CRIT template enables Patroni strict synchronous mode and more detailed connection logging. The synchronous mode targets preservation of acknowledged transactions, but depends on synchronous_commit, synchronous-replica state, and failover conditions. Writes block when no synchronous replica is available.

CRIT configures watchdog as automatic; it activates only when the system has a usable watchdog device. Decide whether required is appropriate according to hardware and availability requirements.

Availability

  • Critical clusters should normally have at least three instances across independent failure domains.
  • Connect through HAProxy, a VIP, or DNS service name instead of binding clients to a fixed primary address.
  • Use an odd number of etcd nodes across independent failure domains.
  • Remove single points of failure in INFRA, DNS, monitoring, and software repositories according to availability requirements.
  • When using pg_rpo and pg_rto, understand their configuration semantics and validate objectives through exercises.

Replicas handle only some node failures; they do not replace backups.

Backup and Recovery

  • The local pgBackRest repository is not encrypted by default and shares a failure domain with the database host.
  • The pgbackrest_method: minio object-storage repository uses AES-256-CBC by default, but cipher_pass: pgBackRest is public and must be replaced.
  • pgBR.${pg_cluster} in ha/safe is also an example and must not be used as the final key.
  • Store important backups in an independent failure domain, and evaluate object locking, versioning, or offline copies.
  • Exercise full restore and PITR regularly to validate WAL, keys, recovery time, and application consistency.

See Data Security and Backup and Recovery for details.

Audit and Response

The default OLTP template logs DDL, slow queries, and PostgreSQL 18 connection-authorization events. CRIT also logs connection and disconnection events.

pgaudit must be installed, preloaded, and configured with an audit policy. Installing the package alone does not produce SQL audit logs. When Vector and VictoriaLogs are enabled, adjust log retention, access, and archive policy to requirements.

Metrics, logs, and alerts are incident inputs only. Production also needs alert classification, on-call ownership, incident determination, response, evidence collection, and post-incident review.

Host and Software Supply Chain

  • Move SELinux from the default permissive to enforcing after compatibility validation.
  • Disable unnecessary SSH password authentication and remote root login; consider a bastion host or multi-factor authentication.
  • Review sudo scope for the administration and database OS users.
  • Keep supported Pigsty and upstream component versions current.
  • Verify the software-repository GPG key fingerprint and enable per-package signature verification where required.

See Compliance: Supply Chain and Vulnerability Response.

3 - Concepts

Understand Pigsty’s core concepts, architecture design, learn how high availability, backup recovery, iac, security works

Pigsty is a portable, extensible open-source PostgreSQL distribution for building production-grade database services in local environments with declarative configuration and automation. It has a vast ecosystem providing a complete set of tools, scripts, and best practices to bring PostgreSQL to enterprise-grade RDS service levels.

Pigsty’s name comes from PostgreSQL In Great STYle, also understood as Postgres, Infras, Graphics, Service, Toolbox, it’s all Yours—a self-hosted PostgreSQL solution with graphical monitoring that’s all yours. You can find the source code on GitHub, visit the official documentation for more information, or experience the Web UI in the online demo.

pigsty-banner

Why Pigsty? What Can It Do?

PostgreSQL is a sufficiently perfect database kernel, but it needs more tools and systems to become a truly excellent database service. In production environments, you need to manage every aspect of your database: high availability, backup recovery, monitoring alerts, access control, parameter tuning, extension installation, connection pooling, load balancing…

Wouldn’t it be easier if all this complex operational work could be automated? This is precisely why Pigsty was created.

Pigsty provides:

  • Out-of-the-Box PostgreSQL Distribution

    Pigsty deeply integrates 575 extensions from the PostgreSQL ecosystem, providing out-of-the-box distributed, time-series, geographic, spatial, graph, vector, search, and other multi-modal database capabilities. From kernel to RDS distribution, providing production-grade database services for versions 14-18 on EL/Debian/Ubuntu.

  • Self-Healing High Availability Architecture

    A high availability architecture built on Patroni, Etcd, and HAProxy enables automatic failover for hardware failures with seamless traffic handoff. Primary failure recovery time RTO < 45s, data recovery point RPO ≈ 0. You can perform rolling maintenance and upgrades on the entire cluster without application coordination.

  • Complete Point-in-Time Recovery Capability

    Based on pgBackRest and an optional Silo object-storage cluster, providing out-of-the-box PITR point-in-time recovery capability. Giving you the ability to quickly return to any point in time, protecting against software defects and accidental data deletion.

  • Flexible Service Access and Traffic Management

    Through HAProxy, Pgbouncer, and VIP, providing flexible service access patterns for read-write separation, connection pooling, and automatic routing. Delivering stable, reliable, auto-routing, transaction-pooled high-performance database services.

  • Stunning Observability

    An observability stack based on VictoriaMetrics and Grafana provides unparalleled monitoring best practices. Over three thousand types of monitoring metrics describe every aspect of the system, from global dashboards to CRUD operations on individual objects.

  • Declarative Configuration Management

    Following the Infrastructure as Code philosophy, using declarative configuration to describe the entire environment. You just tell Pigsty “what kind of database cluster you want” without worrying about how to implement it—the system automatically adjusts to the desired state.

  • Modular Architecture Design

    A modular architecture design that can be freely combined to suit different scenarios. Beyond the core PostgreSQL module, it also provides optional modules for Redis, MINIO (Silo), Etcd, and support for various PG-compatible kernels and modes.

  • Solid Security Best Practices

    Industry-leading security practices: a self-signed CA for encrypted communication, AES-encrypted backups, SCRAM-SHA-256 password hashing, an out-of-the-box ACL model, and least-privilege HBA rules.

  • Simple and Easy Deployment

    All dependencies are pre-packaged for one-click installation in environments without internet access. Local sandbox environments can run on micro VMs with 1 core and 2GB RAM, providing functionality identical to production environments. Provides Vagrant-based local sandboxes and Terraform-based cloud deployments.

What Pigsty Is Not

Pigsty is not a traditional, all-encompassing PaaS (Platform as a Service) system.

  • Pigsty doesn’t provide basic hardware resources. It runs on nodes you provide, whether bare metal, VMs, or cloud instances, but it doesn’t create or manage these resources itself (though it provides Terraform templates to simplify cloud resource preparation).

  • Pigsty is not a container orchestration system. It runs directly on the operating system, not requiring Kubernetes or Docker as infrastructure. Of course, it can coexist with these systems and provides a Docker module for running stateless applications.

  • Pigsty is not a general database management tool. It focuses on PostgreSQL and its ecosystem. While it also supports peripheral components like Redis, Etcd, and Silo, the core is always built around PostgreSQL.

  • Pigsty won’t lock you in. It’s built on open-source components, doesn’t modify the PostgreSQL kernel, and introduces no proprietary protocols. You can continue using your well-managed PostgreSQL clusters anytime without Pigsty.

Pigsty doesn’t restrict how you should or shouldn’t build your database services. For example:

  • Pigsty provides good parameter defaults and configuration templates, but you can override any parameter.
  • Pigsty provides a declarative API, but you can still use underlying tools (Ansible, Patroni, pgBackRest, etc.) for manual management.
  • Pigsty can manage the complete lifecycle, or you can use only its monitoring system to observe existing database instances or RDS.

Pigsty provides a different level of abstraction than the hardware layer—it works at the database service layer, focusing on how to deliver PostgreSQL at its best, rather than reinventing the wheel.

Evolution of PostgreSQL Deployment

To understand Pigsty’s value, let’s review the evolution of PostgreSQL deployment approaches.

Manual Deployment Era

In traditional deployment, DBAs needed to manually install and configure PostgreSQL, manually set up replication, manually configure monitoring, and manually handle failures. The problems with this approach are obvious:

  • Low efficiency: Each instance requires repeating many manual operations, prone to errors.
  • Lack of standardization: Databases configured by different DBAs can vary greatly, making maintenance difficult.
  • Poor reliability: Failure handling depends on manual intervention, with long recovery times and susceptibility to human error.
  • Weak observability: Lack of unified monitoring, making problem discovery and diagnosis difficult.

Managed Database Era

To solve these problems, cloud providers offer managed database services (RDS). Cloud RDS does solve some operational issues, but also brings new challenges:

  • High cost: Managed services typically charge multiples to dozens of times hardware cost as “service fees.”
  • Vendor lock-in: Migration is difficult, tied to specific cloud platforms.
  • Limited functionality: Cannot use certain advanced features, extensions are restricted, parameter tuning is limited.
  • Data sovereignty: Data stored in the cloud, reducing autonomy and control.

Local RDS Era

Pigsty represents a third approach: building database services in local environments that match or exceed cloud RDS.

Pigsty combines the advantages of both approaches:

  • High automation: One-click deployment, automatic configuration, self-healing failures—as convenient as cloud RDS.
  • Complete autonomy: Runs on your own infrastructure, data completely in your own hands.
  • Extremely low cost: Run enterprise-grade database services at near-pure-hardware costs.
  • Complete functionality: Unlimited use of PostgreSQL’s full capabilities and ecosystem extensions.
  • Open architecture: Based on open-source components, no vendor lock-in, free to migrate anytime.

This approach is particularly suitable for:

  • Private and hybrid clouds: Enterprises needing to run databases in local environments.
  • Cost-sensitive users: Organizations looking to reduce database TCO.
  • High-security scenarios: Critical data requiring complete autonomy and control.
  • PostgreSQL power users: Scenarios requiring advanced features and rich extensions.
  • Development and testing: Quickly setting up databases locally that match production environments.

What’s Next

Now that you understand Pigsty’s basic concepts, you can:

3.1 - Architecture

Pigsty’s modular architecture—declarative composition, on-demand customization, flexible deployment.

Pigsty uses a modular architecture with a declarative interface. You can freely combine modules like building blocks as needed.

Modules

Pigsty uses a modular design with six main default modules: PGSQL, INFRA, NODE, ETCD, REDIS, and MINIO.

  • PGSQL: Self-healing HA Postgres clusters powered by Patroni, Pgbouncer, HAproxy, PgBackrest, and more.
  • INFRA: Local software repo, Nginx, Grafana, Victoria, AlertManager, Blackbox Exporter—the complete observability stack.
  • NODE: Tune nodes to desired state—hostname, timezone, NTP, ssh, sudo, haproxy, docker, vector, keepalived.
  • ETCD: Distributed key-value store as DCS for HA Postgres clusters: consensus leader election/config management/service discovery.
  • REDIS: Redis servers supporting standalone primary-replica, sentinel, and cluster modes with full monitoring.
  • MINIO: S3-compatible simple object storage that can serve as an optional backup destination for PG databases.

You can declaratively compose them freely. If you only want host monitoring, installing the INFRA module on infrastructure nodes and the NODE module on managed nodes is sufficient. The ETCD and PGSQL modules are used to build HA PG clusters—installing these modules on multiple nodes automatically forms a high-availability database cluster. You can reuse Pigsty infrastructure and develop your own modules; REDIS and MINIO can serve as examples. Protocol compatibility layers such as PostgreSQL Mongo mode are composed from standard PGSQL and Docker APP workflows.

Note that all modules depend strongly on the NODE module: in Pigsty, nodes must first have the NODE module installed to be managed before deploying other modules. When nodes (by default) use the local software repo for installation, the NODE module has a weak dependency on the INFRA module. Therefore, the admin/infrastructure nodes with the INFRA module complete the bootstrap process in the deploy.yml playbook, resolving the circular dependency.

pigsty-sandbox

Standalone Installation

By default, Pigsty installs on a single node (physical/virtual machine). The deploy.yml playbook installs INFRA, ETCD, PGSQL, and optionally MINIO modules on the current node, giving you a fully-featured observability stack (VictoriaMetrics, VictoriaLogs, VictoriaTraces, Grafana, Alertmanager, Blackbox Exporter, etc.), plus a built-in PostgreSQL standalone instance as a CMDB, ready to use out of the box (cluster name pg-meta, database name meta).

This node now has a complete self-monitoring system, visualization tools, and a Postgres database with PITR auto-configured (HA unavailable since you only have one node). You can use this node as a devbox, for testing, running demos, and data visualization/analysis. Or, use this node as an admin node to deploy and manage more nodes!

pigsty-arch

Monitoring

The installed standalone meta node can serve as an admin node and monitoring center to bring more nodes and database servers under its supervision and control.

Pigsty’s monitoring system can be used independently. If you want to install the VictoriaMetrics/Grafana observability stack, Pigsty provides best practices! It offers rich dashboards for host nodes and PostgreSQL databases. Whether or not these nodes or PostgreSQL servers are managed by Pigsty, with simple configuration, you immediately have a production-grade monitoring and alerting system, bringing existing hosts and PostgreSQL under management.

pigsty-dashboard.jpg

HA PostgreSQL Clusters

Pigsty helps you own your own production-grade HA PostgreSQL RDS service anywhere.

To create such an HA PostgreSQL cluster/RDS service, you simply describe it with a short config and run the playbook to create it:

pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica }
    10.10.10.13: { pg_seq: 3, pg_role: replica }
  vars: { pg_cluster: pg-test }
$ bin/pgsql-add pg-test  # Initialize cluster 'pg-test'

In less than 10 minutes, you’ll have a PostgreSQL database cluster with service access, monitoring, backup PITR, and HA fully configured.

pigsty-ha.png

Hardware failures are covered by the self-healing HA architecture provided by patroni, etcd, and haproxy—in case of primary failure, automatic failover executes within 45 seconds by default. Clients don’t need to modify config or restart applications: Haproxy uses patroni health checks for traffic distribution, and read-write requests are automatically routed to the new cluster primary, avoiding split-brain issues. This process is seamless—for example, in case of replica failure or planned switchover, clients experience only a momentary flash of the current query.

Software failures, human errors, and datacenter-level disasters are covered by pgBackRest and the optional Silo cluster. This provides local/cloud PITR capabilities and, in case of datacenter failure, offers cross-region replication and disaster recovery.

3.1.1 - Nodes

A node is an abstraction of hardware/OS resources—physical machines, bare metal, VMs, or containers/pods.

A node is an abstraction of hardware resources and operating systems. It can be a physical machine, bare metal, virtual machine, or container/pod.

Any machine running a Linux OS (with systemd daemon) and standard CPU/memory/disk/network resources can be treated as a node.

Nodes can have modules installed. Pigsty has several node types, distinguished by which modules are deployed:

Type Description
Regular Node A node managed by Pigsty
ADMIN Node The node that runs Ansible to issue management commands
INFRA Node Nodes with the INFRA module installed
ETCD Node Nodes with the ETCD module for DCS
MINIO Node Nodes with the MINIO module for object storage
PGSQL Node Nodes with the PGSQL module installed
… Nodes with other modules…

In a singleton Pigsty deployment, multiple roles converge on one node: it serves as the regular node, admin node, infra node, ETCD node, and database node simultaneously.

Regular Node

Nodes managed by Pigsty can have modules installed. The node.yml playbook configures nodes to the desired state. A regular node may run the following services:

Component Port Description Status
node_exporter 9100 Host metrics exporter Enabled
haproxy 9101 HAProxy load balancer (admin port) Enabled
vector 9598 Log collection agent Enabled
docker 9323 Container runtime support Optional
keepalived n/a L2 VIP for node cluster Optional
keepalived_exporter 9650 Keepalived status monitor Optional

Here, node_exporter exposes host metrics, vector sends logs to the collection system, and haproxy provides load balancing. These three are enabled by default. Docker, keepalived, and keepalived_exporter are optional and can be enabled as needed.

ADMIN Node

A Pigsty deployment has exactly one admin node—the node that runs Ansible playbooks and issues control/deployment commands.

This node has ssh/sudo access to all other nodes. Admin node security is critical and access must be strictly controlled; see Security Model: Trust Boundaries for its trust scope and critical assets.

During single-node installation and configuration, the current node becomes the admin node. However, alternatives exist. For example, if your laptop can SSH to all managed nodes and has Ansible installed, it can serve as the admin node—though this isn’t recommended for production.

For instance, you might use your laptop to manage a Pigsty VM in the cloud. In this case, your laptop is the admin node.

In serious production environments, the admin node is typically 1-2 dedicated DBA machines. In resource-constrained setups, INFRA nodes often double as admin nodes since all INFRA nodes have Ansible installed by default.

INFRA Node

A Pigsty deployment may have 1 or more INFRA nodes; large production environments typically have 2-3.

The infra group in the inventory defines which nodes are INFRA nodes. These nodes run the INFRA module with these components:

Component Port Description
nginx 80/443 Web UI, local software repository
grafana 3000 Visualization platform
victoriaMetrics 8428 Time-series database (metrics)
victoriaLogs 9428 Log collection server
victoriaTraces 10428 Trace collection server
vmalert 8880 Alerting and derived metrics
alertmanager 9059 Alert aggregation and routing
blackbox_exporter 9115 Blackbox probing (ping nodes/VIPs)
dnsmasq 53 Internal DNS resolution
chronyd 123 NTP time server
ansible - Playbook execution

Nginx serves as the module’s entry point, providing the web UI and local software repository. With multiple INFRA nodes, services on each are independent, but you can access all monitoring data sources from any INFRA node’s Grafana.

Pigsty is licensed under Apache-2.0, though embedded Grafana component uses AGPLv3.

ETCD Node

The ETCD module provides Distributed Consensus Service (DCS) for PostgreSQL high availability.

The etcd group in the inventory defines ETCD nodes. These nodes run etcd servers on two ports:

Component Port Description
etcd 2379 ETCD key-value store (client port)
etcd 2380 ETCD cluster peer communication

MINIO Node

The MINIO module provides optional backup storage for PostgreSQL.

The minio inventory group defines MINIO module nodes. In v4.5.0, these nodes run Silo servers on:

Component Port Description
silo 9000 S3 API endpoint
silo 9001 Silo admin console

PGSQL Node

Nodes with the PGSQL module are called PGSQL nodes. Node and PostgreSQL instance have a 1:1 deployment—one PG instance per node.

PGSQL nodes can borrow identity from their PostgreSQL instance—controlled by node_id_from_pg, defaulting to true, meaning the node name is set to the PG instance name.

PGSQL nodes run these additional components beyond regular node services:

Component Port Description Status
postgres 5432 PostgreSQL database server Enabled
pgbouncer 6432 PgBouncer connection pool Enabled
patroni 8008 Patroni HA management Enabled
pg_exporter 9630 PostgreSQL metrics exporter Enabled
pgbouncer_exporter 9631 PgBouncer metrics exporter Enabled
pgbackrest_exporter 9854 pgBackRest metrics exporter Enabled
vip-manager n/a Binds L2 VIP to cluster primary Optional
{{ pg_cluster }}-primary 5433 HAProxy service: pooled read/write Enabled
{{ pg_cluster }}-replica 5434 HAProxy service: pooled read-only Enabled
{{ pg_cluster }}-default 5436 HAProxy service: primary direct connection Enabled
{{ pg_cluster }}-offline 5438 HAProxy service: offline read Enabled
{{ pg_cluster }}-<service> 543x HAProxy service: custom PostgreSQL services Custom

The vip-manager is only enabled when users configure a PG VIP. Additional custom services can be defined in pg_services, exposed via haproxy using additional service ports.

Node Relationships

Regular nodes typically reference an INFRA node via the admin_ip parameter as their infrastructure provider. For example, with global admin_ip = 10.10.10.10, all nodes use infrastructure services at this IP.

Parameters that reference ${admin_ip}:

Parameter Module Default Value Description
repo_endpoint INFRA http://${admin_ip}:80 Software repo URL
repo_upstream.baseurl INFRA http://${admin_ip}/pigsty Local repo baseurl
infra_portal.endpoint INFRA ${admin_ip}:<port> Nginx proxy backend
dns_records INFRA ["${admin_ip} i.pigsty", ...] DNS records
node_default_etc_hosts NODE ["${admin_ip} i.pigsty"] Default static DNS
node_etc_hosts NODE - Custom static DNS
node_dns_servers NODE ["${admin_ip}"] Dynamic DNS servers
node_ntp_servers NODE - NTP servers (optional)

Typically the admin node and INFRA node coincide. With multiple INFRA nodes, the admin node is usually the first one; others serve as backups.

In large-scale production deployments, you might separate the Ansible admin node from INFRA module nodes. For example, use 1-2 small dedicated hosts under the DBA team as the control hub (ADMIN nodes), and 2-3 high-spec physical machines as monitoring infrastructure (INFRA nodes).

Typical node counts by deployment scale:

Scale ADMIN INFRA ETCD MINIO PGSQL
Single-node 1 1 1 0 1
3-node 1 3 3 0 3
Small prod 1 2 3 0 N
Large prod 2 3 5 4+ N

3.1.2 - Infrastructure

Infrastructure module architecture, components, and functionality in Pigsty.

Running production-grade, highly available PostgreSQL clusters typically requires a comprehensive set of infrastructure services (foundation) for support, such as monitoring and alerting, log collection, time synchronization, DNS resolution, and local software repositories. Pigsty provides the INFRA module to address this—it’s an optional module, but we strongly recommend enabling it.

Overview

The diagram below shows the architecture of a single-node deployment. The right half represents the components included in the INFRA module:

Component Type Description
Nginx Web Server Unified entry for WebUI, local repo, reverse proxy for internal services
Repo Software Repo APT/DNF repository with all RPM/DEB packages needed for deployment
Grafana Visualization Displays metrics, logs, and traces; hosts dashboards, reports, and custom data apps
VictoriaMetrics Time Series DB Scrapes all metrics, Prometheus API compatible, provides VMUI query interface
VictoriaLogs Log Platform Centralized log storage; all nodes run Vector by default, pushing logs here
VictoriaTraces Tracing Collects slow SQL, service traces, and other tracing data
VMAlert Eval Rule/Alert Evaluates alerting rules, pushes events to Alertmanager
AlertManager Alert Manager Aggregates alerts, dispatches notifications via email, Webhook, etc.
BlackboxExporter Blackbox Probe Probes reachability of IPs/VIPs/URLs
DNSMASQ DNS Service Provides DNS resolution for domains used within Pigsty [Optional]
Chronyd Time Sync Provides NTP time synchronization to ensure consistent time across nodes [Optional]
CA Certificate Issues encryption certificates within the environment
Ansible Orchestration Batch, declarative, agentless tool for managing large numbers of servers

pigsty-arch

Nginx

Nginx is the access entry point for all WebUI services in Pigsty, using ports 80 / 443 for HTTP/HTTPS by default. Live Demo

IP Access (replace) Domain (HTTP) Domain (HTTPS) Public Demo
http://10.10.10.10 http://i.pigsty https://i.pigsty https://demo.pigsty.io

Infrastructure components with WebUIs can be exposed uniformly through Nginx, such as Grafana, VictoriaMetrics (VMUI), AlertManager, and HAProxy console. Additionally, the local software repository and other static resources are served via Nginx.

Nginx configures local web servers or reverse proxy servers based on definitions in infra_portal.

infra_portal:
  home : { domain: i.pigsty }

By default, it exposes Pigsty’s admin homepage: i.pigsty. Different endpoints on this page proxy different components:

Endpoint Component Native Port Notes Public Demo
/ Nginx 80/443 Homepage, local repo, file server demo.pigsty.io
/ui/ Grafana 3000 Grafana dashboard entry demo.pigsty.io/ui/
/vmetrics/ VictoriaMetrics 8428 Time series DB Web UI demo.pigsty.io/vmetrics/
/vlogs/ VictoriaLogs 9428 Log DB Web UI demo.pigsty.io/vlogs/
/vtraces/ VictoriaTraces 10428 Tracing Web UI demo.pigsty.io/vtraces/
/vmalert/ VMAlert 8880 Alert rule management demo.pigsty.io/vmalert/
/alertmgr/ AlertManager 9059 Alert management Web UI demo.pigsty.io/alertmgr/
/blackbox/ Blackbox 9115 Blackbox probe

Pigsty online demo homepage

Pigsty allows rich customization of Nginx as a local file server or reverse proxy, with self-signed or real HTTPS certificates.

For more information, see: Tutorial: Nginx—Expose Web Services via Proxy and Tutorial: Certbot—Request and Renew HTTPS Certificates

Repo

Pigsty creates a local software repository on the Infra node during installation to accelerate subsequent software installations. Live Demo

This repository defaults to the /www/pigsty directory, served by Nginx and mounted at the /pigsty path:

IP Access (replace) Domain (HTTP) Domain (HTTPS) Public Demo
http://10.10.10.10/pigsty http://i.pigsty/pigsty https://i.pigsty/pigsty https://demo.pigsty.io/pigsty

Pigsty supports offline installation, which essentially pre-copies a prepared local software repository to the target environment. When Pigsty finds /www/pigsty/repo_complete during deployment, it skips upstream downloads and uses the existing repository directly. The current source has sow generate this file as both a completion marker and a SHA-256 manifest of repository contents. To force a rebuild, run ./infra.yml -t repo_build -e repo_build=true.

repo

For more information, see: Config: INFRA - REPO

Grafana

Grafana is the core component of Pigsty’s monitoring system, used for visualizing metrics, logs, and various information. Live Demo

Grafana listens on port 3000 by default and is proxied via Nginx at the /ui path:

IP Access (replace) Domain (HTTP) Domain (HTTPS) Public Demo
http://10.10.10.10/ui http://i.pigsty/ui https://i.pigsty/ui https://demo.pigsty.io/ui

Pigsty provides pre-built dashboards based on VictoriaMetrics / Logs / Traces, with one-click drill-down and roll-up via URL jumps for rapid troubleshooting.

Grafana can also serve as a low-code visualization platform, so ECharts, victoriametrics-datasource, victorialogs-datasource plugins are installed by default, with Vector / Victoria datasources registered uniformly as vmetrics-*, vlogs-*, vtraces-* for easy custom dashboard extension.

dashboard

For more information, see: Config: INFRA - GRAFANA.

VictoriaMetrics

VictoriaMetrics is Pigsty’s time series database, responsible for scraping and storing all monitoring metrics. Live Demo

It listens on port 8428 by default, mounted at Nginx /vmetrics path, and also accessible via the p.pigsty domain:

IP Access (replace) Domain (HTTP) Domain (HTTPS) Public Demo
http://10.10.10.10/vmetrics http://p.pigsty https://i.pigsty/vmetrics https://demo.pigsty.io/vmetrics

VictoriaMetrics is fully compatible with the Prometheus API, supporting PromQL queries, remote read/write protocols, and the Alertmanager API. The built-in VMUI provides an ad-hoc query interface for exploring metrics data directly, and also serves as a Grafana datasource.

vmetrics

For more information, see: Config: INFRA - VMETRICS

VictoriaLogs

VictoriaLogs is Pigsty’s log platform, centrally storing structured logs from all nodes. Live Demo

It listens on port 9428 by default, mounted at Nginx /vlogs path:

IP Access (replace) Domain (HTTP) Domain (HTTPS) Public Demo
http://10.10.10.10/vlogs http://i.pigsty/vlogs https://i.pigsty/vlogs https://demo.pigsty.io/vlogs

All managed nodes run Vector Agent by default, collecting system logs, PostgreSQL logs, Patroni logs, Pgbouncer logs, etc., processing them into structured format and pushing to VictoriaLogs. The built-in Web UI supports log search and filtering, and can be integrated with Grafana’s victorialogs-datasource plugin for visual analysis.

vlogs

For more information, see: Config: INFRA - VLOGS

VictoriaTraces

VictoriaTraces is used for collecting trace data and slow SQL records. Live Demo

It listens on port 10428 by default, mounted at Nginx /vtraces path:

IP Access (replace) Domain (HTTP) Domain (HTTPS) Public Demo
http://10.10.10.10/vtraces http://i.pigsty/vtraces https://i.pigsty/vtraces https://demo.pigsty.io/vtraces

VictoriaTraces provides a Jaeger-compatible interface for analyzing service call chains and database slow queries. Combined with Grafana dashboards, it enables rapid identification of performance bottlenecks and root cause tracing.

For more information, see: Config: INFRA - VTRACES

VMAlert

VMAlert is the alerting rule computation engine, responsible for evaluating alert rules and pushing triggered events to Alertmanager. Live Demo

It listens on port 8880 by default, mounted at Nginx /vmalert path:

IP Access (replace) Domain (HTTP) Domain (HTTPS) Public Demo
http://10.10.10.10/vmalert http://i.pigsty/vmalert https://i.pigsty/vmalert https://demo.pigsty.io/vmalert

VMAlert reads metrics data from VictoriaMetrics and periodically evaluates alerting rules. Pigsty provides pre-built alerting rules for PGSQL, NODE, REDIS, and other modules, covering common failure scenarios out of the box.

vmalert

For more information, see: Config: INFRA - VMALERT

AlertManager

AlertManager handles alert event aggregation, deduplication, grouping, and dispatch. Live Demo

It listens on port 9059 by default, mounted at Nginx /alertmgr path, and also accessible via the a.pigsty domain:

IP Access (replace) Domain (HTTP) Domain (HTTPS) Public Demo
http://10.10.10.10/alertmgr http://a.pigsty https://i.pigsty/alertmgr https://demo.pigsty.io/alertmgr

AlertManager supports multiple notification channels: email, Webhook, Slack, PagerDuty, WeChat Work, etc. Through alert routing rules, differentiated dispatch based on severity level and module type is possible, with support for silencing, inhibition, and other advanced features.

alertmanager

For more information, see: Config: INFRA - AlertManager

BlackboxExporter

Blackbox Exporter is used for active probing of target reachability, enabling blackbox monitoring.

It listens on port 9115 by default, mounted at Nginx /blackbox path:

IP Access (replace) Domain (HTTP) Domain (HTTPS) Public Demo
http://10.10.10.10/blackbox http://i.pigsty/blackbox https://i.pigsty/blackbox https://demo.pigsty.io/blackbox

It supports multiple probe methods including ICMP Ping, TCP ports, and HTTP/HTTPS endpoints. Useful for monitoring VIP reachability, service port availability, external dependency health, etc.—an important tool for assessing failure impact scope.

blackbox

For more information, see: Config: INFRA - BLACKBOX

Ansible

Ansible is Pigsty’s core orchestration tool; all deployment, configuration, and management operations are performed through Ansible Playbooks.

Pigsty automatically installs Ansible on the admin node (Infra node) during installation. It adopts a declarative configuration style and idempotent playbook design: the same playbook can be run repeatedly, and the system automatically converges to the desired state without side effects.

Ansible’s core advantages:

  • Agentless: Executes remotely via SSH, no additional software needed on target nodes.
  • Declarative: Describes the desired state rather than execution steps; configuration is documentation.
  • Idempotent: Multiple executions produce consistent results; supports retry after partial failures.

For more information, see: Playbooks: Pigsty Playbook

DNSMASQ

DNSMASQ provides DNS resolution on INFRA nodes, resolving domain names to their corresponding IP addresses.

DNSMASQ listens on port 53 (UDP/TCP) by default, providing DNS resolution for all nodes. Records are stored in the /etc/dnsmasq.d/pigsty directory.

Other modules automatically register their domain names with DNSMASQ during deployment, which you can use as needed. DNS is completely optional—Pigsty works normally without it. Client nodes can configure INFRA nodes as their DNS servers, allowing access to services via domain names without remembering IP addresses.

For more information, see: Config: INFRA - DNS and Tutorial: DNS—Configure Domain Resolution

Chronyd

Chronyd provides NTP time synchronization, ensuring consistent clocks across all nodes. It listens on port 123 (UDP) by default as the time source.

Time synchronization is critical for distributed systems: log analysis requires aligned timestamps, certificate validation depends on accurate clocks, and PostgreSQL streaming replication is sensitive to clock drift. In isolated network environments, the INFRA node can serve as an internal NTP server with other nodes synchronizing to it.

In Pigsty, all nodes run chronyd by default for time sync. The default upstream is pool.ntp.org public NTP servers. Chronyd is essentially managed by the Node module, but in isolated networks, you can use admin_ip to point to the INFRA node’s Chronyd service as the internal time source. In this case, the Chronyd service on the INFRA node serves as the internal time synchronization infrastructure.

For more information, see: Config: NODE - TIME

INFRA Node vs Regular Node

In Pigsty, the relationship between nodes and infrastructure is a weak circular dependency: node_monitor → infra → node

The NODE module itself doesn’t depend on the INFRA module, but the monitoring functionality (node_monitor) requires the monitoring platform and services provided by the infrastructure module.

Therefore, in the infra.yml and deploy playbooks, an “interleaved deployment” technique is used:

  • First, initialize the NODE module on all regular nodes, but skip monitoring config since infrastructure isn’t deployed yet.
  • Then, initialize the INFRA module on the INFRA node—monitoring is now available.
  • Finally, reconfigure monitoring on all regular nodes, connecting to the now-deployed monitoring platform.

If you don’t need “one-shot” deployment of all nodes, you can use phased deployment: initialize INFRA nodes first, then regular nodes.

How Are Nodes Coupled to Infrastructure?

Regular nodes reference an INFRA node via the admin_ip parameter as their infrastructure provider.

For example, when you configure global admin_ip = 10.10.10.10, all nodes will typically use infrastructure services at this IP.

This design allows quick, batch switching of infrastructure providers. Parameters that may reference ${admin_ip}:

Parameter Module Default Value Description
repo_endpoint INFRA http://${admin_ip}:80 Software repo URL
repo_upstream.baseurl INFRA http://${admin_ip}/pigsty Local repo baseurl
infra_portal.endpoint INFRA ${admin_ip}:<port> Nginx proxy backend
dns_records INFRA ["${admin_ip} i.pigsty", ...] DNS records
node_default_etc_hosts NODE ["${admin_ip} i.pigsty"] Default static DNS
node_etc_hosts NODE [] Custom static DNS
node_dns_servers NODE ["${admin_ip}"] Dynamic DNS servers
node_ntp_servers NODE ["pool pool.ntp.org iburst"] NTP servers (optional)

For example, when a node installs software, the local repo points to the Nginx local software repository at admin_ip:80/pigsty. The DNS server also points to DNSMASQ at admin_ip:53. However, this isn’t mandatory—nodes can ignore the local repo and install directly from upstream internet sources (most single-node config templates); DNS servers can also remain unconfigured, as Pigsty has no DNS dependency.

INFRA Node vs ADMIN Node

The management-initiating ADMIN node typically coincides with the INFRA node. In single-node deployment, this is exactly the case. In multi-node deployment with multiple INFRA nodes, the admin node is usually the first in the infra group; others serve as backups. However, exceptions exist. You might separate them for various reasons:

For example, in large-scale production deployments, a classic pattern uses 1-2 dedicated management hosts (tiny VMs suffice) belonging to the DBA team as the control hub, with 2-3 high-spec physical machines (or more!) as monitoring infrastructure. Here, admin nodes are separate from infrastructure nodes. In this case, the admin_ip in your config should point to an INFRA node’s IP, not the current ADMIN node’s IP. This is for historical reasons: initially ADMIN and INFRA nodes were tightly coupled concepts, with separation capabilities evolving later, so the parameter name wasn’t changed.

Another common scenario is managing cloud nodes locally. For example, you can install Ansible on your laptop and specify cloud nodes as “managed targets.” In this case, your laptop acts as the ADMIN node, while cloud servers act as INFRA nodes.

all:
  children:
    infra:   { hosts: { 10.10.10.10: { infra_seq: 1 , ansible_host: your_ssh_alias } } }  # <--- Use ansible_host to point to cloud node (fill in ssh alias)
    etcd:    { hosts: { 10.10.10.10: { etcd_seq: 1 } }, vars: { etcd_cluster: etcd } }    # SSH connection will use: ssh your_ssh_alias
    pg-meta: { hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }, vars: { pg_cluster: pg-meta } }
  vars:
    version: v4.5.0
    admin_ip: 10.10.10.10
    region: default

Multiple INFRA Nodes

By default, Pigsty only needs one INFRA node for most requirements. Even if the INFRA module goes down, it won’t affect database services on other nodes.

However, in production environments with high monitoring and alerting requirements, you may want multiple INFRA nodes to improve infrastructure availability. A common deployment uses two Infra nodes for redundancy, monitoring each other… or more nodes to deploy a distributed Victoria cluster for unlimited horizontal scaling.

Each Infra node is independent—Nginx points to services on the local machine. VictoriaMetrics independently scrapes metrics from all services in the environment, and logs are pushed to all VictoriaLogs collection endpoints by default. The only exception is Grafana: every Grafana instance registers all VictoriaMetrics / Logs / Traces / PostgreSQL instances as datasources. Therefore, each Grafana instance can see complete monitoring data.

If you modify Grafana—such as adding new dashboards or changing datasource configs—these changes only affect the Grafana instance on that node. To keep Grafana consistent across all nodes, use a PostgreSQL database as shared storage. See Tutorial: Configure Grafana High Availability for details.

INFRA overview dashboard

3.1.3 - PGSQL Arch

PostgreSQL module component interactions and data flow.

The PGSQL module organizes PostgreSQL in production as clusters—logical entities composed of a group of database instances associated by primary-replica relationships.

Overview

The PGSQL module includes the following components, working together to provide production-grade PostgreSQL HA cluster services:

Component Type Description
postgres Database The world’s most advanced open-source relational database, PGSQL core
patroni HA Manages PostgreSQL, coordinates failover, leader election, config changes
pgbouncer Pool Lightweight connection pooling middleware, reduces overhead, adds flexibility
pgbackrest Backup Full/incremental backup and WAL archiving, supports local and object storage
pg_exporter Metrics Exports PostgreSQL monitoring metrics in a Prometheus-compatible format
pgbouncer_exporter Metrics Exports Pgbouncer connection pool metrics
pgbackrest_exporter Metrics Exports backup status metrics
vip-manager VIP Binds L2 VIP to current primary node for transparent failover [Optional]

The vip-manager is an on-demand component. Additionally, PGSQL uses components from other modules:

Component Module Type Description
haproxy NODE LB Exposes service ports, routes traffic to primary or replicas
vector NODE Logging Collects PostgreSQL, Patroni, Pgbouncer logs and ships to center
etcd ETCD DCS Distributed consistent store for cluster metadata and leader info

By analogy, the PostgreSQL database kernel is the CPU, while the PGSQL module packages it as a complete computer. Patroni and Etcd form the HA subsystem, while pgBackRest and optional Silo form the backup subsystem. HAProxy, Pgbouncer, and vip-manager form the access subsystem. Various Exporters and Vector build the observability subsystem; finally, you can swap different kernel CPUs and extension cards.

Pigsty PostgreSQL cluster architecture
Subsystem Components Function
HA Subsystem Patroni + etcd Failure detection, auto-failover, config management
Access Subsystem HAProxy + Pgbouncer + vip-manager Service exposure, load balancing, pooling, VIP
Backup Subsystem pgBackRest (+ Silo) Full/incremental backup, WAL archiving, PITR
Observability Subsystem pg_exporter / pgbouncer_exporter / pgbackrest_exporter + Vector Metrics collection, log aggregation

Component Interaction

pigsty-arch

  • Cluster DNS is resolved by DNSMASQ on infra nodes
  • Cluster VIP is managed by vip-manager, which binds pg_vip_address to the cluster primary node.
  • Cluster services are exposed by HAProxy on nodes, different services distinguished by node ports (543x).
  • Pgbouncer is connection pooling middleware, listening on port 6432 by default, buffering connections, exposing additional metrics, and providing extra flexibility.
  • PostgreSQL listens on port 5432, providing relational database services
    • Installing PGSQL module on multiple nodes with the same cluster name automatically forms an HA cluster via streaming replication
    • PostgreSQL process is managed by patroni by default.
  • Patroni listens on port 8008 by default, supervising PostgreSQL server processes
    • Patroni starts Postgres server as child process
    • Patroni uses etcd as DCS: stores config, failure detection, and leader election.
    • Patroni provides Postgres info (e.g., primary/replica) via health checks, HAProxy uses this to distribute traffic
  • pg_exporter exposes postgres monitoring metrics on port 9630
  • pgbouncer_exporter exposes pgbouncer metrics on port 9631
  • pgBackRest uses local backup repository by default (pgbackrest_method = local)
    • If using local (default), pgBackRest creates local repository under pg_fs_bkup on primary node
    • If using minio, pgBackRest creates the backup repository on dedicated Silo or an external S3 service
  • Vector collects Postgres-related logs (postgres, pgbouncer, patroni, pgbackrest)
    • vector listens on port 9598, also exposes its own metrics to VictoriaMetrics on infra nodes
    • vector sends logs to VictoriaLogs on infra nodes

HA Subsystem

The HA subsystem consists of Patroni and etcd, responsible for PostgreSQL cluster failure detection, automatic failover, and configuration management.

How it works: Patroni runs on each node, managing the local PostgreSQL process and writing cluster state (leader, members, config) to etcd. When the primary fails, Patroni coordinates election via etcd, promoting the healthiest replica to new primary. The entire process is automatic, with RTO typically under 45 seconds.

Key Interactions:

  • PostgreSQL: Starts, stops, reloads PG as parent process, controls its lifecycle
  • etcd: External dependency, writes/watches leader key for distributed consensus and failure detection
  • HAProxy: Provides health checks via REST API (:8008), reporting instance role
  • vip-manager: Watches leader key in etcd, auto-migrates VIP

For more information, see: High Availability and Config: PGSQL - PG_BOOTSTRAP

Access Subsystem

The access subsystem consists of HAProxy, Pgbouncer, and vip-manager, responsible for service exposure, traffic routing, and connection pooling.

There are multiple access methods. A typical traffic path is: Client → DNS/VIP → HAProxy (543x) → Pgbouncer (6432) → PostgreSQL (5432)

Layer Component Port Role
L2 VIP vip-manager - Binds L2 VIP to primary (optional)
L4 Load Bal HAProxy 543x Service exposure, load balancing, health checks
L7 Pool Pgbouncer 6432 Connection reuse, session management, transaction pooling

Service Ports:

  • 5433 primary: Read-write service, routes to primary Pgbouncer
  • 5434 replica: Read-only service, routes to replica Pgbouncer
  • 5436 default: Default service, direct to primary (bypasses pool)
  • 5438 offline: Offline service, direct to offline replica (ETL/analytics)

Key Features:

  • HAProxy uses Patroni REST API to determine instance role, auto-routes traffic
  • Pgbouncer uses transaction-level pooling, absorbs connection spikes, reduces PG connection overhead
  • vip-manager watches etcd leader key, auto-migrates VIP during failover

For more information, see: Service Access and Config: PGSQL - PG_ACCESS

Backup Subsystem

The backup subsystem consists of pgBackRest (optionally with Silo or external S3 as a remote repository), responsible for data backup and point-in-time recovery (PITR).

Backup Types:

  • Full backup: Complete database copy
  • Incremental/differential backup: Only backs up changed data blocks
  • WAL archiving: Continuous transaction log archiving, enables any point-in-time recovery

Storage Backends:

  • local (default): Local disk, backups stored at pg_fs_bkup mount point
  • minio: S3-compatible object storage, supports centralized backup management and off-site DR

Key Interactions:

  • pgBackRest → PostgreSQL: Executes backup commands, manages WAL archiving
  • pgBackRest → Patroni: Recovery can bootstrap replicas as new primary or standby
  • pgbackrest_exporter → VictoriaMetrics: Exports backup status metrics through the Prometheus-compatible protocol to monitor backup health

For more information, see: PITR, Backup & Recovery, and Config: PGSQL - PG_BACKUP

Observability Subsystem

The observability subsystem consists of three Exporters and Vector, responsible for metrics collection and log aggregation.

Component Port Target Key Metrics
pg_exporter 9630 PostgreSQL Sessions, transactions, replication lag, buffer hits
pgbouncer_exporter 9631 Pgbouncer Pool utilization, wait queue, hit rate
pgbackrest_exporter 9854 pgBackRest Latest backup time, size, type
vector 9598 postgres/patroni/pgbouncer logs Structured log stream

Data Flow:

  • Metrics: Exporter → VictoriaMetrics (INFRA) → Grafana dashboards
  • Logs: Vector → VictoriaLogs (INFRA) → Grafana log queries

pg_exporter / pgbouncer_exporter connect to target services via local Unix socket, decoupled from HA topology. In slim install mode, these components can be disabled.

For more information, see: Config: PGSQL - PG_MONITOR

PostgreSQL

PostgreSQL is the PGSQL module core, listening on port 5432 by default for relational database services, deployed 1:1 with nodes.

Pigsty currently supports PostgreSQL 14-18 (lifecycle major versions), installed via binary packages from the PGDG official repo. Pigsty also allows you to use other PG kernel forks to replace the default PostgreSQL kernel, and install up to 575 extension plugins on top of the PG kernel.

PostgreSQL processes are managed by default by the HA agent—Patroni. When a cluster has only one node, that instance is the primary; when the cluster has multiple nodes, other instances automatically join as replicas: through physical replication, syncing data changes from the primary in real-time. Replicas can handle read-only requests and automatically take over when the primary fails.

pigsty-ha.png

You can access PostgreSQL directly, or through HAProxy and Pgbouncer connection pool.

For more information, see: Config: PGSQL - PG_BOOTSTRAP

Patroni

Patroni is the PostgreSQL HA control component, listening on port 8008 by default.

Patroni takes over PostgreSQL startup, shutdown, configuration, and health status, writing leader and member information to etcd. It handles automatic failover, maintains replication factor, coordinates parameter changes, and provides a REST API for HAProxy, monitoring, and administrators.

HAProxy uses Patroni health check endpoints to determine instance roles and route traffic to the correct primary or replica. vip-manager monitors the leader key in etcd and automatically migrates the VIP when the primary changes.

patroni

For more information, see: Config: PGSQL - PG_BOOTSTRAP

Pgbouncer

Pgbouncer is a lightweight connection pooling middleware, listening on port 6432 by default, deployed 1:1 with PostgreSQL database and node.

Pgbouncer runs statelessly on each instance, connecting to PostgreSQL via local Unix socket, using Transaction Pooling by default for pool management, absorbing burst client connections, stabilizing database sessions, reducing lock contention, and significantly improving performance under high concurrency.

Pigsty routes production traffic (read-write service 5433 / read-only service 5434) through Pgbouncer by default, while only the default service (5436) and offline service (5438) bypass the pool for direct PostgreSQL connections.

Pool mode is controlled by pgbouncer_poolmode, defaulting to transaction (transaction-level pooling). Connection pooling can be disabled via pgbouncer_enabled.

pgbouncer.png

For more information, see: Config: PGSQL - PG_ACCESS

pgBackRest

pgBackRest is a professional PostgreSQL backup/recovery tool, one of the strongest in the PG ecosystem, supporting full/incremental/differential backup and WAL archiving.

Pigsty uses pgBackRest for PostgreSQL PITR capability, allowing you to roll back clusters to any point within the backup retention window.

pgBackRest works with PostgreSQL to create backup repositories on the primary, executing backup and archive tasks. By default, it uses local backup repository (pgbackrest_method = local), but can be configured for Silo or external S3 object storage for centralized backup management.

After initialization, pgbackrest_init_backup can automatically trigger the first full backup. Recovery integrates with Patroni, supporting bootstrapping replicas as new primaries or standbys.

pgbackrest

For more information, see: Backup & Recovery and Config: PGSQL - PG_BACKUP

HAProxy

HAProxy is the service entry point and load balancer, exposing multiple database service ports.

Port Service Target Description
9101 Admin - HAProxy statistics and admin page
5433 primary Primary Pgbouncer Read-write service, routes to primary pool
5434 replica Replica Pgbouncer Read-only service, routes to replica pool
5436 default Primary Postgres Default service, direct to primary (bypasses pool)
5438 offline Offline Postgres Offline service, direct to offline replica (ETL/analytics)

HAProxy uses Patroni REST API health checks to determine instance roles and route traffic to the appropriate primary or replica. Service definitions are composed from pg_default_services and pg_services.

A dedicated HAProxy node group can be specified via pg_service_provider to handle higher traffic; by default, HAProxy on local nodes publishes services.

haproxy

For more information, see: Service Access and Config: PGSQL - PG_ACCESS

vip-manager

vip-manager binds L2 VIP to the current primary node. This is an optional component; enable it if your network supports L2 VIP.

vip-manager runs on each PG node, monitoring the leader key written by Patroni in etcd, and binds pg_vip_address to the current primary node’s network interface. When cluster failover occurs, vip-manager immediately releases the VIP from the old primary and rebinds it on the new primary, switching traffic to the new primary.

This component is optional, enabled via pg_vip_enabled. When enabled, ensure all nodes are in the same VLAN; otherwise, VIP migration will fail. Public cloud networks typically don’t support L2 VIP; it’s recommended only for on-premises and private cloud environments.

node-vip

For more information, see: Tutorial: VIP Configuration and Config: PGSQL - PG_ACCESS

pg_exporter

pg_exporter exports PostgreSQL monitoring metrics, listening on port 9630 by default.

pg_exporter runs on each PG node, connecting to PostgreSQL via local Unix socket, exporting rich metrics covering sessions, buffer hits, replication lag, transaction rates, etc., scraped by VictoriaMetrics on INFRA nodes.

Collection configuration is specified by pg_exporter_config, with support for automatic database discovery (pg_exporter_auto_discovery), and tiered cache strategies via pg_exporter_cache_ttls.

You can disable this component via parameters; in slim install, this component is not enabled.

pg-exporter

For more information, see: Config: PGSQL - PG_MONITOR

pgbouncer_exporter

pgbouncer_exporter exports Pgbouncer connection pool metrics, listening on port 9631 by default.

pgbouncer_exporter uses the same pg_exporter binary but with a dedicated metrics config file, supporting pgbouncer 1.8-1.25+. pgbouncer_exporter reads Pgbouncer statistics views, providing pool utilization, wait queue, and hit rate metrics.

If Pgbouncer is disabled, this component is also disabled. In slim install, this component is not enabled.

For more information, see: Config: PGSQL - PG_MONITOR

pgbackrest_exporter

pgbackrest_exporter exports backup status metrics, listening on port 9854 by default.

pgbackrest_exporter parses pgBackRest status, generating metrics for most recent backup time, size, type, etc. Combined with alerting policies, it quickly detects expired or failed backups, ensuring data safety. Note that when there are many backups or using large network repositories, collection overhead can be significant, so pgbackrest_exporter has a default 2-minute collection interval. In the worst case, you may see the latest backup status in the monitoring system 2 minutes after a backup completes.

For more information, see: Config: PGSQL - PG_MONITOR

etcd

etcd is a distributed consistent store (DCS), providing cluster metadata storage and leader election capability for Patroni.

etcd is deployed and managed by the independent ETCD module, not part of the PGSQL module itself, but critical for PostgreSQL HA. Patroni writes cluster state, leader info, and config parameters to etcd; all nodes reach consensus through etcd. vip-manager also reads the leader key from etcd to enable automatic VIP migration.

For more information, see: ETCD Module

vector

Vector is a high-performance log collection component, deployed by the NODE module, responsible for collecting PostgreSQL-related logs.

Vector runs on nodes, tracking PostgreSQL, Pgbouncer, Patroni, and pgBackRest log directories, sending structured logs to VictoriaLogs on INFRA nodes for centralized storage and querying.

For more information, see: NODE Module

3.2 - ER Model

How Pigsty abstracts different functionality into modules, and the E-R diagrams for these modules.

The largest entity concept in Pigsty is a Deployment. The main entities and relationships (E-R diagram) in a deployment are shown below:

Pigsty full data model ER diagram

A deployment can also be understood as an Environment. For example, Production (Prod), User Acceptance Testing (UAT), Staging, Testing, Development (Devbox), etc. Each environment corresponds to a Pigsty inventory that describes all entities and attributes in that environment.

Typically, an environment includes shared infrastructure (INFRA), which broadly includes ETCD (HA DCS) and MINIO (centralized backup repository), serving multiple PostgreSQL database clusters (and other database module components). (Exception: there are also deployments without infrastructure)

In Pigsty, almost all database modules are organized as “Clusters”. Each cluster is an Ansible group containing several node resources. For example, PostgreSQL HA database clusters, Redis, Etcd, and Silo all exist as clusters. An environment can contain multiple clusters.

3.2.1 - E-R Model of Infra Cluster

Entity-Relationship model for INFRA infrastructure nodes in Pigsty, component composition, and naming conventions.

The INFRA module plays a special role in Pigsty: it’s not a traditional “cluster” but rather a management hub composed of a group of infrastructure nodes, providing core services for the entire Pigsty deployment. Each INFRA node is an autonomous infrastructure service unit running core components like Nginx, Grafana, and VictoriaMetrics, collectively providing observability and management capabilities for managed database clusters.

There are two core entities in Pigsty’s INFRA module:

  • Node: A server running infrastructure components—can be bare metal, VM, container, or Pod.
  • Component: Various infrastructure services running on nodes, such as Nginx, Grafana, VictoriaMetrics, etc.

INFRA nodes typically serve as Admin Nodes, the control plane of Pigsty.

Component Composition

Each INFRA node runs the following core components:

Component Port Description
Nginx 80/443 Web portal, local repo, unified reverse proxy
Grafana 3000 Visualization platform, dashboards, data apps
VictoriaMetrics 8428 Time-series database, Prometheus API compatible
VictoriaLogs 9428 Log database, receives structured logs from Vector
VictoriaTraces 10428 Trace storage for slow SQL / request tracing
VMAlert 8880 Alert rule evaluator based on VictoriaMetrics
Alertmanager 9059 Alert aggregation and dispatch
Blackbox Exporter 9115 ICMP/TCP/HTTP black-box probing
DNSMASQ 53 DNS server for internal domain resolution
Chronyd 123 NTP time server

These components together form Pigsty’s observability infrastructure.

Examples

Let’s look at a concrete example with a two-node INFRA deployment:

infra:
  hosts:
    10.10.10.10: { infra_seq: 1 }
    10.10.10.11: { infra_seq: 2 }

The above config fragment defines a two-node INFRA deployment:

Group Description
infra INFRA infrastructure node group
Node Description
infra-1 10.10.10.10 INFRA node #1
infra-2 10.10.10.11 INFRA node #2

For production environments, deploying at least two INFRA nodes is recommended for infrastructure component redundancy.

Identity Parameters

Pigsty uses the INFRA_ID parameter group to assign deterministic identities to each INFRA module entity. One parameter is required:

Parameter Type Level Description Format
infra_seq int Node INFRA node sequence, required Natural number, starting from 1, unique within group

With node sequence assigned at node level, Pigsty automatically generates unique identifiers for each entity based on rules:

Entity Generation Rule Example
Node infra-{{ infra_seq }} infra-1, infra-2

The INFRA module assigns infra-N format identifiers to nodes for distinguishing multiple infrastructure nodes in the monitoring system. However, this doesn’t change the node’s hostname or system identity; nodes still use their existing hostname or IP address for identification.

Service Portal

INFRA nodes provide unified web service entry through Nginx. The infra_portal parameter defines services exposed through Nginx.

The default configuration only defines the home server:

infra_portal:
  home : { domain: i.pigsty }

Pigsty automatically configures reverse proxy endpoints for enabled components (Grafana, VictoriaMetrics, AlertManager, etc.). If you need to access these services via separate domains, you can explicitly add configurations:

infra_portal:
  home         : { domain: i.pigsty }
  grafana      : { domain: g.pigsty, endpoint: "${admin_ip}:3000", websocket: true }
  prometheus   : { domain: p.pigsty, endpoint: "${admin_ip}:8428" }   # VMUI
  alertmanager : { domain: a.pigsty, endpoint: "${admin_ip}:9059" }
Domain Service Description
i.pigsty Home Pigsty homepage
g.pigsty Grafana Monitoring dashboard
p.pigsty VictoriaMetrics TSDB Web UI
a.pigsty Alertmanager Alert management UI

Accessing Pigsty services via domain names is recommended over direct IP + port.

Deployment Scale

The number of INFRA nodes depends on deployment scale and HA requirements:

Scale INFRA Nodes Description
Dev/Test 1 Single-node deployment, all on one node
Small Prod 1-2 Single or dual node, can share with other services
Medium Prod 2-3 Dedicated INFRA nodes, redundant components
Large Prod 3+ Multiple INFRA nodes, component separation

In singleton deployment, INFRA components share the same node with PGSQL, ETCD, etc. In small-scale deployments, INFRA nodes typically also serve as “Admin Node” / backup admin node and local software repository (/www/pigsty). In larger deployments, these responsibilities can be separated to dedicated nodes.

Monitoring Label System

Pigsty’s monitoring system collects metrics from INFRA components themselves. Unlike database modules, each component in the INFRA module is treated as an independent monitoring object, distinguished by the cls (class) label.

Label Description Example
cls Component type, each forming a “class” nginx
ins Instance name, format {component}-{infra_seq} nginx-1
ip INFRA node IP running the component 10.10.10.10
job VictoriaMetrics scrape job, fixed as infra infra

Using a two-node INFRA deployment (infra_seq: 1 and infra_seq: 2) as example, component monitoring labels are:

Component cls ins Example Port
Nginx nginx nginx-1, nginx-2 9113
Grafana grafana grafana-1, grafana-2 3000
VictoriaMetrics vmetrics vmetrics-1, vmetrics-2 8428
VictoriaLogs vlogs vlogs-1, vlogs-2 9428
VictoriaTraces vtraces vtraces-1, vtraces-2 10428
VMAlert vmalert vmalert-1, vmalert-2 8880
Alertmanager alertmanager alertmanager-1, alertmanager-2 9059
Blackbox blackbox blackbox-1, blackbox-2 9115

All INFRA component metrics use a unified job="infra" label, distinguished by the cls label:

nginx_up{cls="nginx", ins="nginx-1", ip="10.10.10.10", job="infra"}
grafana_info{cls="grafana", ins="grafana-1", ip="10.10.10.10", job="infra"}
vm_app_version{cls="vmetrics", ins="vmetrics-1", ip="10.10.10.10", job="infra"}
vlogs_rows_ingested_total{cls="vlogs", ins="vlogs-1", ip="10.10.10.10", job="infra"}
alertmanager_alerts{cls="alertmanager", ins="alertmanager-1", ip="10.10.10.10", job="infra"}

3.2.2 - E-R Model of PostgreSQL Cluster

Entity-Relationship model for PostgreSQL clusters in Pigsty, including E-R diagram, entity definitions, and naming conventions.

The PGSQL module organizes PostgreSQL in production as clusters—logical entities composed of a group of database instances associated by primary-replica relationships.

Each cluster is an autonomous business unit consisting of at least one primary instance, exposing capabilities through services.

There are four core entities in Pigsty’s PGSQL module:

  • Cluster: An autonomous PostgreSQL business unit serving as the top-level namespace for other entities.
  • Service: A named abstraction that exposes capabilities, routes traffic, and exposes services using node ports.
  • Instance: A single PostgreSQL server consisting of running processes and database files on a single node.
  • Node: A hardware resource abstraction running Linux + Systemd environment—can be bare metal, VM, container, or Pod.

Along with two business entities—“Database” and “Role”—these form the complete logical view as shown below:

er-pgsql

Examples

Let’s look at two concrete examples. Using the four-node Pigsty sandbox, there’s a three-node pg-test cluster:

    pg-test:
      hosts:
        10.10.10.11: { pg_seq: 1, pg_role: primary }
        10.10.10.12: { pg_seq: 2, pg_role: replica }
        10.10.10.13: { pg_seq: 3, pg_role: replica }
      vars: { pg_cluster: pg-test }

The above config fragment defines a high-availability PostgreSQL cluster with these related entities:

Cluster Description
pg-test PostgreSQL 3-node HA cluster
Instance Description
pg-test-1 PostgreSQL instance #1, default primary
pg-test-2 PostgreSQL instance #2, initial replica
pg-test-3 PostgreSQL instance #3, initial replica
Service Description
pg-test-primary Read-write service (routes to primary pgbouncer)
pg-test-replica Read-only service (routes to replica pgbouncer)
pg-test-default Direct read-write service (routes to primary postgres)
pg-test-offline Offline read service (routes to dedicated postgres)
Node Description
node-1 10.10.10.11 Node #1, hosts pg-test-1 PG instance
node-2 10.10.10.12 Node #2, hosts pg-test-2 PG instance
node-3 10.10.10.13 Node #3, hosts pg-test-3 PG instance
ha

Identity Parameters

Pigsty uses the PG_ID parameter group to assign deterministic identities to each PGSQL module entity. Three parameters are required:

Parameter Type Level Description Format
pg_cluster string Cluster PG cluster name, required Valid DNS name, regex [a-zA-Z0-9-]+
pg_seq int Instance PG instance number, required Natural number, starting from 0 or 1, unique within cluster
pg_role enum Instance PG instance role, required Enum: primary, replica, offline

With cluster name defined at cluster level and instance number/role assigned at instance level, Pigsty automatically generates unique identifiers for each entity based on rules:

Entity Generation Rule Example
Instance {{ pg_cluster }}-{{ pg_seq }} pg-test-1, pg-test-2, pg-test-3
Service {{ pg_cluster }}-{{ pg_role }} pg-test-primary, pg-test-replica, pg-test-offline
Node Explicitly specified or borrowed from PG pg-test-1, pg-test-2, pg-test-3

Because Pigsty adopts a 1:1 exclusive deployment model for nodes and PG instances, by default the host node identifier borrows from the PG instance identifier (node_id_from_pg). You can also explicitly specify nodename to override, or disable nodename_overwrite to use the current default.

Sharding Identity Parameters

When using multiple PostgreSQL clusters (sharding) to serve the same business, two additional identity parameters are used: pg_shard and pg_group.

In this case, this group of PostgreSQL clusters shares the same pg_shard name with their own pg_group numbers, like this Citus cluster:

In this case, pg_cluster cluster names are typically composed of: {{ pg_shard }}{{ pg_group }}, e.g., pg-citus0, pg-citus1, etc.

all:
  children:
    pg-citus0: # citus shard 0
      hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
      vars: { pg_cluster: pg-citus0 , pg_group: 0 }
    pg-citus1: # citus shard 1
      hosts: { 10.10.10.11: { pg_seq: 1, pg_role: primary } }
      vars: { pg_cluster: pg-citus1 , pg_group: 1 }
    pg-citus2: # citus shard 2
      hosts: { 10.10.10.12: { pg_seq: 1, pg_role: primary } }
      vars: { pg_cluster: pg-citus2 , pg_group: 2 }
    pg-citus3: # citus shard 3
      hosts: { 10.10.10.13: { pg_seq: 1, pg_role: primary } }
      vars: { pg_cluster: pg-citus3 , pg_group: 3 }

Pigsty provides dedicated monitoring dashboards for horizontal sharding clusters, making it easy to compare performance and load across shards, but this requires using the above entity naming convention.

There are also other identity parameters for special scenarios, such as pg_upstream for specifying backup clusters/cascading replication upstream, gp_role for Greenplum cluster identity, pg_exporters for external monitoring instances, pg_offline_query for offline query instances, etc. See PG_ID parameter docs.

Monitoring Label System

Pigsty provides an out-of-box monitoring system that uses the above identity parameters to identify various PostgreSQL entities.

pg_up{cls="pg-test", ins="pg-test-1", ip="10.10.10.11", job="pgsql"}
pg_up{cls="pg-test", ins="pg-test-2", ip="10.10.10.12", job="pgsql"}
pg_up{cls="pg-test", ins="pg-test-3", ip="10.10.10.13", job="pgsql"}

For example, the cls, ins, ip labels correspond to cluster name, instance name, and node IP—the identifiers for these three core entities. They appear along with the job label in all native monitoring metrics collected by VictoriaMetrics and VictoriaLogs log streams.

The job name for collecting PostgreSQL metrics is fixed as pgsql; The job name for monitoring remote PG instances is fixed as pgrds. The job name for collecting PostgreSQL CSV logs is fixed as postgres; The job name for collecting pgbackrest logs is fixed as pgbackrest, other PG components collect logs via job: syslog.

Additionally, some entity identity labels appear in specific entity-related monitoring metrics, such as:

  • datname: Database name, if a metric belongs to a specific database.
  • relname: Table name, if a metric belongs to a specific table.
  • idxname: Index name, if a metric belongs to a specific index.
  • funcname: Function name, if a metric belongs to a specific function.
  • seqname: Sequence name, if a metric belongs to a specific sequence.
  • query: Query fingerprint, if a metric belongs to a specific query.

3.2.3 - E-R Model of Etcd Cluster

Entity-Relationship model for ETCD clusters in Pigsty, including E-R diagram, entity definitions, and naming conventions.

The ETCD module organizes ETCD in production as clusters—logical entities composed of a group of ETCD instances associated through the Raft consensus protocol.

Each cluster is an autonomous distributed key-value storage unit consisting of at least one ETCD instance, exposing service capabilities through client ports.

There are three core entities in Pigsty’s ETCD module:

  • Cluster: An autonomous ETCD service unit serving as the top-level namespace for other entities.
  • Instance: A single ETCD server process running on a node, participating in Raft consensus.
  • Node: A hardware resource abstraction running Linux + Systemd environment, implicitly declared.

Compared to PostgreSQL clusters, the ETCD cluster model is simpler, without Services or complex Role distinctions. All ETCD instances are functionally equivalent, electing a Leader through the Raft protocol while others become Followers. During scale-out intermediate states, non-voting Learner instance members are also allowed.

Examples

Let’s look at a concrete example with a three-node ETCD cluster:

etcd:
  hosts:
    10.10.10.10: { etcd_seq: 1 }
    10.10.10.11: { etcd_seq: 2 }
    10.10.10.12: { etcd_seq: 3 }
  vars:
    etcd_cluster: etcd

The above config fragment defines a three-node ETCD cluster with these related entities:

Cluster Description
etcd ETCD 3-node HA cluster
Instance Description
etcd-1 ETCD instance #1
etcd-2 ETCD instance #2
etcd-3 ETCD instance #3
Node Description
10.10.10.10 Node #1, hosts etcd-1 instance
10.10.10.11 Node #2, hosts etcd-2 instance
10.10.10.12 Node #3, hosts etcd-3 instance

Identity Parameters

Pigsty uses the ETCD parameter group to assign deterministic identities to each ETCD module entity. Two parameters are required:

Parameter Type Level Description Format
etcd_cluster string Cluster ETCD cluster name, required Valid DNS name, defaults to fixed etcd
etcd_seq int Instance ETCD instance number, required Natural number, starting from 1, unique within cluster

With cluster name defined at cluster level and instance number assigned at instance level, Pigsty automatically generates unique identifiers for each entity based on rules:

Entity Generation Rule Example
Instance {{ etcd_cluster }}-{{ etcd_seq }} etcd-1, etcd-2, etcd-3

The ETCD module does not assign additional identity to host nodes; nodes are identified by their existing hostname or IP address.

Ports & Protocols

Each ETCD instance listens on the following two ports:

Port Parameter Purpose
2379 etcd_port Client port, accessed by Patroni, vip-manager, etc.
2380 etcd_peer_port Peer communication port, used for Raft consensus

ETCD clusters enable TLS-encrypted communication by default and use RBAC authentication. Clients need the correct certificates and passwords to access ETCD services.

Cluster Size

As a distributed coordination service, ETCD cluster size directly affects availability, requiring more than half (quorum) of nodes to be alive to maintain service.

Cluster Size Quorum Fault Tolerance Use Case
1 node 1 0 Dev, test, demo
3 nodes 2 1 Small-medium production
5 nodes 3 2 Large-scale production

Even-member ETCD clusters are technically valid, but they do not tolerate more failures than an odd cluster with one fewer member and add deployment and quorum cost. Production clusters therefore usually have one, three, or five members; clusters larger than five are uncommon.

Monitoring Label System

Pigsty provides an out-of-box monitoring system that uses the above identity parameters to identify various ETCD entities.

etcd_up{cls="etcd", ins="etcd-1", ip="10.10.10.10", job="etcd"}
etcd_up{cls="etcd", ins="etcd-2", ip="10.10.10.11", job="etcd"}
etcd_up{cls="etcd", ins="etcd-3", ip="10.10.10.12", job="etcd"}

For example, the cls, ins, ip labels correspond to cluster name, instance name, and node IP—the identifiers for these three core entities. They appear along with the job label in all ETCD monitoring metrics collected by VictoriaMetrics. The job name for collecting ETCD metrics is fixed as etcd.

3.2.4 - MINIO Cluster Model

The cluster, instance, and node identity model used when Pigsty’s MINIO module deploys Silo.

MINIO is Pigsty’s compatibility module name for object storage. The current v4.5.0 source deploys Silo through minio_type: silo and organizes a group of object-storage instances into a cluster.

Each cluster is an autonomous S3-compatible object-storage unit consisting of at least one instance and exposing service through the S3 API port.

There are three core entities in Pigsty’s MINIO module:

  • Cluster: An autonomous object-storage service unit serving as the top-level namespace for other entities.
  • Instance: A single Silo server process running on a node and managing local disks.
  • Node: A hardware resource abstraction running Linux + Systemd environment, implicitly declared.

Silo also retains the Storage Pool concept for expansion.

Deployment Modes

Silo supports Pigsty’s three inventory deployment modes:

Mode Code Description Use Case
Single-Node Single-Drive SNSD Single node, single data directory or disk Dev, test, demo
Single-Node Multi-Drive SNMD Single node, multiple disks, typically 4+ Resource-constrained small deployments
Multi-Node Multi-Drive MNMD Multiple nodes, multiple disks per node Production recommended

SNSD mode can use a regular directory for quick experimentation. Multi-drive Silo deployments should use real disk mount points or the service will refuse to start.

Examples

The following example explicitly selects the current default Silo backend and defines a four-node multi-drive cluster:

minio:
  hosts:
    10.10.10.10: { minio_seq: 1 }
    10.10.10.11: { minio_seq: 2 }
    10.10.10.12: { minio_seq: 3 }
    10.10.10.13: { minio_seq: 4 }
  vars:
    minio_cluster: minio
    minio_type: silo
    minio_data: '/data{1...4}'
    minio_node: '${minio_cluster}-${minio_seq}.pigsty'

This config fragment defines a four-node Silo cluster with four disks per node. Instance identifiers retain the MINIO module’s compatibility naming:

Cluster Description
minio Silo 4-node HA cluster
Instance Description
minio-1 Object-storage instance #1, managing 4 disks
minio-2 Object-storage instance #2, managing 4 disks
minio-3 Object-storage instance #3, managing 4 disks
minio-4 Object-storage instance #4, managing 4 disks
Node Description
10.10.10.10 Node #1, hosts minio-1 instance
10.10.10.11 Node #2, hosts minio-2 instance
10.10.10.12 Node #3, hosts minio-3 instance
10.10.10.13 Node #4, hosts minio-4 instance

Identity Parameters

Pigsty uses the MINIO parameter group to assign deterministic identities to each MinIO module entity. Two parameters are required:

Parameter Type Level Description Format
minio_cluster string Cluster Object-storage cluster name, required Valid non-empty name, no default
minio_seq int Instance Object-storage instance number, required Natural number, starting from 1, unique within cluster

With cluster name defined at cluster level and instance number assigned at instance level, Pigsty automatically generates unique identifiers for each entity based on rules:

Entity Generation Rule Example
Instance {{ minio_cluster }}-{{ minio_seq }} minio-1, minio-2, minio-3, minio-4

The MINIO module does not assign additional identity to host nodes; nodes are identified by their existing hostname or IP address. The minio_node parameter generates node names for internal Silo cluster use (written to /etc/hosts for cluster discovery), not host-node identity.

Roles locate actual members across the entire inventory by minio_cluster; the Ansible group name does not need to match the cluster name. minio_type is a retained backend selector and currently must be silo.

Core Configuration Parameters

Beyond identity parameters, the following parameters are critical for Silo cluster configuration:

Parameter Type Description
minio_type enum Retained selector; currently only silo
minio_data path Data directory, use {x...y} for multi-drive
minio_node string Node name pattern for multi-node deployment
minio_domain string Service domain, defaults to sss.pigsty

These parameters determine minio_volumes, which the role writes to Silo’s MINIO_VOLUMES:

  • SNSD: Direct minio_data value, e.g., /data/minio
  • SNMD: Expanded minio_data directories, e.g., /data{1...4}
  • MNMD: Combined minio_node and minio_data, e.g., https://minio-{1...4}.pigsty:9000/data{1...4}

Ports & Services

Each object-storage instance listens on the following ports:

Port Parameter Purpose
9000 minio_port S3 API service port
9001 minio_admin_port Web admin console port

The MINIO module enables HTTPS by default, controlled by minio_https. Keep HTTPS enabled with the default pgBackRest S3 repository configuration and install the Pigsty CA correctly.

Clients can reach a multi-node Silo cluster through any member. For a stable entry point, use a load balancer such as HAProxy with a VIP.

Resource Provisioning

After Silo cluster deployment, Pigsty automatically creates the following resources (controlled by minio_provision):

Default Buckets (defined by minio_buckets):

Bucket Purpose
pgsql PostgreSQL pgBackREST backup storage
meta Metadata storage, versioning enabled
data General data storage

Default Users (defined by minio_users):

User Default Password Policy Purpose
pgbackrest S3User.Backup pgsql PostgreSQL backup dedicated user
s3user_meta S3User.Meta meta Access meta bucket
s3user_data S3User.Data data Access data bucket

These passwords are publicly documented default credentials, intended only for demonstrations and local development. Replace them before production deployment.

pgbackrest is used for PostgreSQL cluster backups; s3user_meta and s3user_data are reserved users not actively used.

Monitoring Label System

Pigsty uses the identity parameters above to identify object-storage entities. A Silo availability series looks like this:

minio_up{cls="minio", ins="minio-1", ip="10.10.10.10", job="minio"}
minio_up{cls="minio", ins="minio-2", ip="10.10.10.11", job="minio"}
minio_up{cls="minio", ins="minio-3", ip="10.10.10.12", job="minio"}
minio_up{cls="minio", ins="minio-4", ip="10.10.10.13", job="minio"}

Here cls, ins, and ip identify the cluster name, instance name, and node IP. Compatible monitoring naming keeps job="minio", while the current backend label is flavor=silo. See the metric list for details.

3.2.5 - E-R Model of Redis Cluster

Entity-Relationship model for Redis clusters in Pigsty, including E-R diagram, entity definitions, and naming conventions.

The Redis module organizes Redis in production as clusters—logical entities composed of a group of Redis instances deployed on one or more nodes.

Each cluster is an autonomous high-performance cache/storage unit consisting of at least one Redis instance, exposing service capabilities through ports.

There are three core entities in Pigsty’s Redis module:

  • Cluster: An autonomous Redis service unit serving as the top-level namespace for other entities.
  • Instance: A single Redis server process running on a specific port on a node.
  • Node: A hardware resource abstraction running Linux + Systemd environment, can host multiple Redis instances, implicitly declared.

Unlike PostgreSQL, Redis uses a single-node multi-instance deployment model: one physical/virtual machine node typically deploys multiple Redis instances to fully utilize multi-core CPUs. Therefore, nodes and instances have a 1:N relationship. Additionally, production typically advises against Redis instances with memory > 12GB.

Operating Modes

Redis has three different operating modes, specified by the redis_mode parameter:

Mode Code Description HA Mechanism
Standalone standalone Classic master-replica, default mode Requires Sentinel
Sentinel sentinel HA monitoring and auto-failover for standalone Multi-node quorum
Native Cluster cluster Redis native distributed cluster, no sentinel needed Built-in auto-failover
  • Standalone: Default mode, replication via replica_of parameter. Requires additional Sentinel cluster for HA.
  • Sentinel: Stores no business data, dedicated to monitoring standalone Redis clusters for auto-failover; multi-node itself provides HA.
  • Native Cluster: Data auto-sharded across multiple primaries, each can have multiple replicas, built-in HA, no sentinel needed.

Examples

Let’s look at concrete examples for each mode:

Standalone Cluster

Classic master-replica on a single node:

redis-ms:
  hosts:
    10.10.10.10:
      redis_node: 1
      redis_instances:
        6379: { }
        6380: { replica_of: '10.10.10.10 6379' }
  vars:
    redis_cluster: redis-ms
    redis_password: 'redis.ms'
    redis_max_memory: 64MB
Cluster Description
redis-ms Redis standalone cluster
Node Description
redis-ms-1 10.10.10.10 Node #1, hosts 2 instances
Instance Description
redis-ms-1-6379 Primary instance, listening on port 6379
redis-ms-1-6380 Replica instance, port 6380, replicates from 6379

Sentinel Cluster

Three sentinel instances on a single node for monitoring standalone clusters. Sentinel clusters specify monitored standalone clusters via redis_sentinel_monitor:

redis-sentinel:
  hosts:
    10.10.10.11:
      redis_node: 1
      redis_instances: { 26379: {}, 26380: {}, 26381: {} }
  vars:
    redis_cluster: redis-sentinel
    redis_password: 'redis.sentinel'
    redis_mode: sentinel
    redis_max_memory: 16MB
    redis_sentinel_monitor:
      - { name: redis-ms, host: 10.10.10.10, port: 6379, password: redis.ms, quorum: 2 }

Native Cluster

A Redis native distributed cluster with two nodes and six instances (minimum spec: 3 primaries, 3 replicas):

redis-test:
  hosts:
    10.10.10.12: { redis_node: 1, redis_instances: { 6379: {}, 6380: {}, 6381: {} } }
    10.10.10.13: { redis_node: 2, redis_instances: { 6379: {}, 6380: {}, 6381: {} } }
  vars:
    redis_cluster: redis-test
    redis_password: 'redis.test'
    redis_mode: cluster
    redis_max_memory: 32MB

This creates a 3 primary 3 replica native Redis cluster.

Cluster Description
redis-test Redis native cluster (3P3R)
Instance Description
redis-test-1-6379 Instance on node 1, port 6379
redis-test-1-6380 Instance on node 1, port 6380
redis-test-1-6381 Instance on node 1, port 6381
redis-test-2-6379 Instance on node 2, port 6379
redis-test-2-6380 Instance on node 2, port 6380
redis-test-2-6381 Instance on node 2, port 6381
Node Description
redis-test-1 10.10.10.12 Node #1, hosts 3 instances
redis-test-2 10.10.10.13 Node #2, hosts 3 instances

Identity Parameters

Pigsty uses the REDIS parameter group to assign deterministic identities to each Redis module entity. Three parameters are required:

Parameter Type Level Description Format
redis_cluster string Cluster Redis cluster name, required Valid DNS name, regex [a-z][a-z0-9-]*
redis_node int Node Redis node number, required Natural number, starting from 1, unique within cluster
redis_instances dict Node Redis instance definition, required JSON object, key is port, value is instance config

With cluster name defined at cluster level and node number/instance definition assigned at node level, Pigsty automatically generates unique identifiers for each entity:

Entity Generation Rule Example
Instance {{ redis_cluster }}-{{ redis_node }}-{{ port }} redis-ms-1-6379, redis-ms-1-6380

The Redis module does not assign additional identity to host nodes; nodes are identified by their existing hostname or IP address. redis_node is used for instance naming, not host node identity.

Instance Definition

redis_instances is a JSON object with port number as key and instance config as value:

redis_instances:
  6379: { }                                      # Primary instance, no extra config
  6380: { replica_of: '10.10.10.10 6379' }       # Replica, specify upstream primary
  6381: { replica_of: '10.10.10.10 6379' }       # Replica, specify upstream primary

Each Redis instance listens on a unique port within the node. You can choose any port number, but avoid system reserved ports (< 1024) or conflicts with Pigsty used ports. The replica_of parameter sets replication relationship in standalone mode, format '<ip> <port>', specifying upstream primary address and port.

Additionally, each Redis node runs a Redis Exporter collecting metrics from all local instances:

Port Parameter Purpose
9121 redis_exporter_port Redis Exporter port

Redis’s single-node multi-instance deployment model has some limitations:

  • Node Exclusive: A node can only belong to one Redis cluster, not assigned to different clusters simultaneously.
  • Port Unique: Redis instances on the same node must use different ports to avoid conflicts.
  • Password Shared: Multiple instances on the same node cannot have different passwords (redis_exporter limitation).
  • Manual HA: Standalone Redis clusters require additional Sentinel configuration for auto-failover.

Monitoring Label System

Pigsty provides an out-of-box monitoring system that uses the above identity parameters to identify various Redis entities.

redis_up{cls="redis-ms", ins="redis-ms-1-6379", ip="10.10.10.10", job="redis"}
redis_up{cls="redis-ms", ins="redis-ms-1-6380", ip="10.10.10.10", job="redis"}

For example, the cls, ins, ip labels correspond to cluster name, instance name, and node IP—the identifiers for these three core entities. They appear along with the job label in all Redis monitoring metrics collected by VictoriaMetrics. The job name for collecting Redis metrics is fixed as redis.

3.3 - Infra as Code

Pigsty uses Infrastructure as Code (IaC) philosophy to manage all components, providing declarative management for large-scale clusters.

Pigsty follows the IaC and GitOPS philosophy: use a declarative config inventory to describe the entire environment, and materialize it through idempotent playbooks.

Users describe their desired state declaratively through parameters, and playbooks idempotently adjust target nodes to reach that state. This is similar to Kubernetes CRDs & Operators, but Pigsty implements this functionality on bare metal and virtual machines through Ansible.

Pigsty was born to solve the operational management problem of ultra-large-scale PostgreSQL clusters. The idea behind it is simple — we need the ability to replicate the entire infrastructure (100+ database clusters + PG/Redis + observability) on ready servers within ten minutes. No GUI + ClickOps can complete such a complex task in such a short time, making CLI + IaC the only choice — it provides precise, efficient control.

The config inventory pigsty.yml file describes the state of the entire deployment. Whether it’s production (prod), staging, test, or development (devbox) environments, the difference between infrastructures lies only in the config inventory, while the deployment delivery logic is exactly the same.

You can use git for version control and auditing of this deployment “seed/gene”, and Pigsty even supports storing the config inventory as database tables in PostgreSQL CMDB, further achieving Infra as Data capability. Seamlessly integrate with your existing workflows.

IaC is designed for professional users and enterprise scenarios but is also deeply optimized for individual developers and SMBs. Even if you’re not a professional DBA, you don’t need to understand these hundreds of adjustment knobs and switches. All parameters come with well-performing default values. You can get an out-of-the-box single-node database with zero configuration; Simply add two more IP addresses to get an enterprise-grade high-availability PostgreSQL cluster.

Declare Modules

Take the following default config snippet as an example. This config describes a node 10.10.10.10 with INFRA, NODE, ETCD, and PGSQL modules installed.

# monitoring, alerting, DNS, NTP and other infrastructure cluster...
infra: { hosts: { 10.10.10.10: { infra_seq: 1 } } }

# minio cluster, s3 compatible object storage
minio: { hosts: { 10.10.10.10: { minio_seq: 1 } }, vars: { minio_cluster: minio } }

# etcd cluster, used as DCS for PostgreSQL high availability
etcd: { hosts: { 10.10.10.10: { etcd_seq: 1 } }, vars: { etcd_cluster: etcd } }

# PGSQL example cluster: pg-meta
pg-meta: { hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }, vars: { pg_cluster: pg-meta } }

To actually install these modules, execute the following playbooks:

./infra.yml -l 10.10.10.10  # Initialize infra module on node 10.10.10.10
./etcd.yml  -l 10.10.10.10  # Initialize etcd module on node 10.10.10.10
./minio.yml -l 10.10.10.10  # Initialize minio module on node 10.10.10.10
./pgsql.yml -l 10.10.10.10  # Initialize pgsql module on node 10.10.10.10

Declare Clusters

You can declare PostgreSQL database clusters by installing the PGSQL module on multiple nodes, making them a service unit:

For example, to deploy a three-node high-availability PostgreSQL cluster using streaming replication on the following three Pigsty-managed nodes, you can add the following definition to the all.children section of the config file pigsty.yml:

pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica }
    10.10.10.13: { pg_seq: 3, pg_role: offline }
  vars:  { pg_cluster: pg-test }

After defining, you can use playbooks to create the cluster:

bin/pgsql-add pg-test   # Create the pg-test cluster
pigsty-iac.jpg

You can use different instance roles such as primary, replica, offline, delayed, sync standby; as well as different clusters: such as standby clusters, Citus clusters, and even Redis / MINIO (Silo) / Etcd clusters

Customize Cluster Content

Not only can you define clusters declaratively, but you can also define databases, users, services, and HBA rules within the cluster. For example, the following config file deeply customizes the content of the default pg-meta single-node database cluster:

Including: declaring six business databases and seven business users, adding an extra standby service (synchronous standby, providing read capability with no replication delay), defining some additional pg_hba rules, an L2 VIP address pointing to the cluster primary, and a customized backup strategy.

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary , pg_offline_query: true } }
  vars:
    pg_cluster: pg-meta
    pg_databases:                       # define business databases on this cluster, array of database definition
      - name: meta                      # REQUIRED, `name` is the only mandatory field of a database definition
        baseline: cmdb.sql              # optional, database sql baseline path, (relative path among ansible search path, e.g files/)
        pgbouncer: true                 # optional, add this database to pgbouncer database list? true by default
        schemas: [pigsty]               # optional, additional schemas to be created, array of schema names
        extensions:                     # optional, additional extensions to be installed: array of `{name[,schema]}`
          - { name: postgis , schema: public }
          - { name: timescaledb }
        comment: pigsty meta database   # optional, comment string for this database
        owner: postgres                # optional, database owner, postgres by default
        template: template1            # optional, which template to use, template1 by default
        encoding: UTF8                 # optional, database encoding, UTF8 by default. (MUST same as template database)
        locale: C                      # optional, database locale, C by default.  (MUST same as template database)
        lc_collate: C                  # optional, database collate, C by default. (MUST same as template database)
        lc_ctype: C                    # optional, database ctype, C by default.   (MUST same as template database)
        tablespace: pg_default         # optional, default tablespace, 'pg_default' by default.
        allowconn: true                # optional, allow connection, true by default. false will disable connect at all
        revokeconn: false              # optional, revoke public connection privilege. false by default. (leave connect with grant option to owner)
        register_datasource: true      # optional, register this database to grafana datasources? true by default
        connlimit: -1                  # optional, database connection limit, default -1 disable limit
        pool_auth_user: dbuser_meta    # optional, all connection to this pgbouncer database will be authenticated by this user
        pool_mode: transaction         # optional, pgbouncer pool mode at database level, default transaction
        pool_size: 64                  # optional, pgbouncer pool size at database level, default 64
        pool_reserve: 32          # optional, pgbouncer pool size reserve at database level, default 32
        pool_size_min: 0               # optional, pgbouncer pool size min at database level, default 0
        pool_connlimit: 100          # optional, max database connections at database level, default 100
      - { name: grafana  ,owner: dbuser_grafana  ,revokeconn: true ,comment: grafana primary database }
      - { name: bytebase ,owner: dbuser_bytebase ,revokeconn: true ,comment: bytebase primary database }
      - { name: kong     ,owner: dbuser_kong     ,revokeconn: true ,comment: kong the api gateway database }
      - { name: gitea    ,owner: dbuser_gitea    ,revokeconn: true ,comment: gitea meta database }
      - { name: wiki     ,owner: dbuser_wiki     ,revokeconn: true ,comment: wiki meta database }
    pg_users:                           # define business users/roles on this cluster, array of user definition
      - name: dbuser_meta               # REQUIRED, `name` is the only mandatory field of a user definition
        password: DBUser.Meta           # optional, password, can be a scram-sha-256 hash string or plain text
        login: true                     # optional, can log in, true by default  (new biz ROLE should be false)
        superuser: false                # optional, is superuser? false by default
        createdb: false                 # optional, can create database? false by default
        createrole: false               # optional, can create role? false by default
        inherit: true                   # optional, can this role use inherited privileges? true by default
        replication: false              # optional, can this role do replication? false by default
        bypassrls: false                # optional, can this role bypass row level security? false by default
        pgbouncer: true                 # optional, add this user to pgbouncer user-list? false by default (production user should be true explicitly)
        connlimit: -1                   # optional, user connection limit, default -1 disable limit
        expire_in: 3650                 # optional, now + n days when this role is expired (OVERWRITE expire_at)
        expire_at: '2030-12-31'         # optional, YYYY-MM-DD 'timestamp' when this role is expired  (OVERWRITTEN by expire_in)
        comment: pigsty admin user      # optional, comment string for this user/role
        roles: [dbrole_admin]           # optional, belonged roles. default roles are: dbrole_{admin,readonly,readwrite,offline}
        parameters: {}                  # optional, role level parameters with `ALTER ROLE SET`
        pool_mode: transaction          # optional, pgbouncer pool mode at user level, transaction by default
        pool_connlimit: -1              # optional, max database connections at user level, default -1 disable limit
      - {name: dbuser_view     ,password: DBUser.Viewer   ,pgbouncer: true ,roles: [dbrole_readonly], comment: read-only viewer for meta database}
      - {name: dbuser_grafana  ,password: DBUser.Grafana  ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for grafana database   }
      - {name: dbuser_bytebase ,password: DBUser.Bytebase ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for bytebase database  }
      - {name: dbuser_kong     ,password: DBUser.Kong     ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for kong api gateway   }
      - {name: dbuser_gitea    ,password: DBUser.Gitea    ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for gitea service      }
      - {name: dbuser_wiki     ,password: DBUser.Wiki     ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for wiki.js service    }
    pg_services:                        # extra services in addition to pg_default_services, array of service definition
      # standby service will route {ip|name}:5435 to sync replica's pgbouncer (5435->6432 standby)
      - name: standby                   # required, service name, the actual svc name will be prefixed with `pg_cluster`, e.g: pg-meta-standby
        port: 5435                      # required, service exposed port (work as kubernetes service node port mode)
        ip: "*"                         # optional, service bind ip address, `*` for all ip by default
        selector: "[]"                  # required, service member selector, use JMESPath to filter inventory
        dest: default                   # optional, destination port, default|postgres|pgbouncer|<port_number>, 'default' by default
        check: /sync                    # optional, health check url path, / by default
        backup: "[? pg_role == `primary`]"  # backup server selector
        maxconn: 3000                   # optional, max allowed front-end connection
        balance: roundrobin             # optional, haproxy load balance algorithm (roundrobin by default, other: leastconn)
        options: 'inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100'
    pg_hba_rules:
      - {user: dbuser_view , db: all ,addr: infra ,auth: pwd ,title: 'allow grafana dashboard access cmdb from infra nodes'}
    pg_vip_enabled: true
    pg_vip_address: 10.10.10.2/24
    pg_vip_interface: eth1
    pg_crontab:  # full backup daily at 1am (installed in the postgres user crontab)
      - '00 01 * * * /pg/bin/pg-backup full'

Declare Access Control

You can also customize Pigsty’s access control through declarative configuration. For example, the following config file provides deep security customization for the pg-meta cluster:

Uses the three-node core cluster template: crit.yml, to ensure data consistency is prioritized with zero data loss during failover. Enables L2 VIP and restricts database and connection pool listening addresses to local loopback IP + internal network IP + VIP three specific addresses. The template enables TLS for the Patroni API and PgBouncer, and requires SSL for database access through HBA. It also enables $libdir/passwordcheck in pg_libs to enforce a password-strength policy.

Finally, a separate pg-meta-delay cluster is declared as pg-meta’s delayed replica from one hour ago, for emergency data deletion recovery.

pg-meta:      # 3 instance postgres cluster `pg-meta`
  hosts:
    10.10.10.10: { pg_seq: 1, pg_role: primary }
    10.10.10.11: { pg_seq: 2, pg_role: replica }
    10.10.10.12: { pg_seq: 3, pg_role: replica , pg_offline_query: true }
  vars:
    pg_cluster: pg-meta
    pg_conf: crit.yml
    pg_users:
      - { name: dbuser_meta , password: DBUser.Meta   , pgbouncer: true , roles: [ dbrole_admin ] , comment: pigsty admin user }
      - { name: dbuser_view , password: DBUser.Viewer , pgbouncer: true , roles: [ dbrole_readonly ] , comment: read-only viewer for meta database }
    pg_databases:
      - {name: meta ,baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty] ,extensions: [{name: postgis, schema: public}, {name: timescaledb}]}
    pg_default_service_dest: postgres
    pg_services:
      - { name: standby ,src_ip: "*" ,port: 5435 , dest: default ,selector: "[]" , backup: "[? pg_role == `primary`]" }
    pg_vip_enabled: true
    pg_vip_address: 10.10.10.2/24
    pg_vip_interface: eth1
    pg_listen: '${ip},${vip},${lo}'
    patroni_ssl_enabled: true
    pgbouncer_sslmode: require
    pgbackrest_method: minio
    pg_libs: 'timescaledb, $libdir/passwordcheck, pg_stat_statements, auto_explain' # add passwordcheck extension to enforce strong password
    pg_default_roles:                 # default roles and users in postgres cluster
      - { name: dbrole_readonly  ,login: false ,comment: role for global read-only access     }
      - { name: dbrole_offline   ,login: false ,comment: role for restricted read-only access }
      - { name: dbrole_readwrite ,login: false ,roles: [dbrole_readonly]               ,comment: role for global read-write access }
      - { name: dbrole_admin     ,login: false ,roles: [pg_monitor, dbrole_readwrite]  ,comment: role for object creation }
      - { name: postgres     ,superuser: true  ,expire_in: 7300                        ,comment: system superuser }
      - { name: replicator ,replication: true  ,expire_in: 7300 ,roles: [pg_monitor, dbrole_readonly]   ,comment: system replicator }
      - { name: dbuser_dba   ,superuser: true  ,expire_in: 7300 ,roles: [dbrole_admin]  ,pgbouncer: true ,pool_mode: session, pool_connlimit: 16 , comment: pgsql admin user }
      - { name: dbuser_monitor ,roles: [pg_monitor] ,expire_in: 7300 ,pgbouncer: true ,parameters: {log_min_duration_statement: 1000 } ,pool_mode: session ,pool_connlimit: 8 ,comment: pgsql monitor user }
    pg_default_hba_rules:             # postgres host-based auth rules by default
      - {user: '${dbsu}'    ,db: all         ,addr: local     ,auth: ident ,title: 'dbsu access via local os user ident'  }
      - {user: '${dbsu}'    ,db: replication ,addr: local     ,auth: ident ,title: 'dbsu replication from local os ident' }
      - {user: '${repl}'    ,db: replication ,addr: localhost ,auth: ssl   ,title: 'replicator replication from localhost'}
      - {user: '${repl}'    ,db: replication ,addr: intra     ,auth: ssl   ,title: 'replicator replication from intranet' }
      - {user: '${repl}'    ,db: postgres    ,addr: intra     ,auth: ssl   ,title: 'replicator postgres db from intranet' }
      - {user: '${monitor}' ,db: all         ,addr: localhost ,auth: pwd   ,title: 'monitor from localhost with password' }
      - {user: '${monitor}' ,db: all         ,addr: infra     ,auth: ssl   ,title: 'monitor from infra host with password'}
      - {user: '${admin}'   ,db: all         ,addr: infra     ,auth: ssl   ,title: 'admin @ infra nodes with pwd & ssl'   }
      - {user: '${admin}'   ,db: all         ,addr: world     ,auth: cert  ,title: 'admin @ everywhere with ssl & cert'   }
      - {user: '+dbrole_readonly',db: all    ,addr: localhost ,auth: ssl   ,title: 'pgbouncer read/write via local socket'}
      - {user: '+dbrole_readonly',db: all    ,addr: intra     ,auth: ssl   ,title: 'read/write biz user via password'     }
      - {user: '+dbrole_offline' ,db: all    ,addr: intra     ,auth: ssl   ,title: 'allow etl offline tasks from intranet'}
    pgb_default_hba_rules:            # pgbouncer host-based authentication rules
      - {user: '${dbsu}'    ,db: pgbouncer   ,addr: local     ,auth: peer  ,title: 'dbsu local admin access with os ident'}
      - {user: 'all'        ,db: all         ,addr: localhost ,auth: pwd   ,title: 'allow all user local access with pwd' }
      - {user: '${monitor}' ,db: pgbouncer   ,addr: intra     ,auth: ssl   ,title: 'monitor access via intranet with pwd' }
      - {user: '${monitor}' ,db: all         ,addr: world     ,auth: deny  ,title: 'reject all other monitor access addr' }
      - {user: '${admin}'   ,db: all         ,addr: intra     ,auth: ssl   ,title: 'admin access via intranet with pwd'   }
      - {user: '${admin}'   ,db: all         ,addr: world     ,auth: deny  ,title: 'reject all other admin access addr'   }
      - {user: 'all'        ,db: all         ,addr: intra     ,auth: ssl   ,title: 'allow all user intra access with pwd' }

# OPTIONAL delayed cluster for pg-meta
pg-meta-delay:                    # delayed instance for pg-meta (1 hour ago)
  hosts: { 10.10.10.13: { pg_seq: 1, pg_role: primary, pg_upstream: 10.10.10.10, pg_delay: 1h } }
  vars: { pg_cluster: pg-meta-delay }

Citus Distributed Cluster

Below is a declarative configuration for a four-node Citus distributed cluster:

all:
  children:
    pg-citus0: # citus coordinator, pg_group = 0
      hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
      vars: { pg_cluster: pg-citus0 , pg_group: 0 }
    pg-citus1: # citus data node 1
      hosts: { 10.10.10.11: { pg_seq: 1, pg_role: primary } }
      vars: { pg_cluster: pg-citus1 , pg_group: 1 }
    pg-citus2: # citus data node 2
      hosts: { 10.10.10.12: { pg_seq: 1, pg_role: primary } }
      vars: { pg_cluster: pg-citus2 , pg_group: 2 }
    pg-citus3: # citus data node 3, with an extra replica
      hosts:
        10.10.10.13: { pg_seq: 1, pg_role: primary }
        10.10.10.14: { pg_seq: 2, pg_role: replica }
      vars: { pg_cluster: pg-citus3 , pg_group: 3 }
  vars:                               # global parameters for all citus clusters
    pg_mode: citus                    # pgsql cluster mode: citus
    pg_shard: pg-citus                # citus shard name: pg-citus
    patroni_citus_db: meta            # citus distributed database name
    pg_dbsu_password: DBUser.Postgres # all dbsu password access for citus cluster
    pg_users: [ { name: dbuser_meta ,password: DBUser.Meta ,pgbouncer: true ,roles: [ dbrole_admin ] } ]
    pg_databases: [ { name: meta ,extensions: [ { name: citus }, { name: postgis }, { name: timescaledb } ] } ]
    pg_hba_rules:
      - { user: 'all' ,db: all  ,addr: 127.0.0.1/32 ,auth: ssl ,title: 'all user ssl access from localhost' }
      - { user: 'all' ,db: all  ,addr: intra        ,auth: ssl ,title: 'all user ssl access from intranet'  }

Redis Clusters

Below are declarative configuration examples for Redis primary-replica cluster, sentinel cluster, and Redis Cluster:

redis-ms: # redis classic primary & replica
  hosts: { 10.10.10.10: { redis_node: 1 , redis_instances: { 6379: { }, 6380: { replica_of: '10.10.10.10 6379' } } } }
  vars: { redis_cluster: redis-ms ,redis_password: 'redis.ms' ,redis_max_memory: 64MB }

redis-meta: # redis sentinel x 3
  hosts: { 10.10.10.11: { redis_node: 1 , redis_instances: { 26379: { } ,26380: { } ,26381: { } } } }
  vars:
    redis_cluster: redis-meta
    redis_password: 'redis.meta'
    redis_mode: sentinel
    redis_max_memory: 16MB
    redis_sentinel_monitor: # primary list for redis sentinel, use cls as name, primary ip:port
      - { name: redis-ms, host: 10.10.10.10, port: 6379 ,password: redis.ms, quorum: 2 }

redis-test: # redis native cluster: 3m x 3s
  hosts:
    10.10.10.12: { redis_node: 1 ,redis_instances: { 6379: { } ,6380: { } ,6381: { } } }
    10.10.10.13: { redis_node: 2 ,redis_instances: { 6379: { } ,6380: { } ,6381: { } } }
  vars: { redis_cluster: redis-test ,redis_password: 'redis.test' ,redis_mode: cluster, redis_max_memory: 32MB }

ETCD Cluster

Below is a declarative configuration example for a three-node Etcd cluster:

etcd: # dcs service for postgres/patroni ha consensus
  hosts:  # 1 node for testing, 3 or 5 for production
    10.10.10.10: { etcd_seq: 1 }  # etcd_seq required
    10.10.10.11: { etcd_seq: 2 }  # assign from 1 ~ n
    10.10.10.12: { etcd_seq: 3 }  # three-member cluster keeps an odd voter count
  vars: # cluster level parameter override roles/etcd
    etcd_cluster: etcd  # mark etcd cluster name etcd
    etcd_safeguard: false # safeguard against purging
    etcd_clean: true # purge etcd during init process

MINIO (Silo) Cluster

Below is a declarative configuration example for a three-node Silo cluster. The inventory group and parameters retain the MINIO module’s compatibility names:

minio:
  hosts:
    10.10.10.10: { minio_seq: 1 }
    10.10.10.11: { minio_seq: 2 }
    10.10.10.12: { minio_seq: 3 }
  vars:
    minio_cluster: minio
    minio_type: silo
    minio_data: '/data{1...2}'          # use two disks per node
    minio_node: '${minio_cluster}-${minio_seq}.pigsty' # node name pattern
    haproxy_services:
      - name: minio                     # [required] service name, must be unique
        port: 9002                      # [required] service port, must be unique
        options:
          - option httpchk
          - option http-keep-alive
          - http-check send meth OPTIONS uri /minio/health/live
          - http-check expect status 200
        servers:
          - { name: minio-1 ,ip: 10.10.10.10 , port: 9000 , options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
          - { name: minio-2 ,ip: 10.10.10.11 , port: 9000 , options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
          - { name: minio-3 ,ip: 10.10.10.12 , port: 9000 , options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }

3.3.1 - Inventory

Describe your infrastructure and clusters using declarative configuration files

Every Pigsty deployment corresponds to an Inventory that describes key properties of the infrastructure and database clusters.

Configuration File

Pigsty uses Ansible YAML configuration format by default, with a single YAML configuration file pigsty.yml as the inventory.

~/pigsty
  ^---- pigsty.yml   # <---- Default configuration file

You can directly edit this configuration file to customize your deployment, or use the configure wizard script provided by Pigsty to automatically generate an appropriate configuration file.

Configuration Structure

The inventory uses standard Ansible YAML configuration format, consisting of two parts: global parameters (all.vars) and multiple groups (all.children).

You can define new clusters in all.children and describe the infrastructure using global variables: all.vars, which looks like this:

all:                  # Top-level object: all
  vars: {...}         # Global parameters
  children:           # Group definitions
    infra:            # Group definition: 'infra'
      hosts: {...}        # Group members: 'infra'
      vars:  {...}        # Group parameters: 'infra'
    etcd:    {...}    # Group definition: 'etcd'
    pg-meta: {...}    # Group definition: 'pg-meta'
    pg-test: {...}    # Group definition: 'pg-test'
    redis-test: {...} # Group definition: 'redis-test'
    # ...

Cluster Definition

Each Ansible group may represent a cluster, which can be a node cluster, PostgreSQL cluster, Redis cluster, Etcd cluster, Silo cluster, etc.

A cluster definition consists of two parts: cluster members (hosts) and cluster parameters (vars). You can define cluster members in <cls>.hosts and describe the cluster using configuration parameters in <cls>.vars. Here’s an example of a 3-node high-availability PostgreSQL cluster definition:

all:
  children:    # Ansible group list
    pg-test:   # Ansible group name
      hosts:   # Ansible group instances (cluster members)
        10.10.10.11: { pg_seq: 1, pg_role: primary } # Host 1
        10.10.10.12: { pg_seq: 2, pg_role: replica } # Host 2
        10.10.10.13: { pg_seq: 3, pg_role: offline } # Host 3
      vars:    # Ansible group variables (cluster parameters)
        pg_cluster: pg-test

Cluster-level vars (cluster parameters) override global parameters, and instance-level vars override both cluster parameters and global parameters.

Splitting Configuration

If your deployment is large or you want to better organize configuration files, you can split the inventory into multiple files for easier management and maintenance.

inventory/
├── hosts.yml              # Host and cluster definitions
├── group_vars/
│   ├── all.yml            # Global default variables (corresponds to all.vars)
│   ├── infra.yml          # infra group variables
│   ├── etcd.yml           # etcd group variables
│   └── pg-meta.yml        # pg-meta cluster variables
└── host_vars/
    ├── 10.10.10.10.yml    # Specific host variables
    └── 10.10.10.11.yml

You can place cluster member definitions in the hosts.yml file and put cluster-level configuration parameters in corresponding files under the group_vars directory.

Switching Configuration

You can temporarily specify a different inventory file when running playbooks using the -i parameter.

./pgsql.yml -i another_config.yml
./infra.yml -i nginx_config.yml

Additionally, Ansible supports multiple configuration methods. You can use local yaml|ini configuration files, or use CMDB and any dynamic configuration scripts as configuration sources.

In Pigsty, we specify pigsty.yml in the same directory as the default inventory through ansible.cfg in the Pigsty home directory. You can modify it as needed.

[defaults]
inventory = pigsty.yml

Additionally, Pigsty supports using a CMDB metabase to store the inventory, facilitating integration with existing systems.

3.3.2 - Configure

Use the configure script to automatically generate recommended configuration files based on your environment.

Pigsty provides a configure script as a configuration wizard that automatically generates an appropriate pigsty.yml configuration file based on your current environment.

This is an optional script: if you already understand how to configure Pigsty, you can directly edit the pigsty.yml configuration file and skip the wizard.

Quick Start

Enter the pigsty source home directory and run ./configure to automatically start the configuration wizard. Without any arguments, it defaults to the meta single-node configuration template:

cd ~/pigsty
./configure          # Interactive configuration wizard, auto-detect environment and generate config

This command will use the selected template as a base, detect the current node’s IP address and region, and generate a pigsty.yml configuration file suitable for the current environment.

demo/configure.cast

Features

The configure script performs the following adjustments based on environment and input, generating pigsty.yml in the Pigsty directory by default.

  • Detects the current node IP address; if multiple IPs exist, prompts the user to input a primary IP address as the node’s identity
  • Uses the IP address to replace the placeholder 10.10.10.10 in the configuration template and sets it as the admin_ip parameter value
  • Detects the current region, setting region to default (global default repos) or china (using Chinese mirror repos)
  • For micro instances (vCPU < 4), uses the tiny parameter template for node_tune and pg_conf to optimize resource usage
  • If -v is specified, switches pg_version and pg18-* package-group aliases in the template to that major version; fixed-kernel templates mssql, polar, and pg19 are excluded from this replacement
  • If -g is specified, replaces default passwords recognized by the configuration wizard with randomly generated strong passwords; review uncovered values against the Default Credentials Checklist (strongly recommended)
  • When PG major version ≥ 17, prioritizes the built-in C.UTF-8 locale, or the OS-supported C.UTF-8
  • Checks if the core dependency ansible for deployment is available in the current environment
  • Also checks if the deployment target node is SSH-reachable and can execute commands with sudo (-s to skip)

Usage Examples 

# Basic usage
./configure                       # Interactive configuration wizard
./configure -i 10.10.10.10        # Specify primary IP address

# Specify configuration template
./configure -c meta               # Use default single-node template (default)
./configure -c rich               # Use feature-rich single-node template
./configure -c slim               # Use minimal template (PGSQL + ETCD only)
./configure -c ha/full            # Use 4-node HA sandbox template
./configure -c ha/trio            # Use 3-node HA template
./configure -c supabase           # Use Supabase self-hosted template
./configure -c app/immich         # Use Immich photo-management template

# Specify PostgreSQL version
./configure -v 18                 # Use PostgreSQL 18
./configure -v 16                 # Use PostgreSQL 16
./configure -c rich -v 15         # rich template + PG 15
./configure -c pg19               # Use the dedicated PostgreSQL 19 Beta template

# Region and proxy
./configure -r china              # Use Chinese mirrors
./configure -r europe             # Use European mirrors
./configure -x                    # Import current proxy environment variables

# Skip and automation
./configure -s                    # Skip IP detection, keep placeholder
./configure -n -i 10.10.10.10     # Non-interactive mode with specified IP
./configure -c ha/full -s         # 4-node template, skip IP replacement

# Security enhancement
./configure -g                    # Generate random passwords
./configure -c meta -g -i 10.10.10.10  # Complete production configuration

# Specify output and SSH port
./configure -o prod.yml           # Output to prod.yml
./configure -p 2222               # Use SSH port 2222

Command Arguments 

./configure
    [-c|--conf <template>]      # Configuration template name (meta|rich|slim|ha/full|...)
    [-i|--ip <ipaddr>]          # Specify primary IP address
    [-v|--version <pgver>]      # PostgreSQL major version (14|15|16|17|18|19)
    [-r|--region <region>]      # Upstream software repo region (default|china|europe)
    [-o|--output <file>]        # Output configuration file path (default: pigsty.yml)
    [-s|--skip]                 # Skip IP address detection and replacement
    [-x|--proxy]                # Import proxy settings from environment variables
    [-n|--non-interactive]      # Non-interactive mode (don't ask any questions)
    [-p|--port <port>]          # Specify SSH port
    [-g|--generate]             # Generate random passwords
    [-h|--help]                 # Display help information

Argument Details

Argument Description
-c, --conf Generate config from conf/<template>.yml, supports subdirectories like ha/full
-i, --ip Replace placeholder 10.10.10.10 in config template with specified IP
-v, --version Specify PostgreSQL major version (14-19); PG19 is Beta, so prefer the dedicated pg19 template
-r, --region Set software repo mirror region: default, china (Chinese mirrors), europe (European)
-o, --output Output path, default pigsty.yml; relative paths use Pigsty home, absolute paths are used as given
-s, --skip Skip IP probing, target SSH/Sudo checks, and effective IP replacement; keep 10.10.10.10
-x, --proxy Write current environment proxy variables (HTTP_PROXY, HTTPS_PROXY, ALL_PROXY, NO_PROXY) to config
-n, --non-interactive Non-interactive mode; a single/demo IP is auto-selected, while ambiguous multi-IP hosts require -i
-p, --port SSH port used by readiness checks only; it does not write ansible_port into the generated config
-g, --generate Generate random values for passwords in config file, improving security (strongly recommended)

Execution Flow

The configure script executes detection and configuration in the following order:

┌─────────────────────────────────────────────────────────────┐
│                  configure Execution Flow                   │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  1. check_region          Detect network region (GFW check) │
│         ↓                                                   │
│  2. check_version         Validate PostgreSQL version       │
│         ↓                                                   │
│  3. check_kernel          Detect OS kernel (Linux/Darwin)   │
│         ↓                                                   │
│  4. check_machine         Detect CPU arch (x86_64/aarch64)  │
│         ↓                                                   │
│  5. check_package_manager Detect package manager (dnf/yum/apt) │
│         ↓                                                   │
│  6. check_vendor_version  Detect OS distro and version      │
│         ↓                                                   │
│  7. check_sudo            Detect passwordless sudo          │
│         ↓                                                   │
│  8. check_ssh             Detect passwordless SSH to self   │
│         ↓                                                   │
│  9. check_proxy_env       Handle proxy environment vars     │
│         ↓                                                   │
│ 10. check_ipaddr          Detect/input primary IP address   │
│         ↓                                                   │
│ 11. check_admin           Validate admin SSH + Sudo access  │
│         ↓                                                   │
│ 12. check_conf            Select configuration template     │
│         ↓                                                   │
│ 13. check_config          Generate configuration file       │
│         ↓                                                   │
│ 14. check_utils           Check if Ansible etc. installed   │
│         ↓                                                   │
│     ✓ Configuration complete, output pigsty.yml             │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Automatic Behaviors

Region Detection

The script automatically detects the network environment to determine if you’re in mainland China (behind GFW):

# The actual probe uses HTTPS with a two-second total timeout
curl -I -s --max-time 2 https://www.google.com
  • If Google is reachable, uses the region: default repositories
  • If Google is unreachable but https://pigsty.cc is reachable, sets region: china
  • If neither endpoint is reachable, falls back to region: default and emits an internet-unreachable warning
  • Can manually specify region via -r argument

IP Address Handling

The script determines the primary IP address in the following priority:

  1. Command line argument: If IP is specified via -i, use it directly
  2. Single IP detection: If the current node has only one IP, use it automatically
  3. Demo IP detection: If 10.10.10.10 is detected, select it automatically (for sandbox environments)
  4. Interactive input: When multiple IPs exist, prompt user to choose or input
[WARN] Multiple IP address candidates found:
    (1) 192.168.1.100   inet 192.168.1.100/24 scope global eth0
    (2) 10.10.10.10     inet 10.10.10.10/24 scope global eth1
[ IN ] INPUT primary_ip address (of current meta node, e.g 10.10.10.10):
=> 10.10.10.10

Low-End Hardware Optimization

When fewer than 4 CPU cores are detected (1-3 cores), the script automatically adjusts configuration:

[WARN] replace oltp template with tiny due to cpu < 4

This ensures smooth operation on low-spec virtual machines.

Locale Settings

The script automatically enables C.UTF-8 as the default locale when:

  • PostgreSQL version ≥ 17 (built-in Locale Provider support)
  • Or the current system supports C.UTF-8 / C.utf8 locale
pg_locale: C.UTF-8
pg_lc_collate: C.UTF-8
pg_lc_ctype: C.UTF-8

China Region Special Handling

When region is set to china, the script automatically:

  • Enables docker_registry_mirrors Docker mirror acceleration
  • Enables PIP_MIRROR_URL Python mirror acceleration

Password Generation

When using the -g argument, the script generates 24-character random strings for the following passwords:

Password Parameter Description
grafana_admin_password Grafana admin password
pg_admin_password PostgreSQL admin password
pg_monitor_password PostgreSQL monitor user password
pg_replication_password PostgreSQL replication user password
patroni_password Patroni API password
haproxy_admin_password HAProxy admin password
minio_secret_key Silo Root Secret
etcd_root_password ETCD Root password

It also replaces the following placeholder passwords:

  • DBUser.Meta → random password
  • DBUser.Viewer → random password
  • S3User.Backup → random password
  • S3User.Meta → random password
  • S3User.Data → random password
  • DBUser.Supa → random password
  • Vibe.Coding → random password
$ ./configure -g
[INFO] generating random passwords...
    grafana_admin_password   : xK9mL2nP4qR7sT1vW3yZ5bD8
    pg_admin_password        : aB3cD5eF7gH9iJ1kL2mN4oP6
    ...
[INFO] random passwords generated, check and save them

Configuration Templates

The script reads templates from conf/. The value of -c is a path relative to that directory without the .yml suffix, such as ha/full or app/immich.

Core Templates

Template Description
meta Default template: Single-node installation with INFRA + NODE + ETCD + PGSQL
rich Feature-rich version: Includes almost all extensions, Silo, local repo
slim Minimal version: PostgreSQL + ETCD only, no monitoring infrastructure
fat Complete version: rich base with more extensions installed
pgsql Pure PostgreSQL template
pg19 Single-node PostgreSQL 19 Beta evaluation template
infra Pure infrastructure template

HA Templates (ha/)

Template Description
ha/dual 2-node HA cluster
ha/trio 3-node HA cluster
ha/full 4-node complete sandbox environment
ha/safe Security-hardened HA configuration
ha/octo Compact 8-node HA simulation
ha/simu 20-node production simulation environment
ha/citus 13-node Citus distributed cluster

Application Templates

Template Description
supabase Supabase self-hosted configuration
app/dify Dify AI platform configuration
app/odoo Odoo ERP configuration
app/electric Electric sync engine configuration
app/insforge Insforge backend platform configuration
app/hindsight Hindsight application configuration
app/teable Teable table database configuration
app/mattermost Mattermost collaboration platform configuration
app/maybe Maybe finance application configuration
app/registry Docker Registry configuration
app/immich Immich photo and video management
app/jumpserver JumpServer bastion host

Special Kernel Templates

Template Description
ivory IvorySQL: Oracle-compatible PostgreSQL
mssql Babelfish: SQL Server-compatible PostgreSQL
polar PolarDB: Alibaba Cloud open-source distributed PostgreSQL
ha/citus Citus: Distributed PostgreSQL HA cluster
mysql OpenHalo: MySQL protocol-compatible PostgreSQL
pgtde Percona PostgreSQL Server: transparent encryption
oriole OrioleDB: Next-generation storage engine
agens AgensGraph: graph database kernel
pgedge pgEdge: distributed PostgreSQL kernel
mongo MongoDB-compatible stack template

Demo and Build Templates

Template Description
vibe Vibe Coding development environment
docker Run Pigsty inside a Docker container
demo/bare Minimal readable single-node example
demo/el Full parameter example for EL distributions
demo/debian Full parameter example for Debian/Ubuntu
demo/demo Multi-module demo environment
demo/kernel Ten-node database-kernel matrix
demo/redis Redis replica, Sentinel, and native Cluster demo
demo/minio Multi-node, multi-drive Silo demo (source default)
demo/kafka Kafka KRaft development and secure-cluster demo
demo/mysql Native MySQL 8.4 pilot demo
demo/remote Remote PostgreSQL/RDS monitoring example
demo/saas Legacy single-node SaaS component bundle
demo/wool Small cloud-instance example for China
build/oss Cross-distribution open-source package build env
build/dev Three-node development and build environment

Output Example 

$ ./configure
configure pigsty v4.5.0 begin
[ OK ] region = china
[ OK ] kernel  = Linux
[ OK ] machine = x86_64
[ OK ] package = rpm,dnf
[ OK ] vendor  = rocky (Rocky Linux)
[ OK ] version = 9 (9.5)
[ OK ] sudo = vagrant ok
[ OK ] ssh = [email protected] ok
[WARN] Multiple IP address candidates found:
    (1) 192.168.121.193	    inet 192.168.121.193/24 brd 192.168.121.255 scope global dynamic noprefixroute eth0
    (2) 10.10.10.10	    inet 10.10.10.10/24 brd 10.10.10.255 scope global noprefixroute eth1
[ OK ] primary_ip = 10.10.10.10 (from demo)
[ OK ] admin = [email protected] ok
[ OK ] mode = meta (el9)
[ OK ] locale  = C.UTF-8
[ OK ] ansible = ready
[ OK ] pigsty configured
[WARN] don't forget to check it and change passwords!
proceed with ./deploy.yml

Environment Variables

The script supports the following environment variables:

Environment Variable Description Default
PIGSTY_HOME Pigsty installation directory ~/pigsty
METADB_URL Metabase connection URL service=meta
HTTP_PROXY HTTP proxy -
HTTPS_PROXY HTTPS proxy -
ALL_PROXY Universal proxy -
NO_PROXY Proxy whitelist Built-in default

Notes

  1. Passwordless access: Before running configure, ensure the current user has passwordless sudo privileges and passwordless SSH to localhost. This can be automatically configured via the bootstrap script.

  2. IP address selection: Choose an internal IP as the primary IP address, not a public IP or 127.0.0.1.

  3. Password security: In production, always change default passwords in the configuration file. Use -g to randomize recognized credentials, then review the Default Credentials Checklist for remaining values.

  4. Configuration review: After the script completes, it’s recommended to review the generated pigsty.yml file to confirm the configuration meets expectations.

  5. Multiple executions: You can run configure multiple times to regenerate configuration; each run will overwrite the existing pigsty.yml.

  6. macOS limitations: When running on macOS, the script skips some Linux-specific checks and uses placeholder IP 10.10.10.10. macOS can only serve as an admin node.

FAQ

How to use a custom configuration template?

Place your configuration file in the conf/ directory, then specify it with the -c argument:

cp my-config.yml ~/pigsty/conf/myconf.yml
./configure -c myconf

How to generate different configurations for multiple clusters?

Use the -o argument to specify different output files:

./configure -c ha/full -o cluster-a.yml
./configure -c ha/trio -o cluster-b.yml

Then specify the configuration file when running playbooks:

./deploy.yml -i cluster-a.yml

How to handle multiple IPs in non-interactive mode?

You must explicitly specify the IP address using the -i argument:

./configure -n -i 10.10.10.10

How to keep the placeholder IP in the template?

Use the -s argument to skip IP replacement:

./configure -c ha/full -s   # Keep 10.10.10.10 placeholder

Related Documentation

  • Inventory: Understand the Ansible inventory structure
  • Parameters: Understand Pigsty parameter hierarchy and priority
  • Templates: View all available configuration templates
  • Installation: Understand the complete installation process
  • Metabase: Use PostgreSQL as a dynamic configuration source

3.3.3 - Parameters

Fine-tune Pigsty customization using configuration parameters

In the inventory, you can use various parameters to fine-tune Pigsty customization. These parameters cover everything from infrastructure settings to database configuration.

Parameter List

According to the current source and parameter reference pages, Pigsty’s 10 official modules expose 373 public parameters for fine-grained control. See Reference - Parameter List for the complete list. The native MySQL 8.4 pilot module exposes 13 additional public parameters that are listed separately and excluded from this total.

Module Groups Params Description
PGSQL 9 124 PostgreSQL high-availability cluster configuration
INFRA 10 73 Software repositories and Victoria observability infrastructure
NODE 11 73 Node initialization, system tuning, and operations baseline
ETCD 2 13 ETCD cluster and removal protection parameters
MINIO 2 22 Silo deployment, observability, and removal parameters
REDIS 2 22 Redis/Valkey deployment and removal parameters
DOCKER 1 8 Docker engine parameters
JUICE 1 2 JuiceFS instance and cache parameters
VIBE 1 18 Code/Jupyter/Node.js/Claude/Codex configuration
KAFKA 2 18 Kafka deployment and removal-protection parameters

Parameter Form

Parameters are key-value pairs that describe entities. The Key is a string, and the Value can be one of five types: boolean, string, number, array, or object.

all:                            # <------- Top-level object: all
  vars:
    admin_ip: 10.10.10.10       # <------- Global configuration parameter
  children:
    pg-meta:                    # <------- pg-meta group
      vars:
        pg_cluster: pg-meta     # <------- Cluster-level parameter
      hosts:
        10.10.10.10:            # <------- Host node IP
          pg_seq: 1
          pg_role: primary      # <------- Instance-level parameter

Parameter Priority

Parameters can be set at different levels with the following priority:

Level Location Description Priority
CLI -e command line argument Passed via command line Highest (5)
Host/Instance <group>.hosts.<host> Parameters specific to a single host Higher (4)
Group/Cluster <group>.vars Parameters shared by hosts in group/cluster Medium (3)
Global all.vars Parameters shared by all hosts Lower (2)
Default <roles>/default/main.yml Role implementation defaults Lowest (1)

Here are some examples of parameter priority:

  • Use command line parameter -e grafana_clean=true when running playbooks to wipe Grafana data
  • Use instance-level parameter pg_role on host variables to override pg instance role
  • Use cluster-level parameter pg_cluster on group variables to override pg cluster name
  • Use global parameter node_ntp_servers on global variables to specify global NTP servers
  • If pg_version is not set, Pigsty will use the default value from the pgsql role implementation (default is 18)

Except for identity parameters, every parameter has an appropriate default value, so explicit setting is not required.

Identity Parameters

Identity parameters are special parameters that serve as entity ID identifiers, therefore they have no default values and must be explicitly set.

Module Identity Parameters
PGSQL pg_cluster, pg_seq, pg_role, …
NODE nodename, node_cluster
ETCD etcd_cluster, etcd_seq
MINIO minio_cluster, minio_seq
REDIS redis_cluster, redis_node, redis_instances
INFRA infra_seq

The exception is etcd_cluster, which still defaults to etcd. Object storage minio_cluster no longer has a default and must be defined explicitly in each object-storage cluster’s variables. Do not place it in all.vars, or every host will be marked as a MINIO module member.

3.3.4 - Conf Templates

Use pre-made configuration templates to quickly generate configuration files adapted to your environment

In Pigsty, deployment blueprint details are defined by the inventory, which is the pigsty.yml configuration file. You can customize it through declarative configuration.

However, writing configuration files directly can be daunting for new users. To address this, we provide some ready-to-use configuration templates covering common usage scenarios.

Each template is a predefined pigsty.yml configuration file containing reasonable defaults suitable for specific scenarios.

You can choose a template as your customization starting point, then modify it as needed to meet your specific requirements.

Using Templates

Pigsty provides the configure script as an optional configuration wizard that generates an inventory with good defaults based on your environment and input.

Use ./configure -c <conf> to specify a configuration template, where <conf> is the path relative to the conf directory (the .yml suffix can be omitted).

./configure                     # Default to meta.yml configuration template
./configure -c meta             # Explicitly specify meta.yml single-node template
./configure -c rich             # Use feature-rich template with all extensions and Silo
./configure -c slim             # Use minimal single-node template

# Use different database kernels
./configure -c pgsql            # Native PostgreSQL kernel, basic features (14~18)
./configure -c pg19             # PostgreSQL 19 Beta trial template
./configure -c mssql            # Babelfish kernel, SQL Server protocol compatible (17/18)
./configure -c polar            # PolarDB PG kernel, Aurora/RAC style (17)
./configure -c ivory            # IvorySQL kernel, Oracle syntax compatible (18)
./configure -c mysql            # OpenHalo kernel, MySQL compatible (14)
./configure -c pgtde            # Percona PostgreSQL Server transparent encryption (18)
./configure -c oriole           # OrioleDB kernel, OLTP enhanced (16~18)
./configure -c agens            # AgensGraph graph database kernel (17)
./configure -c pgedge           # pgEdge distributed database kernel (15~18, default 18)
./configure -c ha/citus         # Citus distributed HA PostgreSQL (14~18)
./configure -c supabase         # Supabase self-hosted configuration (15~18)

# Use multi-node HA templates
./configure -c ha/dual          # Use 2-node HA template
./configure -c ha/trio          # Use 3-node HA template
./configure -c ha/full          # Use 4-node HA template

If no template is specified, Pigsty defaults to the meta.yml single-node configuration template.

Template List

Main Templates

The following are single-node configuration templates for installing Pigsty on a single server:

Template Description
meta.yml Default template, single-node PostgreSQL online installation
rich.yml Feature-rich template with local repo, Silo, and more examples
slim.yml Minimal template, PostgreSQL only without monitoring and infrastructure

Database Kernel Templates

Templates for various database management systems and kernels:

Template Description
pgsql.yml Native PostgreSQL kernel, basic features (14~18)
pg19.yml PostgreSQL 19 Beta trial template
mssql.yml Babelfish kernel, SQL Server protocol compatible (17/18)
polar.yml PolarDB PG kernel, Aurora/RAC style (17)
ivory.yml IvorySQL kernel, Oracle syntax compatible (18)
mysql.yml OpenHalo kernel, MySQL compatible (14)
pgtde.yml Percona PostgreSQL Server transparent encryption (18)
oriole.yml OrioleDB kernel, OLTP enhanced (16~18)
agens.yml AgensGraph graph database kernel (17)
pgedge.yml pgEdge distributed database kernel (15~18, default 18)
supabase.yml Supabase self-hosted configuration (15~18)

You can add more nodes later or use HA templates to plan your cluster from the start.

HA Templates

You can configure Pigsty to run on multiple nodes, forming a high-availability (HA) cluster:

Template Description
dual.yml 2-node semi-HA deployment
trio.yml 3-node standard HA deployment
full.yml 4-node standard deployment
safe.yml 4-node security-enhanced deployment with delayed replica
octo.yml Compact 8-node HA simulation
simu.yml 20-node production environment simulation
ha/citus.yml Citus distributed HA PostgreSQL (14~18)

Application Templates

You can use the following templates to run Docker applications/software:

Template Description
supabase.yml Start single-node Supabase
odoo.yml Start Odoo ERP system
dify.yml Start Dify AI workflow system
electric.yml Start Electric sync engine
insforge.yml Start Insforge backend platform
hindsight.yml Start Hindsight application
mattermost.yml Start Mattermost collaboration platform
teable.yml Start Teable spreadsheet database
maybe.yml Start Maybe finance app
registry.yml Start Docker Registry

Demo Templates

Besides main templates, Pigsty provides a set of demo templates for different scenarios:

Template Description
el.yml Full-parameter config file for EL 8/9 systems
debian.yml Full-parameter config file for Debian/Ubuntu systems
remote.yml Example config for monitoring remote PostgreSQL clusters or RDS
redis.yml Redis cluster example configuration
minio.yml 4-node multi-drive Silo cluster example (source default)
kafka.yml Kafka dynamic KRaft example with a single-node dev cluster and a three-node secure cluster
mysql.yml Native MySQL 8.4 single-node/three-node pilot example; distinct from OpenHalo conf/mysql.yml
demo.yml Configuration file for Pigsty public demo site
fat.yml Single-node config with local repo and full feature set
infra.yml Deploy only the infrastructure modules
vibe.yml Vibe Coding / AI application development template
mongo.yml FerretDB / MongoDB-compatible example
docker.yml Docker application host template

Build Templates

The following configuration templates are for development and testing purposes:

Template Description
build/oss.yml Open source build config for EL 9/10, Debian 12/13, Ubuntu 22.04/24.04/26.04
build/dev.yml Development and testing build config

3.3.5 - Use CMDB as Config Inventory

Use PostgreSQL as a CMDB metabase to store Ansible inventory.

Pigsty allows you to use a PostgreSQL metabase as a dynamic configuration source, replacing static YAML configuration files for more powerful configuration management capabilities.

Overview

CMDB (Configuration Management Database) is a method of storing configuration information in a database for management.

In Pigsty, the default configuration source is a static YAML file pigsty.yml, which serves as Ansible’s inventory.

This approach is simple and direct, but when infrastructure scales and requires complex, fine-grained management and external integration, a single static file becomes insufficient.

Feature Static YAML File CMDB Metabase
Querying Manual search/grep SQL queries with any conditions, aggregation analysis
Versioning Depends on Git or manual backup Database transactions, audit logs, time-travel snapshots
Access Control File system permissions, coarse-grained PostgreSQL fine-grained access control
Concurrent Editing Requires file locking or merge conflicts Database transactions naturally support concurrency
External Integration Requires YAML parsing Standard SQL interface, easy integration with any language
Scalability Difficult to maintain when file becomes too large Scales to physical limits
Dynamic Generation Static file, changes require manual application Immediate effect, real-time configuration changes

Pigsty provides the CMDB database schema in the sample database pg-meta.meta schema baseline definition.

How It Works

The core idea of CMDB is to replace the static configuration file with a dynamic script. Ansible supports using executable scripts as inventory, as long as the script outputs inventory data in JSON format. When you enable CMDB, Pigsty creates a dynamic inventory script named inventory.sh:

#!/bin/bash
psql ${METADB_URL} -AXtwc 'SELECT text FROM pigsty.inventory;'

This script’s function is simple: every time Ansible needs to read the inventory, it queries configuration data from the PostgreSQL database’s pigsty.inventory view and returns it in JSON format.

The overall architecture is as follows:

flowchart LR
    conf["bin/inventory_conf"]
    tocmdb["bin/inventory_cmdb"]
    load["bin/inventory_load"]
    ansible["🚀 Ansible"]

    subgraph static["📄 Static Config Mode"]
        yml[("pigsty.yml")]
    end

    subgraph dynamic["🗄️ CMDB Dynamic Mode"]
        sh["inventory.sh"]
        cmdb[("PostgreSQL CMDB")]
    end

    conf -->|"switch"| yml
    yml -->|"load config"| load
    load -->|"write"| cmdb
    tocmdb -->|"switch"| sh
    sh --> cmdb

    yml --> ansible
    cmdb --> ansible

Data Model

The CMDB database schema is defined in files/cmdb.sql, with all objects in the pigsty schema.

Core Tables

Table Description Primary Key
pigsty.group Cluster/group definitions, corresponds to Ansible groups cls
pigsty.host Host definitions, belongs to a group (cls, ip)
pigsty.global_var Global variables, corresponds to all.vars key
pigsty.group_var Group variables, corresponds to all.children.<cls>.vars (cls, key)
pigsty.host_var Host variables, host-level variables (cls, ip, key)
pigsty.default_var Default variable definitions, stores parameter metadata key
pigsty.job Job records table, records executed tasks id

Table Structure Details

Cluster Table pigsty.group

CREATE TABLE pigsty.group (
    cls     TEXT PRIMARY KEY,        -- Cluster name, primary key
    ctime   TIMESTAMPTZ DEFAULT now(), -- Creation time
    mtime   TIMESTAMPTZ DEFAULT now()  -- Modification time
);

Host Table pigsty.host

CREATE TABLE pigsty.host (
    cls    TEXT NOT NULL REFERENCES pigsty.group(cls),  -- Parent cluster
    ip     INET NOT NULL,                               -- Host IP address
    ctime  TIMESTAMPTZ DEFAULT now(),
    mtime  TIMESTAMPTZ DEFAULT now(),
    PRIMARY KEY (cls, ip)
);

Global Variables Table pigsty.global_var

CREATE TABLE pigsty.global_var (
    key   TEXT PRIMARY KEY,           -- Variable name
    value JSONB NULL,                 -- Variable value (JSON format)
    mtime TIMESTAMPTZ DEFAULT now()   -- Modification time
);

Group Variables Table pigsty.group_var

CREATE TABLE pigsty.group_var (
    cls   TEXT NOT NULL REFERENCES pigsty.group(cls),
    key   TEXT NOT NULL,
    value JSONB NULL,
    mtime TIMESTAMPTZ DEFAULT now(),
    PRIMARY KEY (cls, key)
);

Host Variables Table pigsty.host_var

CREATE TABLE pigsty.host_var (
    cls   TEXT NOT NULL,
    ip    INET NOT NULL,
    key   TEXT NOT NULL,
    value JSONB NULL,
    mtime TIMESTAMPTZ DEFAULT now(),
    PRIMARY KEY (cls, ip, key),
    FOREIGN KEY (cls, ip) REFERENCES pigsty.host(cls, ip)
);

Core Views

CMDB provides a series of views for querying and displaying configuration data:

View Description
pigsty.inventory Core view: Generates Ansible dynamic inventory JSON
pigsty.raw_config Raw configuration in JSON format
pigsty.global_config Global config view, merges defaults and global vars
pigsty.group_config Group config view, includes host list and group vars
pigsty.host_config Host config view, merges group and host-level vars
pigsty.pg_cluster PostgreSQL cluster view
pigsty.pg_instance PostgreSQL instance view
pigsty.pg_database PostgreSQL database definition view
pigsty.pg_users PostgreSQL user definition view
pigsty.pg_service PostgreSQL service definition view
pigsty.pg_hba PostgreSQL HBA rules view
pigsty.pg_remote Remote PostgreSQL instance view

pigsty.inventory is the core view that converts database configuration data to the JSON format required by Ansible:

SELECT text FROM pigsty.inventory;

Utility Scripts

Pigsty provides three convenience scripts for managing CMDB:

Script Function
bin/inventory_load Load YAML configuration file into PostgreSQL database
bin/inventory_cmdb Switch configuration source to CMDB (dynamic inventory script)
bin/inventory_conf Switch configuration source to static config file pigsty.yml

inventory_load

Parse and import YAML configuration file into CMDB:

bin/inventory_load                     # Load default pigsty.yml to default CMDB
bin/inventory_load -p /path/to/conf.yml  # Specify configuration file path
bin/inventory_load -d "postgres://..."   # Specify database connection URL
bin/inventory_load -n myconfig           # Specify configuration name

The script performs the following operations:

  1. Clears existing data in the pigsty schema
  2. Parses the YAML configuration file
  3. Writes global variables to the global_var table
  4. Writes cluster definitions to the group table
  5. Writes cluster variables to the group_var table
  6. Writes host definitions to the host table
  7. Writes host variables to the host_var table

Environment Variables

  • PIGSTY_HOME: Pigsty installation directory, defaults to ~/pigsty
  • METADB_URL: Database connection URL, defaults to service=meta

inventory_cmdb

Switch Ansible to use CMDB as the configuration source:

bin/inventory_cmdb

The script performs the following operations:

  1. Creates dynamic inventory script ${PIGSTY_HOME}/inventory.sh
  2. Modifies ansible.cfg to set inventory to inventory.sh

The generated inventory.sh contents:

#!/bin/bash
psql ${METADB_URL} -AXtwc 'SELECT text FROM pigsty.inventory;'

inventory_conf

Switch back to using static YAML configuration file:

bin/inventory_conf

The script modifies ansible.cfg to set inventory back to pigsty.yml.

Usage Workflow

First-time CMDB Setup

  1. Initialize CMDB schema (usually done automatically during Pigsty installation):
psql -f ~/pigsty/files/cmdb.sql
  1. Load configuration to database:
bin/inventory_load
  1. Switch to CMDB mode:
bin/inventory_cmdb
  1. Verify configuration:
ansible all --list-hosts          # List all hosts
ansible-inventory --list          # View complete inventory

Query Configuration

After enabling CMDB, you can flexibly query configuration using SQL:

-- View all clusters
SELECT cls FROM pigsty.group;

-- View all hosts in a cluster
SELECT ip FROM pigsty.host WHERE cls = 'pg-meta';

-- View global variables
SELECT key, value FROM pigsty.global_var;

-- View cluster variables
SELECT key, value FROM pigsty.group_var WHERE cls = 'pg-meta';

-- View all PostgreSQL clusters
SELECT cls, name, pg_databases, pg_users FROM pigsty.pg_cluster;

-- View all PostgreSQL instances
SELECT cls, ins, ip, seq, role FROM pigsty.pg_instance;

-- View all database definitions
SELECT cls, datname, owner, encoding FROM pigsty.pg_database;

-- View all user definitions
SELECT cls, name, login, superuser FROM pigsty.pg_users;

Modify Configuration

You can modify configuration directly via SQL:

-- Add new cluster
INSERT INTO pigsty.group (cls) VALUES ('pg-new');

-- Add cluster variable
INSERT INTO pigsty.group_var (cls, key, value)
VALUES ('pg-new', 'pg_cluster', '"pg-new"');

-- Add host
INSERT INTO pigsty.host (cls, ip) VALUES ('pg-new', '10.10.10.20');

-- Add host variables
INSERT INTO pigsty.host_var (cls, ip, key, value)
VALUES ('pg-new', '10.10.10.20', 'pg_seq', '1'),
       ('pg-new', '10.10.10.20', 'pg_role', '"primary"');

-- Modify global variable
UPDATE pigsty.global_var SET value = '"new-value"' WHERE key = 'some_param';

-- Delete cluster (cascades to hosts and variables)
DELETE FROM pigsty.group WHERE cls = 'pg-old';

Changes take effect immediately without reloading or restarting any service.

Switch Back to Static Configuration

To switch back to static configuration file mode:

bin/inventory_conf

Advanced Usage

Export Configuration

Export CMDB configuration to YAML format:

psql service=meta -AXtwc "SELECT jsonb_pretty(jsonb_build_object('all', jsonb_build_object('children', children, 'vars', vars))) FROM pigsty.raw_config;"

Or use the ansible-inventory command:

ansible-inventory --list --yaml > exported_config.yml

Configuration Auditing

Track configuration changes using the mtime field:

-- View recently modified global variables
SELECT key, value, mtime FROM pigsty.global_var
ORDER BY mtime DESC LIMIT 10;

-- View changes after a specific time
SELECT * FROM pigsty.group_var
WHERE mtime > '2024-01-01'::timestamptz;

Integration with External Systems

CMDB uses standard PostgreSQL, making it easy to integrate with other systems:

  • Web Management Interface: Expose configuration data through REST API (e.g., PostgREST)
  • CI/CD Pipelines: Read/write database directly in deployment scripts
  • Monitoring & Alerting: Generate monitoring rules based on configuration data
  • ITSM Systems: Sync with enterprise CMDB systems

Considerations

  1. Data Consistency: After modifying configuration, you need to re-run the corresponding Ansible playbooks to apply changes to the actual environment

  2. Backup: Configuration data in CMDB is critical, ensure regular backups

  3. Permissions: Configure appropriate database access permissions for CMDB to avoid accidental modifications

  4. Transactions: When making batch configuration changes, perform them within a transaction for rollback on errors

  5. Connection Pooling: The inventory.sh script creates a new connection on each execution; if Ansible runs frequently, consider using connection pooling

Summary

CMDB is Pigsty’s advanced configuration management solution, suitable for scenarios requiring large-scale cluster management, complex queries, external integration, or fine-grained access control. By storing configuration data in PostgreSQL, you can fully leverage the database’s powerful capabilities to manage infrastructure configuration.

Feature Description
Storage PostgreSQL pigsty schema
Dynamic Inventory inventory.sh script
Config Load bin/inventory_load
Switch to CMDB bin/inventory_cmdb
Switch to YAML bin/inventory_conf
Core View pigsty.inventory

3.4 - High Availability

Pigsty uses Patroni to implement PostgreSQL high availability, ensuring automatic failover when the primary becomes unavailable.

Overview

Pigsty’s PostgreSQL clusters come with out-of-the-box high availability, with core capabilities provided by Patroni, Etcd, and HAProxy.

When your PostgreSQL cluster has two or more instances, you automatically have self-healing database high availability without any additional configuration — as long as any instance in the cluster survives, the cluster can provide complete service. Clients only need to connect to any node in the cluster to get full service without worrying about primary-replica topology changes.

The default norm mode targets an RTO under 45 seconds. With asynchronous replication, pg_rpo=1MiB is Patroni’s sampled lag threshold for failover candidates, not a hard upper bound on actual data loss. Strict synchronous mode with crit.yml keeps acknowledged transactions at RPO = 0 during failover. These behaviors can be configured for your hardware and reliability requirements.

Pigsty includes built-in HAProxy load balancers for automatic traffic switching, providing DNS/VIP/LVS and other access methods for clients. Failover and switchover are almost transparent to the business side except for brief interruptions - applications don’t need to modify connection strings or restart. The minimal maintenance window requirements bring great flexibility and convenience: you can perform rolling maintenance and upgrades on the entire cluster without application coordination. The feature that hardware failures can wait until the next day to handle lets developers, operations, and DBAs sleep well during incidents.

pigsty-ha

Many large organizations and core institutions have been using Pigsty in production for extended periods. The largest deployment has 25K CPU cores and 220+ PostgreSQL ultra-large instances (64c / 512g / 3TB NVMe SSD). In this deployment case, dozens of hardware failures and various incidents occurred over five years, yet overall availability of over 99.999% was maintained.

What problems does High Availability solve?

  • Elevates availability in the data security C/IA model: RPO ≈ 0, RTO < 45s.
  • Gains seamless rolling maintenance capability, minimizing maintenance window requirements and bringing great convenience.
  • Hardware failures can self-heal immediately without human intervention, allowing operations and DBAs to sleep well.
  • Replicas can handle read-only requests, offloading primary load and fully utilizing resources.

What are the costs of High Availability?

  • Infrastructure dependency: HA requires DCS (etcd/zk/consul) for consensus.
  • Higher starting threshold: A meaningful HA deployment requires at least three nodes.
  • Extra resource consumption: Each new replica consumes additional resources, though this is usually not a major concern.
  • Significantly increased complexity: Backup costs increase significantly, requiring tools to manage complexity.

Limitations of High Availability

Since replication happens in real-time, all changes are immediately applied to replicas. Therefore, streaming replication-based HA solutions cannot handle data deletion or modification caused by human errors and software defects. (e.g., DROP TABLE or DELETE data) Such failures require using delayed clusters or performing point-in-time recovery using previous base backups and WAL archives.

Configuration Strategy RTO RPO
Standalone + Nothing Data permanently lost, unrecoverable All data lost
Standalone + Base Backup Depends on backup size and bandwidth (hours) Lose data since last backup (hours to days)
Standalone + Base Backup + WAL Archive Depends on backup size and bandwidth (hours) Lose unarchived data (tens of MB)
Primary-Replica + Manual Failover ~10 minutes Lose data in replication lag (~100KB)
Primary-Replica + Auto Failover Within 1 minute Lose data in replication lag (~100KB)
Primary-Replica + Auto Failover + Sync Commit Within 1 minute No data loss

How It Works

In Pigsty, the high availability architecture works as follows:

  • PostgreSQL uses standard streaming replication to build physical replicas; replicas take over when the primary fails.
  • Patroni manages PostgreSQL server processes and handles high availability matters.
  • Etcd provides distributed configuration storage (DCS) capability and is used for leader election after failures.
  • Patroni relies on Etcd to reach cluster leader consensus and provides health check interfaces externally.
  • HAProxy exposes cluster services externally and uses Patroni health check interfaces to automatically distribute traffic to healthy nodes.
  • vip-manager provides an optional Layer 2 VIP, retrieves leader information from Etcd, and binds the VIP to the node where the cluster primary resides.

When the primary fails, a new round of leader election is triggered. The healthiest replica in the cluster (highest LSN position, minimum data loss) wins and is promoted to the new primary. After the winning replica is promoted, read-write traffic is immediately routed to the new primary. The impact of primary failure is brief write service unavailability: write requests will be blocked or fail directly from primary failure until new primary promotion, with unavailability typically lasting 15 to 30 seconds, usually not exceeding 1 minute.

When a replica fails, read-only traffic is routed to other replicas. Only when all replicas fail will read-only traffic ultimately be handled by the primary. The impact of replica failure is partial read-only query interruption: queries currently running on that replica will abort due to connection reset and be immediately taken over by other available replicas.

Failure detection is performed jointly by Patroni and Etcd. The cluster leader holds a lease; if it fails to renew the lease within its TTL (30 seconds in the default norm mode), the lease expires, triggering a Failover and a new election.

Even without any failures, you can proactively change the cluster primary through Switchover. In this case, write queries on the primary will experience a brief interruption and be immediately routed to the new primary. This operation is typically used for rolling maintenance/upgrades of database servers.

3.4.1 - RPO Trade-offs

Trade-off analysis for RPO (Recovery Point Objective), finding the optimal balance between availability and data loss.

RPO (Recovery Point Objective) defines the maximum amount of data loss allowed when the primary fails.

For scenarios where data integrity is critical, such as financial transactions, RPO = 0 is typically required, meaning no data loss is allowed.

However, stricter RPO targets come at a cost: higher write latency, reduced system throughput, and the risk that replica failures may cause primary unavailability. For typical scenarios, some data loss is acceptable in exchange for higher availability and performance.

Trade-offs

In asynchronous replication scenarios, there is typically some replication lag between replicas and the primary (depending on network and throughput, normally in the range of 10KB-100KB / 100µs-10ms). This means when the primary fails, replicas may not have fully synchronized with the latest data. If a failover occurs, the new primary may lose some unreplicated data.

The pg_rpo parameter is written to Patroni’s maximum_lag_on_failover and defaults to 1048576 (1MiB). It is the sampled lag threshold that permits a replica to participate as a failover candidate, not a hard upper bound on actual data loss.

When the cluster primary fails, if any replica has replication lag within this threshold, Pigsty will automatically promote that replica to be the new primary. However, when all replicas exceed this threshold, Pigsty will refuse [automatic failover] to prevent data loss. Manual intervention is then required to decide whether to wait for the primary to recover (which may never happen) or accept the data loss and force-promote a replica.

Because the primary’s WAL position is not sampled continuously, the worst-case loss under asynchronous replication can also include WAL generated during the most recent ttl window (on average, roughly another loop_wait/2 of WAL). Configure this threshold with your workload’s write rate in mind. Increasing it improves the chance of automatic failover but also broadens candidate eligibility.

When you set pg_rpo = 0, Pigsty enables synchronous replication, ensuring the primary only returns write success after at least one replica has persisted the data. This configuration ensures zero replication lag but introduces significant write latency and reduces overall throughput.

flowchart LR
    A([Primary Failure]) --> B{Synchronous<br/>Replication?}

    B -->|No| C{Lag < RPO?}
    B -->|Yes| D{Sync Replica<br/>Available?}

    C -->|Yes| E[Lossy Auto Failover<br/>Sampled candidate lag is within threshold]
    C -->|No| F[Refuse Auto Failover<br/>Wait for Primary Recovery<br/>or Manual Intervention]

    D -->|Yes| G[Lossless Auto Failover<br/>RPO = 0]
    D -->|No| H{Strict Mode?}

    H -->|No| C
    H -->|Yes| F

    style A fill:#dc3545,stroke:#b02a37,color:#fff
    style E fill:#F0AD4E,stroke:#146c43,color:#fff
    style G fill:#198754,stroke:#146c43,color:#fff
    style F fill:#BE002F,stroke:#565e64,color:#fff

Protection Modes

Pigsty provides three protection modes to help users make trade-offs under different RPO requirements, similar to Oracle Data Guard protection modes.

Maximum Performance
  • Default mode, asynchronous replication, transactions commit with only local WAL persistence, no waiting for replicas, replica failures are completely transparent to the primary
  • Primary failure may lose unsent/unreceived WAL. The default sampled candidate-lag threshold is 1MiB, but this is not a hard upper bound on actual loss
  • Optimized for performance, suitable for typical business scenarios that tolerate minor data loss during failures
Maximum Availability
  • Configured with pg_rpo = 0, enables Patroni synchronous commit mode: synchronous_mode: true
  • Under normal conditions, waits for at least one replica confirmation, achieving zero data loss. When all sync replicas fail, automatically degrades to async mode to continue service
  • Balances data safety and service availability, recommended configuration for production critical business
Maximum Protection
  • Uses crit.yml template, enables Patroni strict synchronous mode: synchronous_mode: true / synchronous_mode_strict: true
  • When all sync replicas fail, primary refuses writes to prevent data loss, transactions must be persisted on at least one replica before returning success
  • Suitable for financial transactions, medical records, and other scenarios with extremely high data integrity requirements
Name Maximum Performance Maximum Availability Maximum Protection
Replication Asynchronous Synchronous Strict Synchronous
Data Loss Possible (replication lag) Zero normally, minor when degraded Zero
Write Latency Lowest Medium (+1 network RTT) Medium (+1 network RTT)
Throughput Highest Reduced Reduced
Replica Failure Impact None Auto degrade, service continues Primary stops writes
RPO Possible loss; 1MiB default candidate threshold = 0 normally / possible loss after degradation = 0
Use Case Typical business, performance first Critical business, safety first Financial core, compliance first
Configuration Default config pg_rpo = 0 pg_conf: crit.yml

Implementation

The three protection modes differ in how two core Patroni parameters are configured: synchronous_mode and synchronous_mode_strict:

  • synchronous_mode: Whether Patroni enables synchronous replication. If enabled, check if synchronous_mode_strict enables strict synchronous mode.
  • synchronous_mode_strict = false: Default configuration, allows degradation to async mode when replicas fail, primary continues service (Maximum Availability)
  • synchronous_mode_strict = true: Degradation forbidden, primary stops writes until sync replica recovers (Maximum Protection)
Mode synchronous_mode synchronous_mode_strict Replication Mode Replica Failure Behavior
Max Performance false - Async No impact
Max Availability true false Synchronous Auto degrade to async
Max Protection true true Strict Synchronous Primary refuses writes

Typically, you only need to set the pg_rpo parameter to 0 to enable the synchronous_mode switch, activating Maximum Availability mode. If you use pg_conf = crit.yml template, it additionally enables the synchronous_mode_strict strict mode switch, activating Maximum Protection mode. Additionally, you can enable watchdog to fence the primary directly during node/Patroni freeze scenarios instead of degrading, achieving behavior equivalent to Oracle Maximum Protection mode.

You can also directly configure these Patroni parameters as needed. Refer to Patroni and PostgreSQL documentation to achieve stronger data protection, such as:

  • Specify the synchronous replica list, configure more sync replicas to improve disaster tolerance, use quorum synchronous commit, or even require all replicas to perform synchronous commit.
  • Configure synchronous_commit: 'remote_apply' to strictly ensure primary-replica read-write consistency. (Oracle Maximum Protection mode is equivalent to remote_write)

Recommendations

Maximum Performance mode (asynchronous replication) is the default mode used by Pigsty and is sufficient for the vast majority of workloads. It tolerates some loss during a failure in exchange for higher throughput and availability. In this mode, pg_rpo adjusts the sampled lag threshold for failover candidates; actual worst-case loss also depends on write rate, ttl, and sampling timing.

Maximum Availability mode (synchronous replication) is suitable for scenarios with high data-integrity requirements. Acknowledged transactions have zero loss while a synchronous replica is healthy, but the cluster can degrade when all synchronous replicas are unavailable. In this mode, a minimum of two-node PostgreSQL cluster (one primary, one replica) is required. Set pg_rpo to 0 to enable this mode.

Maximum Protection mode (strict synchronous replication) is suitable for financial transactions, medical records, and other scenarios with extremely high data integrity requirements. We recommend using at least a three-node cluster (one primary, two replicas), because with only two nodes, if the replica fails, the primary will stop writes, causing service unavailability, which reduces overall system reliability. With three nodes, if only one replica fails, the primary can continue to serve.

3.4.2 - Failure Model

Detailed analysis of worst-case, best-case, and average RTO calculation logic and results across three classic failure detection/recovery paths

Patroni failures can be classified into 10 categories by failure target, and further consolidated into five categories based on detection path, which are detailed in this section.

# Failure Scenario Description Final Path
1 PG process crash crash, OOM killed Active Detection
2 PG connection refused max_connections Active Detection
3 PG zombie Process alive but unresponsive Active Detection (timeout)
4 Patroni process crash kill -9, OOM Passive Detection
5 Patroni zombie Process alive but stuck Watchdog
6 Node down Power outage, hardware failure Passive Detection
7 Node zombie IO hang, CPU starvation Watchdog
8 Primary ↔ DCS network failure Firewall, switch failure Network Partition
9 Storage failure Disk failure, disk full, mount failure Active Detection or Watchdog
10 Manual switchover Switchover/Failover Manual Trigger

However, for RTO calculation purposes, all failures ultimately converge to two paths. This section explores the upper bound, lower bound, and average RTO for these two scenarios.

flowchart LR
    A([Primary Failure]) --> B{Patroni<br/>Detected?}

    B -->|PG Crash| C[Attempt Local Restart]
    B -->|Node Down| D[Wait TTL Expiration]

    C -->|Success| E([Local Recovery])
    C -->|Fail/Timeout| F[Release Leader Lock]

    D --> F
    F --> G[Replica Election]
    G --> H[Execute Promote]
    H --> I[HAProxy Detects]
    I --> J([Service Restored])

    style A fill:#dc3545,stroke:#b02a37,color:#fff
    style E fill:#198754,stroke:#146c43,color:#fff
    style J fill:#198754,stroke:#146c43,color:#fff

3.4.2.1 - Model of Patroni Passive Failure

Failover path triggered by node crash causing leader lease expiration and cluster election
infographic list-row-simple-horizontal-arrow
data

  desc Lease Expiration Stages
  items
    - label Lease Expiration
    - label Replica Detect
    - label Elect & Promote
    - label Haproxy Up
theme light
  palette antv

RTO Timeline 

tooltip: { trigger: axis, axisPointer: { type: shadow }, formatter: $fn:fmt }
legend: { top: 0, itemGap: 12, data: [Lease Expiration, Replica Detection, Lock Contest & Promote, Health Check] }
grid: { left: 64, right: 24, bottom: 32, top: 40 }
xAxis: { type: value, name: Seconds, nameLocation: end, max: 160, axisLine: { show: true }, axisTick: { show: true }, splitLine: { show: true, lineStyle: { type: dashed, opacity: 0.5 } }, minorTick: { show: true, splitNumber: 5 }, minorSplitLine: { show: true, lineStyle: { type: dotted, opacity: 0.2 } } }
yAxis: { type: category, axisLine: { show: true }, axisTick: { show: true }, splitLine: { show: false }, axisLabel: { fontSize: 10, fontFamily: monospace }, data: [wide-max, wide-avg, wide-min, "", safe-max, safe-avg, safe-min, "", norm-max, norm-avg, norm-min, "", fast-max, fast-avg, fast-min] }
series:
  - { name: Lease Expire, type: bar, stack: main, barWidth: 20, z: 2, emphasis: { focus: series }, itemStyle: { color: "#e15759" }, data: [120, 110, 100, "-", 60, 55, 50, "-", 30, 27, 25, "-", 20, 17, 15] }
  - { name: Replica Detect, type: bar, stack: main, z: 2, emphasis: { focus: series }, itemStyle: { color: "#edc949" }, data: [20, 10, 0, "-", 10, 5, 0, "-", 5, 3, 0, "-", 5, 3, 0] }
  - { name: Elect & Promote, type: bar, stack: main, z: 2, emphasis: { focus: series }, itemStyle: { color: "#59a14f" }, data: [2, 1, 0, "-", 2, 1, 0, "-", 2, 1, 0, "-", 2, 1, 0] }
  - { name: HAProxy Check, type: bar, stack: main, z: 2, emphasis: { focus: series }, itemStyle: { color: "#4e79a7" }, data: [8, 6, 4, "-", 6, 5, 3, "-", 4, 3, 2, "-", 2, 2, 1] }
  - { name: Total RTO, type: bar, barGap: "-100%", barWidth: 20, z: 1, itemStyle: { color: "#888", opacity: 0 }, emphasis: { itemStyle: { opacity: 0 } }, data: [150, 127, 104, "-", 78, 66, 53, "-", 41, 34, 27, "-", 29, 23, 16] }
  - { name: RTO Budget, type: bar, barGap: "-100%", barWidth: 20, z: 0, itemStyle: { color: "rgba(0,0,0,0.08)" }, emphasis: { itemStyle: { color: "rgba(0,0,0,0.12)" } }, data: [150, 150, 150, "-", 90, 90, 90, "-", 45, 45, 45, "-", 30, 30, 30] }

Failure Model

Phase Best Worst Average Description
Lease Expiration ttl - loop ttl ttl - loop/2 Best: crash just before refresh
Worst: crash right after refresh
Replica Detect 0 loop loop / 2 Best: exactly at check point
Worst: just missed check point
Election Promote 0 2 1 Best: direct lock and promote
Worst: API timeout + Promote
HAProxy Check (rise-1) × fastinter (rise-1) × fastinter + inter (rise-1) × fastinter + inter/2 Best: state change before check
Worst: state change right after check

Key Difference Between Passive and Active Failover:

Scenario Patroni Status Lease Handling Primary Wait Time
Active Failover (PG crash) Alive, healthy Actively tries to restart PG, releases lease on timeout primary_start_timeout
Passive Failover (Node crash) Dies with node Cannot actively release, must wait for TTL expiration ttl

In passive failover scenarios, Patroni dies along with the node and cannot actively release the Leader Key. The lease in DCS can only trigger cluster election after TTL naturally expires.

Timeline Analysis

Phase 1: Lease Expiration

The Patroni primary refreshes the Leader Key every loop_wait cycle, resetting TTL to the configured value.

Timeline:
     t-loop        t          t+ttl-loop    t+ttl
       |           |              |           |
    Last Refresh  Failure      Best Case   Worst Case
       |←── loop ──→|              |           |
       |←──────────── ttl ─────────────────────→|
  • Best case: Failure occurs just before lease refresh (elapsed loop since last refresh), remaining TTL = ttl - loop
  • Worst case: Failure occurs right after lease refresh, must wait full ttl
  • Average case: ttl - loop/2
Texpire={ttl−loopBestttl−loop/2AveragettlWorstT_{expire} = \begin{cases} ttl - loop & \text{Best} \\ ttl - loop/2 & \text{Average} \\ ttl & \text{Worst} \end{cases}

Phase 2: Replica Detection

Replicas wake up on loop_wait cycles and check the Leader Key status in DCS.

Timeline:
    Lease Expired   Replica Wakes
       |            |
       |←── 0~loop ─→|
  • Best case: Replica happens to wake when lease expires, wait 0
  • Worst case: Replica just entered sleep when lease expires, wait loop
  • Average case: loop/2
Tdetect={0Bestloop/2AverageloopWorstT_{detect} = \begin{cases} 0 & \text{Best} \\ loop/2 & \text{Average} \\ loop & \text{Worst} \end{cases}

Phase 3: Lock Contest & Promote

When replicas detect Leader Key expiration, they start the election process. The replica that acquires the Leader Key executes pg_ctl promote to become the new primary.

  1. Via REST API, parallel queries to check each replica’s replication position, typically 10ms, hardcoded 2s timeout.
  2. Compare WAL positions to determine the best candidate, replicas attempt to create Leader Key (CAS atomic operation)
  3. Execute pg_ctl promote to become primary (very fast, typically negligible)
Election Flow:
  ReplicaA ──→ Query replication position ──→ Compare ──→ Contest lock ──→ Success
  ReplicaB ──→ Query replication position ──→ Compare ──→ Contest lock ──→ Fail
  • Best case: Single replica or immediate lock acquisition and promotion, constant overhead 0.1s
  • Worst case: DCS API call timeout: 2s
  • Average case: 1s constant overhead
Telect={0.1Best1Average2WorstT_{elect} = \begin{cases} 0.1 & \text{Best} \\ 1 & \text{Average} \\ 2 & \text{Worst} \end{cases}

Phase 4: Health Check

HAProxy detects the new primary online, requiring rise consecutive successful health checks.

Detection Timeline:
  New Primary    First Check   Second Check  Third Check (UP)
     |          |           |           |
     |←─ 0~inter ─→|←─ fast ─→|←─ fast ─→|
  • Best case: New primary promoted just before check, (rise-1) × fastinter
  • Worst case: New primary promoted right after check, (rise-1) × fastinter + inter
  • Average case: (rise-1) × fastinter + inter/2
Thaproxy={(rise−1)×fastinterBest(rise−1)×fastinter+inter/2Average(rise−1)×fastinter+interWorstT_{haproxy} = \begin{cases} (rise-1) \times fastinter & \text{Best} \\ (rise-1) \times fastinter + inter/2 & \text{Average} \\ (rise-1) \times fastinter + inter & \text{Worst} \end{cases}

RTO Formula

Sum all phase times to get total RTO:

Best Case

RTOmin=ttl−loop+0.1+(rise−1)×fastinterRTO_{min} = ttl - loop + 0.1 + (rise-1) \times fastinter

Average Case

RTOavg=ttl+1+inter/2+(rise−1)×fastinterRTO_{avg} = ttl + 1 + inter/2 + (rise-1) \times fastinter

Worst Case

RTOmax=ttl+loop+2+inter+(rise−1)×fastinterRTO_{max} = ttl + loop + 2 + inter + (rise-1) \times fastinter

Model Calculation

Substitute the four RTO model parameters into the formulas above:

pg_rto_plan:  # [ttl, loop, retry, start, margin, inter, fastinter, downinter, rise, fall]
  fast: [ 20  ,5  ,5  ,15 ,5  ,'1s' ,'0.5s' ,'1s' ,3 ,3 ]  # rto < 30s
  norm: [ 30  ,5  ,10 ,25 ,5  ,'2s' ,'1s'   ,'2s' ,3 ,3 ]  # rto < 45s
  safe: [ 60  ,10 ,20 ,45 ,10 ,'3s' ,'1.5s' ,'3s' ,3 ,3 ]  # rto < 90s
  wide: [ 120 ,20 ,30 ,95 ,15 ,'4s' ,'2s'   ,'4s' ,3 ,3 ]  # rto < 150s

Four Mode Calculation Results (unit: seconds, format: min / avg / max)

Phase fast norm safe wide
Lease Expiration 15 / 17 / 20 25 / 27 / 30 50 / 55 / 60 100 / 110 / 120
Replica Detection 0 / 3 / 5 0 / 3 / 5 0 / 5 / 10 0 / 10 / 20
Lock Contest & Promote 0 / 1 / 2 0 / 1 / 2 0 / 1 / 2 0 / 1 / 2
Health Check 1 / 2 / 2 2 / 3 / 4 3 / 5 / 6 4 / 6 / 8
Total 16 / 23 / 29 27 / 34 / 41 53 / 66 / 78 104 / 127 / 150

3.4.2.2 - Model of Patroni Active Failure

PostgreSQL primary process crashes while Patroni stays alive and attempts restart, triggering failover after timeout
infographic list-row-simple-horizontal-arrow
data
  desc When Patroni is healthy but PostgreSQL crashes
  items
    - label Crash Found
    - label Restart Timeout
    - label Replica Detect
    - label Elect Promote
    - label HAProxy Check
theme light
  palette antv

RTO Timeline 

tooltip: { trigger: axis, axisPointer: { type: shadow }, formatter: $fn:fmt }
legend: { top: 0, itemGap: 12, data: [ Crash Found, Restart Timeout, Replica Detection, Elect Promote, HAProxy Check] }
grid: { left: 64, right: 24, bottom: 32, top: 40 }
xAxis: { type: value, name: Seconds, nameLocation: end, max: 160, axisLine: { show: true }, axisTick: { show: true }, splitLine: { show: true, lineStyle: { type: dashed, opacity: 0.5 } }, minorTick: { show: true, splitNumber: 5 }, minorSplitLine: { show: true, lineStyle: { type: dotted, opacity: 0.2 } } }
yAxis: { type: category, axisLine: { show: true }, axisTick: { show: true }, splitLine: { show: false }, axisLabel: { fontSize: 10, fontFamily: monospace }, data: [wide-max, wide-avg, wide-min, "", safe-max, safe-avg, safe-min, "", norm-max, norm-avg, norm-min, "", fast-max, fast-avg, fast-min] }
series:
  - { name: Crash Found, type: bar, stack: main, barWidth: 20, z: 2, emphasis: { focus: series }, itemStyle: { color: "#b07aa1" }, data: [20, 10, 0, "-", 10, 5, 0, "-", 5, 3, 0, "-", 5, 3, 0] }
  - { name: Restart Timeout, type: bar, stack: main, z: 2, emphasis: { focus: series }, itemStyle: { color: "#f28e2c" }, data: [95, 95, 0, "-", 45, 45, 0, "-", 25, 25, 0, "-", 15, 15, 0] }
  - { name: Replica Detect, type: bar, stack: main, z: 2, emphasis: { focus: series }, itemStyle: { color: "#edc949" }, data: [20, 10, 0, "-", 10, 5, 0, "-", 5, 3, 0, "-", 5, 3, 0] }
  - { name: Elect Promote, type: bar, stack: main, z: 2, emphasis: { focus: series }, itemStyle: { color: "#59a14f" }, data: [2, 1, 0, "-", 2, 1, 0, "-", 2, 1, 0, "-", 2, 1, 0] }
  - { name: HAProxy Check, type: bar, stack: main, z: 2, emphasis: { focus: series }, itemStyle: { color: "#4e79a7" }, data: [8, 6, 4, "-", 6, 5, 3, "-", 4, 3, 2, "-", 2, 2, 1] }
  - { name: RTO Total, type: bar, barGap: "-100%", barWidth: 20, z: 1, itemStyle: { color: "#888", opacity: 0 }, emphasis: { itemStyle: { opacity: 0 } }, data: [145, 122, 4, "-", 73, 61, 3, "-", 41, 35, 2, "-", 29, 24, 1] }
  - { name: RTO Budget, type: bar, barGap: "-100%", barWidth: 20, z: 0, itemStyle: { color: "rgba(0,0,0,0.08)" }, emphasis: { itemStyle: { color: "rgba(0,0,0,0.12)" } }, data: [150, 150, 150, "-", 90, 90, 90, "-", 45, 45, 45, "-", 30, 30, 30] }

Failure Model

Item Best Worst Average Description
Crash Found 0 loop loop/2 Best: PG crashes right before check
Worst: PG crashes right after check
Restart Timeout 0 start start Best: PG recovers instantly
Worst: Wait full start timeout before releasing lease
Replica Detect 0 loop loop/2 Best: Right at check point
Worst: Just missed check point
Elect Promote 0 2 1 Best: Acquire lock and promote directly
Worst: API timeout + Promote
HAProxy Check (rise-1) × fastinter (rise-1) × fastinter + inter (rise-1) × fastinter + inter/2 Best: State changes before check
Worst: State changes right after check

Key Difference Between Active and Passive Failure:

Scenario Patroni Status Lease Handling Main Wait Time
Active Failure (PG crash) Alive, healthy Actively tries to restart PG, releases lease after timeout primary_start_timeout
Passive Failure (node down) Dies with node Cannot actively release, must wait for TTL expiry ttl

In active failure scenarios, Patroni remains alive and can actively detect PG crash and attempt restart. If restart succeeds, service self-heals; if timeout expires without recovery, Patroni actively releases the Leader Key, triggering cluster election.

Timing Analysis

Phase 1: Failure Detection

Patroni checks PostgreSQL status every loop_wait cycle (via pg_isready or process check).

Timeline:
    Last check      PG crash      Next check
       |              |              |
       |←── 0~loop ──→|              |
  • Best case: PG crashes right before Patroni check, detected immediately, wait 0
  • Worst case: PG crashes right after check, wait for next cycle, wait loop
  • Average case: loop/2
Tdetect={0Bestloop/2AverageloopWorstT_{detect} = \begin{cases} 0 & \text{Best} \\ loop/2 & \text{Average} \\ loop & \text{Worst} \end{cases}

Phase 2: Restart Timeout

After Patroni detects PG crash, it attempts to restart PostgreSQL. This phase has two possible outcomes:

Timeline:
  Crash detected     Restart attempt     Success/Timeout
      |                  |                    |
      |←──── 0 ~ start ─────────────────────→|

Path A: Self-healing Success (Best case)

  • PG restarts successfully, service recovers
  • No failover triggered, extremely short RTO
  • Wait time: 0 (relative to Failover path)

Path B: Failover Required (Average/Worst case)

  • PG still not recovered after primary_start_timeout
  • Patroni actively releases Leader Key
  • Wait time: start
Trestart={0Best (self-healing success)startAverage (failover required)startWorstT_{restart} = \begin{cases} 0 & \text{Best (self-healing success)} \\ start & \text{Average (failover required)} \\ start & \text{Worst} \end{cases}

Note: Average case assumes failover is required. If PG can quickly self-heal, overall RTO will be significantly lower.

Phase 3: Standby Detection

Standbys wake up on loop_wait cycle and check Leader Key status in DCS. When primary Patroni releases the Leader Key, standbys discover this and begin election.

Timeline:
    Lease released    Standby wakes
       |                  |
       |←── 0~loop ──────→|
  • Best case: Standby wakes right when lease is released, wait 0
  • Worst case: Standby just went to sleep when lease released, wait loop
  • Average case: loop/2
Tstandby={0Bestloop/2AverageloopWorstT_{standby} = \begin{cases} 0 & \text{Best} \\ loop/2 & \text{Average} \\ loop & \text{Worst} \end{cases}

Phase 4: Lock & Promote

After standbys discover Leader Key vacancy, election begins. The standby that acquires the Leader Key executes pg_ctl promote to become the new primary.

  1. Via REST API, parallel queries to check each standby’s replication position, typically 10ms, hardcoded 2s timeout.
  2. Compare WAL positions to determine best candidate, standbys attempt to create Leader Key (CAS atomic operation)
  3. Execute pg_ctl promote to become primary (very fast, typically negligible)
Election process:
  StandbyA ──→ Query replication position ──→ Compare ──→ Try lock ──→ Success
  StandbyB ──→ Query replication position ──→ Compare ──→ Try lock ──→ Fail
  • Best case: Single standby or direct lock acquisition and promote, constant overhead 0.1s
  • Worst case: DCS API call timeout: 2s
  • Average case: 1s constant overhead
Telect={0.1Best1Average2WorstT_{elect} = \begin{cases} 0.1 & \text{Best} \\ 1 & \text{Average} \\ 2 & \text{Worst} \end{cases}

Phase 5: Health Check

HAProxy detects new primary online, requires rise consecutive successful health checks.

Check timeline:
  New primary    First check    Second check   Third check (UP)
     |              |               |               |
     |←─ 0~inter ──→|←─── fast ────→|←─── fast ────→|
  • Best case: New primary comes up right at check time, (rise-1) × fastinter
  • Worst case: New primary comes up right after check, (rise-1) × fastinter + inter
  • Average case: (rise-1) × fastinter + inter/2
Thaproxy={(rise−1)×fastinterBest(rise−1)×fastinter+inter/2Average(rise−1)×fastinter+interWorstT_{haproxy} = \begin{cases} (rise-1) \times fastinter & \text{Best} \\ (rise-1) \times fastinter + inter/2 & \text{Average} \\ (rise-1) \times fastinter + inter & \text{Worst} \end{cases}

RTO Formula

Sum all phase times to get total RTO:

Best Case (PG instant self-healing)

RTOmin=0+0+0+0.1+(rise−1)×fastinter≈(rise−1)×fastinterRTO_{min} = 0 + 0 + 0 + 0.1 + (rise-1) \times fastinter \approx (rise-1) \times fastinter

Average Case (Failover required)

RTOavg=loop+start+1+inter/2+(rise−1)×fastinterRTO_{avg} = loop + start + 1 + inter/2 + (rise-1) \times fastinter

Worst Case

RTOmax=loop×2+start+2+inter+(rise−1)×fastinterRTO_{max} = loop \times 2 + start + 2 + inter + (rise-1) \times fastinter

Model Calculation

Substituting the four RTO model parameters into the formulas above:

pg_rto_plan:  # [ttl, loop, retry, start, margin, inter, fastinter, downinter, rise, fall]
  fast: [ 20  ,5  ,5  ,15 ,5  ,'1s' ,'0.5s' ,'1s' ,3 ,3 ]  # rto < 30s
  norm: [ 30  ,5  ,10 ,25 ,5  ,'2s' ,'1s'   ,'2s' ,3 ,3 ]  # rto < 45s
  safe: [ 60  ,10 ,20 ,45 ,10 ,'3s' ,'1.5s' ,'3s' ,3 ,3 ]  # rto < 90s
  wide: [ 120 ,20 ,30 ,95 ,15 ,'4s' ,'2s'   ,'4s' ,3 ,3 ]  # rto < 150s

Calculation Results for Four Modes (unit: seconds, format: min / avg / max)

Phase fast norm safe wide
Failure Detection 0 / 3 / 5 0 / 3 / 5 0 / 5 / 10 0 / 10 / 20
Restart Timeout 0 / 15 / 15 0 / 25 / 25 0 / 45 / 45 0 / 95 / 95
Standby Detection 0 / 3 / 5 0 / 3 / 5 0 / 5 / 10 0 / 10 / 20
Lock & Promote 0 / 1 / 2 0 / 1 / 2 0 / 1 / 2 0 / 1 / 2
Health Check 1 / 2 / 2 2 / 3 / 4 3 / 5 / 6 4 / 6 / 8
Total 1 / 24 / 29 2 / 35 / 41 3 / 61 / 73 4 / 122 / 145

Comparison with Passive Failure

Phase Active Failure (PG crash) Passive Failure (node down) Description
Detection Mechanism Patroni active detection TTL passive expiry Active detection discovers failure faster
Core Wait start ttl start is usually less than ttl, but requires additional failure detection time
Lease Handling Active release Passive expiry Active release is more timely
Self-healing Possible Yes No Active detection can attempt local recovery

RTO Comparison (Average case):

Mode Active Failure (PG crash) Passive Failure (node down) Difference
fast 24s 23s +1s
norm 35s 34s +1s
safe 61s 66s -5s
wide 122s 127s -5s

Analysis: In fast and norm modes, active failure RTO is slightly higher than passive failure because it waits for primary_start_timeout (start); but in safe and wide modes, since start < ttl - loop, active failure is actually faster. However, active failure has the possibility of self-healing, with potentially extremely short RTO in best case scenarios.

3.4.2.3 - Network Partition

Primary loses DCS connectivity, causing lease expiration and triggering split-brain protection and failover
infographic list-row-simple-horizontal-arrow
data
  title Network Partition Failover Flow
  desc Primary partitioned from DCS, Patroni proactively demotes to prevent split-brain, waits for TTL expiration before switchover
  items
    - label Primary Demote
      desc Patroni demotes PG after retry timeout
      icon mingcute/shield-fill
    - label Lease Expiration
      desc Leader Key TTL expires
      icon mingcute/close-circle-fill
    - label Replica Detection
      desc Replica detects lease expiration, starts election
      icon mingcute/key-2-fill
    - label Lock & Promote
      desc Replica acquires lock and promotes to new primary
      icon mingcute/radar-fill
    - label Health Check
      desc HAProxy detects new primary online
      icon mingcute/arrow-up-circle-fill
theme light
  palette antv

RTO Timeline 

tooltip: { trigger: axis, axisPointer: { type: shadow }, formatter: $fn:fmt }
legend: { top: 0, itemGap: 12, data: [Primary Demote, Lease Expiration, Replica Detection, Lock & Promote, Health Check] }
grid: { left: 64, right: 24, bottom: 32, top: 40 }
xAxis: { type: value, name: sec, nameLocation: end, max: 160, axisLine: { show: true }, axisTick: { show: true }, splitLine: { show: true, lineStyle: { type: dashed, opacity: 0.5 } }, minorTick: { show: true, splitNumber: 5 }, minorSplitLine: { show: true, lineStyle: { type: dotted, opacity: 0.2 } } }
yAxis: { type: category, axisLine: { show: true }, axisTick: { show: true }, splitLine: { show: false }, axisLabel: { fontSize: 10, fontFamily: monospace }, data: [wide-max, wide-avg, wide-min, "", safe-max, safe-avg, safe-min, "", norm-max, norm-avg, norm-min, "", fast-max, fast-avg, fast-min] }
series:
  - { name: Primary Demote, type: bar, stack: main, barWidth: 20, z: 2, emphasis: { focus: series }, itemStyle: { color: "#76b7b2" }, data: [50, 40, 30, "-", 30, 25, 20, "-", 15, 13, 10, "-", 10, 8, 5] }
  - { name: Lease Expiration, type: bar, stack: main, z: 2, emphasis: { focus: series }, itemStyle: { color: "#e15759" }, data: [70, 70, 70, "-", 30, 30, 30, "-", 15, 15, 15, "-", 10, 10, 10] }
  - { name: Replica Detection, type: bar, stack: main, z: 2, emphasis: { focus: series }, itemStyle: { color: "#edc949" }, data: [20, 10, 0, "-", 10, 5, 0, "-", 5, 3, 0, "-", 5, 3, 0] }
  - { name: Lock & Promote, type: bar, stack: main, z: 2, emphasis: { focus: series }, itemStyle: { color: "#59a14f" }, data: [2, 1, 0, "-", 2, 1, 0, "-", 2, 1, 0, "-", 2, 1, 0] }
  - { name: Health Check, type: bar, stack: main, z: 2, emphasis: { focus: series }, itemStyle: { color: "#4e79a7" }, data: [8, 6, 4, "-", 6, 5, 3, "-", 4, 3, 2, "-", 2, 2, 1] }
  - { name: RTO Total, type: bar, barGap: "-100%", barWidth: 20, z: 1, itemStyle: { color: "#888", opacity: 0 }, emphasis: { itemStyle: { opacity: 0 } }, data: [150, 127, 104, "-", 78, 66, 53, "-", 41, 34, 27, "-", 29, 23, 16] }
  - { name: RTO Budget, type: bar, barGap: "-100%", barWidth: 20, z: 0, itemStyle: { color: "rgba(0,0,0,0.08)" }, emphasis: { itemStyle: { color: "rgba(0,0,0,0.12)" } }, data: [150, 150, 150, "-", 90, 90, 90, "-", 45, 45, 45, "-", 30, 30, 30] }

Failure Model

Phase Best Worst Average Notes
Demote retry loop + retry loop/2 + retry Patroni retries after detecting partition, demotes after timeout
Lease Expiration ttl - loop - retry ttl - loop - retry ttl - loop - retry Remaining TTL time after demotion (approximately constant)
Replica Detection 0 loop loop/2 Best: Right at detection point
Worst: Just missed detection
Lock & Promote 0 2 1 Best: Direct lock and promote
Worst: API timeout + Promote
Health Check (rise-1) × fastinter (rise-1) × fastinter + inter (rise-1) × fastinter + inter/2 Best: State changes before check
Worst: State changes right after check

Key difference between network partition and node crash:

Scenario Patroni State PostgreSQL State Lease Handling Split-brain Risk
Node Crash (Expire) Dies with node Completely unavailable Passive wait for TTL expiration None
Network Partition (This scenario) Alive but cannot access DCS May still be running (needs active demotion) Passive wait for TTL expiration Yes, needs protection

In network partition scenarios, the primary PostgreSQL may still be running and accepting writes, causing split-brain issues. Patroni solves this through active demotion: when unable to refresh Leader Key, proactively demotes PostgreSQL to read-only or shuts it down.

Timeline Analysis

Phase 1: Primary Demotion

When primary Patroni is network-partitioned from DCS, it cannot refresh Leader Key and starts retrying.

Timeline:
  Partition      Detect partition      Retry timeout      Primary demotes
     |               |                    |                    |
     |←── loop ──→|←── retry ──→|
  • Detection delay: After partition occurs, must wait for next loop_wait cycle to detect
  • Retry phase: Patroni continuously retries DCS operations during retry_timeout
  • Active demotion: After retry timeout, Patroni proactively demotes PostgreSQL (prevents split-brain)
Tdemote={retrybest (partition right before detection)loop/2+retryaverageloop+retryworst (partition right after refresh)T_{demote} = \begin{cases} retry & \text{best (partition right before detection)} \\ loop/2 + retry & \text{average} \\ loop + retry & \text{worst (partition right after refresh)} \end{cases}

Key design: Patroni requires constraint loop_wait + 2 × retry_timeout ≤ ttl to ensure primary demotes before TTL expires.

Phase 2: Lease Expiration

After primary demotion, Leader Key still exists in DCS, must wait for TTL to naturally expire.

Timeline:
  Primary demoted                   TTL expires
     |                                 |
     |←── ttl - (loop + retry) ──→|

Since the primary has demoted, waiting time during this phase is the remaining TTL time. Since partition detection and remaining TTL are negatively correlated (earlier partition means slower detection but longer remaining TTL), their sum is constant:

Texpire=ttl−loop−retry(approximately constant)T_{expire} = ttl - loop - retry \quad \text{(approximately constant)}

Note: Primary demotion + lease expiration total time still approximately equals ttl, same as expire failure.

Phase 3: Replica Detection

Replica wakes up in loop_wait cycle and checks Leader Key status in DCS.

Timeline:
    Lease expired      Replica wakes
       |                  |
       |←── 0~loop ─→|
  • Best case: Replica wakes right when lease expires, wait 0
  • Worst case: Replica just entered sleep when lease expires, wait loop
  • Average case: loop/2
Tdetect={0bestloop/2averageloopworstT_{detect} = \begin{cases} 0 & \text{best} \\ loop/2 & \text{average} \\ loop & \text{worst} \end{cases}

Phase 4: Lock & Promote

After replica discovers Leader Key expired, it starts the election process.

Election flow:
  ReplicaA ──→ Query replication position ──→ Compare ──→ Try lock ──→ Success
  ReplicaB ──→ Query replication position ──→ Compare ──→ Try lock ──→ Fail
  • Best case: Single replica or directly acquires lock and promotes, ≈ 0
  • Worst case: DCS API call timeout, 2s
  • Average case: 1s
Telect={0best1average2worstT_{elect} = \begin{cases} 0 & \text{best} \\ 1 & \text{average} \\ 2 & \text{worst} \end{cases}

Phase 5: Health Check

HAProxy detects new primary coming online, requires rise consecutive successful health checks.

Detection timeline:
  New primary    First check    Second check   Third check (UP)
     |              |               |               |
     |←─ 0~inter ─→|←─ fast ─→|←─ fast ─→|
  • Best case: (rise-1) × fastinter
  • Worst case: (rise-1) × fastinter + inter
  • Average case: (rise-1) × fastinter + inter/2
Thaproxy={(rise−1)×fastinterbest(rise−1)×fastinter+inter/2average(rise−1)×fastinter+interworstT_{haproxy} = \begin{cases} (rise-1) \times fastinter & \text{best} \\ (rise-1) \times fastinter + inter/2 & \text{average} \\ (rise-1) \times fastinter + inter & \text{worst} \end{cases}

RTO Formula

Sum all phase times to get total RTO.

Since primary demotion + lease expiration ≈ ttl, network partition RTO formula is same as expire failure:

Best Case

RTOmin=ttl−loop+0.1+(rise−1)×fastinterRTO_{min} = ttl - loop + 0.1 + (rise-1) \times fastinterRTOmin≈ttl−loop+(rise−1)×fastinterRTO_{min} \approx ttl - loop + (rise-1) \times fastinter

Average Case

RTOavg=ttl+1+inter/2+(rise−1)×fastinterRTO_{avg} = ttl + 1 + inter/2 + (rise-1) \times fastinterRTOavg=ttl+1+inter/2+(rise−1)×fastinterRTO_{avg} = ttl + 1 + inter/2 + (rise-1) \times fastinter

Worst Case

RTOmax=ttl+loop+2+inter+(rise−1)×fastinterRTO_{max} = ttl + loop + 2 + inter + (rise-1) \times fastinterRTOmax=ttl+loop+2+inter+(rise−1)×fastinterRTO_{max} = ttl + loop + 2 + inter + (rise-1) \times fastinter

Model Calculation

Substituting the four RTO model parameters into the formulas:

pg_rto_plan:  # [ttl, loop, retry, start, margin, inter, fastinter, downinter, rise, fall]
  fast: [ 20  ,5  ,5  ,15 ,5  ,'1s' ,'0.5s' ,'1s' ,3 ,3 ]  # rto < 30s
  norm: [ 30  ,5  ,10 ,25 ,5  ,'2s' ,'1s'   ,'2s' ,3 ,3 ]  # rto < 45s
  safe: [ 60  ,10 ,20 ,45 ,10 ,'3s' ,'1.5s' ,'3s' ,3 ,3 ]  # rto < 90s
  wide: [ 120 ,20 ,30 ,95 ,15 ,'4s' ,'2s'   ,'4s' ,3 ,3 ]  # rto < 150s

Patroni constraint validation (loop + 2×retry ≤ ttl):

Mode loop retry TTL loop + 2×retry Meets constraint?
fast 5 5 20s 15s ✓ Safe
norm 5 10 30s 25s ✓ Safe
safe 10 20 60s 50s ✓ Safe
wide 20 30 120s 80s ✓ Safe

Four mode calculation results (seconds, format: min / avg / max)

Phase fast norm safe wide
Primary Demote 5 / 8 / 10 10 / 13 / 15 20 / 25 / 30 30 / 40 / 50
Lease Expiration 10 15 30 70
Replica Detection 0 / 3 / 5 0 / 3 / 5 0 / 5 / 10 0 / 10 / 20
Lock & Promote 0 / 1 / 2 0 / 1 / 2 0 / 1 / 2 0 / 1 / 2
Health Check 1 / 2 / 2 2 / 3 / 4 3 / 5 / 6 4 / 6 / 8
Total 16 / 23 / 29 27 / 34 / 41 53 / 66 / 78 104 / 127 / 150

Conclusion: Network partition RTO is same as expire failure (node crash), as the bottleneck is TTL expiration time.

Split-brain Protection

The biggest risk of network partition is split-brain: old primary may still be running and accepting writes. Patroni provides multiple protection mechanisms:

1. Primary Self-Demotion

Patroni’s core protection mechanism: when unable to refresh Leader Key, proactively demotes PostgreSQL.

# Patroni pseudo-code logic
if not can_refresh_leader_key():
    retry_until(retry_timeout)
    if still_cannot_refresh():
        demote_postgresql()  # Demote to read-only or shut down

2. Linux Watchdog

If Patroni process hangs and cannot execute demotion, Linux watchdog will force system restart.

# patroni.yml configuration
watchdog:
  mode: required  # Require watchdog available
  device: /dev/watchdog
  safety_margin: 5

3. Fencing Mechanism

Can configure fencing scripts to forcibly isolate old primary (e.g., disable network interface, stop service, etc.).

Special Scenarios

Scenario A: Primary partitioned from DCS, replicas normal

This is the most common network partition scenario, the main focus of this article.

┌─────────┐         ╳         ┌─────────┐
│ Primary │ ←── Partition ──→ │   DCS   │
│ Patroni │                   │  etcd   │
└─────────┘                   └─────────┘
                                  ↑
                              Normal connection
                                  ↓
                              ┌─────────┐
                              │ Replica │
                              │ Patroni │
                              └─────────┘
  • Primary Patroni cannot refresh Leader Key → Active demotion
  • Replica normally detects TTL expiration → Elected as new primary
  • RTO ≈ Expire failure RTO

Scenario B: Primary normal, replica partitioned from DCS 

┌─────────┐                   ┌─────────┐
│ Primary │ ←── Normal ──→    │   DCS   │
│ Patroni │                   │  etcd   │
└─────────┘                   └─────────┘
                                  ╳
                              Partition
                                  ╳
                              ┌─────────┐
                              │ Replica │
                              │ Patroni │
                              └─────────┘
  • Primary normally refreshes Leader Key
  • Replica cannot participate in election (but replication can continue)
  • No failover triggered, service continues normally

Scenario C: All nodes partitioned from DCS 

┌─────────┐         ╳         ┌─────────┐
│ Primary │ ←── Partition ──→ │   DCS   │
│ Patroni │                   │  etcd   │
└─────────┘                   └─────────┘
                                  ╳
┌─────────┐         ╳             │
│ Replica │ ←── Partition ────────┘
│ Patroni │
└─────────┘
  • Primary demotes, replica cannot elect
  • Cluster completely unavailable
  • Requires manual intervention to restore DCS connectivity

Comparison with Other Failures

Failure Type Primary State Lease Handling RTO Split-brain Risk
Expire Failure Node crash Passive wait TTL expiration 16s ~ 150s None
Crash Failure PG crash, Patroni alive Release after restart timeout 1s ~ 111s None
Network Partition Alive but isolated from DCS Passive wait TTL expiration 16s ~ 150s Yes, needs protection
Manual Switchover Normal or failed Direct release/acquire 1s ~ 11s None

Key Insight: Network partition RTO is same as expire failure, but requires additional split-brain protection mechanisms. Ensuring loop_wait + 2 × retry_timeout ≤ ttl constraint is the key design to prevent split-brain.

3.4.3 - RTO Trade-offs

Trade-off analysis for RTO (Recovery Time Objective), finding the optimal balance between recovery speed and false failover risk.

RTO (Recovery Time Objective) defines the maximum time required for the system to restore write capability when the primary fails.

For critical transaction systems where availability is paramount, the shortest possible RTO is typically required, such as under one minute.

However, shorter RTO comes at a cost: increased false failover risk. Network jitter may be misinterpreted as a failure, leading to unnecessary failovers. For cross-datacenter/cross-region deployments, RTO requirements are typically relaxed (e.g., 1-2 minutes) to reduce false failover risk.

Trade-offs

The upper limit of unavailability during failover is controlled by the pg_rto parameter. Pigsty provides four preset RTO modes: fast, norm, safe, wide, each optimized for different network conditions and deployment scenarios. The default is norm mode (~45 seconds).

When the primary fails, the entire recovery process involves multiple phases: Patroni detects the failure, DCS lock expires, new primary election, promote execution, HAProxy detects the new primary. Reducing RTO means shortening the timeout for each phase, which makes the cluster more sensitive to network jitter, thereby increasing false failover risk.

You need to choose the appropriate mode based on actual network conditions, balancing recovery speed and false failover risk. The worse the network quality, the more conservative mode you should choose; the better the network quality, the more aggressive mode you can choose.

flowchart LR
    A([Primary Failure]) --> B{Patroni<br/>Detected?}

    B -->|PG Crash| C[Attempt Local Restart]
    B -->|Node Down| D[Wait TTL Expiration]

    C -->|Success| E([Local Recovery])
    C -->|Fail/Timeout| F[Release Leader Lock]

    D --> F
    F --> G[Replica Election]
    G --> H[Execute Promote]
    H --> I[HAProxy Detects]
    I --> J([Service Restored])

    style A fill:#dc3545,stroke:#b02a37,color:#fff
    style E fill:#198754,stroke:#146c43,color:#fff
    style J fill:#198754,stroke:#146c43,color:#fff

Four Modes

Pigsty provides four RTO modes to help users make trade-offs under different network conditions.

Name fast norm safe wide
Use Case Same rack Same datacenter (default) Same region, cross-DC Cross-region/continent
Network < 1ms, very stable 1-5ms, normal 10-50ms, cross-DC 100-200ms, public network
Target RTO 30s 45s 90s 150s
False Failover Risk Higher Medium Lower Very Low
Configuration pg_rto: fast pg_rto: norm pg_rto: safe pg_rto: wide
fast: Same Rack/Switch
  • Suitable for scenarios with extremely low network latency (< 1ms) and very stable networks, such as same-rack or same-switch deployments
  • Average RTO: 14s, worst case: 29s, TTL only 20s, check interval 5s
  • Highest network quality requirements, any jitter may trigger failover, higher false failover risk
norm: Same Datacenter (Default)
  • Default mode, suitable for same-datacenter deployment, network latency 1-5ms, normal quality, reasonable packet loss rate
  • Average RTO: 21s, worst case: 43s, TTL is 30s, provides reasonable tolerance window
  • Balances recovery speed and stability, suitable for most production environments
safe: Same Region, Cross-Datacenter
  • Suitable for same-region/same-area cross-datacenter deployment, network latency 10-50ms, occasional jitter possible
  • Average RTO: 43s, worst case: 91s, TTL is 60s, longer tolerance window
  • Primary restart wait time is longer (60s), gives more local recovery opportunities, lower false failover risk
wide: Cross-Region/Continent
  • Suitable for cross-region or even cross-continent deployment, network latency 100-200ms, possible public-network-level packet loss
  • Average RTO: 92s, worst case: 207s, TTL is 120s, very wide tolerance window
  • Sacrifices recovery speed for extremely low false failover rate, suitable for geo-disaster recovery scenarios

RTO Timeline

Patroni / PG HA has two key failure paths: active failure detection (Patroni detects a PG crash and attempts restart) and passive lease expiration (node down waits for TTL expiration to trigger election).

tooltip: { trigger: axis, axisPointer: { type: shadow }, formatter: $fn:fmt }
legend: { top: 0, itemGap: 10, data: [Lease Expiration, Failure Detection, Restart Timeout, Replica Detection, Lock & Promote, Health Check] }
grid: { left: 110, right: 24, bottom: 32, top: 40 }
xAxis: { type: value, name: Seconds, nameLocation: end, max: 160, axisLine: { show: true }, axisTick: { show: true }, splitLine: { show: true, lineStyle: { type: dashed, opacity: 0.5 } }, minorTick: { show: true, splitNumber: 5 }, minorSplitLine: { show: true, lineStyle: { type: dotted, opacity: 0.2 } } }
yAxis: { type: category, axisLine: { show: true }, axisTick: { show: true }, splitLine: { show: false }, axisLabel: { fontSize: 9, fontFamily: monospace }, data: [wide-passive-max, wide-passive-avg, wide-passive-min, wide-active-max, wide-active-avg, wide-active-min, "", safe-passive-max, safe-passive-avg, safe-passive-min, safe-active-max, safe-active-avg, safe-active-min, "", norm-passive-max, norm-passive-avg, norm-passive-min, norm-active-max, norm-active-avg, norm-active-min, "", fast-passive-max, fast-passive-avg, fast-passive-min, fast-active-max, fast-active-avg, fast-active-min] }
series:
  - { name: Lease Expiration, type: bar, stack: main, barWidth: 16, z: 2, emphasis: { focus: series }, itemStyle: { color: "#e15759" }, data: [120, 110, 100, "-", "-", "-", "-", 60, 55, 50, "-", "-", "-", "-", 30, 27, 25, "-", "-", "-", "-", 20, 17, 15, "-", "-", "-"] }
  - { name: Failure Detection, type: bar, stack: main, z: 2, emphasis: { focus: series }, itemStyle: { color: "#b07aa1" }, data: ["-", "-", "-", 20, 10, 0, "-", "-", "-", "-", 10, 5, 0, "-", "-", "-", "-", 5, 3, 0, "-", "-", "-", "-", 5, 3, 0] }
  - { name: Restart Timeout, type: bar, stack: main, z: 2, emphasis: { focus: series }, itemStyle: { color: "#f28e2c" }, data: ["-", "-", "-", 95, 95, 0, "-", "-", "-", "-", 45, 45, 0, "-", "-", "-", "-", 25, 25, 0, "-", "-", "-", "-", 15, 15, 0] }
  - { name: Replica Detection, type: bar, stack: main, z: 2, emphasis: { focus: series }, itemStyle: { color: "#edc949" }, data: [20, 10, 0, 20, 10, 0, "-", 10, 5, 0, 10, 5, 0, "-", 5, 3, 0, 5, 3, 0, "-", 5, 3, 0, 5, 3, 0] }
  - { name: Lock & Promote, type: bar, stack: main, z: 2, emphasis: { focus: series }, itemStyle: { color: "#59a14f" }, data: [2, 1, 0, 2, 1, 0, "-", 2, 1, 0, 2, 1, 0, "-", 2, 1, 0, 2, 1, 0, "-", 2, 1, 0, 2, 1, 0] }
  - { name: Health Check, type: bar, stack: main, z: 2, emphasis: { focus: series }, itemStyle: { color: "#4e79a7" }, data: [8, 6, 4, 8, 6, 4, "-", 6, 5, 3, 6, 5, 3, "-", 4, 3, 2, 4, 3, 2, "-", 2, 2, 1, 2, 2, 1] }
  - { name: RTO Total, type: bar, barGap: "-100%", barWidth: 16, z: 1, itemStyle: { color: "#888", opacity: 0 }, emphasis: { itemStyle: { opacity: 0 } }, data: [150, 127, 104, 145, 122, 4, "-", 78, 66, 53, 73, 61, 3, "-", 41, 34, 27, 41, 35, 2, "-", 29, 23, 16, 29, 24, 1] }
  - { name: RTO Budget, type: bar, barGap: "-100%", barWidth: 16, z: 0, itemStyle: { color: "rgba(0,0,0,0.08)" }, emphasis: { itemStyle: { color: "rgba(0,0,0,0.12)" } }, data: [150, 150, 150, 150, 150, 150, "-", 90, 90, 90, 90, 90, 90, "-", 45, 45, 45, 45, 45, 45, "-", 30, 30, 30, 30, 30, 30] }

Implementation

The four RTO modes differ in how the following 10 Patroni and HAProxy HA-related parameters are configured.

Component Parameter fast norm safe wide Description
patroni ttl 20 30 60 120 Leader lock TTL (seconds)
loop_wait 5 5 10 20 HA loop check interval (seconds)
retry_timeout 5 10 20 30 DCS operation retry timeout (seconds)
primary_start_timeout 15 25 45 95 Primary restart wait time (seconds)
safety_margin 5 5 10 15 Watchdog safety margin (seconds)
haproxy inter 1s 2s 3s 4s Normal state check interval
fastinter 0.5s 1s 1.5s 2s State transition check interval
downinter 1s 2s 3s 4s DOWN state check interval
rise 3 3 3 3 Consecutive successes to mark UP
fall 3 3 3 3 Consecutive failures to mark DOWN

Patroni Parameters

  • ttl: Leader lock TTL. Primary must renew within this time, otherwise lock expires and triggers election. Directly determines passive failure detection delay.
  • loop_wait: Patroni main loop interval. Each loop performs one health check and state sync, affects failure discovery timeliness.
  • retry_timeout: DCS operation retry timeout. During network partition, Patroni retries continuously within this period; after timeout, primary actively demotes to prevent split-brain.
  • primary_start_timeout: Wait time for Patroni to attempt local restart after PG crash. After timeout, releases Leader lock and triggers failover.
  • safety_margin: Watchdog safety margin. Ensures sufficient time to trigger system restart during failures, avoiding split-brain.

HAProxy Parameters

  • inter: Health check interval in normal state, used when service status is stable.
  • fastinter: Check interval during state transition, uses shorter interval to accelerate confirmation when state change detected.
  • downinter: Check interval in DOWN state, uses this interval to probe recovery after service marked DOWN.
  • rise: Consecutive successes required to mark UP. After new primary comes online, must pass rise consecutive checks before receiving traffic.
  • fall: Consecutive failures required to mark DOWN. Service must fail fall consecutive times before being marked DOWN.

Key Constraint

Patroni core constraint: Ensures primary can complete demotion before TTL expires, preventing split-brain.

loop_wait+2×retry_timeout≤ttlloop\_wait + 2 \times retry\_timeout \leq ttl

Data Summary

Recommendations

fast mode is suitable for scenarios with extremely high RTO requirements, but requires sufficiently good network quality (latency < 1ms, very low packet loss). Recommended only for same-rack or same-switch deployments, and should be thoroughly tested in production before enabling.

norm mode (default) is Pigsty’s default configuration, sufficient for the vast majority of same-datacenter deployments. In the model used by this page, the passive and active paths average about 34 and 35 seconds, while still providing a reasonable tolerance window against false failovers caused by network jitter.

safe mode is suitable for same-city cross-datacenter deployments with higher network latency or occasional jitter. The longer tolerance window effectively prevents false failovers from network jitter, making it the recommended configuration for cross-datacenter disaster recovery.

wide mode is suitable for cross-region or even cross-continent deployments with high network latency and possible public-network-level packet loss. In such scenarios, stability is more important than recovery speed, so an extremely wide tolerance window ensures very low false failover rate.

Mode Target RTO Passive RTO Active RTO Scenario
fast 30 16 / 23 / 29 1 / 24 / 29 Same switch, high-quality network
norm 45 27 / 34 / 41 2 / 35 / 41 Default, same DC, standard network
safe 90 53 / 66 / 78 3 / 61 / 73 Same-city active-active / cross-DC DR
wide 150 104 / 127 / 150 4 / 122 / 145 Geo-DR / cross-country
default 326 22 / 34 / 46 2 / 314 / 326 Patroni default params

Typically you only need to set pg_rto to the mode name, and Pigsty will automatically configure Patroni and HAProxy parameters. The current template looks up pg_rto with pg_rto in pg_rto_plan; a numeric or unknown key falls back directly to norm. Do not treat that fallback as a supported “RTO in seconds” configuration.

The mode configuration actually loads the corresponding parameter set from pg_rto_plan. You can modify or override this configuration to implement custom RTO strategies.

pg_rto_plan:  # [ttl, loop, retry, start, margin, inter, fastinter, downinter, rise, fall]
  fast: [ 20  ,5  ,5  ,15 ,5  ,'1s' ,'0.5s' ,'1s' ,3 ,3 ]  # rto < 30s
  norm: [ 30  ,5  ,10 ,25 ,5  ,'2s' ,'1s'   ,'2s' ,3 ,3 ]  # rto < 45s
  safe: [ 60  ,10 ,20 ,45 ,10 ,'3s' ,'1.5s' ,'3s' ,3 ,3 ]  # rto < 90s
  wide: [ 120 ,20 ,30 ,95 ,15 ,'4s' ,'2s'   ,'4s' ,3 ,3 ]  # rto < 150s

3.4.4 - Service Access

Pigsty uses HAProxy to provide service access, with optional pgBouncer for connection pooling, and optional L2 VIP and DNS access.

Split read and write operations, route traffic correctly, and deliver PostgreSQL cluster capabilities reliably.

Service is an abstraction: it represents the form in which database clusters expose their capabilities externally, encapsulating underlying cluster details.

Services are crucial for stable access in production environments, showing their value during automatic failover in high availability clusters. Personal users typically don’t need to worry about this concept.

Personal Users

The concept of “service” is for production environments. Personal users with single-node clusters can skip the complexity and directly use instance names or IP addresses to access the database.

For example, Pigsty’s default single-node pg-meta.meta database can be connected directly using three different users:

psql postgres://dbuser_dba:[email protected]/meta     # Connect directly with DBA superuser
psql postgres://dbuser_meta:[email protected]/meta   # Connect with default business admin user
psql postgres://dbuser_view:DBUser.Viewer@pg-meta/meta     # Connect with default read-only user via instance domain name

Service Overview

In real-world production environments, we use primary-replica database clusters based on replication. Within a cluster, one and only one instance serves as the leader (primary) that can accept writes. Other instances (replicas) continuously fetch change logs from the cluster leader to stay synchronized. Replicas can also handle read-only requests, significantly offloading the primary in read-heavy, write-light scenarios. Therefore, distinguishing write requests from read-only requests is a common practice.

Additionally, for production environments with high-frequency, short-lived connections, we pool requests through connection pool middleware (Pgbouncer) to reduce connection and backend process creation overhead. However, for scenarios like ETL and change execution, we need to bypass the connection pool and directly access the database. Meanwhile, high-availability clusters may undergo failover during failures, causing cluster leadership changes. Therefore, high-availability database solutions require write traffic to automatically adapt to cluster leadership changes. These varying access needs (read-write separation, pooled vs. direct connections, failover auto-adaptation) ultimately lead to the abstraction of the Service concept.

Typically, database clusters must provide this most basic service:

  • Read-write service (primary): Can read from and write to the database

For production database clusters, at least these two services should be provided:

  • Read-write service (primary): Write data: Can only be served by the primary.
  • Read-only service (replica): Read data: Can be served by replicas; falls back to primary when no replicas are available

Additionally, depending on specific business scenarios, there may be other services, such as:

  • Default direct service (default): Allows (admin) users to bypass the connection pool and directly access the database
  • Offline replica service (offline): Dedicated replica not serving online read traffic, used for ETL and analytical queries
  • Sync replica service (standby): Read-only service with no replication delay, handled by synchronous standby/primary for read queries
  • Delayed replica service (delayed): Access data from the same cluster as it was some time ago, handled by delayed replicas

Access Services

Pigsty’s service delivery boundary stops at the cluster’s HAProxy. Users can access these load balancers through various means.

The typical approach is to use DNS or VIP access, binding them to all or any number of load balancers in the cluster.

pigsty-access.jpg

You can use different host & port combinations, which provide PostgreSQL service in different ways.

Host

Type Sample Description
Cluster Domain Name pg-test Resolved by dnsmasq on INFRA nodes; with pg_dns_target: auto, points to the VIP when enabled, otherwise to the primary IP
Cluster VIP Address 10.10.10.3 When pg_vip_enabled is enabled, an L2 VIP managed by vip-manager and bound to the primary node
Instance Hostname pg-test-1 Access via any instance hostname (resolved by dnsmasq @ infra nodes)
Instance IP Address 10.10.10.11 Access any instance’s IP address

Port

Pigsty uses different ports to distinguish pg services

Port Service Type Description
5432 postgres Database Direct access to postgres server
6432 pgbouncer Middleware Access postgres through connection pool middleware
5433 primary Service Access primary pgbouncer (or postgres)
5434 replica Service Access replica pgbouncer (or postgres)
5436 default Service Access primary postgres
5438 offline Service Access offline postgres

Combinations

# Access via cluster domain (this example assumes a cluster VIP; without one, DNS resolves to the primary IP by default)
postgres://test@pg-test:5432/test # DNS -> L2 VIP -> primary direct connection
postgres://test@pg-test:6432/test # DNS -> L2 VIP -> primary connection pool -> primary
postgres://test@pg-test:5433/test # DNS -> L2 VIP -> HAProxy -> primary connection pool -> primary
postgres://test@pg-test:5434/test # DNS -> L2 VIP -> HAProxy -> replica connection pool -> replica
postgres://dbuser_dba@pg-test:5436/test # DNS -> L2 VIP -> HAProxy -> primary direct connection (for admin)
postgres://dbuser_stats@pg-test:5438/test # DNS -> L2 VIP -> HAProxy -> offline direct connection (for ETL/personal queries)

# Access via cluster VIP directly
postgres://[email protected]:5432/test # L2 VIP -> primary direct access
postgres://[email protected]:6432/test # L2 VIP -> primary connection pool -> primary
postgres://[email protected]:5433/test # L2 VIP -> HAProxy -> primary connection pool -> primary
postgres://[email protected]:5434/test # L2 VIP -> HAProxy -> replica connection pool -> replica
postgres://[email protected]:5436/test # L2 VIP -> HAProxy -> primary direct connection (for admin)
postgres://[email protected]:5438/test # L2 VIP -> HAProxy -> offline direct connection (for ETL/personal queries)

# Directly specify any cluster instance name
postgres://test@pg-test-1:5432/test # DNS -> database instance direct connection (singleton access)
postgres://test@pg-test-1:6432/test # DNS -> connection pool -> database
postgres://test@pg-test-1:5433/test # DNS -> HAProxy -> connection pool -> database read/write
postgres://test@pg-test-1:5434/test # DNS -> HAProxy -> connection pool -> database read-only
postgres://dbuser_dba@pg-test-1:5436/test # DNS -> HAProxy -> database direct connection
postgres://dbuser_stats@pg-test-1:5438/test # DNS -> HAProxy -> database offline read/write

# Directly specify any cluster instance IP access
postgres://[email protected]:5432/test # Database instance direct connection (directly specify instance, no automatic traffic distribution)
postgres://[email protected]:6432/test # Connection pool -> database
postgres://[email protected]:5433/test # HAProxy -> connection pool -> database read/write
postgres://[email protected]:5434/test # HAProxy -> connection pool -> database read-only
postgres://[email protected]:5436/test # HAProxy -> database direct connection
postgres://[email protected]:5438/test # HAProxy -> database offline read-write

# Smart client: read/write separation via URL
postgres://[email protected]:6432,10.10.10.12:6432,10.10.10.13:6432/test?target_session_attrs=primary
postgres://[email protected]:6432,10.10.10.12:6432,10.10.10.13:6432/test?target_session_attrs=prefer-standby

3.5 - Point-in-Time Recovery — A Time Machine for PostgreSQL

High availability handles machine failure; point-in-time recovery handles incorrect data. Pigsty uses pgBackRest to provide PITR out of the box, allowing a cluster to return to any recoverable point covered by its backup and WAL history.

If data, a table, or even a database is deleted accidentally, Point-in-Time Recovery (PITR) can return the cluster to an earlier state.

This capability, once treated as specialist DBA work, is enabled by Pigsty’s standard PostgreSQL configuration.

Replication Is Not Backup

High availability can fail over to another instance when hardware fails. It has a natural blind spot, however: replication is not backup.

Streaming replication faithfully sends every primary change to every replica within milliseconds, including a DELETE without a WHERE clause or a DROP TABLE issued against the wrong database. Failover handles a broken machine; when the data itself is wrong, every replica can contain the same error.

Database disasters therefore fall into two broad classes. Redundancy handles physical service failure through multiple copies and automatic failover. Logical errors require history: a base backup plus continuous WAL archives from which PostgreSQL can reconstruct a state before the mistake.

Threat High Availability Delayed Cluster PITR
Hardware or instance failure ✔ Automatic failover ✘ ✔, with a longer RTO
Accidental DML, table drop, or database drop ✘ The error is replicated ✔ Within the delay ✔ At any recoverable point
Defective software corrupts data over time ✘ The error is replicated ✔ Within the delay ✔ Try different recovery targets
Entire cluster or site is lost ✘ ✘ ✔ Only if the repository survives that failure domain

These mechanisms complement one another: HA restores service quickly, a delayed cluster provides a short undo window, and PITR is the final historical recovery path.

How the Time Machine Works

A database can be viewed as a state machine. A base backup is a complete physical snapshot at one point, while WAL (Write-Ahead Log) records every subsequent state change. With a snapshot and an unbroken WAL history starting from it, PostgreSQL can replay the database to any target covered by that history. The backup determines how far back recovery can start; the latest archived WAL determines how close to the present it can reach.

Base backup + WAL archive = point-in-time recovery

Pigsty orchestrates both inputs. Cluster initialization attempts an initial full backup by default, and the primary continuously sends completed WAL segments to the selected repository. See How PITR Works for the complete model of backups, archives, targets, and timelines.

Available Out of the Box

PITR is enabled in Pigsty’s standard PostgreSQL configuration. Each cluster is prepared with a backup repository, WAL archiving, and recovery tooling powered by pgBackRest. The policy remains declarative and can be customized with a few parameters:

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pgbackrest_method: minio       # Silo / S3-compatible storage; local is the default
    pg_crontab: [ '00 01 * * * /pg/bin/pg-backup full' ]  # daily full backup at 01:00

The default local method stores backups under /pg/backup and retains two full backups. With one successful full backup per day, the resulting window is roughly 24–48 hours. Selecting the remote minio preset places the repository in Silo or compatible S3 storage, enables AES-256-CBC repository encryption, and uses time-based retention. With a 14-day retention setting and weekly full backups, the steady-state recovery window is roughly 14–21 days. Treat both ranges as policy estimates: actual coverage starts at the oldest usable backup and ends at the latest WAL that reached the repository.

Recovery is declarative too: specify a target, then let the playbook stop the cluster, restore files, replay WAL, and rebuild HA. An operator must still verify the recovered business state.

./pgsql-pitr.yml -l pg-meta -e '{"pg_pitr": { "time": "2026-07-11 10:00:00+08", "action": "promote" }}'

This follows Pigsty’s declarative configuration model: backup policy is part of the cluster definition, and a recovery target is another declared parameter.

Benefits and Costs

PITR materially improves data integrity and availability:

  • RPO (maximum data loss) is usually reduced to minutes, bounded by WAL that had not reached a surviving repository.
  • RTO (time to restore service) becomes tens of minutes to hours rather than permanent loss, depending on backup size, WAL replay distance, and disk or network throughput.
Standalone strategy Event RTO RPO
No backup Host and local data are lost Permanent loss All data
Base backups only Host and local data are lost Backup size and bandwidth, often hours Changes since the latest backup
Base backups + WAL archives Host and local data are lost Backup size, replay distance, and bandwidth WAL not yet present in the surviving repository

The costs fall mainly into three areas:

  • Confidentiality: backups are another copy of business data and need encryption and access control. Pigsty’s remote preset enables repository encryption, but its default password must be changed.
  • Resources: backups consume storage and archiving consumes bandwidth. Compression, bundling, and block incremental backup reduce this cost but do not eliminate capacity planning.
  • Operations: backup status must be monitored and recovery must be rehearsed. A green backup job alone is not proof that the data can be restored within the required RTO.

PITR by itself does not replace HA. A production design normally combines HA for physical failures with PITR for logical errors and site-level recovery.

Next Steps

  • How PITR Works: snapshots, WAL history, recovery windows, targets, and timelines
  • PITR Architecture: pgBackRest, repository selection, archive flow, scheduling, and failover behavior
  • PITR Tradeoffs: failure domains, capacity, retention, and backup frequency
  • Declarative Recovery: the pg_pitr parameter, pgsql-pitr.yml, and pig pitr
  • PITR Scenarios: accidental deletion, bad releases, investigation, and site loss

For the operational runbooks, see PGSQL Backup and Recovery.

3.5.1 - How PITR Works

Snapshots, WAL history, recovery windows, recovery targets, and timelines: the five concepts needed to reason accurately about PostgreSQL PITR.

If a database is a state machine, WAL (Write-Ahead Log) is its ordered change history. PostgreSQL records each modification in WAL before applying it to data files. Save a physical snapshot at one point, preserve all later WAL, and PostgreSQL can replay that history to a selected consistent state.

PITR is therefore the combination of three simple elements: a snapshot (base backup), history (WAL archive), and a target (where replay should stop).

Snapshot: Base Backup

A base backup is a physical snapshot of the whole PostgreSQL cluster and supplies a starting point for recovery. Pigsty uses pgBackRest to create and manage three backup types:

Type Contents Recovery characteristics
Full All database-cluster files Self-contained, shortest chain, largest backup
Differential Changes since the latest full backup Restore uses the full plus the differential
Incremental Changes since the latest backup of any type Smallest backup, restore depends on its complete chain

The wrapper pg-backup [full|diff|incr] triggers a backup. With no argument it requests incr; pgBackRest creates a full backup instead when no valid full exists. pg_crontab declares recurring jobs and installs them in the postgres user’s crontab.

Backup frequency affects recovery time: the newer the usable backup, the less WAL must be replayed to reach a given target. See PITR Tradeoffs.

WAL History

A snapshot reaches only its own state. WAL archiving preserves every later change needed to advance beyond it. Pigsty’s standard Patroni templates enable archiving and ask PostgreSQL to hand each completed WAL segment to pgBackRest:

archive_mode: 'on'
archive_command: 'pgbackrest --stanza=pg-meta archive-push %p'
archive_timeout: 300

Two implementation details matter:

  • archive_timeout: 300: on a low-write cluster, PostgreSQL can force a segment switch after five minutes so a partially filled segment does not wait indefinitely. This normally keeps the right edge of the recovery window within minutes when WAL is being generated; it is not a promise that every commit is already remote.
  • Asynchronous archive: pgBackRest uses /pg/spool with archive-async=y to batch transfers. Pigsty sets archive-push-queue-max=4GiB; if repository failure lets the queue cross that bound, pgBackRest can drop the queued WAL to protect local disk. That creates an archive gap, so a new full backup is required to establish a fresh recoverable chain.

Expiration is automatic. When old backups expire under the repository policy, pgBackRest also expires archived WAL that no remaining backup needs, unless archive retention is overridden explicitly.

Recovery Window

The backup and its continuous WAL history form a recovery window:

  • Left boundary: the start of the oldest usable remaining backup chain. In practical time-based descriptions, this is usually summarized by the oldest retained full backup’s time.
  • Right boundary: the latest WAL successfully archived to a repository that survives the incident.

The window moves forward as new backups arrive and old chains expire. Pigsty’s local preset keeps two full backups; with one successful full per day, coverage is roughly one to two days. The minio preset uses retention_full_type: time with retention_full: 14; with weekly full backups, the oldest retained chain normally yields roughly 14–21 days of steady-state coverage. These are estimates, not SLAs: missed backups, archive gaps, explicit archive-retention overrides, or repository loss change the actual window. Verify it with pig pb info and restore drills.

See PITR Tradeoffs and Backup Policy.

Targets: Where Replay Stops

PostgreSQL supports several ways to locate a state inside the recovery window. Pigsty exposes six target types through pg_pitr:

pg_pitr type Meaning Typical use
default Replay through all WAL available from the repository Restore the newest archived state after total loss
time Stop at a timestamp Recover from accidental DML or DDL
xid Stop at a transaction ID Exclude a precisely identified bad transaction
lsn Stop at a WAL location Low-level exact targeting
name Stop at a restore point created with pg_create_restore_point() Planned change checkpoint
immediate Stop as soon as the selected backup becomes consistent Validate or expose the selected backup state quickly

The set field is different: it chooses which backup set pgBackRest restores as the starting snapshot; it is not itself a replay stop target.

Boundary Semantics

Targets are inclusive by default: the transaction at the target is retained. To stop immediately before a known bad target, set exclusive: true, which maps to recovery_target_inclusive = false.

Transactions remain atomic. Committed transactions before the effective target survive; transactions not committed at that point are rolled back. Recovery produces a consistent database state rather than half of a transaction.

Timelines

Restoring to the past and accepting new writes creates a fork in history. PostgreSQL uses a timeline to distinguish each branch. PITR promotion, replica promotion, and failover can all create a new timeline; new WAL does not overwrite the old timeline’s files.

gitGraph
    commit id: "Full backup"
    commit id: "Normal writes"
    commit id: "Bad change"
    commit id: "More writes"
    branch Timeline-2
    checkout Timeline-2
    commit id: "PITR before bad change"
    commit id: "New writes"

Keeping the old history allows another attempt if the first target was wrong. The timeline field can select a timeline; Pigsty’s recovery declaration defaults to latest.

Continue with PITR Architecture to see how these concepts map to Pigsty components and configuration.

3.5.2 - PITR Architecture

Pigsty implements PITR with pgBackRest: repository selection, archive flow, scheduling, primary-aware backup execution, performance defaults, and observability.

The PITR principle is compact; the engineering is not. WAL archiving must not stall production writes, object-storage backups need encryption, backup jobs must follow the primary after failover, shared repositories must isolate clusters, and large numbers of small objects can limit throughput.

Pigsty uses pgBackRest as its backup engine and ships production-oriented defaults for those concerns. This page describes the engine, repository abstraction, archive path, scheduler, and primary-aware execution model.

Backup Engine: pgBackRest

Pigsty uses pgBackRest for three responsibilities: create base backups with backup, receive WAL with archive-push, and restore data with restore plus archive-get.

Relevant capabilities include:

  • Parallelism: backup, archive, and restore operations can use multiple processes.
  • Backup chains: full, differential, incremental, and block incremental backups reduce repeated transfer and storage.
  • Compression and encryption: zstd compression and AES-256-CBC repository encryption are built in.
  • Repository backends: POSIX filesystems, S3-compatible services such as Silo and MinIO, Azure, GCS, and SFTP are supported by pgBackRest.
  • Bundling: small files can be packed into larger repository objects, reducing object-storage overhead.

pgBackRest separates cluster histories using a stanza. Pigsty maps the stanza name directly to pg_cluster, allowing multiple clusters to share one storage service without sharing a backup identity:

repository
├── backup/
│   ├── pg-meta/          # base backups for pg-meta
│   └── pg-test/          # base backups for pg-test
└── archive/
    ├── pg-meta/          # archived WAL for pg-meta
    └── pg-test/          # archived WAL for pg-test

Repository Abstraction

Two parameters define repository selection. pgbackrest_method chooses one repository name, and pgbackrest_repo is a dictionary of candidate definitions. Pigsty v4.5.0 renders only the selected pgbackrest_repo[pgbackrest_method] entry as pgBackRest repo1; listing both local and minio does not enable two active repositories.

pgbackrest_method: local          # local, minio, or a custom key below
pgbackrest_repo:
  local:
    path: /pg/backup
    retention_full_type: count
    retention_full: 2             # retain two full backups; a third may exist before expiration
  minio:
    type: s3
    s3_endpoint: sss.pigsty
    s3_region: us-east-1
    s3_bucket: pgsql
    s3_key: pgbackrest
    s3_key_secret: S3User.Backup
    s3_uri_style: path
    path: /pgbackrest
    storage_port: 9000
    storage_ca_file: /etc/pki/ca.crt
    block: y
    bundle: y
    bundle_limit: 20MiB
    bundle_size: 128MiB
    cipher_type: aes-256-cbc
    cipher_pass: pgBackRest       # replace this default secret in production
    retention_full_type: time
    retention_full: 14

The presets intentionally differ. local favors simplicity and fast local restore; it is unencrypted, unbundled, and retained by full-backup count. minio targets a remote Silo or compatible S3 repository, enabling encryption, bundles, block incremental backup, and time-based retention.

Rendering is mechanical: underscores in the chosen repository’s keys become hyphens and each key gets a repo1- prefix in /etc/pgbackrest/pgbackrest.conf. A custom cloud repository can therefore use pgBackRest options directly:

pgbackrest_method: s3
pgbackrest_repo:
  s3:
    type: s3                         # repo1-type=s3
    s3_endpoint: s3.us-west-1.amazonaws.com
    s3_region: us-west-1
    s3_bucket: <your_bucket>
    s3_key: <your_access_key>
    s3_key_secret: <your_secret>
    s3_uri_style: host
    path: /pgbackrest
    cipher_type: aes-256-cbc
    cipher_pass: <your_password>
    retention_full_type: time
    retention_full: 90

See Backup Repository for Silo, external S3-compatible storage, versioning, object locking, TLS, and credential details.

Archiving and Scheduling

When pgbackrest_enabled is true, as it is by default, the Patroni templates configure:

archive_mode: 'on'
archive_timeout: 300
archive_command: 'pgbackrest --stanza=<cluster> archive-push %p'

Base backups enter the system in two ways:

  • Initial backup: after bootstrapping a top-level primary, Pigsty attempts a backup when pgbackrest_init_backup is true. The task ignores backup failure and writes /etc/pgbackrest/initial.done only after success, so the marker means “completed,” not merely “attempted.”
  • Scheduled backup: pg_crontab installs jobs in the database superuser’s crontab. Its role default is an empty list; standard example configurations usually add a daily 01:00 full backup.
pg_crontab: [ '00 01 * * * /pg/bin/pg-backup full' ]

pg-backup [full|diff|incr] is a small wrapper around pgbackrest backup. With no argument it requests an incremental backup, which pgBackRest promotes to a full backup if no usable full exists.

Backups Follow the Primary

pgBackRest and the same scheduled job are installed on every PostgreSQL node, but pg-backup checks /pg/bin/pg-role and only proceeds on the current primary. Replicas fail fast rather than writing a competing backup.

That design decouples the backup schedule from the HA topology:

  • all members receive the same repository configuration and crontab;
  • after failover, the new primary becomes eligible for subsequent backups and WAL archiving without rewriting the schedule;
  • one current primary owns the authoritative write flow to a stanza.

With a non-local repository, Pigsty also adds pgBackRest after basebackup in Patroni’s create_replica_methods. Patroni tries basebackup first; if that method fails, it can restore a replica from the repository with pgbackrest --delta restore, shifting the copy load away from the primary.

Performance Defaults

The shipped pgBackRest template favors light production overhead and aggressive restore throughput:

Setting v4.5.0 behavior Rationale
Compression compress-type=zst Balance compression ratio and throughput
Backup/archive workers One quarter of CPU, clamped to 2–4 Limit competition with the database
Restore workers All detected CPU, capped at 8 Minimize restore time
Asynchronous archive archive-async=y, spool under /pg/spool Batch transfer without synchronous object-store latency
Archive queue limit archive-push-queue-max=4GiB Bound local spool growth
Fast backup start start-fast=y Request an immediate checkpoint
Incremental restore delta=y Reuse destination files that already match

The 4 GiB queue is a safety tradeoff: if the repository remains unavailable and the queue exceeds the limit, pgBackRest can discard queued archive files. PostgreSQL continues running, but the WAL archive becomes incomplete and a new full backup is needed to establish a new recovery chain. See How PITR Works.

Observability

When both backup and exporter settings are enabled, pgbackrest_exporter runs on each PostgreSQL node and exposes metrics on port 9854. The monitoring stack uses those metrics for backup age, type, size, duration, and error visibility.

Useful diagnostic entry points include:

Entry Purpose
pb info Shell helper for pgbackrest info using the configured stanza
/pg/log/pgbackrest/ pgBackRest backup, archive, and restore logs
`pg-backup full diff

See Backup Administration for operational checks, then PITR Tradeoffs for policy design.

3.5.3 - PITR Tradeoffs

Repository location determines the failure domain, retention determines the recovery window, and backup frequency shapes restore time. Together they define a backup policy.

A backup is an insurance policy. Its premium is storage, network traffic, and operational work; its benefit is how much data can be recovered and how quickly service can return. There is no universal free policy: more history normally needs more capacity, while a shorter RTO normally needs newer backups and tested procedures.

Designing a policy means answering three questions: where is the repository, how long is history retained, and how often are backups taken?

Where: Choose the Failure Domain

Repository location is the most important decision because it defines which disasters the backup survives.

A local repository (pgbackrest_method: local) stores backups on the primary’s local filesystem. It is simple, fast, and has no remote service dependency. But data and backup normally share one host failure domain: loss of the machine, disk, or filesystem can destroy both. Local backup protects well against logical errors, but not total host loss unless /pg/backup is deliberately placed on independent storage.

An object-storage repository (pgbackrest_method: minio or a custom S3 definition) sends backups to Silo or S3. It becomes an independent disaster-recovery copy only when deployed outside the database host or site failure domain. Pigsty’s minio preset also enables AES-256-CBC repository encryption, bundling, and block incremental backup. Recovery throughput then depends on the network and storage service, and that service adds operational responsibility.

Scenario Recommended repository Reason
Development, test, demo local Minimal dependencies; rebuild is acceptable
Production Dedicated Silo or compatible S3 storage Independent failure domain and encrypted repository
Cloud deployment Managed S3-compatible or cloud object storage supported by pgBackRest Independent storage and lower operational burden
Ransomware/compliance Versioned storage plus correctly configured object lock/retention Prevent privileged database-host access from deleting protected versions

The backup repository is itself sensitive business data. Change the default access keys and cipher_pass, restrict access, protect credentials separately from the database hosts, and verify any object-lock policy. See Backup Repository.

How Long: Capacity and Recovery Window

Longer retained history generally consumes more storage, but compression, deduplication, block incremental backup, database change rate, and the mix of full/differential/incremental backups determine the actual amount. Measure real backup and WAL growth instead of relying on a fixed multiplier.

For an illustrative 100 GB database changing by 10 GB per day, before compression:

  • Daily full, retain two (local preset policy): about 200 GB of full backups plus WAL, commonly giving roughly a one-to-two-day window when every job succeeds.
  • Weekly full, daily incremental, retain full history by 14 days (minio preset policy): the oldest surviving weekly chain commonly produces roughly 14–21 days of coverage. Capacity must include multiple full backups, their incrementals, archived WAL, and transient retention-plus-one behavior during expiration.

The precise window is not the configuration number alone. It runs from the oldest usable backup chain to the newest WAL present in the surviving repository. pgBackRest’s time retention removes an old full only when another qualifying full can satisfy the period, and related incrementals and WAL follow the retained full chains. Check pig pb info, monitor archive health, and prove coverage with a restore.

Choose a window long enough to cover the delay between an error occurring and being detected. A dropped table may be noticed in minutes; slow corruption or a month-end reconciliation failure can take weeks to surface.

How Often: Backup Frequency and RTO

Restore time has two main components: restore a backup chain, then replay WAL to the target. Backup size and storage throughput shape the first; the distance between the chosen backup and target shapes the second.

WAL replay is largely serial. On a write-heavy database, restoring from a weekly full immediately before the next full can require nearly a week of replay. Daily incremental backups reduce that replay distance while transferring only changes since the previous backup. They still depend on a valid chain, so monitor and test the entire chain rather than only the newest file.

A useful rule is: within the available backup window and production load budget, take backups often enough that measured restore time meets the RTO.

Pigsty Presets

Pigsty provides two candidate repository definitions, but pgbackrest_method selects one for the generated repo1 configuration.

Standard policy: local repository and daily full backup. It is simple and restores through local I/O, making it suitable for development or environments where host-level disaster recovery is provided separately:

pgbackrest_method: local
pg_crontab: [ '00 01 * * * /pg/bin/pg-backup full' ]
# Local preset retains two full backups; actual coverage depends on successful jobs and WAL continuity.

Production policy: remote Silo/S3 repository, weekly full, daily incremental. It separates the repository failure domain and uses the encrypted minio preset:

pgbackrest_method: minio
pg_crontab:
  - '00 01 * * 1 /pg/bin/pg-backup full'
  - '00 01 * * 2,3,4,5,6,7 /pg/bin/pg-backup'
# The preset retains full history by 14 days; weekly fulls commonly yield about 14–21 days.

Do not describe the default pgbackrest_repo dictionary as a “dual-repository” setup: it contains alternative definitions, and the template renders only pgbackrest_repo[pgbackrest_method] as repo1. A real multi-repository pgBackRest design requires explicit advanced configuration and an independently tested backup, expiration, and restore workflow; the two Pigsty presets alone do not create it.

Use Backup Policy for capacity modelling and schedule details.

A Backup Is Proven by Restore

Monitoring a successful backup job is necessary but insufficient. Add clone restore drills to routine operations so you can answer:

  1. Is the chain usable? Restore it end to end and validate data.
  2. What is the measured RTO? Database size and WAL volume change over time.
  3. Can the on-call operator execute the runbook? The first full exercise should not happen during an incident.

A clone recovery leaves the source cluster online but overwrites the designated destination cluster, so verify the exact target and use disposable infrastructure. See Declarative Recovery for the recovery interface.

3.5.4 - Declarative Recovery

Declare the desired pg_pitr recovery target and let pgsql-pitr.yml or pig orchestrate the recovery workflow.

The value of a backup system is realized at restore time, often during an incident when every minute matters. A traditional PITR procedure requires a long sequence of coupled manual steps: pause HA, stop PostgreSQL, prepare recovery settings, restore the backup, replay WAL, validate the target, rebuild metadata, and start the cluster again.

Pigsty applies the same approach used by declarative configuration to recovery: declare the recovery target, then let the orchestration tools stop the cluster, restore the data, replay WAL, and return control to the operator.

Declare a Recovery Target

Describe the target with the pg_pitr parameter and execute it with pgsql-pitr.yml. The most common form restores a cluster to a specific time:

./pgsql-pitr.yml -l pg-meta -e '{"pg_pitr": { "time": "2026-07-11 10:00:00+08", "action": "promote" }}'

The six recovery target types and the rest of the recovery behavior are expressed through fields in this parameter:

pg_pitr:                           # Recovery declaration; every field is optional
  cluster: pg-meta                 # Source backup stanza; defaults to this cluster
  type: time                       # default | time | xid | lsn | name | immediate
  time: '2026-07-11 10:00:00+08'   # Mutually exclusive with xid, lsn, and name
  exclusive: false                 # Stop before the target; inclusive by default
  action: promote                  # Explicit promotion; a targeted restore defaults to pause
  timeline: latest                 # Target timeline; latest by default
  set: latest                      # Starting backup set; selected automatically by default
  repo: { ... }                    # Temporary repository definition when not using local config
  backup: false                    # Move the old data directory to /pg/data-backup first
  archive: true                    # Preserve archiving; exploratory recovery can set false
  db_include: [ ... ]              # Restore only selected databases
  data: /pg/data                   # Destination data directory

See Restore Operations for the complete field reference and examples.

What the Playbook Does

pgsql-pitr.yml turns the manual recovery workflow into six stages and supports Ansible tags for staged execution:

Stage Action
print Print the source cluster, target, and restore command; this stage reports the plan and does not prompt for confirmation
pause Run patronictl pause so Patroni does not intervene during maintenance
stop Stop Patroni and PostgreSQL on replicas, then on the primary
pitr Render recovery settings, run an incremental pgBackRest restore, start PostgreSQL to replay WAL, wait for consistency, and print control data
etcd Remove stale cluster metadata from etcd so old and new timelines are not mixed
start Start Patroni again, resume HA management, and rebuild replicas

Several details are important:

  • Incremental restore: pgBackRest uses delta, so it rewrites only files that differ from the backup. For large databases, this can reduce RTO substantially.
  • Verification, not assumption: the playbook prints checkpoint LSN, timeline, and NextXID data from pg_controldata; an operator must still verify that the recovered business state is correct.
  • Rollback copy: with backup: true, the original data directory is moved to /pg/data-backup before recovery. A later run with backup: true removes an existing /pg/data-backup, so this is not a versioned snapshot store.
  • Staged execution: run -t down, -t pitr, and -t up separately when you want an operator checkpoint between phases. Completion of the pitr phase means PostgreSQL reached a consistent recovery state; for a time, XID, LSN, or named target, also confirm WAL replay reached that target.

The action field controls what happens at the target: promote opens a new timeline, pause waits at the target for inspection, and shutdown stops PostgreSQL there. A targeted recovery defaults to pause when action is omitted. To preserve a manual gate for pause or shutdown, run the stages separately; a one-shot recovery should choose promote explicitly. The playbook performs the mechanical workflow, but it cannot decide whether the recovered data is correct.

Command-Line Recovery with pig

The pig CLI provides single-instance PITR orchestration directly on a database node, without requiring the management node or an Ansible environment:

pig pitr -t "2026-07-11 10:00:00+08"    # Recover to a point in time
pig pitr --xid 250000 -X                # Stop before transaction 250000
pig pitr -d                             # Replay through the WAL archive
pig pitr -I --no-restart                # Prepare immediate recovery and leave PostgreSQL stopped

pig pitr validates the target, stanza, and available backups; stops Patroni and PostgreSQL; performs the restore; optionally starts PostgreSQL; and prints post-recovery instructions. For a Patroni-managed data directory, Patroni remains stopped afterward. Validate the data, then use pig pt start to return the instance to HA management. This single-node workflow does not clear etcd, rebuild replicas, or automatically rejoin the cluster, and it refuses destructive forced shutdown unless --force-stop is supplied explicitly.

The lower-level pig pb commands wrap pgBackRest: pb info lists backups, pb backup creates a backup, and pb restore performs a raw restore. There is a deliberate safety boundary: pig pb restore refuses to run while Patroni still manages the instance, because Patroni could restart PostgreSQL during the restore. Use pig pitr or pgsql-pitr.yml for Patroni-managed instances.

In-Place and Clone Recovery

The same mechanism supports two different workflows:

Dimension In-place recovery Clone recovery
Method Roll the production cluster back Restore a source backup into a different cluster
Downtime Required during recovery The source production cluster remains online
Effect Discards all writes after the target Does not affect the source; the destination is overwritten and can be retried
Best for Whole-cluster corruption or disaster recovery Recovering deleted objects, audit work, and recovery drills

For a clone recovery, the cluster field names the source backup stanza. This example restores the historical state of pg-meta into pg-test without stopping the source cluster:

./pgsql-pitr.yml -l pg-test -e '{"pg_pitr": { "cluster": "pg-meta", "time": "2026-07-11 10:00:00+08", "archive": false, "action": "promote" }}'

Exporting an accidentally deleted table from the clone and importing it into production is generally safer than rolling the entire production cluster back. See Clone a Database Cluster for the complete workflow and cleanup steps.

After Recovery

Recovery completion is not the end of the incident. Include these steps in the closeout checklist:

  1. New timeline, new backup: after promotion, create a full backup with pg-backup full so a recoverable window exists on the new timeline.
  2. Archiving state: if an exploratory restore used archive: false, restore normal archiving as described in Post-Recovery.
  3. Clone cleanup: a clone’s cluster identity and source backup stanza do not match. Recreate the destination stanza before enabling its own backups; see Clone a Database Cluster.

The tools execute the procedure; operators still decide the target, whether to restore in place or into a clone, and whether the recovered data is correct. Continue with PITR Scenarios for that decision framework.

3.5.5 - PITR Scenarios

How to choose a recovery target and workflow for accidental DML, dropped objects, defective releases, investigations, and site loss — and why recovery drills must be routine.

During an incident, the most expensive resource is often decision time. Pigsty can orchestrate the mechanical recovery steps, but an operator must still answer three questions: what is the target, should recovery be in place or into a clone, and how will the result be validated?

Read and rehearse this framework before an incident.

Decision Framework

Scenario Typical problem Recommended workflow Target
Accidental DML DELETE or UPDATE affects the wrong rows Clone, validate, then copy back data time / xid
Dropped table, schema, or database DROP or an incorrect migration Clone, validate, then copy back objects time / name
Defective release or batch corruption Software writes incorrect data for a period Clone and compare before choosing repair or cutover time / xid
Audit, investigation, or forensics Inspect historical state Clone and hold at the target for inspection time / lsn
Whole-cluster or site loss Hosts or storage are gone or encrypted Recover in place on replacement infrastructure default / time

Two principles apply throughout:

  • Stop the damage first. Pause the defective application or remove its write access before choosing a target. The window is moving, but a rushed restore to the wrong cluster can cause a second incident.
  • Prefer a clone while production is usable. It leaves the source untouched, supports repeated target selection, and allows validation before export or cutover. It does overwrite the designated destination cluster. In-place recovery is appropriate when the whole cluster is unusable or the business has explicitly accepted rolling every database back.
flowchart TD
    A["Data error detected"] --> B["Contain the source of bad writes"]
    B --> C{"Can production still serve?"}
    C -->|Yes| D["Clone recovery<br/>validate and copy back or cut over"]
    C -->|No| E["In-place recovery<br/>or rebuild on new infrastructure"]
    D --> F["Validate, take a new backup, review the incident"]
    E --> F

Accidental DML

A DELETE without WHERE, an incorrect UPDATE, or a defective batch job is the most common PITR use case.

First locate the error using application logs, PostgreSQL logs, metrics, or audit records. A timestamp is usually sufficient. If the exact transaction ID is known, xid plus exclusive: true can stop immediately before that transaction.

# If the deletion occurred around 10:15, clone the state from 10:14
./pgsql-pitr.yml -l pg-test -e '{"pg_pitr": { "cluster": "pg-meta", "time": "2026-07-11 10:14:00+08", "archive": false, "action": "promote" }}'

# If the deleting transaction was 250000, stop immediately before it
./pgsql-pitr.yml -l pg-test -e '{"pg_pitr": { "cluster": "pg-meta", "xid": "250000", "exclusive": true, "archive": false, "action": "promote" }}'

Validate the recovered rows, then copy only the required data back with pg_dump, COPY, or an application-specific reconciliation procedure. If a configured delayed cluster is still inside its delay window, reading from it may be faster than PITR.

Dropped Objects

The same approach applies to DROP TABLE, DROP DATABASE, or a migration executed in the wrong environment, with an even stronger preference for a clone. Rolling the entire production cluster back to recover one object also discards every legitimate write after the target.

Restore a separate destination to before the DDL, validate the object, export it with pg_dump, and import it into production. For planned high-risk changes, create a named restore point with pg_create_restore_point() beforehand; a name target then removes timestamp ambiguity.

Defective Release or Batch Corruption

When a faulty release corrupts data for hours, the challenge is usually identifying the last clean state and the full impact. A clone provides a clean comparison set. Restore repeatedly to candidate times, compare it with production, and decide whether to copy back corrected rows or cut over to a recovered cluster.

This decision needs application-owner validation: a successful PostgreSQL restore proves consistency at a target, not that the target represents correct business state.

Audit and Investigation

Questions such as “what was this balance at month end?” require historical state. Restore into a separate destination, stop at a time, LSN, XID, or named restore point, and inspect without altering the source.

action: pause is the targeted-restore default and holds recovery at the target for inspection; it does not itself configure read-only access or create a separate cluster. The inventory limit and cluster source field determine the destination workflow. Run -t down, -t pitr, and -t up separately when you need an operator gate before promotion, and enforce read-only access explicitly if the investigation requires it. immediate means “stop at the first consistent point,” not “choose a historical timestamp.”

Site Loss

If every database host is destroyed or encrypted, HA cannot help. Recovery requires a repository and the other control-plane assets to have survived outside that failure domain. That survivor can be Silo/S3, another protected host or filesystem, or another tested pgBackRest backend; a remote object store is recommended but the essential property is independent failure-domain survival.

Rebuild hosts, restore the declarative inventory, credentials, and PKI, point the cluster at the surviving repository, then restore through the end of archived WAL:

./pgsql-pitr.yml -l pg-meta -e '{"pg_pitr": {"action": "promote"}}'

Inventory and backup data are necessary but not sufficient. Preserve installation media or package repositories, repository credentials and encryption passwords, CA material, custom files, DNS dependencies, and an independently accessible runbook. Keep secrets encrypted and separate from both the database hosts and ordinary source control.

Make Recovery a Routine Drill

The first end-to-end execution of any of these workflows should not occur during a production incident. Use a disposable destination to rehearse clone recovery regularly and after material architecture changes. Measure three outcomes:

  1. Usability: can the backup and complete WAL chain be restored and validated?
  2. RTO: how long does the actual restore and replay take now?
  3. Operator readiness: can the on-call engineer identify source and destination, select a target, and follow the safety gates?

See Restore Operations and Clone a Database Cluster for the task-level runbooks.

3.6 - Monitoring System

How Pigsty’s monitoring system is architected and how monitored targets are automatically managed.

Pigsty’s monitoring system has three pillars—metrics, logs, and alerting—and is available out of the box. Logs and alerts are also important inputs for audit and traceability. It can monitor clusters managed by Pigsty, existing PostgreSQL clusters, and external RDS services.

Monitoring Targets

Pigsty monitoring covers these core targets:

  • PostgreSQL clusters and instances (SQL performance, connections, replication, transactions, checkpoints, WAL)
  • Infrastructure components (Grafana, VictoriaMetrics, Alertmanager, Nginx, etc.)
  • Host nodes (CPU, memory, disk, network, kernel)
  • Key middleware (ETCD, MINIO, REDIS, JUICE, VIBE, etc.)

Technology Stack

Component Purpose
Grafana Visualization dashboards, unified entry point, alert views
VictoriaMetrics Time-series metric ingestion, storage, and query
VictoriaLogs Structured log ingestion, indexing, and search
VMAlert + Alertmanager Alert rule evaluation and notification delivery
Exporter / Agent Database/system metric exposure and log forwarding

Onboarding Modes

Pigsty supports three monitoring onboarding modes:

Mode Use Case Entry
FULL Database is deployed and managed directly by Pigsty PGSQL Monitoring System
MANAGED Existing PostgreSQL cluster with SSH-manageable nodes Monitor Existing Cluster
RDS Cloud database accessible only by connection string Monitor RDS

Continue Reading

3.7 - Security and Compliance

Pigsty manages authentication, authorization, encryption, audit, backup, and recovery as code, with a clear path from the default baseline to production hardening.

The database is usually the most sensitive component in an information system: it stores the most valuable data, so attacks and failures can have the most serious consequences. Database security is not a feature that can be enabled with one switch. It is the combined answer to a series of questions: Who can connect? What can they do after connecting? Can traffic be intercepted? Are operations recorded? Can damaged, lost, or deleted data be recovered?

Pigsty turns these answers into an out-of-the-box security baseline and manages it through declarative configuration: HBA rules, roles and privileges, certificates, encryption, backups, and audit policies are declared as parameters in the inventory, then rendered and applied by idempotent playbooks.

This Security as Code approach is itself an important security practice. Policies can be versioned, reviewed, and traced, while one inventory provides a consistent baseline across many instances. When an auditor asks who can access a database, you can start from a readable YAML declaration, then verify the generated HBA rules and database grants against the running system.

Security as Code

In traditional operations, security settings are often scattered across the environment: pg_hba.conf on one server, a GRANT statement executed manually by a DBA, or a firewall rule opened temporarily during an incident. Over time, documentation and actual state can drift, making it difficult to determine which rule set each instance is using.

Pigsty takes a different approach: security policy is part of the cluster definition and lives alongside other cluster properties.

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_users:                     # Who may log in: account, role, and expiration
      - { name: dbuser_app ,password: '<unique-random-password>' ,roles: [dbrole_readwrite] ,expire_in: 365 }
    pg_databases:                 # Databases and their isolation policy
      - { name: app ,owner: dbuser_app ,revokeconn: true }
    pg_hba_rules:                 # Who may connect, from where, and how
      - { user: dbuser_app ,db: app ,addr: 10.1.0.0/16 ,auth: ssl ,order: 50 ,title: 'app access via ssl' }

Users, privileges, and HBA rules are described declaratively, and playbooks apply them idempotently to every cluster instance. New instances inherit the same policy, and Git history records security configuration changes. Manual GRANT statements, runtime parameter changes, and edits to node files can still cause drift, so production environments should compare declared and actual state regularly.

Default Security Baseline

Reasonable defaults reduce omissions. The following capabilities are enabled in the default Pigsty configuration:

Capability Default Behavior Related Parameter
Password hashing New or updated PostgreSQL passwords use SCRAM-SHA-256 pg_pwd_enc
Data checksums Page checksums are enabled during cluster initialization to detect silent corruption pg_checksum
Server-side TLS PostgreSQL server certificates are installed and ssl is enabled, so TLS connections are accepted —
Local CA A self-signed CA is created automatically for managed component certificates ca_create
etcd encryption and authentication TLS for client and peer traffic, plus RBAC password authentication etcd_root_password
MINIO object storage HTTPS Silo backup traffic uses HTTPS by default minio_https
Nginx HTTPS Web ingress listens on both ports 80 and 443 by default nginx_sslmode
HBA rules Layered access: local ident, intranet password authentication, and SSL required for public administrator access pg_default_hba_rules
Roles and privileges A four-tier role model and default privilege templates provide a least-privilege baseline pg_default_roles
Backup and recovery pgBackRest is enabled by default, with two full backups retained in the local repository pgbackrest_enabled
Firewall Zone mode trusts intranet CIDRs and exposes only required ports to public networks node_firewall_mode
Restricted sudo Sudo access for the database OS user is limited to the required command set pg_dbsu_sudo

Hardening with Trade-offs

The default configuration targets deployments on a trusted intranet. Some controls require explicit enablement because they impose performance or compatibility costs, or require decisions from the operator:

  • Default configurations and examples contain publicly documented default passwords for quick starts and local testing. Before production deployment, use ./configure -g to randomize the credentials it recognizes, then check the pgBackRest encryption passphrase, Silo users in ha/safe, and all custom values.
  • TLS is disabled by default for the Patroni REST API and PgBouncer (patroni_ssl_enabled, pgbouncer_sslmode); enable it explicitly with the certificates already issued.
  • Password strength checks (passwordcheck) and the audit extension (pgaudit) are disabled by default. Confirm package availability, then configure preloading and policy before use.
  • SELinux defaults to permissive. Demo configurations also expose port 5432 through the firewall; remove that exception in production.
  • The local backup repository is not encrypted by default. The remote minio repository preset uses AES-256 encryption by default, but its default encryption passphrase must be changed.

The ha/safe hardening template combines TLS, certificate authentication, password checks, and backup encryption. Together with the consistency-first CRIT parameter template, it provides a practical starting point. Public credentials, audit extensions, and the failure model still require explicit review. See the Security Model for the complete upgrade path.

This Chapter

Section Question Answered
Security Model Where is the root of trust? How many defensive layers exist? How should the baseline be hardened?
Authentication Who can connect? How is identity proven? How are HBA rules declared and applied?
Access Control What can a connected user do? How does least privilege become the default?
Encrypted Communication How is traffic encrypted? Who issues, distributes, and rotates certificates?
Data Security How is data kept intact, recoverable, confidential, and traceable?
Compliance How do security capabilities map to MLPS and SOC 2 controls?

Related Topics

Beyond the conceptual model, these pages provide operational security guidance:

3.7.1 - Security Model

Pigsty trust boundaries and defense in depth, with the admin node as a high-trust control plane and a path from the default baseline to production hardening.

Before examining individual security features, answer two more fundamental questions: Where is the root of trust? and How many defensive layers exist? The first determines what deserves the strongest protection. The second determines what remains when one layer fails.

Trust Boundaries

Pigsty is an Ansible-based declarative deployment system. Like other control-plane systems, its admin node is the control plane and the node that requires the strongest protection.

Role Assets and Privileges
Admin node The pigsty.yml inventory, which normally contains system and application credentials; the CA private key; SSH administration access to every node
INFRA nodes Monitoring and alerts, DNS, Nginx ingress, and software repositories
Database nodes Database instances, local dbsu, and restricted sudo
Clients Database credentials or client certificates; access through service ports, HBA, and authentication

These roles hold different capabilities; they do not form a simple linear hierarchy. Three assets are especially important:

  1. The pigsty.yml inventory contains component passwords and credentials. Strictly control access to the admin node and to the configuration repository when Git is used.
  2. The CA private key, files/pki/ca/ca.key, is the trust anchor for the deployment. Anyone holding it can issue an arbitrary trusted certificate. The file uses mode 0600 inside a 0700 directory; keep an offline backup.
  3. The administration user’s SSH private key lets the admin node manage every enrolled node with passwordless sudo. It is effectively root access to the managed fleet.

Pigsty’s security policy states this boundary explicitly: an attack that requires admin-node access, or possession of both pigsty.yml and the CA private key, is not treated as a product vulnerability. These are high-trust control-plane assets by design and must be protected accordingly.

Seven Defensive Layers

Defense in depth does not ask one mechanism to solve every problem. It combines controls so that one failure does not remove all protection. Pigsty’s security capabilities can be summarized as seven layers:

# Layer Mechanisms Details
1 Network boundary Firewall zones, constrained listen addresses, centralized ingress This page
2 Transport encryption Local CA and TLS between components Encrypted Communication
3 Authentication HBA rules, SCRAM passwords, client certificates Authentication
4 Access control Role model, default privileges, database isolation Access Control
5 Host security SELinux, restricted sudo, dedicated OS users This page
6 Data security Checksums, backup and encryption, PITR, deletion safeguards Data Security
7 Audit trail DDL and connection logs, audit extensions, centralized logs Data Security

Layers 2, 3, 4, 6, and 7 have dedicated chapters. The following sections cover the network and host layers.

Network Boundaries

Pigsty enables a firewall during node provisioning (node_firewall_mode defaults to zone), using firewalld or ufw according to the operating system. Intranet CIDRs (10.0.0.0/8, 172.16.0.0/12, and 192.168.0.0/16, defined by node_firewall_intranet) enter the trusted zone. Public networks can reach only ports declared in node_firewall_public_port, which defaults to 22 for SSH and 80/443 for web traffic.

The default demo inventory, pigsty.yml, also exposes port 5432 for local evaluation. Remove it in production. If direct database access is required, restrict sources to explicit CIDRs with security groups, host firewalls, and HBA.

PostgreSQL listens on all addresses by default (pg_listen: 0.0.0.0). The effective access boundary is the combination of listen addresses, firewall rules, and HBA. Stricter environments can constrain the listener:

pg_listen: '${ip},${vip},${lo}'   # Host IP, cluster VIP, and loopback only

The default firewall does not expose Grafana, VictoriaMetrics, or other web infrastructure directly to public networks. External web access normally enters through the Nginx portal. Database traffic enters through HAProxy service ports. Fewer entry points are easier to harden and audit.

Host Security

The central host-level rule is: each OS user receives only the privileges required for its job.

  • The database superuser postgres (pg_dbsu) has no password by default and can enter the database only through local ident authentication. pg_dbsu_sudo defaults to limit, allowing passwordless systemctl operations for database services and log viewing rather than unrestricted root access.
  • The administration user (node_admin_username, default dba) is used by operators and playbooks and receives passwordless sudo (nopass) by default. Security-sensitive environments can set node_admin_sudo to all, which requires a sudo password, or limit, which restricts the command set.
  • node_selinux_mode defaults SELinux to permissive: violations are logged but not blocked, providing a baseline before moving to enforcing.

Pigsty does not manage the SSH server configuration. Disabling password login, restricting remote root login, and similar operating-system hardening belong in your host security baseline.

Hardening Levels

Security does not have to jump to its final state in one step. Pigsty provides an upgrade path in which each level builds on the previous one:

Level 1: default baseline. Out-of-the-box controls include SCRAM passwords, data checksums, a local CA and component certificates, layered HBA, a four-tier role model, default backups, and firewall zones. This level suits development, testing, and evaluation on a trusted intranet. Production still requires credential review, network-boundary review, and client verification.

Level 2: randomized credentials. Default passwords are documented publicly and must be changed in every network-exposed deployment. Add -g when generating configuration to randomize built-in parameters and example credentials recognized by the configuration wizard:

./configure -g    # --generate: randomize recognized default credentials

This option does not replace the pgBackRest cipher_pass, every Silo example credential in ha/safe, or user-defined values. See the Default Credentials Checklist for the complete scope.

Level 3: policy hardening with the ha/safe template. conf/ha/safe.yml combines several controls into a starting point for further customization:

  • TLS and certificate authentication: the main TCP HBA rules use ssl, public administrator access uses a client certificate, PgBouncer uses require, and the Patroni API uses HTTPS. Local ident and selected localhost password rules remain.
  • Password policy: passwordcheck is preloaded explicitly, and built-in users declare expire_in. Example passwords in the template still require review and replacement.
  • Reduced attack surface: listen addresses are limited to ${ip},${vip},${lo}, and public connection-pool access by monitoring and administration accounts is denied explicitly.
  • Backup encryption: pgBackRest uses the remote minio repository preset with AES-256-CBC. pgBR.${pg_cluster} is a predictable example value and must be replaced.
  • Security extensions: passwordcheck, credcheck, pgaudit, pgsodium, anonymizer, and related extensions are installed. Installation does not preload, create, or configure an extension.

Level 4: database hardening with the crit.yml parameter template. The safe template selects the CRIT parameter template for consistency-first workloads. Compared with the general oltp template, it:

  • forces data checksums regardless of pg_checksum;
  • enables strict synchronous replication (synchronous_mode_strict), blocking writes that require synchronous acknowledgment when no synchronous replica is available;
  • logs connection and disconnection events; PostgreSQL 18 also separates connection receipt, authentication, and authorization stages;
  • configures watchdog as automatic, which activates only when a usable device exists.

Strict synchronous mode targets preservation of acknowledged transactions, but still depends on synchronous_commit, synchronous replica state, and failover eligibility. Validate RPO with failure exercises on the target topology.

You can also select individual controls instead of adopting the complete template:

pg-meta:
  hosts:
    10.10.10.10: { pg_seq: 1 , pg_role: primary }
    10.10.10.11: { pg_seq: 2 , pg_role: replica }
    10.10.10.12: { pg_seq: 3 , pg_role: replica }
  vars:
    pg_cluster: pg-meta
    pg_conf: crit.yml                    # Use the CRIT database parameter template
    patroni_ssl_enabled: true            # Enable HTTPS for the Patroni API
    pgbouncer_sslmode: require           # Require TLS for PgBouncer
    pg_listen: '${ip},${vip},${lo}'      # Constrain listen addresses
    pg_libs: '$libdir/passwordcheck, pg_stat_statements, auto_explain'  # Password strength checks

Next

3.7.2 - Authentication

Pigsty manages PostgreSQL and PgBouncer HBA rules declaratively, combining SCRAM passwords and client certificates to define who may connect and how identity is proven.

PostgreSQL uses pg_hba.conf for Host-Based Authentication: who may connect, from where, to which database, and how they must prove their identity.

The mechanism is powerful, but expensive to maintain manually across a cluster. Primary and replica instances may require different rules, and every instance stores its own configuration in the data directory. Without a common declaration and refresh process, rules can drift between instances.

Pigsty applies the same declarative configuration model here: HBA rules are part of the inventory and are rendered and distributed consistently by playbooks.

HBA as Code

Cluster HBA policy combines two parameter groups: the global defaults in pg_default_hba_rules and cluster-specific additions in pg_hba_rules. The PgBouncer connection pool has two independent counterparts: pgb_default_hba_rules and pgb_hba_rules.

A rule can use either of two forms. The recommended alias form keeps one semantic rule on one line:

pg_hba_rules:
  - { user: dbuser_app ,db: app ,addr: 10.1.0.0/16 ,auth: ssl ,order: 50 ,title: 'app user access via ssl' }

The raw form supplies a literal pg_hba.conf line for cases the aliases cannot express.

In addition to user, address, database, and authentication method, each rule has two control fields:

  • order: render order. HBA uses first-match semantics, so order is priority. By convention, 0-99 is reserved for high-priority user rules, 100-999 for defaults, and rules without order come last.
  • role: instance-role filter. common and default apply to every instance; primary, replica, offline, standby, and delayed apply only to matching instances. A role: offline rule is also rendered on instances marked with pg_offline_query. The same declaration therefore produces the appropriate rules for each instance role without maintaining primary and replica files manually.

After editing the declaration, apply it with the wrapper script. The rules are rendered again and reloaded:

bin/pgsql-hba pg-meta          # Render and apply HBA rules for pg-meta

pg_hba_rules appends rules; it does not automatically narrow broader defaults. To establish a stricter boundary, review pg_default_hba_rules as well, then inspect the generated pg_hba.conf on every instance.

Address and Authentication Aliases

The alias form gives common cases semantic names. Values in addr expand into concrete address blocks:

Alias Expands To Meaning
local Unix socket Local socket only
localhost Unix socket, 127.0.0.1/32, and ::1/128 Local host
admin <admin_ip>/32 Admin node
infra /32 address of each INFRA node Infrastructure nodes
cluster /32 address of every cluster member Cluster-internal traffic
intra 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16 Intranet CIDRs, customizable with node_firewall_intranet
world 0.0.0.0/0 and ::/0 Any address
CIDR Unchanged Custom network

Values in auth select the authentication method and whether TLS is mandatory:

Alias Authentication Method Notes
deny reject Explicit rejection
trust trust Unconditional access; use with care
pwd scram-sha-256 or md5 Follows pg_pwd_enc; SCRAM by default
sha scram-sha-256 Force SCRAM
md5 md5 Compatibility for legacy clients
ssl hostssl with password authentication Password authentication over mandatory TLS
ssl-sha hostssl with scram-sha-256 Mandatory TLS and SCRAM
cert hostssl with cert Client certificate authentication
ident, os ident (peer in PgBouncer) OS user mapping
peer peer Local OS user

The user field supports four placeholders, replaced with actual user names during rendering: ${dbsu} (superuser), ${repl} (replication user), ${monitor} (monitoring user), and ${admin} (administration user). A +role prefix matches all members of that role.

Do not confuse transport enforcement with server verification: auth: ssl requires TLS but does not require the client to verify the server identity. Security-sensitive clients should also use sslmode=verify-full with a trusted CA; see Encrypted Communication.

Default Rules Explained

Pigsty’s default HBA policy follows a simple rule: the farther the source, the stronger the requirement. These are the PostgreSQL defaults from the source configuration:

pg_default_hba_rules:             # postgres default host-based authentication rules, order by `order`
  - {user: '${dbsu}'    ,db: all         ,addr: local     ,auth: ident ,title: 'dbsu access via local os user ident'  ,order: 100}
  - {user: '${dbsu}'    ,db: replication ,addr: local     ,auth: ident ,title: 'dbsu replication from local os ident' ,order: 150}
  - {user: '${repl}'    ,db: replication ,addr: localhost ,auth: pwd   ,title: 'replicator replication from localhost',order: 200}
  - {user: '${repl}'    ,db: replication ,addr: intra     ,auth: pwd   ,title: 'replicator replication from intranet' ,order: 250}
  - {user: '${repl}'    ,db: postgres    ,addr: intra     ,auth: pwd   ,title: 'replicator postgres db from intranet' ,order: 300}
  - {user: '${monitor}' ,db: all         ,addr: localhost ,auth: pwd   ,title: 'monitor from localhost with password' ,order: 350}
  - {user: '${monitor}' ,db: all         ,addr: infra     ,auth: pwd   ,title: 'monitor from infra host with password',order: 400}
  - {user: '${admin}'   ,db: all         ,addr: infra     ,auth: ssl   ,title: 'admin @ infra nodes with pwd & ssl'   ,order: 450}
  - {user: '${admin}'   ,db: all         ,addr: world     ,auth: ssl   ,title: 'admin @ everywhere with ssl & pwd'    ,order: 500}
  - {user: '+dbrole_readonly',db: all    ,addr: localhost ,auth: pwd   ,title: 'pgbouncer read/write via local socket',order: 550}
  - {user: '+dbrole_readonly',db: all    ,addr: intra     ,auth: pwd   ,title: 'read/write biz user via password'     ,order: 600}
  - {user: '+dbrole_offline' ,db: all    ,addr: intra     ,auth: pwd   ,title: 'allow etl offline tasks from intranet',order: 650}

Layer by layer:

  • Local access is most trusted: postgres can enter only through a local Unix socket with ident. No password is required, but remote login is impossible. This is why dbsu has no password by default.
  • The intranet comes next: replication and application accounts use SCRAM password authentication on the intranet. Remote monitoring and administration access primarily originates from INFRA nodes.
  • Public sources are strictest: only the administrator may connect from any address by default, and the connection requires both a password and TLS.

PgBouncer defaults are more restrictive: public access for monitoring and administration accounts is explicitly denied, while application users are limited to localhost and intranet sources.

The default +dbrole_offline rule does not set role and therefore applies to every instance. To restrict offline users to pg_role: offline or instances with pg_offline_query: true, add role: offline explicitly to the corresponding HBA rule.

This default policy favors usability: application accounts can connect from the intranet with password authentication. The ha/safe template changes the main TCP rules to ssl and requires administrators outside the intranet to present a client certificate (cert); local ident and selected localhost password rules remain.

Password Policy

Pigsty uses PostgreSQL’s recommended scram-sha-256 password storage by default (pg_pwd_enc). Downgrade to md5 only for legacy client compatibility.

Before executing ALTER USER ... PASSWORD, the password workflow temporarily disables statement logging (SET log_statement TO 'none') to keep passwords out of PostgreSQL logs. Plaintext passwords still appear in the inventory, and rendered user SQL is written to /pg/tmp/pg-user-<name>.sql with mode 0640. The related Ansible tasks do not use no_log consistently. Restrict access to the admin node, configuration repository, and automation output, and avoid --diff on tasks containing credentials.

Password strength is not enforced by default. If required, preload passwordcheck or the more configurable credcheck:

pg_libs: '$libdir/passwordcheck, pg_stat_statements, auto_explain'   # Reject weak passwords

The ha/safe template sets this pg_libs value explicitly. Selecting the CRIT parameter template alone does not load passwordcheck.

Declare account lifetime with expire_in (days after creation) or expire_at (absolute date), then combine it with the organization’s rotation process:

pg_users:
  - { name: dbuser_app ,password: '<unique-random-password>' ,roles: [dbrole_readwrite] ,expire_in: 365 }

Certificate Authentication

Passwords can be phished, reused, or guessed. For privileged accounts such as administrators, use auth: cert in HBA to require client certificate authentication. The client must present a certificate signed by the local CA whose CN matches the database user name. When the HBA rule accepts only cert, a leaked password alone cannot authenticate.

Issue client certificates with the built-in cert.yml playbook:

./cert.yml -e cn=dbuser_dba                  # Issue a 20-year client certificate by default
./cert.yml -e cn=dbuser_dba -e expire=365d  # Or specify a shorter lifetime

The certificate and key are stored in files/pki/misc/<cn>.crt and files/pki/misc/<cn>.key. Deliver the private key through a controlled channel. The client should still use verify-full to authenticate the database server; see Encrypted Communication.

Connection Pool and Component APIs

The database is not the only authenticated entry point.

The PgBouncer connection pool uses an independent HBA policy and user list. pgbouncer_auth_query is disabled by default, so only users declared with pgbouncer: true are written to userlist.txt and can authenticate through the pool. Re-evaluate the login scope before enabling dynamic authentication queries.

The Patroni REST API carries high-availability control operations such as restart, switchover, and configuration reload. Write operations require HTTP Basic authentication (patroni_username and patroni_password) and are restricted by source-address allowlists. When patroni_ssl_enabled is enabled, the API uses HTTPS throughout.

Credentials for Grafana, the HAProxy administration interface, the object-storage backend selected by the MINIO module, etcd, and other components are also declared in the inventory. See the Default Credentials Checklist for the full list and update guidance.

Next

3.7.3 - Access Control

Pigsty turns least privilege into reusable declarative cluster configuration through a built-in four-tier role model and default privilege templates.

Authentication answers “Who are you?” Authorization answers “What may you do?”

Privilege failures rarely result from a lack of mechanisms—PostgreSQL GRANT and REVOKE are sufficiently precise. The usual problem is the absence of conventions that are applied by default: an application account is made the owner at launch, temporary superuser access is not revoked after troubleshooting, or grants are missed when new tables are created and cause failures in production.

Pigsty provides an out-of-the-box access control model as a starting point: four role tiers, default privileges, and database isolation. It reduces per-database manual grants, but operators must still assign roles according to business boundaries and review effective privileges regularly.

pigsty-acl.jpg

Role System

Pigsty creates four business roles by default. They cannot log in and are used as privilege groups:

Role Attribute Inherits Purpose
dbrole_readonly NOLOGIN — Global read-only access
dbrole_readwrite NOLOGIN dbrole_readonly Global DML access; the default choice for application accounts
dbrole_admin NOLOGIN dbrole_readwrite, pg_monitor Object creation and DDL for administration and release workflows
dbrole_offline NOLOGIN — Independent read-only role that can be restricted to offline instances through HBA

Pigsty also creates four system users, each with a specific responsibility:

User Attribute Purpose
postgres SUPERUSER Database superuser; no password and local ident login only
replicator REPLICATION Streaming replication and backup, with pg_monitor and read-only privileges
dbuser_dba SUPERUSER Routine administration user that inherits dbrole_admin
dbuser_monitor — Monitoring user with only pg_monitor and read-only privileges

Application accounts join role groups through the roles field and inherit their privileges:

pg_users:
  - { name: dbuser_app    ,password: '...' ,roles: [dbrole_readwrite] }  # Regular application account
  - { name: dbuser_report ,password: '...' ,roles: [dbrole_readonly]  }  # Read-only reporting account
  - { name: dbuser_etl    ,password: '...' ,roles: [dbrole_offline]   }  # Offline ETL account

The role system is itself declarative (pg_default_roles) and can be customized. This parameter is a complete list. Preserve all required system users and default roles when changing it, and check references from HBA rules, default privileges, and scripts at the same time.

Default Privileges

Roles answer “Who receives a privilege?” The other half of the problem is: How do newly created objects receive the correct privileges automatically?

PostgreSQL provides ALTER DEFAULT PRIVILEGES. Pigsty declares these rules through pg_default_privileges:

pg_default_privileges:            # Apply these privileges to new objects created by managed identities
  - GRANT USAGE      ON SCHEMAS   TO dbrole_readonly
  - GRANT SELECT     ON TABLES    TO dbrole_readonly
  - GRANT SELECT     ON SEQUENCES TO dbrole_readonly
  - GRANT EXECUTE    ON FUNCTIONS TO dbrole_readonly
  - GRANT USAGE      ON SCHEMAS   TO dbrole_offline
  - GRANT SELECT     ON TABLES    TO dbrole_offline
  - GRANT SELECT     ON SEQUENCES TO dbrole_offline
  - GRANT EXECUTE    ON FUNCTIONS TO dbrole_offline
  - GRANT INSERT     ON TABLES    TO dbrole_readwrite
  - GRANT UPDATE     ON TABLES    TO dbrole_readwrite
  - GRANT DELETE     ON TABLES    TO dbrole_readwrite
  - GRANT USAGE      ON SEQUENCES TO dbrole_readwrite
  - GRANT UPDATE     ON SEQUENCES TO dbrole_readwrite
  - GRANT TRUNCATE   ON TABLES    TO dbrole_admin
  - GRANT REFERENCES ON TABLES    TO dbrole_admin
  - GRANT TRIGGER    ON TABLES    TO dbrole_admin
  - GRANT CREATE     ON SCHEMAS   TO dbrole_admin

The read-only role receives query and function execution privileges, the read-write role adds DML, and the administrator role adds the supporting privileges required for object management.

Ownership Convention

Default privileges have an often-missed prerequisite: they apply only to objects created by identities for which those defaults were configured. Pigsty configures default privileges for:

  • the database OS user pg_dbsu, which defaults to postgres;
  • the administration user pg_admin_username, which defaults to dbuser_dba;
  • dbrole_admin;
  • each database owner declared in pg_databases.

Application DDL should normally run as the declared database owner. Platform administration and release workflows can use dbuser_dba or first execute SET ROLE dbrole_admin. Objects created directly by other users do not enter this default privilege model unless ALTER DEFAULT PRIVILEGES is also configured for those users.

This is PostgreSQL behavior, not a Pigsty limitation: default privileges follow the object creator; they do not automatically propagate from the database or the session login name.

Database Isolation

PostgreSQL grants CONNECT on databases to PUBLIC by default. If HBA also permits a connection, a login role may enter a database it does not own. This default is particularly important to tighten when several applications share a cluster.

Set revokeconn in a database definition to revoke public connection access:

pg_databases:
  - { name: app_a ,owner: dbuser_a ,revokeconn: true }
  - { name: app_b ,owner: dbuser_b ,revokeconn: true }

When enabled, CONNECT is revoked from PUBLIC and granted explicitly to the replication, monitoring, and administration users and to the database owner. The owner receives GRANT OPTION and can decide who else may connect. Without additional grants or inherited roles, the app_a account cannot connect to app_b.

Cluster initialization also revokes CREATE from PUBLIC on the database and the public schema:

REVOKE CREATE ON DATABASE app FROM PUBLIC;
REVOKE CREATE ON SCHEMA public FROM PUBLIC;

Ordinary users can no longer create objects freely in public databases or schemas, reducing risks from unsafe search_path settings and object shadowing. PostgreSQL 15 tightened the default CREATE privilege on the public schema; Pigsty applies the same boundary consistently across all supported major versions.

Offline Role and Instance Isolation

dbrole_offline provides an independent set of read-only privileges for ETL, reporting, and ad hoc queries. The role controls object privileges only; it does not automatically restrict which instance a user may connect to.

In the current default HBA rules, the intranet rule for +dbrole_offline does not set role and therefore applies to every instance. To restrict it to a dedicated pg_role: offline instance, or to a regular replica marked with pg_offline_query: true, modify that rule in the complete pg_default_hba_rules list:

pg_default_hba_rules:
  # Copy and retain all other default rules; change only the offline-role rule
  - { user: '+dbrole_offline', db: all, addr: intra, auth: pwd, role: offline, order: 650,
      title: 'allow offline users on offline instances' }

Defining pg_default_hba_rules replaces the entire default list; the example rule cannot be used alone. Expensive queries are limited to offline instances only when HBA filters by instance role and the user does not inherit another role allowed by broader rules. Resource isolation should also use a dedicated service endpoint, connection limits, and query resource controls.

Beyond the Database

Least privilege also applies at the host level:

  • The postgres superuser has no password and can log in only through local ident. Its sudo access defaults to a restricted set of database service and log commands (pg_dbsu_sudo: limit).
  • The monitoring user dbuser_monitor holds pg_monitor, the read-only role, and privileges on the dedicated monitor schema; it cannot write business tables by default.
  • The replication user replicator receives only the directory function privileges required for backup and recovery instead of broad superuser access.

Next

3.7.4 - Encrypted Communication

Pigsty provides a self-signed CA that issues certificates and distributes trust for managed components, creating a unified TLS foundation.

TLS can provide three separate protections: transport encryption, server authentication, and client authentication. Each must be configured independently. Enabling server-side TLS does not mean the client verifies the server identity, nor does it mean the server requires a client certificate.

The main operational cost of TLS is not the encryption algorithm but certificate issuance, distribution, trust, and rotation. Without centralized management, internal services often encrypt traffic while skipping certificate verification—or remain on plaintext connections.

Pigsty brings PKI under declarative management. During deployment it creates a local self-signed CA, issues certificates for managed components, and distributes trust so TLS is ready for use after installation.

Local CA

During the first deployment, Pigsty checks for a CA on the admin node and creates one when required:

File Description Permissions
files/pki/ca/ca.key CA private key and root of trust for the deployment; protect it carefully 0600, with directory mode 0700
files/pki/ca/ca.crt CA root certificate; safe to distribute 0644
  • ca_create controls CA behavior. An existing private key and certificate are reused unchanged; if the certificate is missing but the private key exists, that key is used to issue a replacement certificate. ca_create: false only prevents creation of a missing CA private key. Deployment stops if ca.key is absent, preventing an unexpected trust root. Always back up and restore ca.key and ca.crt together.
  • ca_cn sets the CA certificate CN, which defaults to pigsty-ca. The key is RSA 4096.
  • The root CA is valid for 100 years, while component certificates default to 20 years (cert_validity: 7300d). The browser-facing Nginx certificate is an exception and currently defaults to 397 days.

Long default lifetimes reduce the initial maintenance burden for private infrastructure; they do not remove the need for production rotation. Organizations with an established certificate policy should shorten lifetimes and monitor expiration.

Trust Distribution

Issuing a certificate is only half of PKI. Every node must trust it. When a node is managed, Pigsty distributes the CA certificate to /etc/pki/ca.crt and links it into the operating system trust store:

  • EL family (RHEL, Rocky, Alma): link under /etc/pki/ca-trust/source/anchors/ and run update-ca-trust
  • Debian and Ubuntu: link under /usr/local/share/ca-certificates/ and run update-ca-certificates

Clients that use the OS trust store, such as curl, can then verify certificates signed by the Pigsty CA. The CA certificate is also published as ca.crt at the site root of the Nginx portal for browsers and external clients.

PostgreSQL libpq clients require special attention: by default they look for ~/.postgresql/root.crt and use sslmode=prefer, so they do not directly use the operating system trust store to verify the server identity.

Server Identity Verification

Security-sensitive PostgreSQL clients should use sslmode=verify-full and specify the Pigsty CA:

psql "host=pg-meta dbname=postgres user=dbuser_dba sslmode=verify-full sslrootcert=/etc/pki/ca.crt"

verify-full validates both the certificate chain and the connection host name. The DNS name or IP address used by the client must therefore appear in the server certificate SAN. External clients must install ca.crt or specify it with sslrootcert.

Certificate Matrix

The local CA issues certificates for the following components and places them under one trust chain:

Component Certificate Identity (CN) Deployment Path Encryption State
PostgreSQL <cluster>-<sequence> /pg/cert/server.{crt,key} Server-side SSL enabled by default; HBA determines whether it is mandatory
PgBouncer Reuses the PostgreSQL certificate /pg/cert/ TLS disabled by default (pgbouncer_sslmode)
Patroni Reuses the PostgreSQL certificate /pg/cert/ API HTTPS disabled by default (patroni_ssl_enabled)
etcd <instance-name> /etc/etcd/server.{crt,key} TLS for client and peer traffic
Silo <node-name> ~minio/.minio/certs/ Silo HTTPS is enabled by default (minio_https)
Kafka <cluster>-<sequence> /etc/kafka/pki/kafka.pem SASL_SSL/SSL with kafka_security: scram; defaults to plaintext
MySQL <instance-name> /etc/mysql/pki/server.{crt,key} Secure transport enforced; clients and group replication verify the certificate chain
Nginx pigsty, with portal domains in SAN /etc/nginx/conf.d/cert/ HTTPS enabled by default (nginx_sslmode)
INFRA node <node-name> /etc/pki/infra.{crt,key} Available to infrastructure components

The encryption-state column reflects deliberate defaults:

  • Enabled at deployment: PostgreSQL accepts SSL connections; etcd uses TLS for client and peer traffic.
  • Encrypted by default: Object-storage backup traffic through the MINIO module and Nginx web traffic use HTTPS.
  • Disabled by default, available on demand: TLS for the Patroni REST API and PgBouncer is disabled by default, but certificates are already present. Enable it through the corresponding parameters; both are enabled in the ha/safe template.

Keep three states distinct: server-side SSL support does not force clients to use SSL, and neither state proves that the client verifies the server identity. HBA rules enforce encryption with auth: ssl or cert. Client sslmode and trust settings control server verification. The default rules require TLS only for administrator connections from arbitrary sources. The safe template changes the main TCP rules to ssl or cert while retaining local ident and selected localhost password rules.

Client Certificates

The built-in cert.yml playbook issues client certificates. The certificate CN represents the database user name for HBA cert authentication:

./cert.yml -e cn=dbuser_dba                  # Issue a 20-year client certificate by default
./cert.yml -e cn=dbuser_dba -e expire=365d  # Or specify a shorter lifetime

Results are stored in files/pki/misc/<cn>.key and files/pki/misc/<cn>.crt. Deliver private keys through a controlled channel and make them readable only by the corresponding user. The client certificate lets the server authenticate the client; the client must still use verify-full to authenticate the database server.

Using an Enterprise CA

If the organization already operates a PKI, Pigsty can issue certificates from that CA, or from an intermediate signed by the enterprise root. Place the certificate and private key at the expected paths; playbooks do not regenerate a CA when one already exists:

files/pki/ca/ca.key    # CA or intermediate CA private key
files/pki/ca/ca.crt    # Corresponding CA certificate

Also set ca_create: false. Deployment will then fail explicitly if the private key is missing instead of creating an unexpected trust root. This setting does not stop the role from reissuing the CA certificate when the private key exists but the certificate is missing, so verify and restore both files together.

Key Protection and Rotation

  • The CA private key exists only on the admin node. Together with pigsty.yml, it is one of the highest-trust assets in the deployment; see Trust Boundaries. Keep an offline backup.
  • If the CA private key is compromised, establish a new trust root and reissue every component and client certificate. Plan an overlap period in which both old and new CAs are trusted to avoid interrupting all connections at once.
  • Component certificate sources are stored under files/pki/<component>/ on the admin node; node certificates are deployment copies. Deleting only a node copy restores the same certificate rather than issuing a new one. To rotate, update or remove the corresponding source on the admin node, rerun the relevant playbook, then reload or roll the component as required.

Next

  • 🔑 Authentication: use HBA to decide who must use SSL or client certificates
  • 🔒 Data Security: encryption for stored data and backups
  • ✅ Compliance: evidence for certificate management

3.7.5 - Data Security

Protect PostgreSQL data integrity, recoverability, confidentiality, and traceability with checksums, backup and PITR, encryption, and audit logs.

Network boundaries, authentication, and access control reduce the likelihood of an incident. When hardware fails, credentials leak, or an operator makes a mistake, data-layer controls must limit the impact and support recovery.

Data security answers four questions: Is the data intact? Can it be recovered? If copied, does it remain confidential? Can you determine what happened?

Integrity

Bad disk sectors, memory bit flips, and storage firmware defects can cause silent data corruption: the data is damaged without an immediate error. Pigsty enables page checksums by default (pg_checksum: true). The cluster is initialized with data-checksums, so PostgreSQL calculates a checksum when writing a page and verifies it when reading.

Page checksums primarily detect corruption in storage media, the I/O path, or pages after they were written. They do not detect every memory error, logical error, or incorrect application write, and they do not replace backups.

The CRIT parameter template goes further: checksums are mandatory regardless of the parameter, and strict synchronous replication (synchronous_mode_strict) blocks writes that require synchronous acknowledgment when no synchronous replica is available. This mode targets preservation of acknowledged transactions, but it still assumes clients have not reduced synchronous_commit, a synchronous replica participates in the commit, and failover selects only a node containing the required WAL. Validate RPO through failure exercises on the target topology.

Recoverability

Replicas primarily handle node failures; backups handle accidental deletion, logical errors, cluster corruption, and broader disasters. High availability can shorten an interruption after primary failure, but replication also copies an accidental deletion to every replica. Backups are therefore indispensable.

Pigsty enables pgBackRest by default (pgbackrest_enabled). Base backups plus continuous WAL archiving provide Point-in-Time Recovery (PITR), allowing recovery to a target time within the retained backup and WAL window.

Select the backup repository with pgbackrest_method:

Repository Location Default Retention Encryption
local (default) Local /pg/backup directory Latest 2 full backups None
minio Silo or external S3-compatible object storage 14 days AES-256-CBC

Two additional controls reduce damage from accidental deletion:

  • Delayed replica: declare a pg_delay: 1h replica for a critical cluster. Before an erroneous operation is replayed, pause replication and extract the required data. A delayed replica eventually catches up and does not replace a backup.
  • Removal safeguards: when pg_safeguard or etcd_safeguard is enabled, the corresponding removal playbook refuses to run, reducing the risk of accidental cluster removal.

Having a backup is not the same as being able to restore. Recovery exercises should be routine; see Backup and Recovery for mechanisms and procedures.

Confidentiality

Protect data at rest at three layers:

Backup encryption. pgbackrest_method: minio denotes an S3-compatible repository. It can be provided by Silo deployed through the MINIO module, or independently managed MinIO, RustFS, and external S3 services. The preset uses AES-256-CBC by default, but the public pgBackRest passphrase must be changed in production. The ha/safe template derives an example passphrase from the cluster name:

pgbackrest_repo:
  minio:
    cipher_type: aes-256-cbc
    cipher_pass: 'pgBR.${pg_cluster}'   # Example only; replace before deployment

pgBR.${pg_cluster} is predictable, and configure -g does not replace it. Use a unique random passphrase in production and store it separately from the backup. Losing the passphrase makes the backup unrecoverable.

The local backup repository is not encrypted by default. Encryption reduces disclosure if backup files or media are copied separately, but offers limited protection when the key and backup remain on the same host.

Transport encryption. Backup uploads to Silo or external S3 services use HTTPS. PostgreSQL client and replication traffic can require SSL through HBA. Clients should also verify the server certificate; see Encrypted Communication.

Encryption at rest. Upstream PostgreSQL currently has no general built-in transparent data encryption (TDE). Pigsty provides two practical options: use the pg_tde extension with Percona Distribution for PostgreSQL for table-level transparent encryption (see the pgtde configuration template); or use security extensions such as pgsodium, pgcrypto, and anonymizer for column-level encryption and masking. The safe template installs this extension category. Full-disk encryption such as LUKS or dm-crypt protects against stolen media at the operating-system layer and complements database-level controls.

Audit and Traceability

After an incident, you must be able to answer who did what and when. Pigsty provides layered logging:

Default baseline: all DDL is logged (log_statement: ddl), and statements taking longer than 100 ms are logged (log_min_duration_statement: 100). PostgreSQL 18 and later also record connection authorization events.

CRIT template: connection and disconnection events are recorded with log_connections and log_disconnections. PostgreSQL 18 can distinguish connection receipt, authentication, and authorization stages.

pgaudit extension: for fine-grained statement auditing such as object reads and writes or role-based audit classes, install pgaudit and add it to pg_libs for preloading. The safe template installs the extension, but loading and audit policy must be declared explicitly.

When INFRA logging is enabled and Vector is configured, PostgreSQL logs are sent to VictoriaLogs for centralized storage. The default retention is 15 days and can be adjusted for compliance. Logs and metrics support search, alerts, and incident reconstruction, but incident classification, response, and evidence preservation still require an operational process.

Next

3.7.6 - Compliance

Compliance combines configuration, process, and evidence. This page covers launch hardening, MLPS and SOC 2 control mappings, supply-chain integrity, and vulnerability response.

Compliance is not a product you can buy. It is a state that must be demonstrated continuously through three elements:

  • Configuration: whether security controls are enabled. Pigsty directly provides this part.
  • Process: access approval, change management, recovery exercises, and related procedures. The organization must establish these.
  • Evidence: records showing that configuration and process remain effective. Pigsty’s inventory, runtime logs, and monitoring system can provide part of this evidence.

This page begins with a pre-launch hardening checklist and then maps Pigsty security capabilities to common compliance frameworks. The mappings support architecture and gap analysis; they are not an MLPS assessment conclusion, a SOC 2 audit opinion, or legal advice.

Default Credentials Checklist

Pigsty default credentials are public in the documentation and source code. They are intended only for demonstrations and local development. Change every applicable default before any production or network-exposed deployment goes live:

Scope Example Default configure -g
Grafana administrator and viewer pigsty, DBUser.Viewer Yes
HAProxy administration interface pigsty Yes
PostgreSQL administration, monitoring, and replication users DBUser.DBA, DBUser.Monitor, DBUser.Replicator Yes
Patroni REST API Patroni.API Yes
etcd root Etcd.Root Yes
MINIO module object-storage root S3User.MinIO Yes
Object-storage backup and example application users S3User.Backup, S3User.Meta, S3User.Data Yes
Example database users DBUser.Meta, DBUser.Supa, Vibe.Coding Yes
pgBackRest encryption passphrase cipher_pass: pgBackRest No
Silo users and pgBR.${pg_cluster} in ha/safe Template example values No
User-defined credentials Custom values No

Use -g while generating configuration to randomize built-in parameters and example strings recognized by the configuration wizard:

./configure -g     # Generate the inventory and randomize recognized default credentials

The wizard prints generated passwords to the terminal, so protect terminal history and automation logs as sensitive data. After generation, inspect the configuration and replace pgBackRest cipher_pass, MINIO module example values in ha/safe that were not covered, and all custom credentials.

Launch Hardening Checklist

Before deployment:

After deployment:

  • Confirm that credentials covered by configure -g and uncovered backup, object-storage, and custom credentials have all been changed
  • Review the effective HBA rules in /pg/data/pg_hba.conf against the declaration and intended boundary
  • Query effective users, roles, default privileges, and database CONNECT grants, and compare them with the inventory
  • Run one full backup and a recovery exercise to validate the backup path
  • Confirm log collection, monitoring alerts, and notification channels

Periodically:

  • Audit privileges: compare pg_users declarations with effective grants, and remove expired or departed-user accounts
  • Rotate credentials and certificates
  • Exercise recovery and failover
  • Track security updates for Pigsty and upstream components

Compliance Evidence

Declarative configuration provides a stable starting point for audit evidence. Retain runtime state as well to show that the configuration was applied and remains effective.

Evidence Source
Security baseline and change history The pigsty.yml inventory and Git history
Access-control matrix pg_default_roles, pg_users, and pg_hba_rules declarations
Effective authentication policy Rendered pg_hba.conf on each instance, compared with declarations to detect drift
Effective users and privileges PostgreSQL catalogs, database ACLs, \du+, and \ddp+
Operation and connection logs PostgreSQL DDL, slow-query, and connection logs retained in VictoriaLogs
Backup records pgBackRest information and monitoring dashboards
Security incidents and alerts Monitoring alert history
Certificate inventory files/pki/ and deployed component certificates

MLPS Level 3 Mapping

The following maps database-related Pigsty capabilities to controls in the “secure computing environment” section of GB/T 22239-2019 Level 3:

Control Pigsty Capability Additional Requirement
Unique identity Independent accounts and SCRAM-SHA-256 password storage Real-name account management process
Password complexity and rotation passwordcheck, credcheck, and expire_in Enable extensions and establish a rotation process
Login failure handling Can be implemented with credcheck and related extensions Enable and configure as required
Access control and least privilege Four-tier roles, default privileges, and database isolation Privilege approval workflow
Security audit DDL, connection, and slow-query logs; pgaudit; centralized retention CRIT or manual connection logging; required retention period
Communication confidentiality Local CA and TLS; HBA-enforced ssl or cert Enforce TLS, client verify-full, and certificate rotation
Data integrity Page checksums by default and strict synchronous replication with CRIT Storage protection, defined failure model, and exercises
Data confidentiality AES-encrypted backup plus TDE and column-encryption options Enable as required
Backup and recovery pgBackRest, PITR, and a remote S3-compatible repository Recovery exercise process
Residual information protection — Media destruction and erasure process

MLPS also covers physical security, communication networks, and management systems beyond the scope of a database distribution. Pigsty can support database-related technical controls in a secure computing environment; facilities, network devices, and governance must be addressed in the overall system.

SOC 2 Mapping

Database-related controls in the SOC 2 Trust Services Criteria (TSC) include:

Criterion Pigsty Capability Additional Requirement
CC6.1 Logical access security HBA, RBAC, default privileges, and database isolation Privilege design, approval, and periodic review
CC6.2 User registration and authorization Declarative users, roles, and expiration Joiner, mover, leaver, and identity-verification process
CC6.3 Access changes and revocation pg_users, role changes, REVOKE, and expiration Tickets, approval evidence, and timely revocation
CC6.6 External boundary threats Firewalls, listen addresses, HBA, and restricted management ingress Network architecture, boundary devices, and continuous validation
CC6.7 Information transmission and movement TLS, client verification, and backup encryption Policies for exports, media, and third-party transfer
CC7.2 System monitoring Victoria observability stack with extensive metrics and alerts Alert-response process
CC7.3 Incident traceability Centralized logs and audit extensions Log-review process
A1.2 Availability and recovery High Availability and PITR Exercise records and RTO/RPO objectives

Supply Chain and Vulnerability Response

Compliance reviews increasingly cover the software supply chain. Pigsty provides the following distribution and response controls:

Package integrity: RPM and DEB packages in the Pigsty repositories (repo.pigsty.io and repo.pigsty.cc) are GPG-signed. The public-key fingerprint is 9592 A7BC 7A68 2E73 3337 6E09 E793 5D8D B9BD 8B20 (B9BD8B20) and can be verified before trust is established. Repository definitions written during deployment and the local repository on the INFRA node do not enforce signature verification for every package by default; review package-manager repository trust and signature settings in production.

Vulnerability response: report security issues privately through GitHub private vulnerability reporting or email, as documented in SECURITY.md. The project targets acknowledgment within three business days and an initial assessment within seven days.

Version support: security fixes ship with the latest stable release. Staying current is the standard way to receive them. Users who must remain on a version for longer can obtain extended support through subscription services.

Next

4 - About

Learn about Pigsty itself in every aspect - features, history, license, privacy policy, community, and news.

4.1 - Features

Pigsty’s value propositions and highlight features.

“PostgreSQL In Great STYle”: Postgres, Infras, Graphics, Service, Toolbox, it’s all Yours.

—— Battery-included, local-first PostgreSQL distribution, open-source RDS alternative

Value Propositions

Pigsty feature overview

Overview

Pigsty is a better local open-source RDS for PostgreSQL alternative:

  • Battery-Included RDS: From kernel to RDS distribution, providing production-grade PG database services for versions 14-18 on EL/Debian/Ubuntu.
  • Rich Extensions: Providing unparalleled 575 extensions with out-of-the-box distributed, time-series, geospatial, graph, vector, multi-modal database capabilities.
  • Flexible Modular Architecture: Compose Redis, Etcd, and Silo object-storage modules with PostgreSQL modes such as Mongo; monitor existing RDS, hosts, and databases independently.
  • Stunning Observability: Based on modern observability stack Prometheus/Grafana, providing stunning, unparalleled database observability capabilities.
  • Battle-Tested Reliability: Self-healing high-availability architecture: automatic failover on hardware failure, seamless traffic switching. With auto-configured PITR as safety net for accidental data deletion!
  • Easy to Use and Maintain: Declarative API, GitOps ready, foolproof operation, Database/Infra-as-Code and management SOPs encapsulating management complexity!
  • Solid Security Practices: HBA, ACL, TLS, backup, logging, and host-firewall foundations, with explicit default boundaries and production hardening requirements.
  • Broad Application Scenarios: Low-code data application development, or use preset Docker Compose templates to spin up massive software using PostgreSQL with one click!
  • Open-Source Free Software: Own better database services at less than 1/10 the cost of cloud databases! Truly “own” your data and achieve autonomy!

PostgreSQL integrates ecosystem tools and best practices:

  • Out-of-the-box PostgreSQL distribution, deeply integrating 575 packaged extensions for geospatial, time-series, distributed, graph, vector, search, and AI!
  • Runs on bare operating systems without container support, supporting mainstream operating systems: EL 8/9/10, Ubuntu 22.04/24.04/26.04, and Debian 12/13.
  • Based on patroni, haproxy, and etcd, creating a self-healing high-availability architecture: automatic failover on hardware failure, seamless traffic switching.
  • Combines pgBackRest with optional Silo object storage to provide out-of-the-box point-in-time recovery (PITR), protecting against software defects and accidental data deletion.
  • Based on Ansible providing declarative APIs to abstract complexity, greatly simplifying daily operations management in a Database-as-Code manner.
  • Pigsty has broad applications, can be used as complete application runtime, develop demo data/visualization applications, and massive software using PG can be spun up with Docker templates.
  • Provides Vagrant-based local development and testing sandbox environment, and Terraform-based cloud auto-deployment solutions, keeping development, testing, and production environments consistent.
  • Run PostgreSQL in Mongo-compatible mode with DocumentDB and the FerretDB Docker APP

Battery-Included RDS

Get production-grade PostgreSQL database services locally immediately!

PostgreSQL is a near-perfect database kernel, but it needs more tools and systems to become a good enough database service (RDS). Pigsty helps PostgreSQL make this leap. Pigsty solves various challenges you’ll encounter when using PostgreSQL: kernel extension installation, connection pooling, load balancing, service access, high availability / automatic failover, log collection, metrics monitoring, alerting, backup recovery, PITR, access control, parameter tuning, security encryption, certificate issuance, NTP, DNS, parameter tuning, configuration management, CMDB, management playbooks… You no longer need to worry about these details!

Pigsty supports PostgreSQL 14 ~ 18 mainline kernels and other compatible forks, running on EL / Debian / Ubuntu and compatible OS distributions, available on x86_64 and ARM64 chip architectures, without container support required. Besides database kernels and many out-of-the-box extension plugins, Pigsty also provides complete infrastructure and runtime required for database services, as well as local sandbox / production environment / cloud IaaS auto-deployment solutions.

Pigsty can bootstrap an entire environment from bare metal with one click, reaching the last mile of software delivery. Ordinary developers and operations engineers can quickly get started and manage databases part-time, building enterprise-grade RDS services without database experts!

pigsty-arch.jpg

Rich Extensions

Hyper-converged multi-modal, use PostgreSQL for everything, one PG to replace all databases!

PostgreSQL’s soul lies in its rich extension ecosystem, and Pigsty uniquely deeply integrates 575 extensions from the PostgreSQL ecosystem, providing you with an out-of-the-box hyper-converged multi-modal database!

Extensions can create synergistic effects, producing 1+1 far greater than 2 results. You can use PostGIS for geospatial data, TimescaleDB for time-series/event stream data analysis, and Citus to upgrade it in-place to a distributed geospatial-temporal database; You can use PGVector to store and search AI embeddings, ParadeDB for ElasticSearch-level full-text search, and simultaneously use precise SQL, full-text search, and fuzzy vector for hybrid search. You can also achieve dedicated OLAP database/data lakehouse analytical performance through pg_duckdb, pg_mooncake and other analytical extensions.

Using PostgreSQL as a single component to replace MySQL, Kafka, ElasticSearch, MongoDB, and big data analytics stacks has become a best practice — a single database choice can significantly reduce system complexity, greatly improve development efficiency and agility, achieving remarkable software/hardware and development/operations cost reduction and efficiency improvement.

pigsty-ecosystem.jpg

Flexible Modular Architecture

Flexible composition, free extension, multi-database support, monitor existing RDS/hosts/databases

Components in Pigsty are abstracted as independently deployable modules, which can be freely combined to address varying requirements. The INFRA module comes with a complete modern monitoring stack, while the NODE module tunes nodes to desired state and brings them under management. Installing the PGSQL module on multiple nodes automatically forms a high-availability database cluster based on primary-replica replication, while the ETCD module provides consensus and metadata storage for database high availability.

Beyond these four core modules, Pigsty also provides a series of optional feature modules: The MINIO module can deploy Silo to provide local object storage and serve as a centralized database backup repository. The REDIS module can provide auxiliary services for databases in standalone primary-replica, sentinel, or native cluster modes. The DOCKER module can be used to spin up stateless application software.

Additionally, Pigsty provides PG-compatible / derivative kernel support. You can use Babelfish for MS SQL Server compatibility, IvorySQL for Oracle compatibility, OpenHaloDB for MySQL compatibility, and OrioleDB for ultimate OLTP performance.

Furthermore, you can use PostgreSQL Mongo mode for MongoDB compatibility, Supabase for Firebase compatibility, and PolarDB to meet domestic compliance requirements. Message queues are covered by the KAFKA module, which deploys Kafka 4.x dynamic KRaft clusters. More professional/pilot modules will be continuously introduced to Pigsty, such as GPSQL, DUCKDB, TIGERBEETLE, KUBERNETES, CONSUL, GREENPLUM, CLOUDBERRY, MYSQL, …

pigsty-sandbox.jpg

Stunning Observability

Using modern open-source observability stack, providing unparalleled monitoring best practices!

Pigsty provides best practices for monitoring based on the open-source Grafana / Prometheus modern observability stack: Grafana for visualization, VictoriaMetrics for metrics collection, VictoriaLogs for log collection and querying, Alertmanager for alert notifications. Blackbox Exporter for checking service availability. The entire system is also designed for one-click deployment as the out-of-the-box INFRA module.

Pigsty automatically monitors every managed component: host nodes, HAProxy load balancers, PostgreSQL databases, PgBouncer connection pools, Etcd metadata stores, Redis-compatible caches, Silo object storage, and the monitoring infrastructure itself. Grafana dashboards and preset alert rules provide immediate operational visibility. The same stack can also monitor applications, existing database instances, and cloud RDS services.

Whether for failure analysis or slow query optimization, capacity assessment or resource planning, Pigsty provides comprehensive data support, truly achieving data-driven operations. In Pigsty, over three thousand types of monitoring metrics are used to describe all aspects of the entire system, and are further processed, aggregated, analyzed, refined, and presented in intuitive visualization modes. From global overview dashboards to CRUD details of individual objects (tables, indexes, functions) in a database instance, everything is visible at a glance. You can drill down, roll up, or jump horizontally freely, browsing current system status and historical trends, and predicting future evolution.

pigsty-dashboard.jpg

Additionally, Pigsty’s monitoring system module can be used independently — to monitor existing host nodes and database instances, or cloud RDS services. With just one connection string and one command, you can get the ultimate PostgreSQL observability experience.

Visit the Screenshot Gallery and Online Demo for more details.

Battle-Tested Reliability

Out-of-the-box high availability and point-in-time recovery capabilities ensure your database is rock-solid!

For table/database drops caused by software defects or human error, Pigsty provides out-of-the-box PITR point-in-time recovery capability, enabled by default without additional configuration. As long as storage space allows, base backups and WAL archiving based on pgBackRest let you quickly return to any point within the recovery window. You can use local directories/disks, Silo deployed by the MINIO module, or external S3-compatible object-storage services to retain longer recovery windows, according to your budget.

Pigsty provides a high-availability self-healing architecture based on Patroni, etcd, and HAProxy. When the node, network, quorum, and synchronous-replica assumptions hold, it can fail over the primary automatically. Actual RTO and RPO depend on replication mode, failure type, timeout settings, and client reconnection behavior.

Pigsty includes built-in HAProxy load balancers for automatic traffic switching, providing DNS/VIP/LVS and other access methods for clients. Failover and active switchover are almost imperceptible to the business side except for brief interruptions, and applications don’t need to modify connection strings or restart. The minimal maintenance window requirements bring great flexibility and convenience: you can perform rolling maintenance and upgrades on the entire cluster without application coordination. The feature that hardware failures can wait until the next day to handle lets developers, operations, and DBAs sleep well. Many large organizations and core institutions have been using Pigsty in production for extended periods. The largest deployment has 25K CPU cores and 200+ PostgreSQL ultra-large instances; in this deployment case, dozens of hardware failures and various incidents occurred over six to seven years, DBAs changed several times, but still maintained availability higher than 99.999%.

pigsty-ha.png

Easy to Use and Maintain

Infra as Code, Database as Code, declarative APIs encapsulate database management complexity.

Pigsty provides services through declarative interfaces, elevating system controllability to a new level: users tell Pigsty “what kind of database cluster I want” through configuration inventories, without worrying about how to do it. In effect, this is similar to CRDs and Operators in K8S, but Pigsty can be used for databases and infrastructure on any node: whether containers, virtual machines, or physical machines.

Whether creating/destroying clusters, adding/removing replicas, or creating new databases/users/services/extensions/whitelist rules, you only need to modify the configuration inventory and run the idempotent playbooks provided by Pigsty, and Pigsty adjusts the system to your desired state. Users don’t need to worry about configuration details — Pigsty automatically tunes based on machine hardware configuration. You only need to care about basics like cluster name, how many instances on which machines, what configuration template to use: transaction/analytics/critical/tiny — developers can also self-serve. But if you’re willing to dive into the rabbit hole, Pigsty also provides rich and fine-grained control parameters to meet the demanding customization needs of the most meticulous DBAs.

Beyond that, Pigsty’s own installation and deployment is also one-click foolproof, with all dependencies pre-packaged, requiring no internet access during installation. The machine resources needed for installation can also be automatically obtained through Vagrant or Terraform templates, allowing you to spin up a complete Pigsty deployment from scratch on a local laptop or cloud VM in about ten minutes. The local sandbox environment can run on a 1-core 2GB micro VM, providing the same functional simulation as production environments, usable for development, testing, demos, and learning.

pigsty-iac.jpg

Solid Security Practices

Pigsty provides the security foundations required for database deployment: layered HBA, built-in roles and default privileges, SCRAM-SHA-256, page checksums, a local CA, component certificates, backup, PITR, centralized logs, and firewall configuration.

The defaults target development, testing, and demonstrations on a trusted intranet. Production deployments must replace public credentials, review network boundaries, enforce TLS where required, configure server-certificate verification, and establish backup recovery, privilege review, and incident-response processes.

Security and Compliance documents each mechanism’s default state and boundary. Security Considerations provides production hardening guidance, and Compliance maps relevant controls to MLPS and SOC 2. Whether a deployment meets a specific requirement depends on scope, organizational process, continuous evidence, and the auditor’s conclusion.

pigsty-acl.jpg

Broad Application Scenarios

Use preset Docker templates to spin up massive software using PostgreSQL with one click!

In various data-intensive applications, the database is often the trickiest part. For example, the core difference between GitLab Enterprise and Community Edition is the underlying PostgreSQL database monitoring and high availability. If you already have a good enough local PG RDS, you can refuse to pay for software’s homemade database components.

Pigsty provides the Docker module and many out-of-the-box Compose templates. You can use Pigsty-managed high-availability PostgreSQL (as well as Redis and Silo) as backend storage, spinning up these software in stateless mode with one click: GitLab, Gitea, Wiki.js, NocoDB, Odoo, Jira, Confluence, Harbor, Mastodon, Discourse, KeyCloak, Mattermost, etc. If your application needs a reliable PostgreSQL database, Pigsty is perhaps the simplest way to get one.

Pigsty also provides application development toolsets closely related to PostgreSQL: PGAdmin4, PGWeb, ByteBase, PostgREST, Kong, as well as EdgeDB, FerretDB, Supabase — these “upper-layer databases” using PostgreSQL as storage. More wonderfully, you can build interactive data applications quickly in a low-code manner based on the Grafana and Postgres built into Pigsty, and even use Pigsty’s built-in ECharts panels to create more expressive interactive visualization works.

Pigsty provides a powerful runtime for your AI applications. Your agents can leverage PostgreSQL and the powerful capabilities of the observability world in this environment to quickly build data-driven intelligent agents.

pigsty-app.jpg

Open-Source Free Software

Pigsty is free software open-sourced under Apache-2.0, watered by the passion of PostgreSQL-loving community members

Pigsty is completely open-source and free software, allowing you to run enterprise-grade PostgreSQL database services at nearly pure hardware cost without database experts. For comparison, database vendors’ “enterprise database services” and public cloud vendors’ RDS charge premiums several to over ten times the underlying hardware resources as “service fees.”

Many users choose the cloud precisely because they can’t handle databases themselves; many users use RDS because there’s no other choice. We will break cloud vendors’ monopoly, providing users with a cloud-neutral, better open-source RDS alternative: Pigsty follows PostgreSQL upstream closely, with no vendor lock-in, no annoying “licensing fees,” no node count limits, and no data collection. All your core assets — data — can be “autonomously controlled,” in your own hands.

Pigsty itself aims to replace tedious manual database operations with database autopilot software, but even the best software can’t solve all problems. There will always be some rare, low-frequency edge cases requiring expert intervention. This is why we also provide professional subscription services to provide safety nets for enterprise users who need them. Subscription consulting fees of tens of thousands are less than one-thirtieth of a top DBA’s annual salary, completely eliminating your concerns and putting costs where they really matter. For community users, we also contribute with love, providing free support and daily Q&A.

pigsty-price.jpg

tooltip: { trigger: axis, formatter: $fn:ttfmt }
legend: { top: 4, itemGap: 16, data: [Oracle, Open-Source PG, Cloud RDS, Pigsty over IaaS, Pigsty over IDC ] }
grid: { left: 96, right: 36, bottom: 70, top: 50 }
xAxis:
  type: category
  name: CPU Cores
  nameLocation: middle
  nameGap: 36
  boundaryGap: false
  data: [2, 4, 8, 12, 16, 24, 32, 52, 64, 104, 128, 196, 256, 384, 512]
yAxis:
  type: log
  logBase: 10
  min: 10
  name: Monthly Cost (CNY)
  axisLabel: { formatter: $fn:yfmt }
  splitLine: { show: true, lineStyle: { type: dashed, opacity: 0.5 } }
series:
  - { name: Oracle, type: line, symbolSize: 7, lineStyle: { width: 3 }, itemStyle: { color: "#d62728" }, data: [45000, 65000, 105000, 145000, 185000, 265000, 345000, 545000, 665000, 1065000, 1305000, 1985000, 2585000, 3865000, 5145000] }
  - { name: Cloud RDS, type: line, symbolSize: 6, lineStyle: { width: 2 }, itemStyle: { color: "#ff7f0e" }, data: [800, 1600, 3200, 4800, 6400, 9600, 12800, 20800, 25600, 41600, 51200, 78400, 102400, 153600, 204800] }
  - { name: Pigsty over IaaS, type: line, symbolSize: 6, lineStyle: { width: 2 }, itemStyle: { color: "#2ca02c" }, data: [360, 720, 1440, 2160, 2880, 4320, 5760, 9360, 11520, 18720, 23040, 35280, 46080, 69120, 92160] }
  - { name: Pigsty over IDC, type: line, symbolSize: 6, lineStyle: { width: 2 }, itemStyle: { color: "#9467bd" }, data: [38, 76, 152, 228, 304, 456, 608, 988, 1216, 1976, 2432, 3724, 4864, 7296, 9728] }

4.2 - History

The origin and motivation of the Pigsty project, its development history, and future goals and vision.

Historical Origins

The Pigsty project began in 2018-2019, originating from Tantan. Tantan is an internet dating app — China’s Tinder, now acquired by Momo. Tantan was a Nordic-style startup with a Swedish engineering founding team.

Tantan had excellent technical taste, using PostgreSQL and Go as its core technology stack. The entire Tantan system architecture was modeled after Instagram, designed entirely around the PostgreSQL database. Up to several million daily active users, millions of TPS, and hundreds of TB of data, the data component used only PostgreSQL. Almost all business logic was implemented using PG stored procedures — even including 100ms recommendation algorithms! It was arguably the most complex PostgreSQL-at-scale use case in China at the time.

This atypical development model of deeply using PostgreSQL features placed extremely high demands on the capabilities of engineers and DBAs. And Pigsty is the open-source project we forged in this real-world large-scale, high-standard database cluster scenario — embodying our experience and best practices as top PostgreSQL experts.

Development Process

In the beginning, Pigsty did not have the vision, goals, and scope it has today. It started as a PostgreSQL monitoring system for our own use. We surveyed all available solutions — open-source, commercial, cloud-based, datadog, pgwatch, etc. — and none could meet our observability needs. So I decided to build one myself based on Grafana and Prometheus. This became Pigsty’s predecessor and prototype. Pigsty as a monitoring system was quite impressive, helping us solve countless management problems.

Subsequently, developers wanted such a monitoring system on their local development machines, so we used Ansible to write provisioning playbooks, transforming this system from a one-time construction task into reusable, replicable software. New versions allowed users to use Vagrant and Terraform, using Infrastructure as Code to quickly spin up local DevBox development machines or production environment servers, automatically completing PostgreSQL and monitoring system deployment.

Next, we redesigned the production environment PostgreSQL architecture, introducing Patroni and pgBackRest to solve database high availability and point-in-time recovery issues. We developed a zero-downtime migration solution based on logical replication, rolling upgrading two hundred production database clusters to the latest major version through blue-green deployment. And we incorporated these capabilities into Pigsty.

Pigsty is software we built for ourselves. The biggest benefit of “eating our own dog food” is that we are both developers and users — as client users, we know exactly what we need, do not cut corners, and never worry about automating ourselves out of jobs.

We solved problem after problem, depositing the solutions into Pigsty. Pigsty’s positioning also gradually evolved from a monitoring system into an out-of-the-box PostgreSQL database distribution. We then decided to open-source Pigsty and began a series of technical sharing and publicity, and external users from various industries began using Pigsty and providing feedback.

Full-Time Entrepreneurship

In 2022, the Pigsty project received seed funding from Miracle Plus, initiated by Dr. Qi Lu, allowing me to work on this full-time.

As an open-source project, Pigsty has developed quite well. In these years of full-time work, Pigsty’s GitHub stars grew from a few hundred to 5,213 as of 2026-07-11; it made the HN front page, and growth began snowballing. In November 2025, Pigsty won the Magneto Award at the PostgreSQL Ecosystem Conference. In 2026, Pigsty’s subproject PGEXT.CLOUD was selected for a PGCon.Dev 2026 talk. Pigsty became the first Chinese open-source project to appear on the stage of this core PostgreSQL ecosystem conference.

Previously, Pigsty could only run on CentOS 7, but now it covers all mainstream Linux distributions (EL, Debian, Ubuntu) across 16 operating system platforms. Supported PG major versions cover 14-18, and we maintain and integrate 575 extension plugins in the PG ecosystem. Among these, I personally maintain over half (360+) of the extension plugins, providing out-of-the-box RPM/DEB packages. Including Pigsty itself, “based on open source, giving back to open source,” this is our way of contributing to the PG ecosystem.

Pigsty’s positioning has also continuously evolved from a PostgreSQL database distribution to an open-source cloud database. It truly benchmarks against cloud vendors’ entire cloud database brands.

Rebel Against Public Clouds

Public cloud vendors like AWS, Azure, GCP, and Aliyun have provided many conveniences for startups, but they are closed-source and force users to rent infrastructure at exorbitant fees.

We believe that excellent database services, like excellent database kernels, should be accessible to every user, rather than requiring expensive rental from cyber lords.

Cloud computing’s agility and elasticity value proposition is strong, but it should be free, open-source, inclusive, and local-first — We believe the cloud computing universe needs a solution representing open-source values that returns infrastructure control to users without sacrificing the benefits of the cloud.

Therefore, we are also leading a movement and battle to exit the cloud, as rebels against public clouds, to reshape the industry’s values.

Our Vision

I hope that in the future world, everyone will have the de facto right to freely use excellent services, rather than being confined to a few cyber lord public cloud giants’ territories as cyber tenants or even cyber serfs.

This is exactly what Pigsty aims to do — a better, free and open-source RDS alternative. Allowing users to spin up database services better than cloud RDS anywhere (including cloud servers) with one click.

Pigsty is a complete complement to PostgreSQL, and a spicy mockery of cloud databases. It literally means “pigsty,” but it’s also an acronym for Postgres In Great STYle, meaning “PostgreSQL in its full glory.”

Pigsty itself is completely open-source and free software, so you can build a PostgreSQL service that scores 90 without database experts. We sustain operations by providing premium consulting services to take you from 90 to 100, with warranty, Q&A, and a safety net.

A well-built system may run for years without needing a “safety net,” but database problems, once they occur, are never small. Often, expert experience can turn decay into magic, and we provide such premium consulting — we believe this is a more just, reasonable, and sustainable model.

About the Team

I am Feng Ruohang, the author of Pigsty. Almost all of Pigsty’s code is developed by me alone.

Individual heroism still exists in the software field. Only unique individuals can create unique works — I hope Pigsty becomes such a work.

If you’re interested in me, here’s my personal homepage: https://vonng.com/

“Modb Interview with Feng Ruohang” (Chinese)

“Post-90s, Quit to Start Business, Says Will Crush Cloud Databases” (Chinese)



4.3 - News & Events

News and events related to Pigsty and PostgreSQL, including latest announcements!

Recent News

Conferences & Talks

Date Type Event Topic
2025-11-29 Award&Talk The 8th Conf of PG Ecosystem (Hangzhou) PostgreSQL Magneto Award, A World-Grade Postgres Meta Distribution
2025-05-16 Lightning PGConf.Dev 2025, Montreal Extension Delivery: Make your PGEXT accessible to users
2025-05-12 Keynote PGEXT.DAY, PGCon.Dev 2025 The Missing Package Manager and Extension Repo for PostgreSQL Ecosystem
2025-04-19 Workshop PostgreSQL Database Technology Summit Using Pigsty to Deploy PG Ecosystem Partners: Dify, Odoo, Supabase
2025-04-11 Live Host OSCHINA Data Intelligence Talk Is the Viral MCP Hype or Revolutionary?
2025-01-15 Live Stream Open Source Veterans & Newcomers Episode 4 PostgreSQL Extensions Devouring DB World? PG Package Manager pig & Self-hosted RDS
2025-01-09 Award OSCHINA 2024 Outstanding Contribution Expert Outstanding Contribution Expert Award
2025-01-06 Panel China PostgreSQL Database Ecosystem Conference PostgreSQL Extensions are Devouring the Database World
2024-11-23 Podcast Tech Hotpot Podcast From the Linux Foundation: Why the Recent Focus on ‘Chokepoints’?
2024-08-21 Interview Blue Tech Wave Interview with Feng Ruohang: Simplifying PG Management
2024-08-15 Tech Summit GOTC Global Open Source Technology Summit PostgreSQL AI/ML/RAG Extension Ecosystem and Best Practices
2024-07-12 Keynote 13th PG China Technical Conference The Future of Database World: Extensions, Service, and Postgres
2024-05-31 Unconference PGCon.Dev 2024 Global PG Developer Conference Built-in Prometheus Metrics Exporter
2024-05-28 Seminar PGCon.Dev 2024 Extension Summit Extension in Core & Binary Packing
2024-05-10 Live Debate Three-way Talk: Cloud Mudslide Series Episode 3 Is Public Cloud a Scam?
2024-04-17 Live Debate Three-way Talk: Cloud Mudslide Series Episode 2 Are Cloud Databases a Tax on Intelligence?
2024-04-16 Panel Cloudflare Immerse Shenzhen Cyber Bodhisattva Panel Discussion
2024-04-12 Tech Summit 2024 Data Technology Carnival Pigsty: Solving PostgreSQL Operations Challenges
2024-03-31 Live Debate Three-way Talk: Cloud Mudslide Series Episode 1 Luo Selling Cloud While We’re Moving Off Cloud?
2024-01-24 Live Host OSCHINA Open Source Talk Episode 9 Will DBAs Be Eliminated by Cloud?
2023-12-20 Live Debate Open Source Talk Episode 7 To Cloud or Not: Cost Cutting or Value Creation?
2023-11-24 Tech Summit Vector Databases in the LLM Era Panel: New Future of Vector Databases in the AI Age
2023-09-08 Interview Motianlun Feature Interview Feng Ruohang: A Tech Enthusiast Who Makes Great Open Source Founders
2023-08-16 Tech Summit DTCC 2023 DBA Night: PostgreSQL vs MySQL Open Source License Issues
2023-08-09 Live Debate Open Source Talk Episode 1 MySQL vs PostgreSQL: Which is World’s No.1?
2023-07-01 Tech Summit SACC 2023 Workshop 8: FinOps Practice: Cloud Cost Management & Optimization
2023-05-12 Meetup PostgreSQL China Wenzhou Meetup PG With DB4AI: Vector Database PGVECTOR & AI4DB: Self-Driving Database Pigsty
2023-04-08 Tech Summit Database Carnival 2023 A Better Open Source RDS Alternative: Pigsty
2023-04-01 Tech Summit PostgreSQL China Xi’an Meetup PG High Availability & Disaster Recovery Best Practices
2023-03-23 Live Stream Bytebase x Pigsty Best Practices for Managing PostgreSQL: Bytebase x Pigsty
2023-03-04 Tech Summit PostgreSQL China Conference Challenging RDS, Pigsty v2.0 Release
2023-02-01 Tech Summit DTCC 2022 Open Source RDS Alternative: Battery-Included, Self-Driving Database Distro Pigsty
2022-07-21 Live Debate Cloud Swallows Open Source Can Open Source Strike Back Against Cloud?
2022-07-04 Interview Creator’s Story Post-90s Developer Quits to Start Up, Aiming to Challenge Cloud Databases
2022-06-28 Live Stream Bass’s Roundtable DBA’s Gospel: SQL Audit Best Practices
2022-06-12 Demo Day MiraclePlus S22 Demo Day User-Friendly Cost-Effective Database Distribution Pigsty
2022-06-05 Live Stream PG Chinese Community Sharing Pigsty v1.5 Quick Start, New Features & Production Cluster Setup

4.4 - Roadmap

Future feature planning, new feature release schedule, and todo list.

Release Strategy

Pigsty uses semantic versioning: <major>.<minor>.<patch>. Alpha/Beta/RC versions will have suffixes like -a1, -b1, -c1 appended to the version number.

Major version updates signify incompatible foundational changes and major new features; minor version updates typically indicate regular feature updates and small API changes; patch version updates mean bug fixes and package version updates.

Pigsty plans to release one major version update per year. Minor version updates usually follow PostgreSQL’s minor version update rhythm, catching up within a month at the latest after a new PostgreSQL version is released. Pigsty typically plans 4-6 minor versions per year. For complete release history, please refer to Release Notes.

Deploy with Specific Version Numbers

Pigsty develops using the main trunk branch. Please always use Releases with version numbers.

Unless you know what you’re doing, do not use GitHub’s main branch. Always check out and use a specific version.

Features Under Consideration

  • Agent Native CLI - PIG
  • DBA Agent - basic integration
  • Grafana dashboard improvements
  • Boar management console

Here are our Active Issues and Roadmap.

Extensions and Packages

For the extension support roadmap, you can find it here: https://pigsty.io/ext/e/roadmap

Under Consideration

Not Considering for Now

4.5 - Join the Community

Pigsty is a Build in Public project. We are very active on GitHub, and Chinese users are mainly active in WeChat groups.

GitHub

Our GitHub repository is: https://github.com/pgsty/pigsty. Please give us a ⭐️ star!

We welcome anyone to submit new Issues or create Pull Requests, propose feature suggestions, and contribute to Pigsty.

Star History Chart

Please note that for issues related to Pigsty documentation, please submit Issues in the github.com/pgsty/pigsty.cc repository.

Press ⌘K on macOS, or CtrlK, to search the documentation, extension catalog, and blog directly.

Maintainers

Pigsty is built by its maintainers and community.

2 contributors GitHub

WeChat Groups

Chinese users are mainly active in WeChat groups. Currently, there are seven active groups. Groups 1-4 are full; for other groups, you need to add the assistant’s WeChat to be invited.

To join the WeChat community, search for “Pigsty小助手” (WeChat ID: pigsty-cc), note or send “加群” (join group), and the assistant will invite you to the group.

Pigsty Chinese community

International Community

Telegram: https://t.me/joinchat/gV9zfZraNPM3YjFh

Discord: https://discord.gg/j5pG8qfKxU

You can also contact me via email: [email protected]

Community Help

When you encounter problems using Pigsty, you can seek help from the community. The more information you provide, the more likely you are to get help from the community.

Please refer to the Community Help Guide and provide as much information as possible so that community members can help you solve the problem. Here is a reference template for asking for help:

What happened? (Required)

Pigsty version and OS version (Required)

$ grep version pigsty.yml

$ cat /etc/os-release

$ uname -a

Some cloud providers have customized standard OS distributions. You can tell us which cloud provider’s OS image you are using. If you have customized and modified the environment after installing the OS, or if there are specific security rules and firewall configurations in your LAN, please also inform us when asking questions.

Pigsty configuration file

Please don’t forget to redact any sensitive information: passwords, internal keys, sensitive configurations, etc.

cat ~/pigsty/pigsty.yml

What did you expect to happen?

Please describe what should happen under normal circumstances, and how the actual situation differs from expectations.

How to reproduce this issue?

Please tell us in as much detail as possible how to reproduce this issue.

Monitoring screenshots

If you are using the monitoring system provided by Pigsty, you can provide relevant screenshots.

Error logs

Please provide logs related to the error as much as possible. Please do not paste content like “Failed to start xxx service” that has no informational value.

You can query logs from Grafana / VictoriaLogs, or get logs from the following locations:

  • Syslog: /var/log/messages (rhel) or /var/log/syslog (debian)
  • Postgres: /pg/log/postgres/*
  • Patroni: /pg/log/patroni/*
  • Pgbouncer: /pg/log/pgbouncer/*
  • Pgbackrest: /pg/log/pgbackrest/*
journalctl -u patroni
journalctl -u <service name>

Have you searched Issues/Website/FAQ?

In the FAQ, we provide answers to many common questions. Please check before asking.

You can also search for related issues from GitHub Issues and Discussions:

Is there any other information we need to know?

The more information and context you provide, the more likely we can help you solve the problem.

4.6 - Privacy Policy

What user data does Pigsty software and website collect, and how will we process your data and protect your privacy?

Pigsty Software

When you install Pigsty software, if you use offline package installation in a network-isolated environment, we will not receive any data about you.

If you choose online installation, when downloading related packages, our servers or cloud provider servers will automatically log the visiting machine’s IP address and/or hostname in the logs, along with the package names you downloaded.

We will not share this information with other organizations unless required by law. (Honestly, we’d have to be really bored to look at this stuff.)

Pigsty’s primary domain is: pigsty.io. For mainland China, please use the registered mirror site pigsty.cc.

Pigsty Website

When you visit our website, our servers will automatically log your IP address and/or hostname in Nginx logs.

We will only store information such as your email address, name, and location when you decide to send us such information by completing a survey or registering as a user on one of our websites.

We collect this information to help us improve website content, customize web page layouts, and contact people for technical and support purposes. We will not share your email address with other organizations unless required by law.

This website uses Google Analytics, a web analytics service provided by Google, Inc. (“Google”). Google Analytics uses “cookies,” which are text files placed on your computer to help the website analyze how users use the site.

The information generated by the cookie about your use of the website (including your IP address) will be transmitted to and stored by Google on servers in the United States. Google will use this information to evaluate your use of the website, compile reports on website activity for website operators, and provide other services related to website activity and internet usage. Google may also transfer this information to third parties if required by law or where such third parties process the information on Google’s behalf. Google will not associate your IP address with any other data held by Google. You may refuse the use of cookies by selecting the appropriate settings on your browser, however, please note that if you do this, you may not be able to use the full functionality of this website. By using this website, you consent to the processing of data about you by Google in the manner and for the purposes set out above.

If you have any questions or comments about this policy, or request deletion of personal data, you can contact us by sending an email to [email protected]



4.7 - License

Pigsty’s open-source licenses — Apache-2.0 and CC BY 4.0

License Summary

Pigsty core uses Apache-2.0; documentation uses CC BY 4.0.

Official License: https://github.com/pgsty/pigsty/blob/main/LICENSE

Pigsty Core

The Pigsty core is licensed under Apache License 2.0.

Apache-2.0 is a permissive open-source license. You may freely use, modify, and distribute the software for commercial purposes without opening your own source code or adopting the same license.

What This License Grants What This License Does NOT Grant License Conditions
Commercial use Trademark use Include license and copyright notice
Modification Liability & warranty State changes
Distribution
Patent grant
Private use

Pigsty Documentation

Pigsty documentation sites (pigsty.cc, pigsty.io, pgsty.com) use Creative Commons Attribution 4.0 International (CC BY 4.0).

CC BY 4.0 permits free sharing and adaptation with appropriate credit, a license link, and indication of changes.

What This License Grants What This License Does NOT Grant License Conditions
Commercial use Trademark use Attribution
Modification Liability & warranty Indicate changes
Distribution Patent grant Provide license link
Private use

SBOM Inventory

Open-source software used or related to the Pigsty project.

For 575 PostgreSQL extension plugin licenses, refer to PostgreSQL Extension License List.

Module Software Name License Purpose & Description Necessity
PGSQL PostgreSQL PostgreSQL License PostgreSQL kernel Required
PGSQL patroni MIT License PostgreSQL high availability Required
ETCD etcd Apache License 2.0 HA consensus and distributed config storage Required
INFRA Ansible GPLv3 Executes playbooks and management commands Required
INFRA Nginx BSD-2 Exposes Web UI and serves local repo Recommended
PGSQL pgbackrest MIT License PITR backup/recovery management Recommended
PGSQL pgbouncer ISC License PostgreSQL connection pooling Recommended
PGSQL vip-manager BSD 2-Clause License Automatic L2 VIP binding to PG primary Recommended
PGSQL pg_exporter Apache License 2.0 PostgreSQL and PgBouncer monitoring Recommended
NODE node_exporter Apache License 2.0 Host node monitoring metrics Recommended
NODE haproxy HAPROXY’s License (GPLv2) Load balancing and service exposure Recommended
INFRA Grafana AGPLv3 Database visualization platform Recommended
INFRA VictoriaMetrics Apache License 2.0 TSDB, metric collection, alerting Recommended
INFRA VictoriaLogs Apache License 2.0 Centralized log collection, storage, query Recommended
INFRA DNSMASQ GPLv2 / GPLv3 DNS resolution and cluster name lookup Recommended
MINIO Silo AGPLv3 The only object-storage service supported by the current MINIO module Optional
INFRA Historical MinIO branch AGPLv3 Historical/repository package; not a v4.5 MINIO backend Optional
INFRA RustFS Apache License 2.0 Repository-retained package; not a v4.5 MINIO backend Optional
NODE keepalived MIT License VIP binding on node clusters Optional
REDIS Redis BSD 3-Clause Default cache engine, using the Redis 7.2 BSD branch Optional
REDIS Valkey BSD 3-Clause Cache engine selected with redis_type: valkey Optional
REDIS Redis Exporter MIT License Redis monitoring Optional
MONGO FerretDB Apache License 2.0 MongoDB compatibility over PostgreSQL Optional
DOCKER docker-ce Apache License 2.0 Container management Optional
CLOUD SealOS Apache License 2.0 Fast K8S cluster deployment and packaging Optional
DUCKDB DuckDB MIT High-performance analytics Optional
External Vagrant Business Source License 1.1 Local test environment VMs Optional
External Terraform Business Source License 1.1 One-click cloud resource provisioning Optional
External Virtualbox GPLv2 Virtual machine management software Optional

Necessity Levels:

  • Required: Essential core capabilities, no option to disable
  • Recommended: Enabled by default, can be disabled via configuration
  • Optional: Not enabled by default, can be enabled via configuration

Apache-2.0 License Text 

                                 Apache License
                           Version 2.0, January 2004
                        http://www.apache.org/licenses/

   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION

   1. Definitions.

      "License" shall mean the terms and conditions for use, reproduction,
      and distribution as defined by Sections 1 through 9 of this document.

      "Licensor" shall mean the copyright owner or entity authorized by
      the copyright owner that is granting the License.

      "Legal Entity" shall mean the union of the acting entity and all
      other entities that control, are controlled by, or are under common
      control with that entity. For the purposes of this definition,
      "control" means (i) the power, direct or indirect, to cause the
      direction or management of such entity, whether by contract or
      otherwise, or (ii) ownership of fifty percent (50%) or more of the
      outstanding shares, or (iii) beneficial ownership of such entity.

      "You" (or "Your") shall mean an individual or Legal Entity
      exercising permissions granted by this License.

      "Source" form shall mean the preferred form for making modifications,
      including but not limited to software source code, documentation
      source, and configuration files.

      "Object" form shall mean any form resulting from mechanical
      transformation or translation of a Source form, including but
      not limited to compiled object code, generated documentation,
      and conversions to other media types.

      "Work" shall mean the work of authorship, whether in Source or
      Object form, made available under the License, as indicated by a
      copyright notice that is included in or attached to the work
      (an example is provided in the Appendix below).

      "Derivative Works" shall mean any work, whether in Source or Object
      form, that is based on (or derived from) the Work and for which the
      editorial revisions, annotations, elaborations, or other modifications
      represent, as a whole, an original work of authorship. For the purposes
      of this License, Derivative Works shall not include works that remain
      separable from, or merely link (or bind by name) to the interfaces of,
      the Work and Derivative Works thereof.

      "Contribution" shall mean any work of authorship, including
      the original version of the Work and any modifications or additions
      to that Work or Derivative Works thereof, that is intentionally
      submitted to Licensor for inclusion in the Work by the copyright owner
      or by an individual or Legal Entity authorized to submit on behalf of
      the copyright owner. For the purposes of this definition, "submitted"
      means any form of electronic, verbal, or written communication sent
      to the Licensor or its representatives, including but not limited to
      communication on electronic mailing lists, source code control systems,
      and issue tracking systems that are managed by, or on behalf of, the
      Licensor for the purpose of discussing and improving the Work, but
      excluding communication that is conspicuously marked or otherwise
      designated in writing by the copyright owner as "Not a Contribution."

      "Contributor" shall mean Licensor and any individual or Legal Entity
      on behalf of whom a Contribution has been received by Licensor and
      subsequently incorporated within the Work.

   2. Grant of Copyright License. Subject to the terms and conditions of
      this License, each Contributor hereby grants to You a perpetual,
      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
      copyright license to reproduce, prepare Derivative Works of,
      publicly display, publicly perform, sublicense, and distribute the
      Work and such Derivative Works in Source or Object form.

   3. Grant of Patent License. Subject to the terms and conditions of
      this License, each Contributor hereby grants to You a perpetual,
      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
      (except as stated in this section) patent license to make, have made,
      use, offer to sell, sell, import, and otherwise transfer the Work,
      where such license applies only to those patent claims licensable
      by such Contributor that are necessarily infringed by their
      Contribution(s) alone or by combination of their Contribution(s)
      with the Work to which such Contribution(s) was submitted. If You
      institute patent litigation against any entity (including a
      cross-claim or counterclaim in a lawsuit) alleging that the Work
      or a Contribution incorporated within the Work constitutes direct
      or contributory patent infringement, then any patent licenses
      granted to You under this License for that Work shall terminate
      as of the date such litigation is filed.

   4. Redistribution. You may reproduce and distribute copies of the
      Work or Derivative Works thereof in any medium, with or without
      modifications, and in Source or Object form, provided that You
      meet the following conditions:

      (a) You must give any other recipients of the Work or
          Derivative Works a copy of this License; and

      (b) You must cause any modified files to carry prominent notices
          stating that You changed the files; and

      (c) You must retain, in the Source form of any Derivative Works
          that You distribute, all copyright, patent, trademark, and
          attribution notices from the Source form of the Work,
          excluding those notices that do not pertain to any part of
          the Derivative Works; and

      (d) If the Work includes a "NOTICE" text file as part of its
          distribution, then any Derivative Works that You distribute must
          include a readable copy of the attribution notices contained
          within such NOTICE file, excluding those notices that do not
          pertain to any part of the Derivative Works, in at least one
          of the following places: within a NOTICE text file distributed
          as part of the Derivative Works; within the Source form or
          documentation, if provided along with the Derivative Works; or,
          within a display generated by the Derivative Works, if and
          wherever such third-party notices normally appear. The contents
          of the NOTICE file are for informational purposes only and
          do not modify the License. You may add Your own attribution
          notices within Derivative Works that You distribute, alongside
          or as an addendum to the NOTICE text from the Work, provided
          that such additional attribution notices cannot be construed
          as modifying the License.

      You may add Your own copyright statement to Your modifications and
      may provide additional or different license terms and conditions
      for use, reproduction, or distribution of Your modifications, or
      for any such Derivative Works as a whole, provided Your use,
      reproduction, and distribution of the Work otherwise complies with
      the conditions stated in this License.

   5. Submission of Contributions. Unless You explicitly state otherwise,
      any Contribution intentionally submitted for inclusion in the Work
      by You to the Licensor shall be under the terms and conditions of
      this License, without any additional terms or conditions.
      Notwithstanding the above, nothing herein shall supersede or modify
      the terms of any separate license agreement you may have executed
      with Licensor regarding such Contributions.

   6. Trademarks. This License does not grant permission to use the trade
      names, trademarks, service marks, or product names of the Licensor,
      except as required for reasonable and customary use in describing the
      origin of the Work and reproducing the content of the NOTICE file.

   7. Disclaimer of Warranty. Unless required by applicable law or
      agreed to in writing, Licensor provides the Work (and each
      Contributor provides its Contributions) on an "AS IS" BASIS,
      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
      implied, including, without limitation, any warranties or conditions
      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
      PARTICULAR PURPOSE. You are solely responsible for determining the
      appropriateness of using or redistributing the Work and assume any
      risks associated with Your exercise of permissions under this License.

   8. Limitation of Liability. In no event and under no legal theory,
      whether in tort (including negligence), contract, or otherwise,
      unless required by applicable law (such as deliberate and grossly
      negligent acts) or agreed to in writing, shall any Contributor be
      liable to You for damages, including any direct, indirect, special,
      incidental, or consequential damages of any character arising as a
      result of this License or out of the use or inability to use the
      Work (including but not limited to damages for loss of goodwill,
      work stoppage, computer failure or malfunction, or any and all
      other commercial damages or losses), even if such Contributor
      has been advised of the possibility of such damages.

   9. Accepting Warranty or Additional Liability. While redistributing
      the Work or Derivative Works thereof, You may choose to offer,
      and charge a fee for, acceptance of support, warranty, indemnity,
      or other liability obligations and/or rights consistent with this
      License. However, in accepting such obligations, You may act only
      on Your own behalf and on Your sole responsibility, not on behalf
      of any other Contributor, and only if You agree to indemnify,
      defend, and hold each Contributor harmless for any liability
      incurred by, or claims asserted against, such Contributor by reason
      of your accepting any such warranty or additional liability.

   END OF TERMS AND CONDITIONS

   APPENDIX: How to apply the Apache License to your work.

      To apply the Apache License to your work, attach the following
      boilerplate notice, with the fields enclosed by brackets "[]"
      replaced with your own identifying information. (Don't include
      the brackets!)  The text should be enclosed in the appropriate
      comment syntax for the file format. We also recommend that a
      file or class name and description of purpose be included on the
      same "printed page" as the copyright notice for easier
      identification within third-party archives.

   Copyright (C) 2018-2026  Ruohang Feng, @Vonng ([email protected])

   Licensed under the Apache License, Version 2.0 (the "License");
   you may not use this file except in compliance with the License.
   You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.

4.8 - Sponsor Us

Pigsty sponsors and investors list - thank you for your support of this project!

Sponsor Us

Pigsty is a free and open-source software, passionately developed by PostgreSQL community members, aiming to integrate the power of the PostgreSQL ecosystem and promote the widespread adoption of PostgreSQL. If our work has helped you, please consider sponsoring or supporting our project:

  • Sponsor us directly with financial support - express your sincere support in the most direct and powerful way!
  • Consider purchasing our Technical Support Services. We can provide professional PostgreSQL high-availability cluster deployment and maintenance services, making your budget worthwhile!
  • Share your Pigsty use cases and experiences through articles, talks, and videos.
  • Allow us to mention your organization in “Users of Pigsty.”
  • Recommend/refer our project and services to friends, colleagues, and clients in need.
  • Follow our WeChat Official Account and share relevant technical articles to groups and your social media.

Angel Investors

Pigsty is a project invested by Miracle Plus (formerly YC China) S22. We thank Miracle Plus and Dr. Qi Lu for their support of this project!

Sponsors

Special thanks to Vercel for sponsoring pigsty and hosting the Pigsty website.

Vercel OSS Program

Special thanks to JetBrains for sponsoring Pigsty with JetBrains Open Source License

JetBrains logo.

4.9 - User Cases

Pigsty customer and application cases across various domains and industries

According to Google Analytics PV and download statistics, Pigsty currently has approximately 100,000 users, with half from mainland China and half from other regions globally. They span across multiple industries including internet, cloud computing, finance, autonomous driving, manufacturing, tech innovation, ISV, and defense. If you are using Pigsty and are willing to share your case and Logo with us, please contact us - we offer one free consultation session as a token of appreciation.

Internet

Tantan: 200+ physical machines for PostgreSQL and Redis services

Bilibili: Supporting PostgreSQL innovative business

Cloud Vendors

Bitdeer: Providing PG DBaaS

Oracle OCI: Using Pigsty to deliver PostgreSQL clusters

Finance

AirWallex: Monitoring 200+ GCP PostgreSQL databases

Media & Entertainment

Media Storm: Self-hosted PG RDS / Victoria Metrics

Autonomous Driving

Momenta: Autonomous driving, managing self-hosted PostgreSQL clusters

Manufacturing

Huafon Group: Using Pigsty to deliver PostgreSQL clusters as chemical industry time-series data warehouse

Tech Innovation

Beijing Lingwu Technology: Migrating PostgreSQL from cloud to self-hosted

Motphys: Self-hosted PostgreSQL supporting GitLab

Sailong Biotech: Self-hosted Supabase

Hangzhou Lingma Technology: Self-hosted PostgreSQL

ISV

Inner Mongolia Haode Tianmu Technology Co., Ltd.

Shanghai Yuanfang

DSG

4.10 - Subscription

Pigsty Professional/Enterprise subscription service - When you encounter difficulties related to PostgreSQL and Pigsty, our subscription service provides you with comprehensive support.

Pigsty aims to unite the power of the PostgreSQL ecosystem and help users make the most of the world’s most popular database, PostgreSQL, with self-driving database management software.

While Pigsty itself has already resolved many issues in PostgreSQL usage, achieving truly enterprise-grade service quality requires expert support and comprehensive coverage from the original provider. We deeply understand the importance of professional commercial support for enterprise customers. Therefore, Pigsty Enterprise Edition provides a series of value-added services on top of the open-source version, helping users better utilize PostgreSQL and Pigsty for customers to choose according to their needs.

If you have any of the following needs, please consider Pigsty subscription service:

  • Running databases in critical scenarios requiring strict SLA guarantees and comprehensive coverage.
  • Need comprehensive support for complex issues related to Pigsty and PostgreSQL.
  • Seeking guidance on PostgreSQL/Pigsty production environment best practices.
  • Want experts to help interpret monitoring dashboards, analyze and identify performance bottlenecks and fault root causes, and provide recommendations.
  • Need to plan database architectures that meet security/disaster recovery/compliance requirements based on existing resources and business needs.
  • Need to migrate from other databases to PostgreSQL, or migrate and transform legacy instances.
  • Building an observability system, data dashboards, and visualization applications based on the Victoria/Grafana technology stack.
  • Migrating off cloud and seeking open-source alternatives to RDS for PostgreSQL - cloud-neutral, vendor lock-in-free solutions.
  • Want professional support for Redis/ETCD/Silo, as well as extensions like TimescaleDB/Citus.
  • Want to perform secondary development and OEM branding with explicit commercial authorization.
  • Want to sell Pigsty as SaaS/PaaS/DBaaS, or provide technical services/consulting/cloud services based on this distribution.

Subscription Plans

In addition to the Open Source Edition, Pigsty offers two different subscription service tiers: Professional Edition and Enterprise Edition, which you can choose based on your actual situation and needs.

Note on https://pigsty.io/price: The https://pigsty.io/price page is a simplified global pricing landing page (USD pricing, includes the Standard tier and node-cap presets). This page is the detailed subscription reference (CNY pricing, delivery scope, and OS/PG compatibility matrix). For technical compatibility boundaries, this page and Supported Linux prevail.

Pigsty Open Source Edition (OSS)Free and Open Source

No scale limit, no warranty

License: Apache-2.0
PG Support: 18 (default), 14–18 available
Architecture Support: x86_64, Arm64
OS Support: Latest minor versions of three families

  • EL 9.8 / 10.2
  • Debian 12.15 / 13.6
  • Ubuntu 22.04.5 / 24.04.4 / 26.04.0

Features: Core Modules
SLA: No SLA commitment

Community support Q&A:

Support: No person-day support option
Repository: Global Cloudflare hosted repository

Best for self-sufficient open source veterans.

Pigsty Professional Edition (PRO)¥150,000 / year

Default choice for regular users

License: Commercial License
PG Support: 14–18
Architecture Support: x86_64, Arm64
OS Support: Mainstream OS major/minor versions

  • EL 8 / 9 / 10 compatible
  • Debian 12 / 13
  • Ubuntu 22 / 24 / 26

Features: All Modules (except domestic innovation kernels)
SLA: Response within business hours

Expert consulting services:

  • Software bug fixes
  • Complex issue analysis
  • Expert ticket support

Support: 1 person-day included per year
Delivery: Standard offline software package
Repository: China mainland mirror sites

The default choice for regular users.

Pigsty Enterprise Edition (ENTERPRISE)¥400,000 / year

Critical scenarios with strict SLA

License: Commercial License
PG Support: 14–18+ (legacy versions on request)
Architecture Support: x86_64, Arm64
OS Support: Customized on demand

  • EL, Debian, Ubuntu
  • Cloud Linux operating systems
  • Domestic OS and ARM

Features: All Modules
SLA: 7 x 24 (< 1h)

Enterprise-level expert consulting services:

  • Software bug fixes
  • Complex issue analysis
  • Expert Q&A support
  • Backup compliance advice
  • Upgrade path support
  • Performance bottleneck identification
  • Annual architecture review
  • Extension plugin integration
  • DBaaS & OEM use cases

Support: 2 person-days included per year
Repository: China mainland mirror sites
Delivery: Customized offline software package
Domestic Innovation: PolarDB-O support

For critical scenarios with a strict SLA.

Pigsty Open Source Edition (OSS)

Pigsty Open Source Edition uses the Apache-2.0 license, provides complete core functionality, requires no fees, but does not guarantee any warranty service. If you find defects in Pigsty, we welcome you to submit an Issue on Github.

Pigsty Open Source supports seven currently validated baselines: EL 9.8 / 10.2, Debian 12.15 / 13.6, and Ubuntu 22.04.5 / 24.04.4 / 26.04.0, across both x86_64 and aarch64. The historical v4.4.0 Community Edition artifacts comprise six dual-architecture offline bundles built on EL 10.1, Debian 13.6, and Ubuntu 24.04.4. Those build baselines are not the same as the currently recommended operating systems; see the offline installation guide.

Using the Pigsty open source version allows junior development/operations engineers to have 70%+ of the capabilities of professional DBAs. Even without database experts, they can easily set up a highly available, high-performance, easy-to-maintain, secure and reliable PostgreSQL database cluster.

Code OS Distribution Version x86_64 aarch64 PG18 PG17 PG16 PG15 PG14
EL10 RHEL 10 / Rocky10 / Alma10 el10.x86_64 el10.aarch64
EL9 RHEL 9 / Rocky9 / Alma9 el9.x86_64 el9.aarch64
U26 Ubuntu 26.04 (resolute) u26.x86_64 u26.aarch64
U24 Ubuntu 24.04 (noble) u24.x86_64 u24.aarch64
U22 Ubuntu 22.04 (jammy) u22.x86_64 u22.aarch64
D13 Debian 13 (trixie) d13.x86_64 d13.aarch64
D12 Debian 12 (bookworm) d12.x86_64 d12.aarch64

= Primary support, = Optional support

Pigsty Professional Edition (PRO)

Professional Edition Subscription: Starting Price ¥150,000 / year

Pigsty Professional Edition subscription provides complete functional modules and warranty for Pigsty itself. For defects in PostgreSQL itself and extension plugins, we will make our best efforts to provide feedback and fixes through the PostgreSQL global developer community.

Pigsty Professional Edition is built on the open source version, fully compatible with all open source features, and provides additional modules plus broader database/OS compatibility options: we provide build options for all minor versions of eight mainstream Linux releases (EL8/9/10, Debian 12/13, Ubuntu 22/24/26).

Pigsty Professional Edition includes support for PostgreSQL 14 - 18, and tracks upstream PostgreSQL minor updates continuously (for active majors, typically day-zero or near-day availability), ensuring smooth rolling upgrades to newer majors and minors.

Pigsty Professional Edition subscription allows you to use China mainland mirror site software repositories, accessible without VPN/proxy; we will also customize offline software installation packages for your exact operating system major/minor version, ensuring normal installation and delivery in air-gapped environments, achieving autonomous and controllable deployment.

Pigsty Professional Edition subscription provides standard expert consulting services, including complex issue analysis, DBA Q&A support, backup compliance advice, etc. We commit to responding to your issues within business hours (5x8), and provide 1 person-day support per year, with optional person-day add-on options.

Pigsty Professional Edition uses a commercial license, providing additional modules, technical support, and warranty services.

Pigsty Professional Edition starting price is ¥150,000 / year, equivalent to the annual fee for 9 vCPU AWS high-availability RDS PostgreSQL, or a junior operations engineer with a monthly salary of 10,000 yuan.

Code OS Distribution Version x86_64 aarch64 PG18 PG17 PG16 PG15 PG14
EL10 RHEL 10 / Rocky10 / Alma10 el10.x86_64 el10.aarch64
EL9 RHEL 9 / Rocky9 / Alma9 el9.x86_64 el9.aarch64
EL8 RHEL 8 / Rocky8 / Alma8 / Anolis8 el8.x86_64 el8.aarch64
U26 Ubuntu 26.04 (resolute) u26.x86_64 u26.aarch64
U24 Ubuntu 24.04 (noble) u24.x86_64 u24.aarch64
U22 Ubuntu 22.04 (jammy) u22.x86_64 u22.aarch64
D13 Debian 13 (trixie) d13.x86_64 d13.aarch64
D12 Debian 12 (bookworm) d12.x86_64 d12.aarch64

Pigsty Enterprise Edition

Enterprise Edition Subscription: Starting Price ¥400,000 / year

Pigsty Enterprise Edition subscription includes all service content provided by the Pigsty Professional Edition subscription, plus the following value-added service items:

Pigsty Enterprise Edition subscription provides the broadest range of database/operating system version support, including extended support for EOL operating systems (EL7, D11), domestic operating systems, cloud vendor operating systems, and legacy PostgreSQL major versions (PG12+ on request), as well as full support for Arm64 architecture chips.

Pigsty Enterprise Edition subscription provides domestic innovation and localization solutions, allowing you to use PolarDB v2.0 (this kernel license needs to be purchased separately) kernel to replace the native PostgreSQL kernel and meet local compliance requirements.

Pigsty Enterprise Edition subscription provides higher-standard enterprise-level consulting services, committing to 7x24 with (< 1h) response time SLA, and can provide more types of consulting support: version upgrades, performance bottleneck identification, annual architecture review, extension plugin integration, etc.

Pigsty Enterprise Edition subscription includes 2 person-days of support per year, with optional person-day add-on options, for resolving more complex and time-consuming issues.

Pigsty Enterprise Edition allows you to use Pigsty for DBaaS purposes, building cloud database services for external sales.

Pigsty Enterprise Edition starting price is ¥400,000 / year, equivalent to the annual fee for 24 vCPU AWS high-availability RDS, or an operations expert with a monthly salary of 30,000 yuan.

Code OS Distribution Version x86_64 aarch64 PG18 PG17 PG16 PG15 PG14 PG13 PG12
EL10 RHEL 10 / Rocky10 / Alma10 el10.x86_64 el10.aarch64
EL9 RHEL 9 / Rocky9 / Alma9 el9.x86_64 el9.aarch64
EL8 RHEL 8 / Rocky8 / Alma8 / Anolis8 el8.x86_64 el8.aarch64
U26 Ubuntu 26.04 (resolute) u26.x86_64 u26.aarch64
U24 Ubuntu 24.04 (noble) u24.x86_64 u24.aarch64
U22 Ubuntu 22.04 (jammy) u22.x86_64 u22.aarch64
D13 Debian 13 (trixie) d13.x86_64 d13.aarch64
D12 Debian 12 (bookworm) d12.x86_64 d12.aarch64
D11 Debian 11 (bullseye) d11.x86_64 d11.aarch64
EL7 RHEL7 / CentOS7 / UOS … el7.x86_64 -

Pigsty Subscription Notes

Feature Differences

Pigsty Professional/Enterprise Edition includes the following additional features compared to the open source version:

  • Command Line Management Tool: Unlock the full functionality of the Pigsty command line tool (pig)
  • System Customization Capability: Provide pre-built offline installation packages for exact mainstream Linux operating system distribution major/minor versions
  • Offline Installation Capability: Complete Pigsty installation in environments without Internet access (air-gapped environments)
  • Multi-version PG Kernel: Allow users to freely specify and install PostgreSQL major versions within the lifecycle (14 - 18)
  • Kernel Replacement Capability: Allow users to use other PostgreSQL-compatible kernels to replace the native PG kernel, and the ability to install these kernels offline
    • Babelfish: Provides Microsoft SQL Server wire protocol-level compatibility
    • IvorySQL: Based on PG, provides Oracle syntax/type/stored procedure compatibility
    • PolarDB PG: Provides support for open-source PolarDB for PostgreSQL kernel
    • PolarDB O: Domestic innovation database with Oracle-compatible kernel for local compliance requirements (Enterprise Edition subscription only)
  • Extension Support Capability: Provides out-of-the-box installation for 575 available PG extensions for PG 14-18 on mainstream operating systems.
  • Complete Functional Modules: Provides all functional modules:
    • Supabase: Reliably self-host production-grade open-source Firebase
    • Silo: Enterprise PB-level object storage planning and self-hosting
    • DuckDB: Provides comprehensive DuckDB support, and PostgreSQL + DuckDB OLAP extension plugin support
    • Kafka: Provides high-availability Kafka cluster deployment and monitoring
    • Kubernetes, VictoriaMetrics & VictoriaLogs
  • Domestic Operating System Support: Provides domestic innovation OS support options (Enterprise Edition subscription only)
  • Domestic ARM Architecture Support: Provides domestic ARM64 architecture support options (Enterprise Edition subscription only)
  • China Mainland Mirror Repository: Smooth installation without VPN, providing domestic YUM/APT repository mirrors and DockerHub access proxy.
  • Chinese Interface Support: Monitoring system Chinese interface support (Beta)

Payment Model

Pigsty subscription uses an annual payment model. After signing the contract, the one-year validity period is calculated from the contract date. If payment is made before the subscription contract expires, it is considered automatic renewal. Consecutive subscriptions have discounts. The first renewal (second year) enjoys a 95% discount, the second and subsequent renewals enjoy a 90% discount on subscription fees, and one-time subscriptions for three years or more enjoy an overall 85% discount.

After the annual subscription contract terminates, you can choose not to renew the subscription service. Pigsty will no longer provide software updates, technical support, and consulting services, but you can continue to use the already installed version of Pigsty Professional Edition software. If you subscribed to Pigsty professional services and choose not to renew, when re-subscribing you do not need to make up for the subscription fees during the interruption period, but all discounts and benefits will be reset.

Pigsty’s pricing strategy ensures value for money - you can immediately get top DBA’s database architecture construction solutions and management best practices, with their consulting support and comprehensive coverage; while the cost is highly competitive compared to hiring database experts full-time or using cloud databases. Here are market references for enterprise-level database professional service pricing:

The fair price for decent database professional services is 10,000 ~ 20,000 yuan / year, with the billing unit being vCPU, i.e., one CPU thread (1 Intel core = 2 vCPU threads). Pigsty provides top-tier PostgreSQL expert services in China and adopts a per-node billing model. On commonly seen high-core-count server nodes, it brings users an unparalleled cost reduction and efficiency improvement experience.

Pigsty Expert Services

In addition to Pigsty subscription, Pigsty also provides on-demand Pigsty x PostgreSQL expert services - industry-leading database experts available for consultation.

Expert Advisor: ¥300,000 / three years


Within three years, provides 10 complex case handling sessions related to PostgreSQL and Pigsty, and unlimited Q&A.

Expert Support: ¥30,000 / person·day


Industry-leading expert on-site support, available for architecture consultation, fault analysis, problem troubleshooting, database health checks, monitoring interpretation, migration assessment, teaching and training, cloud migration/de-cloud consultation, and other continuous time-consuming scenarios.

Expert Consultation: ¥3,000 / case


Consult on any questions you want to know about Pigsty, PostgreSQL, databases, cloud computing, AI… Database veterans, cloud computing maverick sharing industry-leading insights, cognition, and judgment.

Quick Consultation: ¥300 / question


Get a quick diagnostic opinion and response to questions related to PostgreSQL / Pigsty / databases, not exceeding 5 minutes.

Contact Information

Please send an email to [email protected]. Users in mainland China are welcome to add WeChat ID RuohangFeng.

4.11 - FAQ

Answers to frequently asked questions about the Pigsty project itself.

What is Pigsty, and what is it not?

Pigsty is a PostgreSQL database distribution, a local-first open-source RDS cloud database solution. Pigsty is not a Database Management System (DBMS), but rather a tool, distribution, solution, and best practice for managing DBMS.

Analogy: The database is the car, then the DBA is the driver, RDS is the taxi service, and Pigsty is the autonomous driving software.

What problem does Pigsty solve?

The ability to use databases well is extremely scarce: either hire database experts at high cost to self-build (hire drivers), or rent RDS from cloud vendors at sky-high prices (hail a taxi), but now you have a new option: Pigsty (autonomous driving). Pigsty helps users use databases well: allowing users to self-build higher-quality and more efficient local cloud database services at less than 1/10 the cost of RDS, without a DBA!

Who are Pigsty’s target users?

Pigsty has two typical target user groups. The foundation is medium to large companies building ultra-large-scale enterprise/production-grade PostgreSQL RDS / DBaaS services. Through extreme customizability, Pigsty can meet the most demanding database management needs and provide enterprise-level support and service guarantees.

At the same time, Pigsty also provides “out-of-the-box” PG RDS self-building solutions for individual developers, small and medium enterprises lacking DBA capabilities, and the open-source community.

Why can Pigsty help you use databases well?

Pigsty embodies the experience and best practices of top experts refined in the most complex and largest-scale client PostgreSQL scenarios, productized into replicable software: Solving extension installation, high availability, connection pooling, monitoring, backup and recovery, parameter optimization, IaC batch management, one-click installation, automated operations, and many other issues at once. Avoiding many pitfalls in advance and preventing repeated mistakes.

Why is Pigsty better than RDS?

Pigsty provides a feature set and infrastructure support far beyond RDS, including 575 extension plugins and 12+ kernel support. Pigsty provides a unique professional-grade monitoring system in the PG ecosystem, along with architectural best practices battle-tested in complex scenarios, simple and easy to use.

Moreover, forged in top-tier client scenarios like Tantan, Apple, and Alibaba, continuously nurtured with passion and love, its depth and maturity are incomparable to RDS’s one-size-fits-all approach.

Why is Pigsty cheaper than RDS?

Pigsty allows you to use 10 ¥/core·month pure hardware resources to run 400¥-1400¥/core·month RDS cloud databases, and save the DBA’s salary. Typically, the total cost of ownership (TCO) of a large-scale Pigsty deployment can be over 90% lower than RDS.

Pigsty can simultaneously reduce software licensing/services/labor costs. Self-building requires no additional staff, allowing you to spend costs where it matters most.

How does Pigsty help developers?

Pigsty integrates the most comprehensive extensions in the PG ecosystem (575), providing an All-in-PG solution: a single component replacing specialized components like Redis, Kafka, MySQL, ES, vector databases, OLAP / big data analytics.

Greatly improving R&D efficiency and agility while reducing complexity costs, and developers can achieve self-service management and autonomous DevOps with Pigsty’s support, without needing a DBA.

How does Pigsty help operations?

Pigsty’s self-healing high-availability architecture ensures hardware failures don’t need immediate handling, letting ops and DBAs sleep well; monitoring aids problem analysis and performance optimization; IaC enables automated management of ultra-large-scale clusters.

Operations can moonlight as DBAs with Pigsty’s support, while DBAs can skip the system building phase, saving significant work hours and focusing on high-value work, or relaxing, learning PG.

Who is the author of Pigsty?

Pigsty is primarily developed by Feng Ruohang alone, an open-source contributor, database expert, and evangelist who has focused on PostgreSQL for 10 years, formerly at Alibaba, Tantan, and Apple, a full-stack expert. Now the founder of a one-person company, providing professional consulting services.

He is also a tech KOL, the founder of the top WeChat database personal account “非法加冯” (Illegally Add Feng), with 60,000+ followers across all platforms.

What is Pigsty’s ecosystem position and influence?

Pigsty is the most influential Chinese open-source project in the global PostgreSQL ecosystem, with about 100,000 users, half from overseas. Pigsty is also one of the most active open-source projects in the PostgreSQL ecosystem, currently dominating in extension distribution and monitoring systems.

PGEXT.Cloud is a PostgreSQL extension repository maintained by Pigsty, with the world’s largest PostgreSQL extension distribution volume. It has become an upstream software supply chain for multiple international PostgreSQL vendors.

Pigsty is currently one of the major distributions in the PostgreSQL ecosystem and a challenger to cloud vendor RDS, now widely used in defense, government, healthcare, internet, finance, manufacturing, and other industries.

What scale of customers is Pigsty suitable for?

Pigsty originated from the need for ultra-large-scale PostgreSQL automated management but has been deeply optimized for ease of use. Individual developers and small-medium enterprises lacking professional DBA capabilities can also easily get started.

The largest deployment is 25K vCPU, 4.5 million QPS, 6+ years; the smallest deployment can run completely on a 1c1g VM for Demo / Devbox use.

What capabilities does Pigsty provide?

Pigsty focuses on integrating the PostgreSQL ecosystem and providing PostgreSQL best practices, but also supports a series of open-source software that works well with PostgreSQL. For example:

  • Etcd, Redis, Silo, DuckDB, Prometheus
  • FerretDB, Babelfish, IvorySQL, PolarDB, OrioleDB
  • OpenHalo, Supabase, Greenplum, Dify, Odoo, …

What scenarios is Pigsty suitable for?

  • Running large-scale PostgreSQL clusters for business
  • Self-building RDS, object storage, cache, data warehouse, Supabase, …
  • Self-building enterprise applications like Odoo, Dify, Wiki, GitLab
  • Running monitoring infrastructure, monitoring existing databases and hosts
  • Using multiple PG extensions in combination
  • Dashboard development and interactive data application demos, data visualization, web building

Is Pigsty open source and free?

Pigsty is 100% open-source software + free software. Under the premise of complying with the open-source license, you can use it freely and for various commercial purposes.

We value software freedom. Pigsty uses the Apache-2.0 license. Please see the license for details.

Does Pigsty provide commercial support?

Pigsty software itself is open-source and free, and provides commercial subscriptions for all budgets, providing quality assurance for Pigsty & PostgreSQL. Subscriptions provide broader OS/PG/chip architecture support ranges, as well as expert consulting and support. Pigsty commercial subscriptions deliver industry-leading management/technical experience/solutions, helping you save valuable time, shouldering risks for you, and providing a safety net for difficult problems.

Does Pigsty support domestic innovation (信创)?

Pigsty software itself is not a database and is not subject to domestic innovation catalog restrictions, and already has multiple military use cases. However, the Pigsty open-source edition does not provide any form of domestic innovation support. Commercial subscription provides domestic innovation solutions in cooperation with Alibaba Cloud, supporting the use of PolarDB-O with domestic innovation qualifications (requires separate purchase) as the RDS kernel, capable of running on domestic innovation OS/chip environments.

Can Pigsty run as a multi-tenant DBaaS?

Pigsty uses the Apache-2.0 license. You may use it for DBaaS purposes under the license terms. For explicit commercial authorization, consider the Pigsty Enterprise subscription.

Can Pigsty’s Logo be rebranded as your own product?

When redistributing Pigsty, you must retain copyright notices, patent notices, trademark notices, and attribution notices from the original work, and attach prominent change descriptions in modified files while preserving the content of the LICENSE file. Under these premises, you can replace PIGSTY’s Logo and trademark, but you must not promote it as “your own original work.” We provide commercial licensing support for OEM and rebranding in the enterprise edition.

Pigsty’s Business Entity

Pigsty is a project invested by Miracle Plus S22. The original entity Panji Cloud Data (Beijing) Technology Co., Ltd. has been liquidated and divested of the Pigsty business.

Pigsty is currently independently operated and maintained by author Feng Ruohang. The business entities are:

  • Hainan Zhuxia Cloud Data Co., Ltd. / 91460000MAE6L87B94
  • Haikou Longhua Piji Data Center / 92460000MAG0XJ569B
  • Haikou Longhua Yuehang Technology Center / 92460000MACCYGBQ1N

PIGSTY® and PGSTY® are registered trademarks of Haikou Longhua Yuehang Technology Center.

4.12 - Comparison

This article compares Pigsty with similar products and projects, highlighting feature differences.

Comparison with RDS

Pigsty is a local-first RDS alternative released under Apache-2.0, deployable on your own physical/virtual machines or cloud servers.

We’ve chosen Amazon AWS RDS for PostgreSQL (the global market leader) and Alibaba Cloud RDS for PostgreSQL (China’s market leader) as benchmarks for comparison.

Both Aliyun RDS and AWS RDS are closed-source cloud database services, available only through rental models on public clouds. The following cloud-vendor information is a February 2024 archive based on PostgreSQL 16 at that time. The Pigsty column in the Feature Comparison table is maintained against the current release, while the later Key Extensions version table remains a period snapshot.

Feature Comparison

Feature Pigsty Aliyun RDS AWS RDS
Major Version Support 14 - 18 13 - 18 13 - 18
Read Replicas Supports unlimited read replicas Standby instances not exposed to users Standby instances not exposed to users
Read/Write Splitting Port-based traffic separation Separate paid component Separate paid component
Fast/Slow Separation Supports offline ETL instances Not available Not available
Cross-Region DR Supports standby clusters Multi-AZ deployment supported Multi-AZ deployment supported
Delayed Replicas Supports delayed instances Not available Not available
Load Balancing HAProxy / LVS Separate paid component Separate paid component
Connection Pool Pgbouncer Separate paid component: RDS Separate paid component: RDS Proxy
High Availability Patroni / etcd Requires HA edition Requires HA edition
Point-in-Time Recovery pgBackRest / Silo Backup supported Backup supported
Metrics Monitoring VictoriaMetrics / Exporter Free basic / Paid advanced Free basic / Paid advanced
Log Collection VictoriaLogs / Vector Basic support Basic support
Visualization Grafana / Echarts Basic monitoring Basic monitoring
Alert Aggregation AlertManager Basic support Basic support

Key Extensions

This is a historical PostgreSQL 16 extension-support snapshot based on information visible on 2024-02-28. Its versions and projects—including pg_analytics, which was later archived and removed from the catalog—are not the current Pigsty v4.5.0 or cloud-provider support matrix. Use the extension catalog for current Pigsty coverage and recheck each provider’s documentation for its current service capabilities.

Extension Pigsty RDS / PGDG Official Repo Aliyun RDS AWS RDS
Install Extensions Free to install Not allowed Not allowed
Geospatial PostGIS 3.4.2 PostGIS 3.3.4 / Ganos 6.1 PostGIS 3.4.1
Point Cloud PG PointCloud 1.2.5 Ganos PointCloud 6.1
Vector Embedding PGVector 0.6.1 / Svector 0.5.6 pase 0.0.1 PGVector 0.6
Machine Learning PostgresML 2.8.1
Time Series TimescaleDB 2.14.2
Horizontal Scaling Citus 12.1
Columnar Storage Hydra 1.1.1
Full Text Search pg_bm25 0.5.6
Graph Database Apache AGE 1.5.0
GraphQL PG GraphQL 1.5.0
OLAP pg_analytics 0.5.6
Message Queue pgq 3.5.0
DuckDB duckdb_fdw 1.1
Fuzzy Tokenization zhparser 1.1 / pg_bigm 1.2 zhparser 1.0 / pg_jieba pg_bigm 1.2
CDC Extraction wal2json 2.5.3 wal2json 2.5
Bloat Management pg_repack 1.5.0 pg_repack 1.4.8 pg_repack 1.5.0
AWS RDS PG Available Extensions

AWS RDS for PostgreSQL 16 available extensions (excluding PG built-in extensions)

name pg16 pg15 pg14 pg13 pg12 pg11 pg10
amcheck 1.3 1.3 1.3 1.2 1.2 yes 1
auto_explain yes yes yes yes yes yes yes
autoinc 1 1 1 1 null null null
bloom 1 1 1 1 1 1 1
bool_plperl 1 1 1 1 null null null
btree_gin 1.3 1.3 1.3 1.3 1.3 1.3 1.2
btree_gist 1.7 1.7 1.6 1.5 1.5 1.5 1.5
citext 1.6 1.6 1.6 1.6 1.6 1.5 1.4
cube 1.5 1.5 1.5 1.4 1.4 1.4 1.2
dblink 1.2 1.2 1.2 1.2 1.2 1.2 1.2
dict_int 1 1 1 1 1 1 1
dict_xsyn 1 1 1 1 1 1 1
earthdistance 1.1 1.1 1.1 1.1 1.1 1.1 1.1
fuzzystrmatch 1.2 1.1 1.1 1.1 1.1 1.1 1.1
hstore 1.8 1.8 1.8 1.7 1.6 1.5 1.4
hstore_plperl 1 1 1 1 1 1 1
insert_username 1 1 1 1 null null null
intagg 1.1 1.1 1.1 1.1 1.1 1.1 1.1
intarray 1.5 1.5 1.5 1.3 1.2 1.2 1.2
isn 1.2 1.2 1.2 1.2 1.2 1.2 1.1
jsonb_plperl 1 1 1 1 1 null null
lo 1.1 1.1 1.1 1.1 1.1 1.1 1.1
ltree 1.2 1.2 1.2 1.2 1.1 1.1 1.1
moddatetime 1 1 1 1 null null null
old_snapshot 1 1 1 null null null null
pageinspect 1.12 1.11 1.9 1.8 1.7 1.7 1.6
pg_buffercache 1.4 1.3 1.3 1.3 1.3 1.3 1.3
pg_freespacemap 1.2 1.2 1.2 1.2 1.2 1.2 1.2
pg_prewarm 1.2 1.2 1.2 1.2 1.2 1.2 1.1
pg_stat_statements 1.1 1.1 1.9 1.8 1.7 1.6 1.6
pg_trgm 1.6 1.6 1.6 1.5 1.4 1.4 1.3
pg_visibility 1.2 1.2 1.2 1.2 1.2 1.2 1.2
pg_walinspect 1.1 1 null null null null null
pgcrypto 1.3 1.3 1.3 1.3 1.3 1.3 1.3
pgrowlocks 1.2 1.2 1.2 1.2 1.2 1.2 1.2
pgstattuple 1.5 1.5 1.5 1.5 1.5 1.5 1.5
plperl 1 1 1 1 1 1 1
plpgsql 1 1 1 1 1 1 1
pltcl 1 1 1 1 1 1 1
postgres_fdw 1.1 1.1 1.1 1 1 1 1
refint 1 1 1 1 null null null
seg 1.4 1.4 1.4 1.3 1.3 1.3 1.1
sslinfo 1.2 1.2 1.2 1.2 1.2 1.2 1.2
tablefunc 1 1 1 1 1 1 1
tcn 1 1 1 1 1 1 1
tsm_system_rows 1 1 1 1 1 1 1.1
tsm_system_time 1 1 1 1 1 1 1.1
unaccent 1.1 1.1 1.1 1.1 1.1 1.1 1.1
uuid-ossp 1.1 1.1 1.1 1.1 1.1 1.1 1.1
Aliyun RDS PG Available Extensions

Aliyun RDS for PostgreSQL 16 available extensions (excluding PG built-in extensions)

name pg16 pg15 pg14 pg13 pg12 pg11 pg10 description
bloom 1 1 1 1 1 1 1 Provides a bloom filter-based index access method.
btree_gin 1.3 1.3 1.3 1.3 1.3 1.3 1.2 Provides GIN operator class examples that implement B-tree equivalent behavior for multiple data types and all enum types.
btree_gist 1.7 1.7 1.6 1.5 1.5 1.5 1.5 Provides GiST operator class examples that implement B-tree equivalent behavior for multiple data types and all enum types.
citext 1.6 1.6 1.6 1.6 1.6 1.5 1.4 Provides a case-insensitive string type.
cube 1.5 1.5 1.5 1.4 1.4 1.4 1.2 Provides a data type for representing multi-dimensional cubes.
dblink 1.2 1.2 1.2 1.2 1.2 1.2 1.2 Cross-database table operations.
dict_int 1 1 1 1 1 1 1 Additional full-text search dictionary template example.
earthdistance 1.1 1.1 1.1 1.1 1.1 1.1 1.1 Provides two different methods to calculate great circle distances on the Earth’s surface.
fuzzystrmatch 1.2 1.1 1.1 1.1 1.1 1.1 1.1 Determines similarities and distances between strings.
hstore 1.8 1.8 1.8 1.7 1.6 1.5 1.4 Stores key-value pairs in a single PostgreSQL value.
intagg 1.1 1.1 1.1 1.1 1.1 1.1 1.1 Provides an integer aggregator and an enumerator.
intarray 1.5 1.5 1.5 1.3 1.2 1.2 1.2 Provides some useful functions and operators for manipulating null-free integer arrays.
isn 1.2 1.2 1.2 1.2 1.2 1.2 1.1 Validates input according to a hard-coded prefix list, also used for concatenating numbers during output.
ltree 1.2 1.2 1.2 1.2 1.1 1.1 1.1 For representing labels of data stored in a hierarchical tree structure.
pg_buffercache 1.4 1.3 1.3 1.3 1.3 1.3 1.3 Provides a way to examine the shared buffer cache in real time.
pg_freespacemap 1.2 1.2 1.2 1.2 1.2 1.2 1.2 Examines the free space map (FSM).
pg_prewarm 1.2 1.2 1.2 1.2 1.2 1.2 1.1 Provides a convenient way to load data into the OS buffer or PostgreSQL buffer.
pg_stat_statements 1.1 1.1 1.9 1.8 1.7 1.6 1.6 Provides a means of tracking execution statistics of all SQL statements executed by a server.
pg_trgm 1.6 1.6 1.6 1.5 1.4 1.4 1.3 Provides functions and operators for alphanumeric text similarity, and index operator classes that support fast searching of similar strings.
pgcrypto 1.3 1.3 1.3 1.3 1.3 1.3 1.3 Provides cryptographic functions for PostgreSQL.
pgrowlocks 1.2 1.2 1.2 1.2 1.2 1.2 1.2 Provides a function to show row locking information for a specified table.
pgstattuple 1.5 1.5 1.5 1.5 1.5 1.5 1.5 Provides multiple functions to obtain tuple-level statistics.
plperl 1 1 1 1 1 1 1 Provides Perl procedural language.
plpgsql 1 1 1 1 1 1 1 Provides SQL procedural language.
pltcl 1 1 1 1 1 1 1 Provides Tcl procedural language.
postgres_fdw 1.1 1.1 1.1 1 1 1 1 Cross-database table operations.
sslinfo 1.2 1.2 1.2 1.2 1.2 1.2 1.2 Provides information about the SSL certificate provided by the current client.
tablefunc 1 1 1 1 1 1 1 Contains multiple table-returning functions.
tsm_system_rows 1 1 1 1 1 1 1 Provides the table sampling method SYSTEM_ROWS.
tsm_system_time 1 1 1 1 1 1 1 Provides the table sampling method SYSTEM_TIME.
unaccent 1.1 1.1 1.1 1.1 1.1 1.1 1.1 A text search dictionary that can remove accents (diacritics) from lexemes.
uuid-ossp 1.1 1.1 1.1 1.1 1.1 1.1 1.1 Provides functions to generate universally unique identifiers (UUIDs) using several standard algorithms.
xml2 1.1 1.1 1.1 1.1 1.1 1.1 1.1 Provides XPath queries and XSLT functionality.

Performance Comparison

Metric Pigsty Aliyun RDS AWS RDS
Peak Performance PGTPC on NVME SSD Benchmark sysbench oltp_rw RDS PG Performance Whitepaper sysbench oltp scenario QPS 4000 ~ 8000 per core
Storage Spec: Max Capacity 32TB / NVME SSD 32 TB / ESSD PL3 64 TB / io2 EBS Block Express
Storage Spec: Max IOPS 4K Random Read: Max 3M, Random Write 2000~350K 4K Random Read: Max 1M 16K Random IOPS: 256K
Storage Spec: Max Latency 4K Random Read: 75µs, Random Write: 15µs 4K Random Read: 200µs 500µs / Inferred as 16K random IO
Storage Spec: Max Reliability UBER < 1e-18, equivalent to 18 nines MTBF: 2M hours 5DWPD, 3 years continuous Reliability 9 nines, equivalent to UBER 1e-9 Storage and Data Reliability Durability: 99.999%, 5 nines (0.001% annual failure rate) io2 specification
Storage Spec: Max Cost ¥31.5/TB·month (5-year warranty amortized / 3.2T / Enterprise-grade / MLC) ¥3200/TB·month (original ¥6400, monthly ¥4000) 50% off with 3-year prepaid ¥1900/TB·month using max spec 65536GB / 256K IOPS best discount

Observability

Pigsty provides nearly 3000 monitoring metrics and 50+ monitoring dashboards, covering database monitoring, host monitoring, connection pool monitoring, load balancer monitoring, and more, providing users with an unparalleled observability experience.

Pigsty monitoring dashboard

Pigsty provides 638 PostgreSQL-related monitoring metrics, while AWS RDS only has 99, and Aliyun RDS has only single-digit metrics:

Alibaba Cloud RDS for PostgreSQL metrics

Additionally, some projects provide PostgreSQL monitoring capabilities, but are relatively simple:

Maintainability

Metric Pigsty Aliyun RDS AWS RDS
System Usability Simple Simple Simple
Configuration Management Config files / CMDB based on Ansible Inventory Can use Terraform Can use Terraform
Change Method Idempotent Playbooks based on Ansible Playbook Console click operations Console click operations
Parameter Tuning Auto-adapts to node specs, Four preset templates: OLTP, OLAP, TINY, CRIT
Infra as Code Natively supported Can use Terraform Can use Terraform
Customizable Parameters Pigsty Parameters 283 parameters
Service & Support Commercial subscription support available After-sales ticket support After-sales ticket support
Air-gapped Deployment Offline installation supported N/A N/A
Database Migration Playbooks for zero-downtime migration from existing v10+ PG instances to Pigsty managed instances via logical replication Cloud migration assistance Aliyun RDS Data Sync

Cost

Based on experience, RDS unit cost is 5-15 times that of self-hosted for software and hardware resources, with a rent-to-own ratio typically around one month. For details, see Cost Analysis.

Factor Metric Pigsty Aliyun RDS AWS RDS
Cost Software License/Service Fee Free, hardware ~¥20-40/core·month ¥200-400/core·month ¥400-1300/core·month
Support Service Fee Service ~¥100/core·month Included in RDS cost

Other On-Premises Database Management Software

Some software and vendors providing PostgreSQL management capabilities:

  • Aiven: Closed-source commercial cloud-hosted solution
  • Percona: Commercial consulting, simple PG distribution
  • ClusterControl: Commercial database management software

Other Kubernetes Operators

Pigsty refuses to use Kubernetes for managing databases in production, so there are ecological differences with these solutions.

  • PGO
  • StackGres
  • CloudNativePG
  • TemboOperator
  • PostgresOperator
  • PerconaOperator
  • Kubegres
  • KubeDB
  • KubeBlocks

For more information, see:

4.12.1 - Cost Reference

This article provides cost data to help you evaluate self-hosted Pigsty, cloud RDS costs, and typical DBA salaries.

Overview

The cost data below is intended to illustrate order-of-magnitude differences. Cloud vendor pricing and discounts vary over time, region, instance size, and purchase model.

EC2 Core·Month RDS Core·Month
DHH Self-Hosted Core-Month Price (192C 384G) 25.32 Junior Open Source DB DBA Reference Salary ¥15K/person·month
IDC Self-Hosted (Dedicated Physical: 64C384G) 19.53 Mid-Level Open Source DB DBA Reference Salary ¥30K/person·month
IDC Self-Hosted (Container, 500% Oversold) 7 Senior Open Source DB DBA Reference Salary ¥60K/person·month
UCloud Elastic VM (8C16G, Oversold) 25 ORACLE Database License 10000
Aliyun ECS 2x Memory (Dedicated, No Oversold) 107 Aliyun RDS PG 2x Memory (Dedicated) 260
Aliyun ECS 4x Memory (Dedicated, No Oversold) 138 Aliyun RDS PG 4x Memory (Dedicated) 320
Aliyun ECS 8x Memory (Dedicated, No Oversold) 180 Aliyun RDS PG 8x Memory (Dedicated) 410
AWS C5D.METAL 96C 200G (Monthly No Prepaid) 100 AWS RDS PostgreSQL db.T2 (2x) 440
AWS C5D.METAL 96C 200G (3-Year Prepaid) 80 AWS RDS PostgreSQL db.M5 (4x) 611
AWS C7A.METAL 192C 384G (3-Year Prepaid) 104.8 AWS RDS PostgreSQL db.R6G (8x) 786

RDS Cost Reference

Payment Model Price Annualized (¥10K)
IDC Self-Hosted (Single Physical Machine) ¥75K / 5 years 1.5
IDC Self-Hosted (2-3 Machines for HA) ¥150K / 5 years 3.0 ~ 4.5
Aliyun RDS On-Demand ¥87.36/hour 76.5
Aliyun RDS Monthly (Baseline) ¥42K / month 50
Aliyun RDS Annual (85% off) ¥425,095 / year 42.5
Aliyun RDS 3-Year Prepaid (50% off) ¥750,168 / 3 years 25
AWS On-Demand $25,817 / month 217
AWS 1-Year No Prepaid $22,827 / month 191.7
AWS 3-Year Full Prepaid $120K + $17.5K/month 175
AWS China/Ningxia On-Demand ¥197,489 / month 237
AWS China/Ningxia 1-Year No Prepaid ¥143,176 / month 171
AWS China/Ningxia 3-Year Full Prepaid ¥647K + ¥116K/month 160.6

Here’s a comparison of self-hosted vs cloud database costs:

Method Annualized (¥10K)
IDC Hosted Server 64C / 384G / 3.2TB NVME SSD 660K IOPS (2-3 Machines) 3.0 ~ 4.5
Aliyun RDS PG HA Edition pg.x4m.8xlarge.2c, 64C / 256GB / 3.2TB ESSD PL3 25 ~ 50
AWS RDS PG HA Edition db.m5.16xlarge, 64C / 256GB / 3.2TB io1 x 80k IOPS 160 ~ 217

ECS Cost Reference

Pure Compute Price Comparison (Excluding NVMe SSD / ESSD PL3)

Using Aliyun as an example, the monthly pure compute price is 5-7x the self-hosted baseline, while 5-year prepaid is 2x self-hosted

Payment Model Unit Price (¥/Core·Month) Relative to Standard Self-Hosted Premium Multiple
On-Demand (1.5x) ¥ 202 160 % 9.2 ~ 11.2
Monthly (Standard) ¥ 126 100 % 5.7 ~ 7.0
1-Year Prepaid (65% off) ¥ 83.7 66 % 3.8 ~ 4.7
2-Year Prepaid (55% off) ¥ 70.6 56 % 3.2 ~ 3.9
3-Year Prepaid (44% off) ¥ 55.1 44 % 2.5 ~ 3.1
4-Year Prepaid (35% off) ¥ 45 35 % 2.0 ~ 2.5
5-Year Prepaid (30% off) ¥ 38.5 30 % 1.8 ~ 2.1
DHH @ 2023 ¥ 22.0
Tantan IDC Self-Hosted ¥ 18.0

Equivalent Price Comparison Including NVMe SSD / ESSD PL3

Including common NVMe SSD specs, the monthly pure compute price is 11-14x the self-hosted baseline, while 5-year prepaid is about 9x.

Payment Model Unit Price (¥/Core·Month) + 40GB ESSD PL3 Self-Hosted Premium Multiple
On-Demand (1.5x) ¥ 202 ¥ 362 14.3 ~ 18.6
Monthly (Standard) ¥ 126 ¥ 286 11.3 ~ 14.7
1-Year Prepaid (65% off) ¥ 83.7 ¥ 244 9.6 ~ 12.5
2-Year Prepaid (55% off) ¥ 70.6 ¥ 230 9.1 ~ 11.8
3-Year Prepaid (44% off) ¥ 55.1 ¥ 215 8.5 ~ 11.0
4-Year Prepaid (35% off) ¥ 45 ¥ 205 8.1 ~ 10.5
5-Year Prepaid (30% off) ¥ 38.5 ¥ 199 7.9 ~ 10.2
DHH @ 2023 ¥ 25.3
Tantan IDC Self-Hosted ¥ 19.5

DHH Case: 192 cores with 12.8TB Gen4 SSD (1c:66); Tantan Case: 64 cores with 3.2T Gen3 MLC SSD (1c:50).

Cloud prices calculated at 40GB ESSD PL3 per core (1 core:4x RAM:40x disk).

EBS Cost Reference

Evaluation Factor Local PCI-E NVME SSD Aliyun ESSD PL3 AWS io2 Block Express
Capacity 32TB 32 TB 64 TB
IOPS 4K Random Read: 600K ~ 1.1M, 4K Random Write: 200K ~ 350K 4K Random Read: Max 1M 16K Random IOPS: 256K
Latency 4K Random Read: 75µs, 4K Random Write: 15µs 4K Random Read: 200µs Random IO: ~500µs (contextually inferred as 16K)
Reliability UBER < 1e-18, equivalent to 18 nines, MTBF: 2M hours, 5DWPD for 3 years Data Reliability 9 nines Storage and Data Reliability Durability: 99.999%, 5 nines (0.001% annual failure rate) io2 Specification
Cost ¥16/TB·month (5-year amortized / 3.2T MLC), 5-year warranty, ¥3000 retail ¥3200/TB·month (original ¥6400, monthly ¥4000), 50% off with 3-year full prepaid ¥1900/TB·month using max spec 65536GB 256K IOPS best discount
SLA 5-year warranty, replacement on failure Aliyun RDS SLA Availability 99.99%: 15% monthly fee, 99%: 30% monthly fee, 95%: 100% monthly fee Amazon RDS SLA Availability 99.95%: 15% monthly fee, 99%: 25% monthly fee, 95%: 100% monthly fee

S3 Cost Reference

Date $/GB·Month ¥/TB·5Years HDD ¥/TB SSD ¥/TB
2006.03 0.150 63000 2800
2010.11 0.140 58800 1680
2012.12 0.095 39900 420 15400
2014.04 0.030 12600 371 9051
2016.12 0.023 9660 245 3766
2023.12 0.023 9660 105 280
Other References High-Perf Storage Top-Tier Discounted vs Purchased NVMe SSD Price Ref
S3 Express 0.160 67200 DHH 12T 1400
EBS io2 0.125 + IOPS 114000 Shannon 3.2T 900

Cloud Exit Collection

There was a time when “moving to the cloud” was almost politically correct in tech circles, and an entire generation of app developers had their vision obscured by the cloud. Let’s use real data analysis and firsthand experience to explain the value and pitfalls of the public cloud rental model — for your reference in this era of cost reduction and efficiency improvement — please see “Cloud Computing Mudslide: Collection”

Cloud Infrastructure Basics

Cloud Business Model

Cloud Exit Odyssey

Cloud Failure Post-Mortems

RDS Failures

Cloud Vendor Profiles

4.12.2 - Open-Source Impact

Impact comparison of PostgreSQL ecosystem projects, mainly measured by GitHub star counts.

China PostgreSQL Ecosystem Projects

Sorted by GitHub stars in descending order. Last updated: 2026-08-13 (Beijing time).

Project Star Author Type Summary
pgsty/pigsty 5521 Ruohang Feng @ PGSTY Distribution Out-of-the-box PostgreSQL distribution
polardb/PolarDB-for-PostgreSQL 3191 Alibaba Cloud Kernel Open-source PolarDB for PostgreSQL kernel
tensorchord/pgvecto.rs 2181 TensorChord Extension Vector search extension written in Rust
tensorchord/VectorChord 1770 TensorChord Extension Next-generation vector search extension
Tencent/TBase 1439 Tencent Cloud Kernel Tencent distributed HTAP database kernel
apache/cloudberry 1315 HashData Kernel Open-source MPP data warehouse kernel
IvorySQL/IvorySQL 1051 HighGo Kernel Oracle-compatible PostgreSQL fork
pgplex/pgschema 995 Chen Tianzhou Tool Declarative Postgres schema migration CLI
amutu/zhparser 869 Jov Extension Chinese full-text parser based on SCWS
opengauss-mirror/openGauss-server 784 Huawei Kernel Early PostgreSQL 9.2 kernel fork
HaloTech-Co-Ltd/openHalo 437 HaloTech Kernel PostgreSQL kernel compatible with MySQL wire protocol
jaiminpan/pg_jieba 417 Pan Jiamin Extension Chinese full-text search extension based on Jieba
alitrack/duckdb_fdw 409 Li Hongyan Extension DuckDB foreign data wrapper
tensorchord/VectorChord-bm25 375 TensorChord Extension Native BM25 ranking index for PostgreSQL
pgsty/pg_exporter 359 Ruohang Feng @ PGSTY Tool Metrics exporter for PostgreSQL and Pgbouncer
ChenHuajun/pg_roaringbitmap 286 Chen Huajun @ Suning Extension PostgreSQL RoaringBitmap bitmap extension
pgsty/pig 199 Ruohang Feng @ PGSTY Tool PostgreSQL extension package manager
tensorchord/pg_bestmatch.rs 101 TensorChord Extension BM25 sparse-vector generation in PostgreSQL
wublabdubdub/PDU-PostgreSQLDataUnloader 101 Zhang Chen Tool PostgreSQL database rescue and unloading tool
tensorchord/pg_tokenizer.rs 45 TensorChord Extension Full-text search tokenizer extension
jaiminpan/pg_scws 41 Pan Jiamin Extension Chinese tokenizer extension based on SCWS
pgsty/pgext 31 Ruohang Feng @ PGSTY Tool PostgreSQL extension catalog and metadata tool 
tooltip:
  trigger: axis
  axisPointer: { type: shadow }
  formatter: $fn:tipfmt
grid: { left: 320, right: 72, top: 20, bottom: 26 }
xAxis:
  type: value
  max: 5600
  name: GitHub Stars
  nameLocation: middle
  nameGap: 24
  axisLabel: { formatter: $fn:fnum }
  splitLine: { show: true, lineStyle: { type: dashed, opacity: 0.45 } }
yAxis:
  type: category
  inverse: true
  axisLabel:
    align: right
    margin: 8
    width: 300
    overflow: truncate
    fontSize: 11
    fontFamily: monospace
  data:
    - 'pgsty/pigsty'
    - 'polardb/PolarDB-for-PostgreSQL'
    - 'tensorchord/pgvecto.rs'
    - 'tensorchord/VectorChord'
    - 'Tencent/TBase'
    - 'apache/cloudberry'
    - 'IvorySQL/IvorySQL'
    - 'pgplex/pgschema'
    - 'amutu/zhparser'
    - 'opengauss-mirror/openGauss-server'
    - 'HaloTech-Co-Ltd/openHalo'
    - 'jaiminpan/pg_jieba'
    - 'alitrack/duckdb_fdw'
    - 'tensorchord/VectorChord-bm25'
    - 'pgsty/pg_exporter'
    - 'ChenHuajun/pg_roaringbitmap'
    - 'pgsty/pig'
    - 'tensorchord/pg_bestmatch.rs'
    - 'wublabdubdub/PDU-PostgreSQLDataUnloader'
    - 'tensorchord/pg_tokenizer.rs'
    - 'jaiminpan/pg_scws'
    - 'pgsty/pgext'
series:
  - name: Star
    type: bar
    barWidth: 20
    showBackground: true
    backgroundStyle: { color: "rgba(148, 163, 184, 0.16)" }
    itemStyle:
      color: $fn:barclr
      borderRadius: [0, 5, 5, 0]
    label:
      show: true
      position: right
      formatter: $fn:labfmt
      color: '#334155'
      fontWeight: 600
    data: [5521, 3191, 2181, 1770, 1439, 1315, 1051, 995, 869, 784, 437, 417, 409, 375, 359, 286, 199, 101, 101, 45, 41, 31]

PostgreSQL Distribution Impact Metrics

Sorted by GitHub stars in descending order, with commercial products that do not publish stars listed last. Last updated: 2026-08-13 (Beijing time).

Project Star Vendor Type License Summary
CloudNativePG 9133 EDB K8S Native Apache-2.0 Mainstream PG Operator without Patroni dependency
Pigsty 5521 PGSTY Linux Native Apache-2.0 Ansible-driven integrated PostgreSQL distribution
Zalando Postgres Operator 5222 Zalando K8S Native MIT Long-standing Patroni/Spilo architecture operator
PGO 4436 Crunchy Data K8S Native Apache-2.0 Production-grade operator with backup and monitoring
Autobase 4332 vitabaks Linux Native MIT Automated deployment for Patroni/etcd/Consul
KubeBlocks 3102 ApeCloud K8S Native AGPL-3.0 Unified multi-database operator platform
StackGres 1426 OnGres K8S Native AGPL-3.0 Integrated PG operator with CRD/CLI/Web UI
Kubegres 1350 Reactive Tech K8S Native Apache-2.0 Minimal operator built on native streaming replication
Tembo Operator 1263 Tembo K8S Native Unspecified Scenario-based stacks for PostgreSQL
pgEdge 744 pgEdge Linux Native PostgreSQL Distributed PG distribution focused on Spock multi-master replication
KubeDB 733 AppsCode K8S Native ACL-1.0 Multi-database operator with kubectl plugin
Percona Operator for PostgreSQL 381 Percona K8S Native Apache-2.0 PostgreSQL operator in Percona ecosystem
EDB TPA 86 EDB Linux Native GPL-3.0 EDB official Ansible delivery toolkit
Percona Distribution for PostgreSQL - Percona Linux Native Multi Integrated PostgreSQL distribution bundle
ClusterControl - ServerNines Linux Native Commercial Multi-database deploy, monitoring, backup, and failover platform
CYBERTEC PGEE - CYBERTEC Linux Native Commercial Enterprise PostgreSQL distribution focused on security and performance
Crunchy Postgres for Ansible - Crunchy Data Linux Native Commercial Crunchy bare-metal/VM automation solution
EDB Postgres Advanced Server (EPAS) - EDB Linux Native Commercial EDB flagship distribution with Oracle-compatibility features

Star History Chart

Other Resources

5 - References

Detailed reference information and lists, supported Linux distros, available modules, metrics, extensions, and more.

5.1 - Supported Linux

Pigsty compatible Linux OS distribution major versions and CPU architectures

Pigsty runs on Linux, supporting amd64/x86_64 and arm64/aarch64 arch, plus 3 major distros: EL, Debian, Ubuntu.

Pigsty runs bare-metal without containers. Supports actively maintained mainstream releases across the 3 major distro families and both archs.

Overview

Recommended OS versions: Rocky Linux 9.8 / 10.2, Debian 12.15 / 13.6, Ubuntu 22.04.5 / 24.04.4 / 26.04.0.

Distro Arch OS Code PG18 PG17 PG16 PG15 PG14
RHEL / Rocky / Alma 10 x86_64 el10.x86_64
RHEL / Rocky / Alma 10 aarch64 el10.aarch64
RHEL / Rocky / Alma 9 x86_64 el9.x86_64
RHEL / Rocky / Alma 9 aarch64 el9.aarch64
Ubuntu 26.04 (resolute) x86_64 u26.x86_64
Ubuntu 26.04 (resolute) aarch64 u26.aarch64
Ubuntu 24.04 (noble) x86_64 u24.x86_64
Ubuntu 24.04 (noble) aarch64 u24.aarch64
Ubuntu 22.04 (jammy) x86_64 u22.x86_64
Ubuntu 22.04 (jammy) aarch64 u22.aarch64
Debian 13 (trixie) x86_64 d13.x86_64
Debian 13 (trixie) aarch64 d13.aarch64
Debian 12 (bookworm) x86_64 d12.x86_64
Debian 12 (bookworm) aarch64 d12.aarch64

These seven minor releases are the current validation baselines. The extension repository retains dual-architecture EL8 compatibility, so the complete package matrix covers 16 Linux platforms. EL8 is in its retirement transition and is no longer a recommended deployment baseline.

EL

Pigsty supports RHEL / Rocky / Alma / Anolis / CentOS 8, 9, 10.

EL Distro Arch OS Code PG18 PG17 PG16 PG15 PG14
RHEL10 / Rocky10 / Alma10 x86_64 el10.x86_64
RHEL10 / Rocky10 / Alma10 aarch64 el10.aarch64
RHEL9 / Rocky9 / Alma9 x86_64 el9.x86_64
RHEL9 / Rocky9 / Alma9 aarch64 el9.aarch64
RHEL8 / Rocky8 / Alma8 x86_64 el8.x86_64
RHEL8 / Rocky8 / Alma8 aarch64 el8.aarch64
RHEL7 / CentOS7 x86_64 el7.x86_64
RHEL7 / CentOS7 aarch64 -
Rocky Linux 9.8 / 10.2 Recommended

Rocky Linux 9.8 / 10.2 balances stability and fresh software. Recommended for EL users.

EL8 EOL Soon

EL8 goes EOL in 2029. Plan upgrade ASAP. EL10 support is ready, EL8 will be dropped in next release.

EL 7 EOL @ 2024-06

RHEL 7 EOL since Jun 2024. PGDG stopped providing binary packages for PG 16/17/18 on EL7.

For extended support on legacy OS, consider Enterprise Subscription.

Ubuntu

Pigsty supports Ubuntu 26.04 / 24.04 / 22.04:

Ubuntu Distro Arch OS Code PG18 PG17 PG16 PG15 PG14
Ubuntu 26.04 (resolute) x86_64 u26.x86_64
Ubuntu 26.04 (resolute) aarch64 u26.aarch64
Ubuntu 24.04 (noble) x86_64 u24.x86_64
Ubuntu 24.04 (noble) aarch64 u24.aarch64
Ubuntu 22.04 (jammy) x86_64 u22.x86_64
Ubuntu 22.04 (jammy) aarch64 u22.aarch64
Ubuntu 22.04.5 / 24.04.4 / 26.04.0 LTS Recommended

Ubuntu 26.04 provides the newest LTS baseline, while Ubuntu 24.04 remains the conservative default for Ubuntu users.

Debian

Pigsty supports Debian 12 / 13, latest Debian 13.6 recommended:

Debian Distro Arch OS Code PG18 PG17 PG16 PG15 PG14
Debian 13 (trixie) x86_64 d13.x86_64
Debian 13 (trixie) aarch64 d13.aarch64
Debian 12 (bookworm) x86_64 d12.x86_64
Debian 12 (bookworm) aarch64 d12.aarch64
Debian 11 (bullseye) x86_64 d11.x86_64 (historical)
Debian 11 (bullseye) aarch64 -
Debian 12.15 / 13.6 Recommended
Debian 11 EOL @ 2024-07

Debian 11 EOL since Jul 2024. For extended support on legacy OS, consider Enterprise Subscription.

Vagrant

For local VM deployment, use these Vagrant base images (same as used in Pigsty dev):

Terraform

For cloud deployment, use these Terraform base image prefixes (Aliyun example):

x86_64 Aliyun Image Prefix
Rocky 8.10 rockylinux_8_10_x64
Rocky 9.8 rockylinux_9_8_x64
Rocky 10.2 rockylinux_10_2_x64
Ubuntu 22.04.5 ubuntu_22_04_x64_20G
Ubuntu 24.04.4 ubuntu_24_04_x64_20G
Ubuntu 26.04.0 ubuntu_26_04_x64_20G
Debian 12.15 debian_12_15_x64
Debian 13.6 debian_13_6_x64
aarch64 Aliyun Image Prefix
Rocky 8.10 rockylinux_8_10_arm64
Rocky 9.8 rockylinux_9_8_arm64
Rocky 10.2 rockylinux_10_2_arm64
Ubuntu 22.04.5 ubuntu_22_04_arm64_20G
Ubuntu 24.04.4 ubuntu_24_04_arm64_20G
Ubuntu 26.04.0 ubuntu_26_04_arm64_20G
Debian 12.15 debian_12_15_arm64
Debian 13.6 debian_13_6_arm64

5.2 - Modules

This article lists available Pigsty modules and the current module planning.

Official Modules

Module Category Status Docs Path Summary
PGSQL Core GA /docs/pgsql High-availability PostgreSQL clusters with built-in backup, monitoring, SOP, and extension ecosystem.
INFRA Core GA /docs/infra Local software repository + VictoriaMetrics/Logs/Traces + Grafana infrastructure stack.
NODE Core GA /docs/node Node initialization and convergence: system tuning, admin, HAProxy, Vector, Keepalived, etc.
ETCD Core GA /docs/etcd DCS for PostgreSQL HA (service discovery, config, leader-election metadata).
MINIO Extension GA /docs/minio Deploys Silo S3-compatible object storage, suitable for PostgreSQL backups.
REDIS Extension GA /docs/redis Redis by default, or Valkey, in standalone, Sentinel, or native-cluster mode with monitoring.
DOCKER Extension GA /docs/docker Docker daemon and the runtime capability for containerized apps.
JUICE Extension BETA /docs/juice JuiceFS distributed file system using PostgreSQL as metadata engine.
VIBE Extension BETA /docs/vibe Browser-based dev environment with Code-Server, JupyterLab, Node.js, Claude Code, and Codex CLI.
KAFKA Extension BETA /docs/kafka Apache Kafka 4.x dynamic KRaft cluster deployment, security baseline, and monitoring.

Core Modules

Pigsty provides four core modules that are important for delivering complete highly available PostgreSQL services:

  • PGSQL: Self-healing PostgreSQL clusters with HA, PITR, IaC, SOP, monitoring, and 575 extensions.
  • INFRA: Local software repository, VictoriaMetrics, VictoriaLogs, VictoriaTraces, Grafana, Alertmanager, Blackbox Exporter…
  • NODE: Node convergence for hostname, timezone, NTP, SSH, sudo, HAProxy, Vector, and Keepalived.
  • ETCD: Distributed key-value store used as DCS for HA PostgreSQL clusters: consensus leader election/config management/service discovery.

Although these four modules are usually installed together, separate use is still feasible. In practice, only the NODE module is usually mandatory.

Extension Modules

Pigsty provides six extension modules. They are not mandatory for core functionality, but can enhance PostgreSQL capabilities:

  • MINIO: An S3-compatible object-storage module that deploys Silo and provides PostgreSQL backup integration and monitoring.
  • REDIS: Redis server with standalone/sentinel/cluster production deployment and full monitoring support.
  • DOCKER: Docker daemon service for one-click deployment of stateless software templates on Pigsty.
  • JUICE: JuiceFS distributed filesystem module using PostgreSQL as metadata engine, providing shared POSIX storage.
  • VIBE: Browser-based development environment with Code-Server, JupyterLab, Node.js, Claude Code, and Codex CLI.
  • KAFKA: Apache Kafka 4.x dynamic KRaft clusters with TLS/SCRAM/ACL security baseline, declarative topics/users, and full monitoring.

Ecosystem Modules

The modules below are closely related to the PostgreSQL ecosystem. They are optional ecosystem capabilities and are not counted in the 10 official modules above:

  • SUPABASE, DUCKDB: peripheral ecosystem integration.
  • MSSQL, IVORY, POLAR, CITUS, CLOUDBERRY, PGEDGE: kernel replacement, distributed, and MPP forms.
  • MYSQL-compatible kernel (OpenHalo), ORIOLE, PGTDE, AGENS: protocol compatibility, storage engine, transparent encryption, and graph database kernels. Here, MYSQL means the pg_mode=mysql PostgreSQL-compatible kernel, not a native MySQL service.
  • GREENPLUM, NEON: historical docs retained, no longer default public capabilities.
  • Native MYSQL pilot: the current mysql.yml, mysql-rm.yml, and roles/mysql* manage a fixed native MySQL 8.4 platform with either one node or a three-node single-primary InnoDB Cluster. It remains a PILOT and is not counted among the 10 official modules above.
  • KUBE, VICTORIA, JUPYTER: other pilot modules, currently not open for public use.

5.3 - File Hierarchy

How Pigsty’s file system structure is designed and organized, and directory structures used by each module.

Pigsty FHS

Pigsty’s home directory is located at ~/pigsty by default. The file structure within this directory is as follows:

~/pigsty Source Tree

  • app/
    • Application template resources
  • bin/
    • Management and operations scripts
  • files/
    • victoria/
      • Rules and operations scripts
    • grafana/
      • Grafana dashboards
    • postgres/
      • PostgreSQL management scripts
    • migration/
      • Data-migration task definitions
    • pki/
      • Self-signed CA and certificates
  • roles/
    • Ansible role implementations
  • templates/
    • Ansible templates
  • vagrant/
    • Vagrant sandbox definitions
  • terraform/
    • Terraform cloud-resource templates
  • configure
  • ansible.cfg
  • pigsty.yml
  • *.yml

/infra is a runtime symlink to /data/infra, which keeps observability data and generated configuration together:

/data/infra 
metrics/           # VictoriaMetrics TSDB data
logs/              # VictoriaLogs data
traces/             # VictoriaTraces data
alertmgr/           # AlertManager data
rules/              # Rule definitions, including agent.yml
targets/            # FileSD monitoring targets
dashboards/         # Grafana dashboard definitions
datasources/        # Grafana datasource definitions
prometheus.yml      # Victoria Prometheus-compatible configuration

CA FHS

Pigsty’s self-signed CA is located in files/pki/ under the Pigsty home directory.

You must keep the CA key file secure: files/pki/ca/ca.key. This key is generated by the ca role during deploy.yml or infra.yml execution.

# pigsty/files/pki                           # (local_user) 0755
#  ^-----@ca                                 # (local_user) 0700
#         ^[email protected]                      # 0600, CRITICAL: keep secret
#         ^[email protected]                      # 0644, CRITICAL: trust anchor
#  ^-----@csr                                # (local_user) 0755, CSRs
#  ^-----@misc                               # (local_user) 0755, misc/issued certs
#  ^-----@etcd                               # (local_user) 0755, ETCD certs
#  ^-----@minio                              # (local_user) 0755, MinIO certs
#  ^-----@nginx                              # (local_user) 0755, Nginx SSL certs
#  ^-----@infra                              # (local_user) 0755, infra client certs
#  ^-----@pgsql                              # (local_user) 0755, PostgreSQL certs
#  ^-----@kafka                              # (local_user) 0755, Kafka server certs
#  ^-----@mysql                              # (local_user) 0755, MySQL server certs

Nodes managed by Pigsty will have the following certificate files installed:

/etc/pki/ca.crt                             # root:root 0644, root cert on all nodes
/etc/pki/ca-trust/source/anchors/ca.crt     # EL system trust anchor
/usr/local/share/ca-certificates/ca.crt     # Debian/Ubuntu system trust anchor

All infra nodes will have the following certificates:

/etc/pki/infra.crt                          # root:infra 0644, infra node cert
/etc/pki/infra.key                          # root:infra 0640, infra node key

When your admin node fails, the files/pki directory and pigsty.yml file should be available on the backup admin node. You can use rsync to achieve this:

# run on meta-1, rsync to meta2
cd ~/pigsty;
rsync -avz ./ meta-2:~/pigsty

INFRA FHS

The infra role creates infra_data (default: /data/infra) and creates a symlink /infra -> /data/infra. /data/infra permissions are root:infra 0771; subdirectories default to *:infra 0750 unless overridden:

# /infra -> /data/infra
# /data/infra                              # root:infra 0771
#  ^-----@pgadmin                          # 5050:5050 0700
#  ^-----@alertmgr                         # prometheus:infra 0700
#  ^-----@conf                             # root:infra 0750
#            ^-----patronictl.yml          # root:admin 0640
#  ^-----@tmp                              # root:infra 0750
#  ^-----@hosts                            # dnsmasq:dnsmasq 0755 (DNS records)
#            ^-----default                 # root:root 0644
#  ^-----@datasources                      # root:infra 0750
#            ^-----*.json                  # 0600 (generated by register)
#  ^-----@dashboards                       # grafana:infra 0750
#  ^-----@metrics                          # victoria:infra 0750
#  ^-----@logs                             # victoria:infra 0750
#  ^-----@traces                           # victoria:infra 0750
#  ^-----@bin                              # victoria:infra 0750
#            ^-----check|new|reload|status # root:infra 0755
#  ^-----@rules                            # victoria:infra 0750
#            ^-----agent.yml               # victoria:infra 0644
#            ^-----infra.yml               # victoria:infra 0644
#            ^-----node.yml                # victoria:infra 0644
#            ^-----pgsql.yml               # victoria:infra 0644
#            ^-----redis.yml               # victoria:infra 0644
#            ^-----etcd.yml                # victoria:infra 0644
#            ^-----minio.yml               # victoria:infra 0644
#            ^-----kafka.yml               # victoria:infra 0644
#            ^-----mysql.yml               # victoria:infra 0644
#  ^-----@targets                          # victoria:infra 0750
#            ^-----@infra                  # infra targets (files 0640)
#            ^-----@node                   # node targets (files 0640)
#            ^-----@ping                   # ping targets (files 0640)
#            ^-----@etcd                   # etcd targets (files 0640)
#            ^-----@pgsql                  # pgsql targets (files 0640)
#            ^-----@pgrds                  # pgrds targets (files 0640)
#            ^-----@redis                  # redis targets (files 0640)
#            ^-----@minio                  # minio targets (files 0640)
#            ^-----@juice                  # juicefs targets (files 0640)
#            ^-----@mysql                  # mysql targets (files 0640)
#            ^-----@kafka                  # kafka targets (files 0640)
#            ^-----@docker                 # docker targets (files 0640)
#            ^-----@patroni                # patroni SSL targets (files 0640)
#  ^-----prometheus.yml                    # victoria:infra 0644

This structure is created by: roles/infra/tasks/dir.yml, roles/infra/tasks/victoria.yml, roles/infra/tasks/register.yml, roles/infra/tasks/dns.yml, and roles/infra/tasks/env.yml.

NODE FHS

The node data directory is specified by node_data, defaulting to /data, owned by root:root with mode 0755.

Most core components place their default data directories here. Some pilot modules use fixed paths of their own; native MySQL 8.4 currently uses /var/lib/mysql.

/data                                 # root:root 0755
#  ^-----@postgres                    # postgres:postgres 0700 (default pg_fs_main)
#  ^-----@backups                     # postgres:postgres 0700 (default pg_fs_backup)
#  ^-----@redis                       # redis:redis 0700 (shared by multiple instances)
#  ^-----@minio                       # minio:minio 0750 (single-node single-disk mode)
#  ^-----@etcd                        # etcd:etcd 0700 (etcd_data)
#  ^-----@infra                       # root:infra 0771 (infra module data directory)
#  ^-----@docker                      # root:root 0755 (Docker data directory)
#  ^-----@kafka                       # kafka:kafka 0700 (kafka_data)
#  ^-----@...                         # Other component data directories

HAProxy

Pigsty starts HAProxy with its own systemd unit and manages the main configuration separately from service fragments:

/etc/systemd/system/haproxy.service   # systemd unit rendered by Pigsty
/etc/haproxy/haproxy.cfg              # HAProxy main configuration
/etc/haproxy/conf.d/*.cfg             # node and PostgreSQL service fragments
/etc/default/haproxy                  # optional user environment file; Pigsty does not create it

To append startup arguments in /etc/default/haproxy, use EXTRAOPTS and retain the default -S /run/haproxy-master.sock. The systemd unit already loads configuration with explicit -f arguments, so do not add another -f to EXTRAOPTS.

Victoria FHS

Monitoring config has moved from the legacy /etc/prometheus layout to the /infra runtime layout. The main template is roles/infra/templates/victoria/prometheus.yml, rendered to /infra/prometheus.yml.

files/victoria/bin/* and files/victoria/rules/* are synced to /infra/bin/ and /infra/rules/, while each module registers FileSD targets under /infra/targets/*.

# /infra
#  ^-----prometheus.yml              # Victoria main config (Prometheus-compatible) 0644
#  ^-----@bin                        # Utility scripts (check/new/reload/status) 0755
#  ^-----@rules                      # Recording and alerting rules (*.yml 0644)
#            ^-----agent.yml         # Agent pre-aggregation rules
#            ^-----infra.yml         # infra rules and alerts
#            ^-----etcd.yml          # etcd rules and alerts
#            ^-----node.yml          # node rules and alerts
#            ^-----pgsql.yml         # pgsql rules and alerts
#            ^-----redis.yml         # redis rules and alerts
#            ^-----minio.yml         # minio rules and alerts
#            ^-----kafka.yml         # kafka rules and alerts
#            ^-----mysql.yml         # mysql rules and alerts
#  ^-----@targets                    # FileSD targets (*.yml 0640)
#            ^-----@infra            # infra static targets
#            ^-----@node             # node static targets
#            ^-----@pgsql            # pgsql static targets
#            ^-----@pgrds            # pgsql remote RDS targets
#            ^-----@redis            # redis static targets
#            ^-----@minio            # minio static targets
#            ^-----@mysql            # mysql static targets
#            ^-----@etcd             # etcd static targets
#            ^-----@ping             # ping static targets
#            ^-----@kafka            # kafka static targets
#            ^-----@juice            # juicefs static targets
#            ^-----@docker           # docker static targets
#            ^-----@patroni          # patroni static targets (when SSL enabled)
# /etc/default/vmetrics              # vmetrics startup args (victoria:infra 0644)
# /etc/default/vlogs                 # vlogs startup args (victoria:infra 0644)
# /etc/default/vtraces               # vtraces startup args (victoria:infra 0644)
# /etc/default/vmalert               # vmalert startup args (victoria:infra 0644)
# /etc/alertmanager.yml              # alertmanager main config (prometheus:infra 0644)
# /etc/default/alertmanager          # alertmanager env (prometheus:infra 0640)
# /etc/blackbox.yml                  # blackbox main config (prometheus:infra 0644)
# /etc/default/blackbox_exporter     # blackbox env (prometheus:infra 0644)

Pigsty-rendered INFRA units are consistently stored in /etc/systemd/system/, including vmetrics, vlogs, vtraces, vmalert, alertmanager, blackbox_exporter, nginx_exporter, and dnsmasq. Distribution package unit directories are not write targets for these roles.

PostgreSQL FHS

The following parameters and internal variables are related to PostgreSQL directory layout:

  • pg_dbsu_home: Postgres default user home directory, default: /var/lib/pgsql
  • pg_bin_dir: Postgres binary directory, default: /usr/pgsql/bin/
  • pg_fs_main: Postgres primary data directory, default: /data/postgres
  • pg_fs_backup: Postgres backup disk mount point, default: /data/backups (optional; can also be a subdirectory on primary disk)
  • pg_data: Internal variable, fixed to the Postgres data-directory symlink /pg/data
  • pg_cluster_dir: Derived variable, {{ pg_fs_main }}/{{ pg_cluster }}-{{ pg_version }}
  • pg_backup_dir: Derived variable, {{ pg_fs_backup }}/{{ pg_cluster }}-{{ pg_version }}
#--------------------------------------------------------------#
# Working assumptions:
#   {{ pg_fs_main   }} primary data directory, default: `/data/postgres` [SSD]
#   {{ pg_fs_backup }} backup data disk, default: `/data/backups`        [HDD]
#--------------------------------------------------------------#
# Default config (pg_cluster=pg-test, pg_version=18):
#     pg_fs_main = /data/postgres      High-speed SSD
#     pg_fs_backup = /data/backups     Cheap HDD (optional)
#
#     /pg        -> /data/postgres/pg-test-18
#     /pg/data   -> /data/postgres/pg-test-18/data
#     /pg/backup -> /data/backups/pg-test-18/backup
#--------------------------------------------------------------#
- name: create pgsql directories
  tags: pg_dir
  become: true
  block:

    - name: create pgsql directories
      file: path={{ item.path }} state=directory owner={{ item.owner|default(pg_dbsu) }} group={{ item.group|default('postgres') }} mode={{ item.mode }}
      with_items:
        - { path: "{{ pg_fs_main }}"            ,mode: "0700" }
        - { path: "{{ pg_fs_backup }}"          ,mode: "0700" }
        - { path: "{{ pg_cluster_dir }}"        ,mode: "0700" }
        - { path: "{{ pg_cluster_dir }}/bin"    ,mode: "0700" }
        - { path: "{{ pg_cluster_dir }}/log"    ,mode: "0750" }
        - { path: "{{ pg_cluster_dir }}/tmp"    ,mode: "0700" }
        - { path: "{{ pg_cluster_dir }}/cert"   ,mode: "0700" }
        - { path: "{{ pg_cluster_dir }}/conf"   ,mode: "0700" }
        - { path: "{{ pg_cluster_dir }}/data"   ,mode: "0700" }
        - { path: "{{ pg_cluster_dir }}/spool"  ,mode: "0700" }
        - { path: "{{ pg_backup_dir }}/backup"  ,mode: "0700" }
        - { path: "/var/run/postgresql"         ,owner: root, group: root, mode: "0755" }

    - name: link pgsql directories
      file: src={{ item.src }} dest={{ item.dest }} state=link
      with_items:
        - { src: "{{ pg_backup_dir }}/backup" ,dest: "{{ pg_cluster_dir }}/backup" }
        - { src: "{{ pg_cluster_dir }}"       ,dest: "/pg" }

Data File Structure

# Physical directories
{{ pg_fs_main }}     /data/postgres                    # postgres:postgres 0700, primary data directory
{{ pg_cluster_dir }} /data/postgres/pg-test-18         # postgres:postgres 0700, cluster directory
                     /data/postgres/pg-test-18/bin     # postgres:postgres 0700 (scripts root:postgres 0755)
                     /data/postgres/pg-test-18/log     # postgres:postgres 0750, logs
                     /data/postgres/pg-test-18/tmp     # postgres:postgres 0700, temp files
                     /data/postgres/pg-test-18/cert    # postgres:postgres 0700, certs
                     /data/postgres/pg-test-18/conf    # postgres:postgres 0700, config index
                     /data/postgres/pg-test-18/data    # postgres:postgres 0700, main data
                     /data/postgres/pg-test-18/spool   # postgres:postgres 0700, pgBackRest spool
                     /data/postgres/pg-test-18/backup  # -> /data/backups/pg-test-18/backup

{{ pg_fs_backup  }}  /data/backups                     # postgres:postgres 0700, optional backup mount
{{ pg_backup_dir }}  /data/backups/pg-test-18          # postgres:postgres 0700, cluster backup directory
                     /data/backups/pg-test-18/backup   # postgres:postgres 0700, actual backup location

# Symlinks
/pg             ->   /data/postgres/pg-test-18         # pg root symlink
/pg/data        ->   /data/postgres/pg-test-18/data    # pg data directory
/pg/backup      ->   /data/backups/pg-test-18/backup   # pg backup directory

Binary File Structure

On EL-compatible distributions (using yum), PostgreSQL default installation location is:

/usr/pgsql-${pg_version}/

Pigsty creates a symlink named /usr/pgsql pointing to the actual version specified by the pg_version parameter, for example:

/usr/pgsql -> /usr/pgsql-18

Therefore, the default pg_bin_dir is /usr/pgsql/bin/, and this path is added to the system PATH environment variable, defined in: /etc/profile.d/pgsql.sh.

export PATH="/usr/pgsql/bin:/pg/bin:$PATH"
export PGHOME=/usr/pgsql
export PGDATA=/pg/data

On Ubuntu/Debian, the default PostgreSQL Deb package installation location is:

/usr/lib/postgresql/${pg_version}/bin

Pigsty-rendered PostgreSQL runtime units are likewise stored in /etc/systemd/system/. They primarily include patroni.service, postgres.service, pgbouncer.service, pg_exporter.service, pgbackrest_exporter.service, pgbouncer_exporter.service, and vip-manager.service when VIP is enabled.

Pgbouncer FHS

Pgbouncer runs under the same user as {{ pg_dbsu }} (default postgres), with configs in /etc/pgbouncer.

  • pgbouncer.ini: main pool configuration (postgres:postgres 0640)
  • database.txt: pooled database definitions (postgres:postgres 0600)
  • useropts.txt: per-user connection options (postgres:postgres 0600)
  • userlist.txt: password file maintained by /pg/bin/pgb-user
  • pgb_hba.conf: access control file (postgres:postgres 0600)
/etc/pgbouncer/                # postgres:postgres 0750
/etc/pgbouncer/pgbouncer.ini   # postgres:postgres 0640
/etc/pgbouncer/database.txt    # postgres:postgres 0600
/etc/pgbouncer/useropts.txt    # postgres:postgres 0600
/etc/pgbouncer/userlist.txt    # postgres:postgres (managed by pgb-user)
/etc/pgbouncer/pgb_hba.conf    # postgres:postgres 0600
/pg/log/pgbouncer              # postgres:postgres 0750
/var/run/postgresql            # {{ pg_dbsu }}:postgres 0755 (managed by tmpfiles)

Object Storage FHS

The MINIO module currently deploys only Silo, while retaining minio_* parameter and directory names for compatibility:

/etc/default/silo                             # root:minio 0640, service environment
/etc/systemd/system/silo.service              # root:root 0644, rendered by Pigsty
/data/minio/                                  # minio:minio 0750, default data directory
/infra/targets/minio/<cluster>-<seq>.yml      # victoria:infra 0640, FileSD target
/home/minio/.mcli/config.json                 # mcli alias; also written for the execution user

Silo certificates are stored in /home/minio/.minio/certs/. The module name, role parameters, data directory, and FileSD path retain the compatible MINIO / minio_* naming.

Redis FHS

Pigsty manages Redis or Valkey with the same directory layout and instance naming.

Service units call binaries according to redis_type (/bin/* is compatible with /usr/bin/* on most distributions):

/bin/redis-server  /bin/redis-cli    # redis_type: redis
/bin/valkey-server /bin/valkey-cli   # redis_type: valkey

For a Redis instance named redis-test-1-6379, the related resources are as follows:

/etc/systemd/system/redis-test-1-6379.service         # root:root 0644, rendered by Pigsty
/etc/systemd/system/redis_exporter.service            # root:root 0644, rendered by Pigsty
/etc/redis/                                           # redis:redis 0700
/etc/redis/redis-test-1-6379.conf                     # redis:redis 0600
/data/redis/                                          # redis:redis 0700
/data/redis/redis-test-1-6379                         # redis:redis 0700
/data/redis/redis-test-1-6379/redis-test-1-6379.rdb   # RDB file
/data/redis/redis-test-1-6379/redis-test-1-6379.aof   # AOF file
/var/log/redis/                                       # redis:redis 0700
/var/log/redis/redis-test-1-6379.log                  # logs
/var/run/redis/                                       # redis:redis 0700 (tmpfiles creates 0755 at boot)
/var/run/redis/redis-test-1-6379.pid                  # PID

Pigsty-rendered Redis/Valkey instance and exporter units are consistently stored in /etc/systemd/system/, and instance units use Type=notify. Package-provided units may still live in distribution directories, but those are not role write targets.

5.4 - Parameters

Pigsty v4.x configuration overview and module parameter navigation

This is the parameter navigation page for Pigsty v4.x, without repeating full explanations for each parameter. For parameter details, please read each module’s param page.

Cross-checked against the current source and parameter reference pages, the 10 official modules expose 373 public parameters. Native MySQL 8.4 remains a pilot module; its 13 public parameters are listed separately and are not included in the official-module total.

Module Parameter Navigation

Module Groups Count Description
PGSQL 9 124 PostgreSQL HA cluster configuration
INFRA 10 73 Software repository and Victoria-based observability infra
NODE 11 73 Node initialization, system tuning, and ops baseline
ETCD 2 13 ETCD cluster and removal safeguard parameters
MINIO 2 22 Silo deployment, observability, and removal parameters
REDIS 2 22 Redis/Valkey deployment and removal parameters
DOCKER 1 8 Docker engine parameters
JUICE 1 2 JuiceFS instance and cache parameters
VIBE 1 18 Code/Jupyter/Node.js/Claude/Codex configuration
KAFKA 2 18 Kafka deployment and removal safeguard parameters

Pilot module: native MYSQL 8.4 currently exposes 13 public parameters: 11 for deployment and 2 for protected removal. Fixed ports, paths, software versions, and timers are not public parameters.

Parameter Group Quick View

Module Major Groups
PGSQL PG_ID, PG_BUSINESS, PG_INSTALL, PG_BOOTSTRAP, PG_PROVISION, PG_BACKUP, PG_ACCESS, PG_MONITOR, PG_REMOVE
INFRA META, CA, INFRA_ID, REPO, INFRA_PACKAGE, NGINX, DNS, VICTORIA, PROMETHEUS, GRAFANA
NODE NODE_ID, NODE_DNS, NODE_PACKAGE, NODE_TUNE, NODE_SEC, NODE_ADMIN, NODE_TIME, NODE_VIP
HAPROXY, NODE_EXPORTER, VECTOR
ETCD ETCD, ETCD_REMOVE
MINIO MINIO, MINIO_REMOVE
REDIS REDIS, REDIS_REMOVE
DOCKER DOCKER
JUICE JUICE
VIBE VIBE
KAFKA KAFKA, KAFKA_REMOVE

Recommendations

  • Read in this order for first deployment: NODE, INFRA, PGSQL
  • In production, always review: *_safeguard, password credentials, ports, and network exposure
  • Validate changes on one cluster first, then roll out globally in batches

5.5 - Playbooks

Pigsty v4.x preset Ansible playbook navigation and execution notes

This page summarizes Pigsty v4.x playbook entries and usage guidance by module. For detailed task tags, open each module’s playbook page.

Module Playbook Navigation

Module Count Playbooks
INFRA 3 deploy.yml infra.yml infra-rm.yml
NODE 2 node.yml node-rm.yml
ETCD 2 etcd.yml etcd-rm.yml
PGSQL 7 pgsql.yml pgsql-rm.yml
pgsql-user.yml pgsql-db.yml
pgsql-monitor.yml pgsql-migration.yml pgsql-pitr.yml
REDIS 2 redis.yml redis-rm.yml
MINIO 2 minio.yml minio-rm.yml
DOCKER 1 docker.yml
JUICE 1 juice.yml
VIBE 1 vibe.yml
KAFKA 2 kafka.yml kafka-rm.yml
MYSQL (pilot) 2 mysql.yml mysql-rm.yml

Playbook Matrix

Playbook Module Purpose
deploy.yml INFRA One-pass deployment for the core chain (Infra/Node/Etcd/PGSQL, enabling MINIO by config)
infra.yml INFRA Initialize infrastructure nodes
infra-rm.yml INFRA Remove infrastructure components
node.yml NODE Node onboarding and baseline convergence
node-rm.yml NODE Node offboarding
etcd.yml ETCD ETCD install/scale-out
etcd-rm.yml ETCD ETCD remove/scale-in
pgsql.yml PGSQL Initialize PostgreSQL cluster or add instance
pgsql-rm.yml PGSQL Remove PostgreSQL cluster/instance
pgsql-user.yml PGSQL Add business users
pgsql-db.yml PGSQL Add business databases
pgsql-monitor.yml PGSQL Register remote PostgreSQL for monitoring
pgsql-migration.yml PGSQL Generate migration runbook and scripts
pgsql-pitr.yml PGSQL Point-in-time recovery (PITR)
redis.yml REDIS Deploy Redis
redis-rm.yml REDIS Remove Redis
minio.yml MINIO Deploy Silo
minio-rm.yml MINIO Remove Silo, its configuration, and optional data
docker.yml DOCKER Deploy Docker engine
juice.yml JUICE Deploy/remove JuiceFS instances
vibe.yml VIBE Deploy VIBE dev environment
kafka.yml KAFKA Create or converge a complete dynamic KRaft cluster
kafka-rm.yml KAFKA Remove a Kafka cluster, or safely retire a single member
mysql.yml MYSQL Converge a native MySQL 8.4 single node or three-node InnoDB Cluster (pilot)
mysql-rm.yml MYSQL Stop or retire a native MySQL instance or cluster while preserving local state (pilot)

Auxiliary Playbooks

The following playbooks are cross-module helpers.

Playbook Description
cache.yml Build offline installation package cache
cert.yml Issue certificates using Pigsty CA
app.yml Install Docker Compose app templates
slim.yml Minimal component installation scenario

Playbook Usage Notes

Protection Mechanism

Several modules provide deletion safeguards through *_safeguard parameters:

The PGSQL, ETCD, MINIO, REDIS, and KAFKA role defaults are explicitly false; set them to true for initialized production clusters. Native MySQL is the exception: mysql_safeguard defaults to true, and even after disabling it you must provide a mysql_rm_confirm value that exactly matches the target instance or cluster.

When safeguard is true, corresponding *-rm.yml playbooks abort immediately. You can force override via CLI:

./pgsql-rm.yml -l pg-test -e pg_safeguard=false
./etcd-rm.yml  -l etcd -e etcd_safeguard=false
./minio-rm.yml -l minio -e minio_type=silo -e minio_safeguard=false
./redis-rm.yml -l redis-test -e redis_safeguard=false
./kafka-rm.yml -l kf-main -e kafka_safeguard=false
./mysql-rm.yml -l my-test -e mysql_safeguard=false -e mysql_rm_confirm=my-test

Limiting Execution Scope

Use -l to limit execution targets:

./pgsql.yml -l pg-meta            # run only on pg-meta cluster
./node.yml -l 10.10.10.10         # run only on one node
./redis.yml -l redis-test         # run only on redis-test cluster

For large-scale rollout, validate on one cluster first, then deploy in batches.

Idempotency

Most playbooks are idempotent and safe to rerun, with caveats:

  • infra.yml does not clean data by default; all clean parameters (vmetrics_clean, vlogs_clean, vtraces_clean, grafana_clean, nginx_clean) default to false
  • To rebuild from a clean state, explicitly set relevant clean parameters to true
  • Re-running *-rm.yml deletion playbooks requires extra caution

Task Tags

Use -t to run only selected task subsets:

./pgsql.yml -l pg-test -t pg_service    # refresh services only on pg-test
./node.yml -t haproxy                   # configure haproxy only
./etcd.yml -t etcd_launch               # restart etcd only

Quick Command Reference

INFRA Module 

./deploy.yml                     # deploy the core chain in one pass
./infra.yml                      # initialize infrastructure
./infra-rm.yml                   # remove infrastructure components
./cache.yml -l <infra-host>      # build an offline package from an existing repo on an Infra node
./cert.yml -e cn=<name>          # issue client certificate

NODE Module 

./node.yml -l <cls|ip>           # add node
./node-rm.yml -l <cls|ip>        # remove node
bin/node-add <cls|ip>            # add node (wrapper)
bin/node-rm <cls|ip>             # remove node (wrapper)

ETCD Module 

./etcd.yml                       # initialize etcd cluster
./etcd-rm.yml -l etcd            # remove etcd cluster; deletes local data and configuration by default
bin/etcd-add <ip>                # add etcd member (wrapper)
bin/etcd-rm <ip>                 # remove etcd member (wrapper)

PGSQL Module 

./pgsql.yml -l <cls>                             # initialize PostgreSQL cluster
./pgsql-rm.yml -l <cls>                          # remove PostgreSQL cluster
./pgsql-user.yml -l <cls> -e username=<user>     # create business user
./pgsql-db.yml -l <cls> -e dbname=<db>           # create business database
./pgsql-monitor.yml -e clsname=<cls>             # monitor remote cluster
./pgsql-migration.yml -e@files/migration/<cls>.yml  # generate migration runbook
./pgsql-pitr.yml -l <cls> -e '{"pg_pitr": {}}'  # perform PITR recovery

bin/pgsql-add <cls>              # initialize cluster (wrapper)
bin/pgsql-rm <cls>               # remove cluster (wrapper)
bin/pgsql-user <cls> <user>      # create user (wrapper)
bin/pgsql-db <cls> <db>          # create database (wrapper)
bin/pgsql-svc <cls>              # refresh services (wrapper)
bin/pgsql-hba <cls>              # reload HBA (wrapper)
bin/pgmon-add <cls>              # monitor remote cluster (wrapper)

REDIS Module 

./redis.yml -l <cls>             # initialize Redis cluster
./redis-rm.yml -l <cls>          # remove Redis cluster

MINIO Module 

./minio.yml -l <cls>                       # initialize the MINIO module's Silo cluster
./minio-rm.yml -l <cls> -e minio_type=silo # remove Silo; this value must be confirmed explicitly

DOCKER Module 

./docker.yml -l <host>           # install Docker
./app.yml -e app=<name>          # deploy Docker Compose app

KAFKA Module 

./kafka.yml -l <cls>             # create / converge a complete Kafka cluster
./kafka.yml -l <cls> --check     # read-only precheck
./kafka-rm.yml -l <cls>          # remove the whole cluster
./kafka-rm.yml -l <ip>           # retire a single member from the cluster

For ordinary convergence, -l must cover every declared member of the selected Kafka cluster; only kafka-rm.yml accepts a single member, for retirement.

MYSQL Pilot Module 

./mysql.yml -l <cls> --check
./mysql.yml -l <cls>             # accepts only a complete 1- or 3-member cluster scope
./mysql-rm.yml -l <instance> --check \
  -e mysql_safeguard=false -e mysql_rm_confirm=<instance>
./mysql-rm.yml -l <cls> \
  -e mysql_safeguard=false -e mysql_rm_confirm=<cls>

mysql-rm.yml stops the service, writes a retirement marker, and deregisters monitoring, but does not delete data directories, backups, configuration, certificates, packages, or InnoDB Cluster metadata.

5.6 - Port List

Default ports used by Pigsty components, with related parameters and status.

This page lists default ports used by Pigsty module components. Adjust as needed or use as a reference for fine-grained firewall configuration.

Module Component Port Parameter Status
NODE node_exporter 9100 node_exporter_port Enabled
NODE haproxy 9101 haproxy_exporter_port Enabled
NODE vector 9598 vector_port Enabled
NODE keepalived_exporter 9650 vip_exporter_port Optional
NODE chronyd 123 - Enabled
DOCKER docker 9323 docker_exporter_port Optional
INFRA nginx 80 nginx_port Enabled
INFRA nginx 443 nginx_ssl_port Enabled
INFRA nginx_exporter 9113 nginx_exporter_port Enabled
INFRA grafana 3000 grafana_port Enabled
INFRA victoriaMetrics 8428 vmetrics_port Enabled
INFRA victoriaLogs 9428 vlogs_port Enabled
INFRA victoriaTraces 10428 vtraces_port Enabled
INFRA vmalert 8880 vmalert_port Enabled
INFRA alertmanager 9059 alertmanager_port Enabled
INFRA blackbox_exporter 9115 blackbox_port Enabled
INFRA dnsmasq 53 dns_port Enabled
ETCD etcd 2379 etcd_port Enabled
ETCD etcd 2380 etcd_peer_port Enabled
MINIO Silo S3 API 9000 minio_port Optional
MINIO Silo admin port 9001 minio_admin_port Optional
REDIS Redis / Valkey 6379 redis_instances Optional
REDIS redis_exporter 9121 redis_exporter_port Optional
VIBE code-server 8443 code_port Optional
VIBE jupyterlab 8888 jupyter_port Optional
KAFKA broker 9092 kafka_port 🧪 BETA
KAFKA KRaft controller 9093 kafka_controller_port 🧪 BETA
KAFKA kafka_exporter 9308 kafka_exporter_port 🧪 BETA
KAFKA JMX exporter 9404 kafka_jmx_exporter_port 🧪 BETA
MYSQL mysqld 3306 Fixed value (the current pilot exposes no port parameter) 🧪 PILOT
MYSQL MySQL X Protocol 33060 Fixed value; loopback-only on a single node, member-facing in a 3-node topology 🧪 PILOT
MYSQL Group Replication 33061 Fixed value; three-node InnoDB Cluster only 🧪 PILOT
MYSQL MySQL Router RW 6446 Fixed value; three-node InnoDB Cluster only 🧪 PILOT
MYSQL MySQL Router RO 6447 Fixed value; three-node InnoDB Cluster only 🧪 PILOT
MYSQL mysqld_exporter 9104 Fixed value; controlled by mysql_exporter_enabled 🧪 PILOT
PGSQL postgres 5432 pg_port Enabled
PGSQL pgbouncer 6432 pgbouncer_port Enabled
PGSQL patroni 8008 patroni_port Enabled
PGSQL pg_exporter 9630 pg_exporter_port Enabled
PGSQL pgbouncer_exporter 9631 pgbouncer_exporter_port Enabled
PGSQL pgbackrest_exporter 9854 pgbackrest_exporter_port Enabled
PGSQL {{ pg_cluster }}-primary 5433 pg_default_services Enabled
PGSQL {{ pg_cluster }}-replica 5434 pg_default_services Enabled
PGSQL {{ pg_cluster }}-default 5436 pg_default_services Enabled
PGSQL {{ pg_cluster }}-offline 5438 pg_default_services Enabled
PGSQL {{ pg_cluster }}-<service> 543x pg_services Optional

The native MySQL pilot reuses port 3306 for MySQL Shell AdminAPI. XtraBackup is invoked by a local systemd timer and has no listening port, while the role explicitly disables the MySQL Router REST management interface. The table lists only network endpoints currently managed by the role.

Public Port Recommendations

If you use firewall zone mode, expose only minimum required ports via node_firewall_public_port:

  • Minimal management surface: 22, 80, 443 (recommended)
  • If public direct DB access is required: additionally expose 5432

Avoid exposing internal component ports directly to the public internet: etcd (2379/2380), patroni (8008), exporters (9xxx), object-storage S3/admin endpoints (9000/9001), redis (6379), ferretdb (27017/27018), Kafka (9092/9093), MySQL Group Replication (33061), etc.

node_firewall_mode: zone
node_firewall_public_port: [22, 80, 443]
# node_firewall_public_port: [22, 80, 443, 5432]  # only if public DB access is required

6 - Configuration Templates

Batteries-included configuration templates for specific scenarios, with detailed explanations.

Use -c with configure to select a template. Its value is a path relative to conf/ without the .yml suffix. If omitted, Pigsty uses the default meta template.

Category Templates
Solo Templates meta, rich, fat, slim, infra, vibe, docker
Kernel Templates pgsql, pg19, mssql, polar, ivory, agens, pgedge, mysql (OpenHalo), mongo, pgtde, oriole
HA Templates ha/simu, ha/octo, ha/citus, ha/full, ha/safe, ha/trio, ha/dual
App Templates supabase, app/odoo, app/dify, app/insforge, app/hindsight, app/electric, app/maybe, app/teable, app/mattermost, app/registry, app/immich, app/jumpserver
Misc Templates demo/bare, demo/el, demo/debian, demo/demo, demo/kernel, demo/redis, demo/minio, demo/kafka, demo/mysql (native MySQL pilot), demo/remote, demo/saas, demo/wool, build/oss, build/dev

6.1 - meta

Default single-node installation template with extensive configuration parameter descriptions

The meta configuration template is Pigsty’s default template, designed to fulfill Pigsty’s core functionality—deploying PostgreSQL—on a single node.

To maximize compatibility, meta installs only the minimum required software set to ensure it runs across all operating system distributions and architectures.

Overview

  • Config Name: meta
  • Node Count: Single node
  • Description: Default single-node installation template with extensive configuration parameter descriptions and minimum required feature set.
  • OS Distro: el8, el9, el10, d12, d13, u22, u24, u26
  • OS Arch: x86_64, aarch64
  • Related: meta, slim, fat

Usage: This is the default config template, so there’s no need to specify -c meta explicitly during configure:

./configure [-i <primary_ip>]

For example, if you want to install PostgreSQL 16 rather than the default 18, you can use the -v arg in configure:

./configure -v 16   # or 17, 15, 14; use the pg19 template for PG19 Beta

Content

Source: pigsty/conf/meta.yml

---
#==============================================================#
# File      :   meta.yml
# Desc      :   Pigsty default 1-node online install config
# Ctime     :   2020-05-22
# Mtime     :   2026-07-10
# Docs      :   https://pigsty.io/docs/conf/meta
# License   :   Apache-2.0 @ https://pigsty.io/docs/about/license/
# Copyright :   2018-2026  Ruohang Feng / Vonng ([email protected])
#==============================================================#

# This is the default 1-node configuration template, with:
# INFRA, NODE, PGSQL, ETCD, MINIO, DOCKER, APP
# with basic pg extensions: postgis, pgvector
#
# Work with PostgreSQL 14-18 on all supported platform
# Usage:
#   curl https://repo.pigsty.io/get | bash
#   ./configure -g
#   ./deploy.yml

all:

  #==============================================================#
  # Clusters, Nodes, and Modules
  #==============================================================#
  children:

    #----------------------------------------------#
    # PGSQL : https://pigsty.io/docs/pgsql
    #----------------------------------------------#
    # this is an example single-node postgres cluster with pgvector installed, with one biz database & two biz users
    pg-meta:
      hosts:
        10.10.10.10: { pg_seq: 1, pg_role: primary } # <---- primary instance with read-write capability
        #x.xx.xx.xx: { pg_seq: 2, pg_role: replica } # <---- read only replica for read-only online traffic
        #x.xx.xx.xy: { pg_seq: 3, pg_role: offline } # <---- offline instance of ETL & interactive queries
      vars:
        pg_cluster: pg-meta

        # install, load, create pg extensions: https://pigsty.io/docs/pgsql/ext/
        pg_extensions: [ postgis, pgvector ]

        # define business users/roles : https://pigsty.io/docs/pgsql/config/user
        pg_users:
          - { name: dbuser_meta ,password: DBUser.Meta   ,pgbouncer: true ,roles: [dbrole_admin   ] ,comment: pigsty admin user }
          - { name: dbuser_view ,password: DBUser.Viewer ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer  }

        # define business databases : https://pigsty.io/docs/pgsql/config/db
        pg_databases:
          - name: meta
            baseline: cmdb.sql
            comment: "pigsty meta database"
            schemas: [pigsty]
            # define extensions in database : https://pigsty.io/docs/pgsql/ext/create
            extensions: [ postgis, vector ]

        pg_hba_rules:   # https://pigsty.io/docs/pgsql/config/hba
          - { user: all ,db: all ,addr: intra ,auth: pwd ,title: 'everyone intranet access with password' ,order: 800 }
        pg_crontab:     # https://pigsty.io/docs/pgsql/admin/crontab
          - '00 01 * * * /pg/bin/pg-backup full'

        # define (OPTIONAL) L2 VIP that bind to primary
        #pg_vip_enabled: true
        #pg_vip_address: 10.10.10.2/24


    #----------------------------------------------#
    # INFRA : https://pigsty.io/docs/infra
    #----------------------------------------------#
    infra:
      hosts:
        10.10.10.10: { infra_seq: 1 }
      vars:
        repo_enabled: false   # disable in 1-node mode :  https://pigsty.io/docs/infra/admin/repo
        #repo_extra_packages: [ pg18-main ,pg18-time ,pg18-gis ,pg18-rag ,pg18-fts ,pg18-olap ,pg18-feat ,pg18-lang ,pg18-type ,pg18-util ,pg18-func ,pg18-admin ,pg18-stat ,pg18-sec ,pg18-fdw ,pg18-sim ,pg18-etl]

    #----------------------------------------------#
    # ETCD : https://pigsty.io/docs/etcd
    #----------------------------------------------#
    etcd:
      hosts:
        10.10.10.10: { etcd_seq: 1 }
      vars:
        etcd_cluster: etcd
        etcd_safeguard: false             # prevent purging running etcd instance?

    #----------------------------------------------#
    # MINIO : https://pigsty.io/docs/minio
    #----------------------------------------------#
    #minio:
    #  hosts:
    #    10.10.10.10: { minio_seq: 1 }
    #  vars:
    #    minio_cluster: minio
    #    minio_users:                      # list of minio user to be created
    #      - { access_key: pgbackrest  ,secret_key: S3User.Backup ,policy: pgsql }
    #      - { access_key: s3user_meta ,secret_key: S3User.Meta   ,policy: meta  }
    #      - { access_key: s3user_data ,secret_key: S3User.Data   ,policy: data  }

    #----------------------------------------------#
    # DOCKER : https://pigsty.io/docs/docker
    # APP    : https://pigsty.io/docs/app
    #----------------------------------------------#
    # launch example pgadmin app with: ./app.yml (http://10.10.10.10:8885 [email protected] / pigsty)
    app:
      hosts: { 10.10.10.10: {} }
      vars:
        docker_enabled: true                # enabled docker with ./docker.yml
        #docker_registry_mirrors: ["https://docker.1panel.live","https://docker.1ms.run","https://docker.xuanyuan.me","https://registry-1.docker.io"]
        app: pgadmin                        # specify the default app name to be installed (in the apps)
        apps:                               # define all applications, appname: definition
          pgadmin:                          # pgadmin app definition (app/pgadmin -> /opt/pgadmin)
            conf:                           # override /opt/pgadmin/.env
              PGADMIN_DEFAULT_EMAIL: [email protected]
              PGADMIN_DEFAULT_PASSWORD: pigsty


  #==============================================================#
  # Global Parameters
  #==============================================================#
  vars:

    #----------------------------------------------#
    # INFRA : https://pigsty.io/docs/infra
    #----------------------------------------------#
    version: v4.5.0                   # pigsty version string
    admin_ip: 10.10.10.10             # admin node ip address
    region: default                   # upstream mirror region: default|china|europe
    proxy_env:                        # global proxy env when downloading packages
      no_proxy: "localhost,127.0.0.1,10.0.0.0/8,192.168.0.0/16,*.pigsty,*.aliyun.com,mirrors.*,*.myqcloud.com,*.tsinghua.edu.cn"
      # http_proxy:  # set your proxy here: e.g http://user:[email protected]
      # https_proxy: # set your proxy here: e.g http://user:[email protected]
      # all_proxy:   # set your proxy here: e.g http://user:[email protected]
    infra_portal:                     # infra services exposed via portal
      home : { domain: i.pigsty }     # default domain name
      pgadmin : { domain: adm.pigsty ,endpoint: "${admin_ip}:8885" }
      #minio  : { domain: m.pigsty ,endpoint: "${admin_ip}:9001" ,scheme: https ,websocket: true }

    #----------------------------------------------#
    # NODE : https://pigsty.io/docs/node/param
    #----------------------------------------------#
    nodename_overwrite: false             # do not overwrite node hostname on single node mode
    node_tune: oltp                       # node tuning specs: oltp,olap,tiny,crit
    node_etc_hosts: ['${admin_ip} i.pigsty sss.pigsty']
    node_repo_modules: 'node,infra,pgsql' # add these repos directly to the singleton node
    #node_repo_modules: local             # use this if you want to build & user local repo
    node_repo_remove: true                # remove existing node repo for node managed by pigsty
    #node_packages: [openssh-server]      # packages to be installed current nodes with the latest version
    node_firewall_public_port: [22, 80, 443, 5432]    # expose 5432 for demo convenience, remove in production!

    #----------------------------------------------#
    # PGSQL : https://pigsty.io/docs/pgsql/param
    #----------------------------------------------#
    pg_version: 18                      # default postgres version
    pg_conf: oltp.yml                   # pgsql tuning specs: {oltp,olap,tiny,crit}.yml
    pg_safeguard: false                 # prevent purging running postgres instance?
    pg_packages: [ pgsql-main, pgsql-common ]  # pg kernel and common utils
    #pg_extensions: [ pg18-time ,pg18-gis ,pg18-rag ,pg18-fts ,pg18-olap ,pg18-feat ,pg18-lang ,pg18-type ,pg18-util ,pg18-func ,pg18-admin ,pg18-stat ,pg18-sec ,pg18-fdw ,pg18-sim ,pg18-etl]

    #----------------------------------------------#
    # BACKUP : https://pigsty.io/docs/pgsql/backup
    #----------------------------------------------#
    # if you want to use minio as backup repo instead of 'local' fs, uncomment this, and configure `pgbackrest_repo`
    # you can also use external object storage as backup repo
    #pgbackrest_method: minio          # if you want to use minio as backup repo instead of 'local' fs, uncomment this
    #pgbackrest_repo:                  # pgbackrest repo: https://pgbackrest.org/configuration.html#section-repository
    #  local:                          # default pgbackrest repo with local posix fs
    #    path: /pg/backup              # local backup directory, `/pg/backup` by default
    #    retention_full_type: count    # retention full backups by count
    #    retention_full: 2             # keep 2, at most 3 full backup when using local fs repo
    #  minio:                          # optional minio repo for pgbackrest
    #    type: s3                      # minio is s3-compatible, so s3 is used
    #    s3_endpoint: sss.pigsty       # minio endpoint domain name, `sss.pigsty` by default
    #    s3_region: us-east-1          # minio region, us-east-1 by default, useless for minio
    #    s3_bucket: pgsql              # minio bucket name, `pgsql` by default
    #    s3_key: pgbackrest            # minio user access key for pgbackrest
    #    s3_key_secret: S3User.Backup  # minio user secret key for pgbackrest
    #    s3_uri_style: path            # use path style uri for minio rather than host style
    #    path: /pgbackrest             # minio backup path, default is `/pgbackrest`
    #    storage_port: 9000            # minio port, 9000 by default
    #    storage_ca_file: /etc/pki/ca.crt  # minio ca file path, `/etc/pki/ca.crt` by default
    #    block: y                      # Enable block incremental backup
    #    bundle: y                     # bundle small files into a single file
    #    bundle_limit: 20MiB           # Limit for file bundles, 20MiB for object storage
    #    bundle_size: 128MiB           # Target size for file bundles, 128MiB for object storage
    #    cipher_type: aes-256-cbc      # enable AES encryption for remote backup repo
    #    cipher_pass: pgBackRest       # AES encryption password, default is 'pgBackRest'
    #    retention_full_type: time     # retention full backup by time on minio repo
    #    retention_full: 14            # keep full backup for last 14 days
    #  s3: # aliyun oss (s3 compatible) object storage service
    #    type: s3                      # oss is s3-compatible
    #    s3_endpoint: oss-cn-beijing-internal.aliyuncs.com
    #    s3_region: oss-cn-beijing
    #    s3_bucket: <your_bucket_name>
    #    s3_key: <your_access_key>
    #    s3_key_secret: <your_secret_key>
    #    s3_uri_style: host
    #    path: /pgbackrest
    #    bundle: y                     # bundle small files into a single file
    #    bundle_limit: 20MiB           # Limit for file bundles, 20MiB for object storage
    #    bundle_size: 128MiB           # Target size for file bundles, 128MiB for object storage
    #    cipher_type: aes-256-cbc      # enable AES encryption for remote backup repo
    #    cipher_pass: pgBackRest       # AES encryption password, default is 'pgBackRest'
    #    retention_full_type: time     # retention full backup by time on minio repo
    #    retention_full: 14            # keep full backup for last 14 days

    #----------------------------------------------#
    # PASSWORD : https://pigsty.io/docs/setup/security/
    #----------------------------------------------#
    grafana_admin_password: pigsty
    grafana_view_password: DBUser.Viewer
    pg_admin_password: DBUser.DBA
    pg_monitor_password: DBUser.Monitor
    pg_replication_password: DBUser.Replicator
    patroni_password: Patroni.API
    haproxy_admin_password: pigsty
    minio_secret_key: S3User.MinIO
    etcd_root_password: Etcd.Root
...

Explanation

The meta template is Pigsty’s default getting-started configuration, designed for quick onboarding.

Use Cases:

  • First-time Pigsty users
  • Quick deployment in development and testing environments
  • Small production environments running on a single machine
  • As a base template for more complex deployments

Key Features:

  • Online installation mode without building local software repository (repo_enabled: false)
  • Default installs PostgreSQL 18 with postgis and pgvector extensions
  • Includes complete observability infrastructure (Grafana, VictoriaMetrics, VictoriaLogs, etc.)
  • Preconfigured Docker and pgAdmin application examples
  • Silo backup storage disabled by default, can be enabled as needed

Notes:

  • Default passwords are sample passwords; must be changed for production environments
  • Single-node etcd has no high availability guarantee, suitable for development and testing
  • If you need to build a local software repository, use the rich template

6.2 - rich

Feature-rich single-node configuration with local software repository, all extensions, Silo backup, and complete examples

The rich configuration template is an enhanced version of meta, designed for users who need to experience complete functionality.

If you want to build a local software repository, use Silo for backup storage, run Docker applications, or need preconfigured business databases, use this template.

Overview

  • Config Name: rich
  • Node Count: Single node
  • Description: Feature-rich single-node configuration, adding local software repository, Silo backup, complete extensions, Docker application examples on top of meta
  • OS Distro: el8, el9, el10, d12, d13, u22, u24, u26
  • OS Arch: x86_64, aarch64
  • Related: meta, slim, fat

This template’s main enhancements over meta:

  • Builds local software repository (repo_enabled: true), downloads all PG extensions
  • Enables single-node Silo as PostgreSQL backup storage
  • Preinstalls TimescaleDB, pgvector, pg_wait_sampling and other extensions
  • Includes detailed user/database/service definition comment examples
  • Adds Redis primary-replica instance example
  • Preconfigures pg-test three-node HA cluster configuration stub

Usage:

./configure -c rich [-i <primary_ip>]

Content

Source: pigsty/conf/rich.yml

---
#==============================================================#
# File      :   rich.yml
# Desc      :   Pigsty feature-rich 1-node online install config
# Ctime     :   2020-05-22
# Mtime     :   2025-12-12
# Docs      :   https://pigsty.io/docs/conf/rich
# License   :   Apache-2.0 @ https://pigsty.io/docs/about/license/
# Copyright :   2018-2026  Ruohang Feng / Vonng ([email protected])
#==============================================================#

# This is the enhanced version of default meta.yml, which has:
# - almost all available postgres extensions
# - build local software repo for entire env
# - 1 node minio used as central backup repo
# - cluster stub for 3-node pg-test / redis
# - stub for nginx, certs, and website self-hosting config
# - detailed comments for database / user / service
#
# Usage:
#   curl https://repo.pigsty.io/get | bash
#   ./configure -c rich
#   ./deploy.yml

all:

  #==============================================================#
  # Clusters, Nodes, and Modules
  #==============================================================#
  children:

    #----------------------------------------------#
    # PGSQL : https://pigsty.io/docs/pgsql
    #----------------------------------------------#
    # this is an example single-node postgres cluster with pgvector installed, with one biz database & two biz users
    pg-meta:
      hosts:
        10.10.10.10: { pg_seq: 1, pg_role: primary } # <---- primary instance with read-write capability
        #x.xx.xx.xx: { pg_seq: 2, pg_role: replica } # <---- read only replica for read-only online traffic
        #x.xx.xx.xy: { pg_seq: 3, pg_role: offline } # <---- offline instance of ETL & interactive queries
      vars:
        pg_cluster: pg-meta

        # install, load, create pg extensions: https://pigsty.io/docs/pgsql/ext/
        pg_extensions: [ postgis, timescaledb, pgvector, pg_wait_sampling ]
        pg_libs: 'timescaledb, pg_stat_statements, auto_explain, pg_wait_sampling'

        # define business users/roles : https://pigsty.io/docs/pgsql/config/user
        pg_users:
          - name: dbuser_meta               # REQUIRED, `name` is the only mandatory field of a user definition
            password: DBUser.Meta           # optional, the password. can be a scram-sha-256 hash string or plain text
            pgbouncer: true                 # optional, add this user to the pgbouncer user-list? false by default (production user should be true explicitly)
            comment: pigsty admin user      # optional, comment string for this user/role
            roles: [ dbrole_admin ]         # optional, belonged roles. default roles are: dbrole_{admin|readonly|readwrite|offline}
            #state: create                  # optional, create|absent, 'create' by default, use 'absent' to drop user
            #login: true                    # optional, can log in, true by default (new biz ROLE should be false)
            #superuser: false               # optional, is superuser? false by default
            #createdb: false                # optional, can create databases? false by default
            #createrole: false              # optional, can create role? false by default
            #inherit: true                  # optional, can this role use inherited privileges? true by default
            #replication: false             # optional, can this role do replication? false by default
            #bypassrls: false               # optional, can this role bypass row level security? false by default
            #connlimit: -1                  # optional, user connection limit, default -1 disable limit
            #expire_in: 3650                # optional, now + n days when this role is expired (OVERWRITE expire_at)
            #expire_at: '2030-12-31'        # optional, YYYY-MM-DD 'timestamp' when this role is expired (OVERWRITTEN by expire_in)
            #parameters: {}                 # optional, role level parameters with `ALTER ROLE SET`
            #pool_mode: transaction         # optional, pgbouncer pool mode at user level, transaction by default
            #pool_connlimit: -1             # optional, max database connections at user level, default -1 disable limit
            # Enhanced roles syntax (PG16+): roles can be string or object with options:
            #   - dbrole_readwrite                       # simple string: GRANT role
            #   - { name: role, admin: true }            # GRANT WITH ADMIN OPTION
            #   - { name: role, set: false }             # PG16: REVOKE SET OPTION
            #   - { name: role, inherit: false }         # PG16: REVOKE INHERIT OPTION
            #   - { name: role, state: absent }          # REVOKE membership
          - { name: dbuser_view ,password: DBUser.Viewer ,pgbouncer: true ,roles: [dbrole_readonly], comment: read-only viewer for meta database }
          #- {name: dbuser_bytebase ,password: DBUser.Bytebase ,pgbouncer: true ,roles: [dbrole_admin] ,comment: admin user for bytebase database   }
          #- {name: dbuser_remove ,state: absent }       # use state: absent to remove a user

        # define business databases : https://pigsty.io/docs/pgsql/config/db
        pg_databases:                       # define business databases on this cluster, array of database definition
          - name: meta                      # REQUIRED, `name` is the only mandatory field of a database definition
            #state: create                  # optional, create|absent|recreate, create by default
            baseline: cmdb.sql              # optional, database sql baseline path, (relative path among the ansible search path, e.g.: files/)
            schemas: [ pigsty ]             # optional, additional schemas to be created, array of schema names
            extensions:                     # optional, additional extensions to be installed: array of `{name[,schema]}`
              - vector                      # install pgvector for vector similarity search
              - postgis                     # install postgis for geospatial type & index
              - timescaledb                 # install timescaledb for time-series data
              - { name: pg_wait_sampling, schema: monitor } # install pg_wait_sampling on monitor schema
            comment: pigsty meta database   # optional, comment string for this database
            #pgbouncer: true                # optional, add this database to the pgbouncer database list? true by default
            #owner: postgres                # optional, database owner, current user if not specified
            #template: template1            # optional, which template to use, template1 by default
            #strategy: FILE_COPY            # optional, clone strategy: FILE_COPY or WAL_LOG (PG15+), default to PG's default
            #encoding: UTF8                 # optional, inherited from template / cluster if not defined (UTF8)
            #locale: C                      # optional, inherited from template / cluster if not defined (C)
            #lc_collate: C                  # optional, inherited from template / cluster if not defined (C)
            #lc_ctype: C                    # optional, inherited from template / cluster if not defined (C)
            #locale_provider: libc          # optional, locale provider: libc, icu, builtin (PG15+)
            #icu_locale: en-US              # optional, icu locale for icu locale provider (PG15+)
            #icu_rules: ''                  # optional, icu rules for icu locale provider (PG16+)
            #builtin_locale: C.UTF-8        # optional, builtin locale for builtin locale provider (PG17+)
            #tablespace: pg_default         # optional, default tablespace, pg_default by default
            #is_template: false             # optional, mark database as template, allowing clone by any user with CREATEDB privilege
            #allowconn: true                # optional, allow connection, true by default. false will disable connect at all
            #revokeconn: false              # optional, revoke public connection privilege. false by default. (leave connect with grant option to owner)
            #register_datasource: true      # optional, register this database to grafana datasources? true by default
            #connlimit: -1                  # optional, database connection limit, default -1 disable limit
            #pool_auth_user: dbuser_meta    # optional, all connection to this pgbouncer database will be authenticated by this user
            #pool_mode: transaction         # optional, pgbouncer pool mode at database level, default transaction
            #pool_size: 64                  # optional, pgbouncer pool size at database level, default 64
            #pool_reserve: 32               # optional, pgbouncer pool size reserve at database level, default 32
            #pool_size_min: 0               # optional, pgbouncer pool size min at database level, default 0
            #pool_connlimit: 100            # optional, max database connections at database level, default 100
          #- {name: bytebase ,owner: dbuser_bytebase ,revokeconn: true ,comment: bytebase primary database }

        pg_hba_rules:   # https://pigsty.io/docs/pgsql/config/hba
          - { user: all ,db: all ,addr: intra ,auth: pwd ,title: 'everyone intranet access with password' ,order: 800 }
        pg_crontab:     # https://pigsty.io/docs/pgsql/admin/crontab
          - '00 01 * * * /pg/bin/pg-backup full'

        # define (OPTIONAL) L2 VIP that bind to primary
        #pg_vip_enabled: true
        #pg_vip_address: 10.10.10.2/24

    #----------------------------------------------#
    # PGSQL HA Cluster Example: 3-node pg-test
    #----------------------------------------------#
    #pg-test:
    #  hosts:
    #    10.10.10.11: { pg_seq: 1, pg_role: primary }   # primary instance, leader of cluster
    #    10.10.10.12: { pg_seq: 2, pg_role: replica }   # replica instance, follower of leader
    #    10.10.10.13: { pg_seq: 3, pg_role: replica, pg_offline_query: true } # replica with offline access
    #  vars:
    #    pg_cluster: pg-test           # define pgsql cluster name
    #    pg_users:  [{ name: test , password: test , pgbouncer: true , roles: [ dbrole_admin ] }]
    #    pg_databases: [{ name: test }]
    #    # define business service here: https://pigsty.io/docs/pgsql/service
    #    pg_services:                        # extra services in addition to pg_default_services, array of service definition
    #      # standby service will route {ip|name}:5435 to sync replica's pgbouncer (5435->6432 standby)
    #      - name: standby                   # required, service name, the actual svc name will be prefixed with `pg_cluster`, e.g: pg-meta-standby
    #        port: 5435                      # required, service exposed port (work as kubernetes service node port mode)
    #        ip: "*"                         # optional, service bind ip address, `*` for all ip by default
    #        selector: "[]"                  # required, service member selector, use JMESPath to filter inventory
    #        dest: default                   # optional, destination port, default|postgres|pgbouncer|<port_number>, 'default' by default
    #        check: /sync                    # optional, health check url path, / by default
    #        backup: "[? pg_role == `primary`]"  # backup server selector
    #        maxconn: 3000                   # optional, max allowed front-end connection
    #        balance: roundrobin             # optional, haproxy load balance algorithm (roundrobin by default, other: leastconn)
    #        options: 'inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100'
    #    pg_vip_enabled: true
    #    pg_vip_address: 10.10.10.3/24
    #    pg_crontab:  # make a full backup on monday 1am, and an incremental backup during weekdays
    #      - '00 01 * * 1 /pg/bin/pg-backup full'
    #      - '00 01 * * 2,3,4,5,6,7 /pg/bin/pg-backup'

    #----------------------------------------------#
    # INFRA : https://pigsty.io/docs/infra
    #----------------------------------------------#
    infra:
      hosts:
        10.10.10.10: { infra_seq: 1 }
      vars:
        repo_enabled: true    # build local repo, and install everything from it:  https://pigsty.io/docs/infra/admin/repo
        # and download all extensions into local repo
        repo_extra_packages: [ pg18-main ,pg18-time ,pg18-gis ,pg18-rag ,pg18-fts ,pg18-olap ,pg18-feat ,pg18-lang ,pg18-type ,pg18-util ,pg18-func ,pg18-admin ,pg18-stat ,pg18-sec ,pg18-fdw ,pg18-sim ,pg18-etl]

    #----------------------------------------------#
    # ETCD : https://pigsty.io/docs/etcd
    #----------------------------------------------#
    etcd:
      hosts:
        10.10.10.10: { etcd_seq: 1 }
      vars:
        etcd_cluster: etcd
        etcd_safeguard: false             # prevent purging running etcd instance?

    #----------------------------------------------#
    # MINIO : https://pigsty.io/docs/minio
    #----------------------------------------------#
    minio:
      hosts:
        10.10.10.10: { minio_seq: 1 }
      vars:
        minio_cluster: minio
        minio_users:                      # list of minio user to be created
          - { access_key: pgbackrest  ,secret_key: S3User.Backup ,policy: pgsql }
          - { access_key: s3user_meta ,secret_key: S3User.Meta   ,policy: meta  }
          - { access_key: s3user_data ,secret_key: S3User.Data   ,policy: data  }

    #----------------------------------------------#
    # DOCKER : https://pigsty.io/docs/docker
    # APP    : https://pigsty.io/docs/app
    #----------------------------------------------#
    # OPTIONAL, launch example pgadmin app with: ./app.yml & ./app.yml -e app=bytebase
    app:
      hosts: { 10.10.10.10: {} }
      vars:
        docker_enabled: true                # enabled docker with ./docker.yml
        #docker_registry_mirrors: ["https://docker.1panel.live","https://docker.1ms.run","https://docker.xuanyuan.me","https://registry-1.docker.io"]
        app: pgadmin                        # specify the default app name to be installed (in the apps)
        apps:                               # define all applications, appname: definition

          # Admin GUI for PostgreSQL, launch with: ./app.yml
          pgadmin:                          # pgadmin app definition (app/pgadmin -> /opt/pgadmin)
            conf:                           # override /opt/pgadmin/.env
              PGADMIN_DEFAULT_EMAIL: [email protected]   # default user name
              PGADMIN_DEFAULT_PASSWORD: pigsty         # default password

          # Schema Migration GUI for PostgreSQL, launch with: ./app.yml -e app=bytebase
          bytebase:
            conf:
              BB_DOMAIN: http://ddl.pigsty  # replace it with your public domain name and postgres database url
              BB_PGURL: "postgresql://dbuser_bytebase:[email protected]:5432/bytebase?sslmode=prefer"

    #----------------------------------------------#
    # REDIS : https://pigsty.io/docs/redis
    #----------------------------------------------#
    # OPTIONAL, launch redis clusters with: ./redis.yml
    redis-ms:
      hosts: { 10.10.10.10: { redis_node: 1 , redis_instances: { 6379: { }, 6380: { replica_of: '10.10.10.10 6379' } } } }
      vars: { redis_cluster: redis-ms ,redis_password: 'redis.ms' ,redis_max_memory: 64MB }



  #==============================================================#
  # Global Parameters
  #==============================================================#
  vars:

    #----------------------------------------------#
    # INFRA : https://pigsty.io/docs/infra
    #----------------------------------------------#
    version: v4.5.0                   # pigsty version string
    admin_ip: 10.10.10.10             # admin node ip address
    region: default                   # upstream mirror region: default|china|europe
    proxy_env:                        # global proxy env when downloading packages
      no_proxy: "localhost,127.0.0.1,10.0.0.0/8,192.168.0.0/16,*.pigsty,*.aliyun.com,mirrors.*,*.myqcloud.com,*.tsinghua.edu.cn"
      # http_proxy:  # set your proxy here: e.g http://user:[email protected]
      # https_proxy: # set your proxy here: e.g http://user:[email protected]
      # all_proxy:   # set your proxy here: e.g http://user:[email protected]

    certbot_sign: false               # enable certbot to sign https certificate for infra portal
    certbot_email: [email protected]     # replace your email address to receive expiration notice
    infra_portal:                     # infra services exposed via portal
      home      : { domain: i.pigsty }     # default domain name
      pgadmin   : { domain: adm.pigsty ,endpoint: "${admin_ip}:8885" }
      bytebase  : { domain: ddl.pigsty ,endpoint: "${admin_ip}:8887" }
      minio     : { domain: m.pigsty ,endpoint: "${admin_ip}:9001" ,scheme: https ,websocket: true }

      #website:   # static local website example stub
      #  domain: repo.pigsty              # external domain name for static site
      #  certbot: repo.pigsty             # use certbot to sign https certificate for this static site
      #  path: /www/pigsty                # path to the static site directory

      #supabase:  # dynamic upstream service example stub
      #  domain: supa.pigsty          # external domain name for upstream service
      #  certbot: supa.pigsty         # use certbot to sign https certificate for this upstream server
      #  endpoint: "10.10.10.10:8000" # path to the static site directory
      #  websocket: true              # add websocket support
      #  certbot: supa.pigsty         # certbot cert name, apply with `make cert`

    #----------------------------------------------#
    # PASSWORD : https://pigsty.io/docs/setup/security/
    #----------------------------------------------#
    grafana_admin_password: pigsty
    grafana_view_password: DBUser.Viewer
    pg_admin_password: DBUser.DBA
    pg_monitor_password: DBUser.Monitor
    pg_replication_password: DBUser.Replicator
    patroni_password: Patroni.API
    haproxy_admin_password: pigsty
    minio_secret_key: S3User.MinIO
    etcd_root_password: Etcd.Root

    #----------------------------------------------#
    # NODE : https://pigsty.io/docs/node/param
    #----------------------------------------------#
    nodename_overwrite: false             # do not overwrite node hostname on single node mode
    node_tune: oltp                       # node tuning specs: oltp,olap,tiny,crit
    node_etc_hosts:                       # add static domains to all nodes /etc/hosts
      - '${admin_ip} i.pigsty sss.pigsty'
      - '${admin_ip} adm.pigsty ddl.pigsty repo.pigsty supa.pigsty'
    node_repo_modules: local              # use pre-made local repo rather than install from upstream
    node_repo_remove: true                # remove existing node repo for node managed by pigsty
    #node_packages: [openssh-server]      # packages to be installed current nodes with latest version
    #node_timezone: Asia/Hong_Kong        # overwrite node timezone

    #----------------------------------------------#
    # PGSQL : https://pigsty.io/docs/pgsql/param
    #----------------------------------------------#
    pg_version: 18                      # default postgres version
    pg_conf: oltp.yml                   # pgsql tuning specs: {oltp,olap,tiny,crit}.yml
    pg_safeguard: false                 # prevent purging running postgres instance?
    pg_packages: [ pgsql-main, pgsql-common ]                 # pg kernel and common utils
    #pg_extensions: [ pg18-time ,pg18-gis ,pg18-rag ,pg18-fts ,pg18-olap ,pg18-feat ,pg18-lang ,pg18-type ,pg18-util ,pg18-func ,pg18-admin ,pg18-stat ,pg18-sec ,pg18-fdw ,pg18-sim ,pg18-etl]

    #----------------------------------------------#
    # BACKUP : https://pigsty.io/docs/pgsql/backup
    #----------------------------------------------#
    # if you want to use minio as backup repo instead of 'local' fs, uncomment this, and configure `pgbackrest_repo`
    # you can also use external object storage as backup repo
    pgbackrest_method: minio          # if you want to use minio as backup repo instead of 'local' fs, uncomment this
    pgbackrest_repo:                  # pgbackrest repo: https://pgbackrest.org/configuration.html#section-repository
      local:                          # default pgbackrest repo with local posix fs
        path: /pg/backup              # local backup directory, `/pg/backup` by default
        retention_full_type: count    # retention full backups by count
        retention_full: 2             # keep 2, at most 3 full backups when using local fs repo
      minio:                          # optional minio repo for pgbackrest
        type: s3                      # minio is s3-compatible, so s3 is used
        s3_endpoint: sss.pigsty       # minio endpoint domain name, `sss.pigsty` by default
        s3_region: us-east-1          # minio region, us-east-1 by default, useless for minio
        s3_bucket: pgsql              # minio bucket name, `pgsql` by default
        s3_key: pgbackrest            # minio user access key for pgbackrest [CHANGE ACCORDING to minio_users.pgbackrest]
        s3_key_secret: S3User.Backup  # minio user secret key for pgbackrest [CHANGE ACCORDING to minio_users.pgbackrest]
        s3_uri_style: path            # use path style uri for minio rather than host style
        path: /pgbackrest             # minio backup path, default is `/pgbackrest`
        storage_port: 9000            # minio port, 9000 by default
        storage_ca_file: /etc/pki/ca.crt  # minio ca file path, `/etc/pki/ca.crt` by default
        block: y                      # Enable block incremental backup
        bundle: y                     # bundle small files into a single file
        bundle_limit: 20MiB           # Limit for file bundles, 20MiB for object storage
        bundle_size: 128MiB           # Target size for file bundles, 128MiB for object storage
        cipher_type: aes-256-cbc      # enable AES encryption for remote backup repo
        cipher_pass: pgBackRest       # AES encryption password, default is 'pgBackRest'
        retention_full_type: time     # retention full backup by time on minio repo
        retention_full: 14            # keep full backup for the last 14 days
      s3:                             # you can use cloud object storage as backup repo
        type: s3                      # Add your object storage credentials here!
        s3_endpoint: oss-cn-beijing-internal.aliyuncs.com
        s3_region: oss-cn-beijing
        s3_bucket: <your_bucket_name>
        s3_key: <your_access_key>
        s3_key_secret: <your_secret_key>
        s3_uri_style: host
        path: /pgbackrest
        bundle: y                     # bundle small files into a single file
        bundle_limit: 20MiB           # Limit for file bundles, 20MiB for object storage
        bundle_size: 128MiB           # Target size for file bundles, 128MiB for object storage
        cipher_type: aes-256-cbc      # enable AES encryption for remote backup repo
        cipher_pass: pgBackRest       # AES encryption password, default is 'pgBackRest'
        retention_full_type: time     # retention full backup by time on minio repo
        retention_full: 14            # keep full backup for the last 14 days
...

Explanation

The rich template is Pigsty’s complete functionality showcase configuration, suitable for users who want to deeply experience all features.

Use Cases:

  • Offline environments requiring local software repository
  • Environments needing Silo as PostgreSQL backup storage
  • Pre-planning multiple business databases and users
  • Running Docker applications (pgAdmin, Bytebase, etc.)
  • Learners wanting to understand complete configuration parameter usage

Main Differences from meta:

  • Enables local software repository building (repo_enabled: true)
  • Enables Silo backup storage (compatibility preset pgbackrest_method: minio)
  • Preinstalls TimescaleDB, pg_wait_sampling and other additional extensions
  • Includes detailed parameter comments for understanding configuration meanings
  • Preconfigures HA cluster stub configuration (pg-test)

Notes:

  • Some extensions unavailable on ARM64 architecture, adjust as needed
  • Building local software repository requires longer time and larger disk space
  • Default passwords are sample passwords, must be changed for production

6.3 - slim

Minimal installation template without monitoring infrastructure, installs PostgreSQL directly from internet

The slim configuration template provides minimal installation capability, installing a PostgreSQL high-availability cluster directly from the internet without deploying Infra monitoring infrastructure.

When you only need an available database instance without the monitoring system, consider using the Slim Installation mode.

Overview

  • Config Name: slim
  • Node Count: Single node
  • Description: Minimal installation template without monitoring infrastructure, installs PostgreSQL directly
  • OS Distro: el8, el9, el10, d12, d13, u22, u24, u26
  • OS Arch: x86_64, aarch64
  • Related: meta

Usage:

./configure -c slim [-i <primary_ip>]
./slim.yml   # Execute slim installation

Content

Source: pigsty/conf/slim.yml

---
#==============================================================#
# File      :   slim.yml
# Desc      :   Pigsty slim installation config template
# Ctime     :   2020-05-22
# Mtime     :   2025-12-28
# Docs      :   https://pigsty.io/docs/conf/slim
# License   :   Apache-2.0 @ https://pigsty.io/docs/about/license/
# Copyright :   2018-2026  Ruohang Feng / Vonng ([email protected])
#==============================================================#

# This is the config template for slim / minimal installation
# No monitoring & infra will be installed, just raw postgresql
#
# Usage:
#   curl https://repo.pigsty.io/get | bash
#   ./configure -c slim
#   ./slim.yml

all:
  children:

    etcd: # dcs service for postgres/patroni ha consensus
      hosts: # 1 node for testing, 3 or 5 for production
        10.10.10.10: { etcd_seq: 1 }  # etcd_seq required
        #10.10.10.11: { etcd_seq: 2 }  # assign from 1 ~ n
        #10.10.10.12: { etcd_seq: 3 }  # three-member cluster keeps an odd voter count
      vars: # cluster level parameter override roles/etcd
        etcd_cluster: etcd  # mark etcd cluster name etcd

    #----------------------------------------------#
    # PostgreSQL Cluster
    #----------------------------------------------#
    pg-meta:
      hosts:
        10.10.10.10: { pg_seq: 1, pg_role: primary }
        #10.10.10.11: { pg_seq: 2, pg_role: replica } # you can add more!
        #10.10.10.12: { pg_seq: 3, pg_role: replica, pg_offline_query: true }
      vars:
        pg_cluster: pg-meta
        pg_users:
          - { name: dbuser_meta ,password: DBUser.Meta   ,pgbouncer: true ,roles: [dbrole_admin   ] ,comment: pigsty admin user }
          - { name: dbuser_view ,password: DBUser.Viewer ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer  }
        pg_databases:
          - { name: meta, baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty] ,extensions: [ vector ]}
        pg_hba_rules:   # https://pigsty.io/docs/pgsql/config/hba
          - { user: all ,db: all ,addr: intra ,auth: pwd ,title: 'everyone intranet access with password' ,order: 800 }
        pg_crontab:     # https://pigsty.io/docs/pgsql/admin/crontab
          - '00 01 * * * /pg/bin/pg-backup full'

  vars:
    version: v4.5.0                   # pigsty version string
    admin_ip: 10.10.10.10             # admin node ip address
    region: default                   # upstream mirror region: default,china,europe
    nodename_overwrite: false           # do not overwrite node hostname on single node mode
    node_repo_modules: node,infra,pgsql # add these repos directly to the singleton node
    node_tune: oltp                     # node tuning specs: oltp,olap,tiny,crit
    pg_conf: oltp.yml                   # pgsql tuning specs: {oltp,olap,tiny,crit}.yml
    pg_version: 18                      # Default PostgreSQL Major Version is 18
    pg_packages: [ pgsql-main, pgsql-common ]   # pg kernel and common utils
    #pg_extensions: [ pg18-time ,pg18-gis ,pg18-rag ,pg18-fts ,pg18-olap ,pg18-feat ,pg18-lang ,pg18-type ,pg18-util ,pg18-func ,pg18-admin ,pg18-stat ,pg18-sec ,pg18-fdw ,pg18-sim ,pg18-etl]

    #----------------------------------------------#
    # PASSWORD : https://pigsty.io/docs/setup/security/
    #----------------------------------------------#
    grafana_admin_password: pigsty
    grafana_view_password: DBUser.Viewer
    pg_admin_password: DBUser.DBA
    pg_monitor_password: DBUser.Monitor
    pg_replication_password: DBUser.Replicator
    patroni_password: Patroni.API
    haproxy_admin_password: pigsty
    minio_secret_key: S3User.MinIO
    etcd_root_password: Etcd.Root
...

Explanation

The slim template is Pigsty’s minimal installation configuration, designed for quick deployment of bare PostgreSQL clusters.

Use Cases:

  • Only need PostgreSQL database, no monitoring system required
  • Resource-limited small servers or edge devices
  • Quick deployment of temporary test databases
  • Already have monitoring system, only need PostgreSQL HA cluster

Key Features:

  • Uses slim.yml playbook instead of deploy.yml for installation
  • Installs software directly from internet, no local software repository
  • Retains core PostgreSQL HA capability (Patroni + etcd + HAProxy)
  • Minimized package downloads, faster installation
  • Default uses PostgreSQL 18

Differences from meta:

  • slim uses dedicated slim.yml playbook, skips Infra module installation
  • Faster installation, less resource usage
  • Suitable for “just need a database” scenarios

Notes:

  • After slim installation, cannot view database status through Grafana
  • If monitoring is needed, use meta or rich template
  • Can add replicas as needed for high availability

6.4 - fat

Feature-All-Test template, single-node installation of all extensions, builds local repo with PG 14-18 all versions

The fat configuration template is Pigsty’s Feature-All-Test template, installing all extension plugins on a single node and building a local software repository containing all extensions for PostgreSQL 14-18 (five major versions).

This is a full-featured configuration for testing and development, suitable for scenarios requiring complete software package cache or testing all extensions.

Overview

  • Config Name: fat
  • Node Count: Single node
  • Description: Feature-All-Test template, installs all extensions, builds local repo with PG 14-18 all versions
  • OS Distro: el8, el9, el10, d12, d13, u22, u24, u26
  • OS Arch: x86_64, aarch64
  • Related: meta, slim, fat

Usage:

./configure -c fat [-i <primary_ip>]

To specify a particular PostgreSQL version:

./configure -c fat -v 16   # Use PostgreSQL 16

Content

Source: pigsty/conf/fat.yml

---
#==============================================================#
# File      :   fat.yml
# Desc      :   Pigsty Feature-All-Test config template
# Ctime     :   2020-05-22
# Mtime     :   2025-12-28
# Docs      :   https://pigsty.io/docs/conf/fat
# License   :   Apache-2.0 @ https://pigsty.io/docs/about/license/
# Copyright :   2018-2026  Ruohang Feng / Vonng ([email protected])
#==============================================================#

# This is the 4-node sandbox for pigsty
#
# Usage:
#   curl https://repo.pigsty.io/get | bash
#   ./configure -c fat [-v 18|17|16|15]
#   ./deploy.yml

all:

  #==============================================================#
  # Clusters, Nodes, and Modules
  #==============================================================#
  children:

    #----------------------------------------------#
    # PGSQL : https://pigsty.io/docs/pgsql
    #----------------------------------------------#
    # this is an example single-node postgres cluster with pgvector installed, with one biz database & two biz users
    pg-meta:
      hosts:
        10.10.10.10: { pg_seq: 1, pg_role: primary } # <---- primary instance with read-write capability
        #x.xx.xx.xx: { pg_seq: 2, pg_role: replica } # <---- read only replica for read-only online traffic
        #x.xx.xx.xy: { pg_seq: 3, pg_role: offline } # <---- offline instance of ETL & interactive queries
      vars:
        pg_cluster: pg-meta

        # install, load, create pg extensions: https://pigsty.io/docs/pgsql/ext/
        pg_extensions: [ pg18-main ,pg18-time ,pg18-gis ,pg18-rag ,pg18-fts ,pg18-olap ,pg18-feat ,pg18-lang ,pg18-type ,pg18-util ,pg18-func ,pg18-admin ,pg18-stat ,pg18-sec ,pg18-fdw ,pg18-sim ,pg18-etl]
        pg_libs: 'timescaledb, pg_stat_statements, auto_explain, pg_wait_sampling'

        # define business users/roles : https://pigsty.io/docs/pgsql/config/user
        pg_users:
          - name: dbuser_meta               # REQUIRED, `name` is the only mandatory field of a user definition
            password: DBUser.Meta           # optional, the password. can be a scram-sha-256 hash string or plain text
            pgbouncer: true                 # optional, add this user to the pgbouncer user-list? false by default (production user should be true explicitly)
            comment: pigsty admin user      # optional, comment string for this user/role
            roles: [ dbrole_admin ]         # optional, belonged roles. default roles are: dbrole_{admin|readonly|readwrite|offline}
            #state: create                   # optional, create|absent, 'create' by default, use 'absent' to drop user
            #login: true                     # optional, can log in, true by default (new biz ROLE should be false)
            #superuser: false                # optional, is superuser? false by default
            #createdb: false                 # optional, can create databases? false by default
            #createrole: false               # optional, can create role? false by default
            #inherit: true                   # optional, can this role use inherited privileges? true by default
            #replication: false              # optional, can this role do replication? false by default
            #bypassrls: false                # optional, can this role bypass row level security? false by default
            #connlimit: -1                   # optional, user connection limit, default -1 disable limit
            #expire_in: 3650                 # optional, now + n days when this role is expired (OVERWRITE expire_at)
            #expire_at: '2030-12-31'         # optional, YYYY-MM-DD 'timestamp' when this role is expired (OVERWRITTEN by expire_in)
            #parameters: {}                  # optional, role level parameters with `ALTER ROLE SET`
            #pool_mode: transaction          # optional, pgbouncer pool mode at user level, transaction by default
            #pool_connlimit: -1              # optional, max database connections at user level, default -1 disable limit
            # Enhanced roles syntax (PG16+): roles can be string or object with options:
            #   - dbrole_readwrite                       # simple string: GRANT role
            #   - { name: role, admin: true }            # GRANT WITH ADMIN OPTION
            #   - { name: role, set: false }             # PG16: REVOKE SET OPTION
            #   - { name: role, inherit: false }         # PG16: REVOKE INHERIT OPTION
            #   - { name: role, state: absent }          # REVOKE membership
          - { name: dbuser_view ,password: DBUser.Viewer ,pgbouncer: true ,roles: [dbrole_readonly], comment: read-only viewer for meta database }
          #- {name: dbuser_bytebase ,password: DBUser.Bytebase ,pgbouncer: true ,roles: [dbrole_admin] ,comment: admin user for bytebase database   }
          #- {name: dbuser_remove ,state: absent }       # use state: absent to remove a user

        # define business databases : https://pigsty.io/docs/pgsql/config/db
        pg_databases:                       # define business databases on this cluster, array of database definition
          - name: meta                      # REQUIRED, `name` is the only mandatory field of a database definition
            #state: create                  # optional, create|absent|recreate, create by default
            baseline: cmdb.sql              # optional, database sql baseline path, (relative path among the ansible search path, e.g.: files/)
            schemas: [ pigsty ]             # optional, additional schemas to be created, array of schema names
            extensions:                     # optional, additional extensions to be installed: array of `{name[,schema]}`
              - vector                      # install pgvector for vector similarity search
              - postgis                     # install postgis for geospatial type & index
              - timescaledb                 # install timescaledb for time-series data
              - { name: pg_wait_sampling, schema: monitor } # install pg_wait_sampling on monitor schema
            comment: pigsty meta database   # optional, comment string for this database
            #pgbouncer: true                # optional, add this database to the pgbouncer database list? true by default
            #owner: postgres                # optional, database owner, current user if not specified
            #template: template1            # optional, which template to use, template1 by default
            #strategy: FILE_COPY            # optional, clone strategy: FILE_COPY or WAL_LOG (PG15+), default to PG's default
            #encoding: UTF8                 # optional, inherited from template / cluster if not defined (UTF8)
            #locale: C                      # optional, inherited from template / cluster if not defined (C)
            #lc_collate: C                  # optional, inherited from template / cluster if not defined (C)
            #lc_ctype: C                    # optional, inherited from template / cluster if not defined (C)
            #locale_provider: libc          # optional, locale provider: libc, icu, builtin (PG15+)
            #icu_locale: en-US              # optional, icu locale for icu locale provider (PG15+)
            #icu_rules: ''                  # optional, icu rules for icu locale provider (PG16+)
            #builtin_locale: C.UTF-8        # optional, builtin locale for builtin locale provider (PG17+)
            #tablespace: pg_default         # optional, default tablespace, pg_default by default
            #is_template: false             # optional, mark database as template, allowing clone by any user with CREATEDB privilege
            #allowconn: true                # optional, allow connection, true by default. false will disable connect at all
            #revokeconn: false              # optional, revoke public connection privilege. false by default. (leave connect with grant option to owner)
            #register_datasource: true      # optional, register this database to grafana datasources? true by default
            #connlimit: -1                  # optional, database connection limit, default -1 disable limit
            #pool_auth_user: dbuser_meta    # optional, all connection to this pgbouncer database will be authenticated by this user
            #pool_mode: transaction         # optional, pgbouncer pool mode at database level, default transaction
            #pool_size: 64                  # optional, pgbouncer pool size at database level, default 64
            #pool_reserve: 32               # optional, pgbouncer pool size reserve at database level, default 32
            #pool_size_min: 0               # optional, pgbouncer pool size min at database level, default 0
            #pool_connlimit: 100            # optional, max database connections at database level, default 100
          #- {name: bytebase ,owner: dbuser_bytebase ,revokeconn: true ,comment: bytebase primary database }

        pg_hba_rules:   # https://pigsty.io/docs/pgsql/config/hba
          - { user: all ,db: all ,addr: intra ,auth: pwd ,title: 'everyone intranet access with password' ,order: 800 }
        pg_crontab:     # https://pigsty.io/docs/pgsql/admin/crontab
          - '00 01 * * * /pg/bin/pg-backup full'

        # define (OPTIONAL) L2 VIP that bind to primary
        pg_vip_enabled: true
        pg_vip_address: 10.10.10.2/24


    #----------------------------------------------#
    # INFRA : https://pigsty.io/docs/infra
    #----------------------------------------------#
    infra:
      hosts:
        10.10.10.10: { infra_seq: 1 }
      vars:
        repo_enabled: true # build local repo:  https://pigsty.io/docs/infra/admin/repo
        #repo_extra_packages: [ pg18-main ,pg18-time ,pg18-gis ,pg18-rag ,pg18-fts ,pg18-olap ,pg18-feat ,pg18-lang,pg18-type ,pg18-util ,pg18-func ,pg18-admin ,pg18-stat ,pg18-sec ,pg18-fdw ,pg18-sim ,pg18-etl]
        repo_packages: [
          node-bootstrap, infra-package, infra-addons, node-package1, node-package2, node-package3, pgsql-utility, extra-modules,
          pg18-full,pg18-time,pg18-gis,pg18-rag,pg18-fts,pg18-olap,pg18-feat,pg18-lang,pg18-type,pg18-util,pg18-func,pg18-admin,pg18-stat,pg18-sec,pg18-fdw,pg18-sim,pg18-etl,
          pg17-full,pg17-time,pg17-gis,pg17-rag,pg17-fts,pg17-olap,pg17-feat,pg17-lang,pg17-type,pg17-util,pg17-func,pg17-admin,pg17-stat,pg17-sec,pg17-fdw,pg17-sim,pg17-etl,
          pg16-full,pg16-time,pg16-gis,pg16-rag,pg16-fts,pg16-olap,pg16-feat,pg16-lang,pg16-type,pg16-util,pg16-func,pg16-admin,pg16-stat,pg16-sec,pg16-fdw,pg16-sim,pg16-etl,
          pg15-full,pg15-time,pg15-gis,pg15-rag,pg15-fts,pg15-olap,pg15-feat,pg15-lang,pg15-type,pg15-util,pg15-func,pg15-admin,pg15-stat,pg15-sec,pg15-fdw,pg15-sim,pg15-etl,
          pg14-full,pg14-time,pg14-gis,pg14-rag,pg14-fts,pg14-olap,pg14-feat,pg14-lang,pg14-type,pg14-util,pg14-func,pg14-admin,pg14-stat,pg14-sec,pg14-fdw,pg14-sim,pg14-etl,
          infra-extra, kafka-stack, java-runtime, sealos, tigerbeetle, polardb, ivorysql
        ]

    #----------------------------------------------#
    # ETCD : https://pigsty.io/docs/etcd
    #----------------------------------------------#
    etcd:
      hosts:
        10.10.10.10: { etcd_seq: 1 }
      vars:
        etcd_cluster: etcd
        etcd_safeguard: false             # prevent purging running etcd instance?

    #----------------------------------------------#
    # MINIO : https://pigsty.io/docs/minio
    #----------------------------------------------#
    minio:
      hosts:
        10.10.10.10: { minio_seq: 1 }
      vars:
        minio_cluster: minio
        minio_users:                      # list of minio user to be created
          - { access_key: pgbackrest  ,secret_key: S3User.Backup ,policy: pgsql }
          - { access_key: s3user_meta ,secret_key: S3User.Meta   ,policy: meta  }
          - { access_key: s3user_data ,secret_key: S3User.Data   ,policy: data  }

    #----------------------------------------------#
    # DOCKER : https://pigsty.io/docs/docker
    # APP    : https://pigsty.io/docs/app
    #----------------------------------------------#
    # OPTIONAL, launch example pgadmin app with: ./app.yml & ./app.yml -e app=bytebase
    app:
      hosts: { 10.10.10.10: {} }
      vars:
        docker_enabled: true                # enabled docker with ./docker.yml
        #docker_registry_mirrors: ["https://docker.1panel.live","https://docker.1ms.run","https://docker.xuanyuan.me","https://registry-1.docker.io"]
        app: pgadmin                        # specify the default app name to be installed (in the apps)
        apps:                               # define all applications, appname: definition

          # Admin GUI for PostgreSQL, launch with: ./app.yml
          pgadmin:                          # pgadmin app definition (app/pgadmin -> /opt/pgadmin)
            conf:                           # override /opt/pgadmin/.env
              PGADMIN_DEFAULT_EMAIL: [email protected]   # default user name
              PGADMIN_DEFAULT_PASSWORD: pigsty         # default password

          # Schema Migration GUI for PostgreSQL, launch with: ./app.yml -e app=bytebase
          bytebase:
            conf:
              BB_DOMAIN: http://ddl.pigsty  # replace it with your public domain name and postgres database url
              BB_PGURL: "postgresql://dbuser_bytebase:[email protected]:5432/bytebase?sslmode=prefer"


  #==============================================================#
  # Global Parameters
  #==============================================================#
  vars:

    #----------------------------------------------#
    # INFRA : https://pigsty.io/docs/infra
    #----------------------------------------------#
    version: v4.5.0                   # pigsty version string
    admin_ip: 10.10.10.10             # admin node ip address
    region: default                   # upstream mirror region: default|china|europe
    proxy_env:                        # global proxy env when downloading packages
      no_proxy: "localhost,127.0.0.1,10.0.0.0/8,192.168.0.0/16,*.pigsty,*.aliyun.com,mirrors.*,*.myqcloud.com,*.tsinghua.edu.cn"
      # http_proxy:  # set your proxy here: e.g http://user:[email protected]
      # https_proxy: # set your proxy here: e.g http://user:[email protected]
      # all_proxy:   # set your proxy here: e.g http://user:[email protected]

    certbot_sign: false               # enable certbot to sign https certificate for infra portal
    certbot_email: [email protected]     # replace your email address to receive expiration notice
    infra_portal:                     # domain names and upstream servers
      home         : { domain: i.pigsty }
      pgadmin      : { domain: adm.pigsty ,endpoint: "${admin_ip}:8885" }
      bytebase     : { domain: ddl.pigsty ,endpoint: "${admin_ip}:8887" ,websocket: true}
      minio        : { domain: m.pigsty ,endpoint: "${admin_ip}:9001" ,scheme: https ,websocket: true }

      #website:   # static local website example stub
      #  domain: repo.pigsty              # external domain name for static site
      #  certbot: repo.pigsty             # use certbot to sign https certificate for this static site
      #  path: /www/pigsty                # path to the static site directory

      #supabase:  # dynamic upstream service example stub
      #  domain: supa.pigsty          # external domain name for upstream service
      #  certbot: supa.pigsty         # use certbot to sign https certificate for this upstream server
      #  endpoint: "10.10.10.10:8000" # path to the static site directory
      #  websocket: true              # add websocket support
      #  certbot: supa.pigsty         # certbot cert name, apply with `make cert`

    #----------------------------------------------#
    # NODE : https://pigsty.io/docs/node/param
    #----------------------------------------------#
    nodename_overwrite: true              # overwrite node hostname on multi-node template
    node_tune: oltp                       # node tuning specs: oltp,olap,tiny,crit
    node_etc_hosts:                       # add static domains to all nodes /etc/hosts
      - 10.10.10.10 i.pigsty sss.pigsty
      - 10.10.10.10 adm.pigsty ddl.pigsty repo.pigsty supa.pigsty
    node_repo_modules: local,node,infra,pgsql # use pre-made local repo rather than install from upstream
    node_repo_remove: true                # remove existing node repo for node managed by pigsty
    #node_packages: [openssh-server]      # packages to be installed current nodes with latest version
    #node_timezone: Asia/Hong_Kong        # overwrite node timezone

    #----------------------------------------------#
    # PGSQL : https://pigsty.io/docs/pgsql/param
    #----------------------------------------------#
    pg_version: 18                      # default postgres version
    pg_conf: oltp.yml                   # pgsql tuning specs: {oltp,olap,tiny,crit}.yml
    pg_safeguard: false                 # prevent purging running postgres instance?
    pg_packages: [ pgsql-main, pgsql-common ] # pg kernel and common utils
    #pg_extensions: [ pg18-time ,pg18-gis ,pg18-rag ,pg18-fts ,pg18-olap ,pg18-feat ,pg18-lang ,pg18-type ,pg18-util ,pg18-func ,pg18-admin ,pg18-stat ,pg18-sec ,pg18-fdw ,pg18-sim ,pg18-etl]

    #----------------------------------------------#
    # BACKUP : https://pigsty.io/docs/pgsql/backup
    #----------------------------------------------#
    # if you want to use minio as backup repo instead of 'local' fs, uncomment this, and configure `pgbackrest_repo`
    # you can also use external object storage as backup repo
    pgbackrest_method: minio          # if you want to use minio as backup repo instead of 'local' fs, uncomment this
    pgbackrest_repo:                  # pgbackrest repo: https://pgbackrest.org/configuration.html#section-repository
      local:                          # default pgbackrest repo with local posix fs
        path: /pg/backup              # local backup directory, `/pg/backup` by default
        retention_full_type: count    # retention full backups by count
        retention_full: 2             # keep 2, at most 3 full backups when using local fs repo
      minio:                          # optional minio repo for pgbackrest
        type: s3                      # minio is s3-compatible, so s3 is used
        s3_endpoint: sss.pigsty       # minio endpoint domain name, `sss.pigsty` by default
        s3_region: us-east-1          # minio region, us-east-1 by default, useless for minio
        s3_bucket: pgsql              # minio bucket name, `pgsql` by default
        s3_key: pgbackrest            # minio user access key for pgbackrest [CHANGE ACCORDING to minio_users.pgbackrest]
        s3_key_secret: S3User.Backup  # minio user secret key for pgbackrest [CHANGE ACCORDING to minio_users.pgbackrest]
        s3_uri_style: path            # use path style uri for minio rather than host style
        path: /pgbackrest             # minio backup path, default is `/pgbackrest`
        storage_port: 9000            # minio port, 9000 by default
        storage_ca_file: /etc/pki/ca.crt  # minio ca file path, `/etc/pki/ca.crt` by default
        block: y                      # Enable block incremental backup
        bundle: y                     # bundle small files into a single file
        bundle_limit: 20MiB           # Limit for file bundles, 20MiB for object storage
        bundle_size: 128MiB           # Target size for file bundles, 128MiB for object storage
        cipher_type: aes-256-cbc      # enable AES encryption for remote backup repo
        cipher_pass: pgBackRest       # AES encryption password, default is 'pgBackRest'
        retention_full_type: time     # retention full backup by time on minio repo
        retention_full: 14            # keep full backup for the last 14 days
      s3:                             # you can use cloud object storage as backup repo
        type: s3                      # Add your object storage credentials here!
        s3_endpoint: oss-cn-beijing-internal.aliyuncs.com
        s3_region: oss-cn-beijing
        s3_bucket: <your_bucket_name>
        s3_key: <your_access_key>
        s3_key_secret: <your_secret_key>
        s3_uri_style: host
        path: /pgbackrest
        bundle: y                     # bundle small files into a single file
        bundle_limit: 20MiB           # Limit for file bundles, 20MiB for object storage
        bundle_size: 128MiB           # Target size for file bundles, 128MiB for object storage
        cipher_type: aes-256-cbc      # enable AES encryption for remote backup repo
        cipher_pass: pgBackRest       # AES encryption password, default is 'pgBackRest'
        retention_full_type: time     # retention full backup by time on minio repo
        retention_full: 14            # keep full backup for the last 14 days

    #----------------------------------------------#
    # PASSWORD : https://pigsty.io/docs/setup/security/
    #----------------------------------------------#
    grafana_admin_password: pigsty
    grafana_view_password: DBUser.Viewer
    pg_admin_password: DBUser.DBA
    pg_monitor_password: DBUser.Monitor
    pg_replication_password: DBUser.Replicator
    patroni_password: Patroni.API
    haproxy_admin_password: pigsty
    minio_secret_key: S3User.MinIO
    etcd_root_password: Etcd.Root
...

Explanation

The fat template is Pigsty’s full-featured test configuration, designed for completeness testing and offline package building.

Key Features:

  • All Extensions: Installs all categorized extension packages for PostgreSQL 18
  • Multi-version Repository: Local repo contains all five major versions of PostgreSQL 14-18
  • Complete Component Stack: Includes Silo backup, Docker applications, VIP, etc.
  • Enterprise Components: Includes Kafka, PolarDB, IvorySQL, TigerBeetle, etc.

Repository Contents:

Category Description
PostgreSQL 14-18 Five major versions’ kernels and all extensions
Extension Categories time, gis, rag, fts, olap, feat, lang, type, util, func, admin, stat, sec, fdw, sim, etl
Enterprise Components kafka-stack, Java Runtime, Sealos, TigerBeetle
Database Kernels PolarDB, IvorySQL

Differences from rich:

  • fat contains all five versions of PostgreSQL 14-18, rich only contains current default version
  • fat contains additional enterprise components (Kafka, PolarDB, IvorySQL, etc.)
  • fat requires larger disk space and longer build time

Use Cases:

  • Pigsty development testing and feature validation
  • Building complete multi-version offline software packages
  • Testing all extension compatibility scenarios
  • Enterprise environments pre-caching all software packages

Notes:

  • Requires large disk space (100GB+ recommended) for storing all packages
  • Building local software repository requires longer time
  • Some extensions unavailable on ARM64 architecture
  • Default passwords are sample passwords, must be changed for production

6.5 - infra

Only installs observability infrastructure, dedicated template without PostgreSQL and etcd

The infra configuration template only deploys Pigsty’s observability infrastructure components (VictoriaMetrics/Grafana/VictoriaLogs/Nginx, etc.), without PostgreSQL and etcd.

Suitable for scenarios requiring a standalone monitoring stack, such as monitoring external PostgreSQL/RDS instances or other data sources.

Overview

  • Config Name: infra
  • Node Count: Single or multiple nodes
  • Description: Only installs observability infrastructure, without PostgreSQL and etcd
  • OS Distro: el8, el9, el10, d12, d13, u22, u24, u26
  • OS Arch: x86_64, aarch64
  • Related: meta

Usage:

./configure -c infra [-i <primary_ip>]
./infra.yml    # Only execute infra playbook

Content

Source: pigsty/conf/infra.yml

---
#==============================================================#
# File      :   infra.yml
# Desc      :   Infra Only Config
# Ctime     :   2025-12-16
# Mtime     :   2025-12-30
# Docs      :   https://pigsty.io/docs/conf/infra
# License   :   Apache-2.0 @ https://pigsty.io/docs/about/license/
# Copyright :   2018-2026  Ruohang Feng / Vonng ([email protected])
#==============================================================#

# This is the config template for deploy victoria stack alone
# tutorial: https://pigsty.io/docs/infra
#
# Usage:
#   curl https://repo.pigsty.io/get | bash
#   ./configure -c infra
#   ./infra.yml

all:
  children:
    infra:
      hosts:
        10.10.10.10: { infra_seq: 1 }
        #10.10.10.11: { infra_seq: 2 } # you can add more nodes if you want
        #10.10.10.12: { infra_seq: 3 } # don't forget to assign unique infra_seq for each node
      vars:
        docker_enabled: true            # enabled docker with ./docker.yml
        docker_registry_mirrors: ["https://docker.1panel.live","https://docker.1ms.run","https://docker.xuanyuan.me","https://registry-1.docker.io"]
        pg_exporters:     # bin/pgmon-add pg-rds
          20001: { pg_cluster: pg-rds ,pg_seq: 1 ,pg_host: 10.10.10.10 ,pg_exporter_url: 'postgres://postgres:[email protected]:5432/postgres' }

  vars:                                 # global variables
    version: v4.5.0                     # pigsty version string
    admin_ip: 10.10.10.10               # admin node ip address
    region: default                     # upstream mirror region: default,china,europe
    node_tune: oltp                     # node tuning specs: oltp,olap,tiny,crit
    infra_portal:                       # infra services exposed via portal
      home : { domain: i.pigsty }       # default domain name
    repo_enabled: false                 # online installation without repo
    node_repo_modules: node,infra,pgsql # add these repos directly
    #haproxy_enabled: false              # enable haproxy on infra node?
    #vector_enabled: false               # enable vector on infra node?

    # DON't FORGET TO CHANGE DEFAULT PASSWORDS!
    grafana_admin_password: pigsty
    haproxy_admin_password: pigsty
...

Explanation

The infra template is Pigsty’s pure monitoring stack configuration, designed for standalone deployment of observability infrastructure.

Use Cases:

  • Monitoring external PostgreSQL instances (RDS, self-hosted, etc.)
  • Need standalone monitoring/alerting platform
  • Already have PostgreSQL clusters, only need to add monitoring
  • As a central console for multi-cluster monitoring

Included Components:

  • VictoriaMetrics: Time series database for storing metrics
  • VictoriaLogs: Log aggregation system
  • VictoriaTraces: Distributed tracing system
  • Grafana: Visualization dashboards
  • Alertmanager: Alert management
  • Nginx: Reverse proxy and web entry

Not Included:

  • PostgreSQL database cluster
  • etcd distributed coordination service
  • Silo object storage

Monitoring External Instances: After configuration, add monitoring for external PostgreSQL instances via the pgsql-monitor.yml playbook:

pg_exporters:
  20001: { pg_cluster: pg-foo, pg_seq: 1, pg_host: 10.10.10.100 }
  20002: { pg_cluster: pg-bar, pg_seq: 1, pg_host: 10.10.10.101 }

Notes:

  • This template will not install any databases
  • For full functionality, use meta or rich template
  • Can add multiple infra nodes for high availability as needed

6.6 - vibe

VIBE AI coding sandbox config template, integrating Code-Server, JupyterLab, Claude Code, Codex CLI, and JuiceFS

The vibe config template provides a ready-to-use AI coding sandbox, integrating Code-Server (Web VS Code), JupyterLab, Claude Code observability, Codex CLI, JuiceFS distributed filesystem, and a feature-rich PostgreSQL database.

Overview

  • Config Name: vibe
  • Node Count: Single node
  • Description: VIBE AI coding sandbox with Code-Server + JupyterLab + Claude Code + Codex CLI + JuiceFS + PostgreSQL
  • OS Distro: el8, el9, el10, d12, d13, u22, u24, u26
  • OS Arch: x86_64, aarch64
  • Related: meta

Usage:

./configure -c vibe [-i <primary_ip>]

Content

Source: pigsty/conf/vibe.yml

---
#==============================================================#
# File      :   vibe.yml
# Desc      :   Pigsty ai vibe coding sandbox
# Ctime     :   2026-01-19
# Mtime     :   2026-06-28
# Docs      :   https://pigsty.io/docs/conf/vibe
# License   :   Apache-2.0 @ https://pigsty.io/docs/about/license/
# Copyright :   2018-2026  Ruohang Feng / Vonng ([email protected])
#==============================================================#

# VIBE CODING SANDBOX
# PostgreSQL with related extensions
# Code-Server, Jupyter, Claude Code, optional Codex CLI
#
# Usage:
#   curl https://repo.pigsty.io/get | bash
#   ./configure -c vibe
#   ./deploy.yml
#   ./juice.yml     # pgfs: juicefs on pgsql, mount on /fs
#   ./vibe.yml      # code-server, jupyter, claude-code, and codex cli

all:
  children:
    infra: { hosts: { 10.10.10.10: { infra_seq: 1 }} ,vars: { repo_enabled: false }}
    etcd:  { hosts: { 10.10.10.10: { etcd_seq: 1  }} ,vars: { etcd_cluster: etcd  }}
    pgsql: { hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } } ,vars: { pg_cluster: pgsql }}

    # optional modules
    #minio: { hosts: { 10.10.10.10: { minio_seq: 1 }} ,vars: { minio_cluster: minio }}
    #redis-ms:
    #  hosts: { 10.10.10.10: { redis_node: 1 , redis_instances: { 6379: { }, 6380: { replica_of: '10.10.10.10 6379' } } } }
    #  vars: { redis_cluster: redis-ms ,redis_password: 'redis.ms' ,redis_max_memory: 64MB }

  vars:
    #----------------------------------------------#
    # INFRA: https://pigsty.io/docs/infra
    #----------------------------------------------#
    version: v4.5.0                     # pigsty version string
    admin_ip: 10.10.10.10               # admin node ip address
    region: default                     # upstream mirror region: default,china,europe
    infra_portal:                       # infra services exposed via portal
      home : { domain: i.pigsty }       # default domain name
    dns_enabled: false                  # disable dns service
    #blackbox_enabled: false            # disable blackbox exporter
    #alertmanager_enabled: false        # disable alertmanager
    infra_extra_services:               # home page navigation entries
      - { name: Code Server  ,url: '/code'             ,desc: 'VS Code Server'       ,icon: 'code'     }
      - { name: Jupyter      ,url: '/jupyter'          ,desc: 'Jupyter Notebook'     ,icon: 'jupyter'  }
      - { name: Claude Code  ,url: '/ui/d/claude-code' ,desc: 'Claude Observability' ,icon: 'claude'   }

    #----------------------------------------------#
    # NODE: https://pigsty.io/docs/node
    #----------------------------------------------#
    nodename_overwrite: false           # do not overwrite node hostname on single node mode
    node_tune: oltp                     # node tuning specs: oltp,olap,tiny,crit
    node_dns_method: none               # do not setup dns
    node_repo_modules: node,infra,pgsql # add these repos directly to the singleton node
    node_packages: [ openssh-server, juicefs, restic, rclone, uv, opencode, golang, asciinema, tmux ]
    docker_enabled: true                # enable docker service
    node_firewall_mode: zone            # default: trust intranet, expose selected public ports
    node_firewall_public_port: [22, 80, 443, 5432]    # expose 5432 for remote access, remove in production!
    #docker_registry_mirrors: ["https://docker.1panel.live","https://docker.1ms.run","https://docker.xuanyuan.me","https://registry-1.docker.io"]

    #----------------------------------------------#
    # PGSQL: https://pigsty.io/docs/pgsql
    #----------------------------------------------#
    pg_version: 18                      # Default PostgreSQL Major Version is 18
    pg_conf: oltp.yml                   # pgsql tuning specs: {oltp,olap,tiny,crit}.yml
    pg_packages: [ pgsql-main, patroni, pgbackrest, pg-exporter, pgbackrest-exporter ]
    pg_extensions: [ pg18-main ,pg18-time ,pg18-gis ,pg18-rag ,pg18-fts ,pg18-olap ,pg18-feat ,pg18-lang ,pg18-type ,pg18-util ,pg18-func ,pg18-admin ,pg18-stat ,pg18-sec ,pg18-fdw ,pg18-sim ,pg18-etl]
    pg_users:
      - { name: dbuser_meta ,password: DBUser.Meta   ,pgbouncer: true ,roles: [dbrole_admin   ] ,comment: pigsty admin user }
      - { name: dbuser_view ,password: DBUser.Viewer ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer  }
    pg_databases:
      - { name: meta, baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty] ,extensions: [ postgis, timescaledb, vector, age ]}
    pg_libs: 'timescaledb, pg_stat_statements, auto_explain, pg_wait_sampling'
    pg_hba_rules:
      - { user: all ,db: all ,addr: intra ,auth: pwd ,title: 'everyone intranet access with password' ,order: 800 }
      # WARNING: devbox only. Remove world access in production.
      - { user: all ,db: all ,addr: world ,auth: pwd ,title: 'everyone world access with password'    ,order: 900 }
    pg_crontab: [ '00 01 * * * /pg/bin/pg-backup full' ] # make a full backup every 1am
    patroni_mode: remove                # remove patroni after deployment
    pgbouncer_enabled: false            # disable pgbouncer pool
    pgbouncer_exporter_enabled: false   # disable pgbouncer_exporter on pgsql hosts?
    pgbackrest_exporter_enabled: false  # disable pgbackrest_exporter
    pg_default_services: []             # do not provision pg services
    #pg_reload: false                   # do not reload patroni/service

    #----------------------------------------------#
    # PASSWORD : https://pigsty.io/docs/setup/security/
    #----------------------------------------------#
    grafana_admin_password: pigsty
    grafana_view_password: DBUser.Viewer
    pg_admin_password: DBUser.DBA
    pg_monitor_password: DBUser.Monitor
    pg_replication_password: DBUser.Replicator
    patroni_password: Patroni.API
    haproxy_admin_password: pigsty

    #----------------------------------------------#
    # OPTIONAL VIBE COMPONENTS
    #----------------------------------------------#
    code_enabled: true                # install & enable code-server via vibe role
    code_password: DBUser.Meta
    jupyter_enabled: true             # enable jupyter (disabled by default, enable for vibe sandbox)
    jupyter_password: DBUser.Meta
    juice_instances:
      jfs:
        path  : /fs
        meta  : postgres://dbuser_meta:[email protected]:5432/meta
        data  : --storage postgres --bucket 10.10.10.10:5432/meta --access-key dbuser_meta --secret-key DBUser.Meta
        port  : 9567

    # Claude Code is the default coding agent. Node.js is installed on demand when agents are enabled.
    nodejs_enabled: true              # standalone nodejs runtime
    npm_packages: []                  # extra global npm packages
    codex_enabled: true
    claude_enabled: true
    #claude_env:                      # use 3rd party Anthropic-compatible API service
    #  ANTHROPIC_BASE_URL: https://open.bigmodel.cn/api/anthropic
    #  ANTHROPIC_API_URL: https://open.bigmodel.cn/api/anthropic
    #  ANTHROPIC_AUTH_TOKEN: your_api_service_token
    #  ANTHROPIC_DEFAULT_OPUS_MODEL: "glm-5.2[1m]"
    #  ANTHROPIC_DEFAULT_SONNET_MODEL: "glm-5.2[1m]"
    #  ANTHROPIC_DEFAULT_HAIKU_MODEL: "glm-4.7"
    #  CLAUDE_CODE_AUTO_COMPACT_WINDOW: "1000000"

...

Explanation

The vibe template is an AI-era Web coding sandbox, enabling development, data analysis, AI app building all in browser.

Core Components:

Component Description Access Method
Code-Server Web version of VS Code, full-featured code editor http://<ip>/code
JupyterLab Interactive data science notebook, Python/SQL http://<ip>/jupyter
Claude Code AI coding runtime and observability entrypoint (claude_env customizable) Terminal / Dashboard
Codex CLI OpenAI agentic coding CLI; VIBE installs it but does not manage its configuration Terminal
JuiceFS PostgreSQL-based distributed filesystem Mount point /fs
PostgreSQL 18 Feature-rich database with pg18-main + categorized extension package groups Port 5432

Node tools explicitly installed by this template (node_packages):

  • openssh-server, juicefs, restic, rclone
  • uv, opencode, golang
  • asciinema, tmux

PostgreSQL Extensions:

This template installs PostgreSQL 18 extension groups by category:

pg18-main, pg18-time, pg18-gis, pg18-rag, pg18-fts, pg18-olap,
pg18-feat, pg18-lang, pg18-type, pg18-util, pg18-func, pg18-admin,
pg18-stat, pg18-sec, pg18-fdw, pg18-sim, pg18-etl

By default, the meta database enables postgis, timescaledb, and vector; other extensions can be enabled as needed.

VIBE Module Components

The VIBE module provides AI coding sandbox capability; vibe.yml explicitly enables Code-Server and Jupyter and installs Claude Code and Codex CLI by default.

Code-Server: VS Code in browser

  • Full VS Code functionality, extension support
  • HTTPS access via Nginx reverse proxy
  • Supports Open VSX and Microsoft extension marketplaces
  • Explicit template params: code_enabled, code_password
  • Optional params: code_port, code_data, code_gallery

JupyterLab: Interactive computing environment

  • Python/SQL/Markdown notebook support
  • Pre-configured Python venv with data science libraries
  • HTTPS access via Nginx reverse proxy
  • Explicit template params: jupyter_enabled, jupyter_password
  • Optional params: jupyter_port, jupyter_data, jupyter_venv

Claude Code: AI coding assistant runtime

  • Uses module default behavior to bootstrap Claude runtime
  • Supports endpoint/API key overrides through claude_env
  • Provides claude-code dashboard for usage monitoring

Codex CLI: AI coding assistant

  • Controlled by codex_enabled, which defaults to true
  • VIBE installs @openai/codex only; it does not write Codex configuration or connect Codex to the Claude Code dashboard

JuiceFS Filesystem

This template uses JuiceFS for distributed filesystem capability, with a special feature: both metadata and data stored in PostgreSQL.

Architecture Features:

  • Metadata Engine: Uses PostgreSQL for filesystem metadata storage
  • Data Storage: Uses PostgreSQL Large Object for file data storage
  • Mount Point: Default mount at /fs (controlled by juice_instances.jfs.path)
  • Monitoring Port: 9567 provides Prometheus metrics

Use Cases:

  • Persistent storage for code projects
  • Working directory for Jupyter Notebooks
  • Storage for AI models and datasets
  • File sharing across instances (when scaled to multiple nodes)

Config Example:

juice_instances:
  jfs:
    path  : /fs
    meta  : postgres://dbuser_meta:[email protected]:5432/meta
    data  : --storage postgres --bucket 10.10.10.10:5432/meta --access-key dbuser_meta --secret-key DBUser.Meta
    port  : 9567

Deployment Steps 

# 1. Download Pigsty
curl -fsSL https://repo.pigsty.io/get | bash; cd ~/pigsty

# 2. Use vibe config template
./configure -c vibe

# 3. Modify passwords (important!)
vi pigsty.yml
# Change code_password, jupyter_password, database, and infrastructure defaults

# 4. Deploy infrastructure and PostgreSQL
./deploy.yml

# 5. Optional: deploy the JuiceFS filesystem
./juice.yml -l 10.10.10.10

# 6. Deploy VIBE (Code-Server, JupyterLab, Claude Code, Codex CLI)
./vibe.yml -l 10.10.10.10

Access Methods

After deployment, access via browser:

# Code-Server (VS Code Web)
https://<domain>/code/
# Use the rotated code_password

# JupyterLab
https://<domain>/jupyter/
# Use the rotated jupyter_password token

# Claude Code Dashboard
https://<domain>/ui/d/claude-code
# Use the rotated Grafana administrator credentials

# PostgreSQL
psql 'host=<ip> port=5432 dbname=meta user=dbuser_meta sslmode=require'

Use Cases

  • AI App Development: Build RAG, Agent, LLM applications
  • Data Science: Use JupyterLab for data analysis and visualization
  • Remote Development: Setup Web IDE environment on cloud servers
  • Teaching Demos: Provide consistent dev environment for students
  • Rapid Prototyping: Quickly validate ideas without local env setup
  • Claude Code Observability: Monitor AI coding assistant usage

Notes

  • Must change passwords: code_password and jupyter_password defaults are for testing only
  • Jupyter boundary: The template listens on 0.0.0.0:8888, allows any Origin, disables XSRF checks, and relies on the token by default; restrict the port and portal sources and never expose it directly to the Internet
  • Network security: This template exposes 5432 (node_firewall_public_port) and includes addr: world HBA by default; remove those public paths for production and add portal Basic Auth when appropriate
  • Resource requirements: Recommend at least 2 cores 4GB memory, SSD disk
  • Simplified architecture: This template disables Patroni, PgBouncer etc HA components, suitable for single-node dev env
  • Claude API: Using Claude Code requires configuring API key in claude_env

6.7 - docker

Pigsty Docker single-node template for quickly bootstrapping Pigsty in containers.

The docker configuration template runs Pigsty inside a Docker container and provides a minimal single-node stack for infrastructure and PostgreSQL.

For full workflow details, see Docker Deployment.

Overview

  • Config Name: docker
  • Node Count: Single node (container runtime)
  • Description: Quick-start container template using 127.0.0.1 and trimmed system capabilities for Docker scenarios
  • OS Distro: Container image runtime (official Pigsty Docker image recommended)
  • OS Arch: x86_64, aarch64
  • Related: meta, vibe

Usage:

./configure -c docker -i 127.0.0.1 -g

Content

Source: pigsty/conf/docker.yml

---
#==============================================================#
# File      :   docker.yml
# Desc      :   Pigsty docker coding environment
# Ctime     :   2026-01-19
# Mtime     :   2026-01-27
# Docs      :   https://pigsty.io/docs/conf/docker
# License   :   Apache-2.0 @ https://pigsty.io/docs/about/license/
# Copyright :   2018-2026  Ruohang Feng / Vonng ([email protected])
#==============================================================#

# DOCKER CONFIG, use 127.0.0.1 inside docker
# mount the /data volume when running docker container
#
# Usage:
#   curl https://repo.pigsty.io/get | bash
#   ./configure -c docker -i 127.0.0.1 -g
#   ./deploy.yml

all:
  children:
    infra: { hosts: { 10.10.10.10: { infra_seq: 1 }} ,vars: { repo_enabled: false }}
    etcd:  { hosts: { 10.10.10.10: { etcd_seq: 1  }} ,vars: { etcd_cluster: etcd  }}
    pgsql: { hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary  }} ,vars: { pg_cluster: pgsql }}
    #minio: { hosts: { 10.10.10.10: { minio_seq: 1 }} ,vars: { minio_cluster: minio }}

  vars:

    #----------------------------------------------#
    # Infra
    #----------------------------------------------#
    version: v4.5.0                   # pigsty version string
    admin_ip: 10.10.10.10               # admin node ip address
    region: china                     # upstream mirror region: default|china|europe
    dns_enabled: false                # disable dnsmasq service on single node
    infra_portal:
      home : { domain: i.pigsty }
    proxy_env:                        # global proxy env when downloading packages
      no_proxy: "localhost,10.10.10.10,10.0.0.0/8,192.168.0.0/16,*.pigsty,*.aliyun.com,mirrors.*,*.myqcloud.com,*.tsinghua.edu.cn"
      # http_proxy:  # set your proxy here: e.g http://user:[email protected]
      # https_proxy: # set your proxy here: e.g http://user:[email protected]
      # all_proxy:   # set your proxy here: e.g http://user:[email protected]

    #----------------------------------------------#
    # Node
    #----------------------------------------------#
    nodename: pigsty
    node_id_from_pg: false
    node_tune: oltp
    node_write_etc_hosts: false
    node_dns_method: none
    node_ntp_enabled: false
    node_kernel_modules: []
    node_repo_remove: true
    node_repo_modules: 'node,infra,pgsql'


    #----------------------------------------------#
    # PGSQL: https://pigsty.io/docs/pgsql
    #----------------------------------------------#
    pg_version: 18                      # Default PostgreSQL Major Version is 18
    pg_conf: oltp.yml                   # pgsql tuning specs: {oltp,olap,tiny,crit}.yml
    pg_extensions: [ pg18-main ,pg18-time ,pg18-gis ,pg18-rag ,pg18-fts ,pg18-olap ,pg18-feat ,pg18-lang ,pg18-type ,pg18-util ,pg18-func ,pg18-admin ,pg18-stat ,pg18-sec ,pg18-fdw ,pg18-sim ,pg18-etl]
    pg_users:
      - { name: dbuser_meta ,password: DBUser.Meta   ,pgbouncer: true ,roles: [dbrole_admin   ] ,comment: pigsty admin user }
      - { name: dbuser_view ,password: DBUser.Viewer ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer  }
    pg_databases:
      - { name: meta, baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty] ,extensions: [ postgis, timescaledb, vector ]}
    pg_libs: 'timescaledb, pg_stat_statements, auto_explain, pg_wait_sampling'
    pg_hba_rules:
      - { user: all ,db: all ,addr: intra ,auth: pwd ,title: 'everyone intranet access with password' ,order: 800 }
      - { user: all ,db: all ,addr: world ,auth: pwd ,title: 'everyone world access with password'    ,order: 900 }
    pg_crontab: [ '00 01 * * * /pg/bin/pg-backup full' ] # make a full backup every 1am
    #pg_reload: false                   # do not reload patroni/service

    #----------------------------------------------#
    # PASSWORD : https://pigsty.io/docs/setup/security/
    #----------------------------------------------#
    grafana_admin_password: pigsty
    grafana_view_password: DBUser.Viewer
    pg_admin_password: DBUser.DBA
    pg_monitor_password: DBUser.Monitor
    pg_replication_password: DBUser.Replicator
    patroni_password: Patroni.API
    haproxy_admin_password: pigsty
    minio_secret_key: S3User.MinIO
    etcd_root_password: Etcd.Root

    #----------------------------------------------#
    # OPTIONAL
    #----------------------------------------------#
    #code_password: DBUser.Meta
    #jupyter_password: DBUser.Meta
    #juice_instances:  # dict of juicefs filesystems to deploy
    #  jfs:
    #    path  : /fs
    #    meta  : postgres://dbuser_meta:[email protected]:5432/meta
    #    data  : --storage postgres --bucket 10.10.10.10:5432/meta --access-key dbuser_meta --secret-key DBUser.Meta
    #    port  : 9567
    #node_packages: [ openssh-server, tmux, juicefs, restic, rclone, uv, code-server ]
    #npm_packages: [ '@anthropic-ai/claude-code' , 'happy-coder' ]
    #claude_env:
    #  ANTHROPIC_BASE_URL: https://open.bigmodel.cn/api/anthropic
    #  ANTHROPIC_API_URL: https://open.bigmodel.cn/api/anthropic
    #  ANTHROPIC_AUTH_TOKEN: your_api_service_token
    #  ANTHROPIC_MODEL: glm-4.7
    #  ANTHROPIC_SMALL_FAST_MODEL: glm-4.5-air
...

Explanation

The docker template is optimized for development and validation inside containers.

Key Features:

  • Disables local repo build (repo_enabled: false) to avoid extra build overhead in containers
  • Simplifies node behavior by disabling NTP, kernel module loading, and /etc/hosts rewrite
  • Uses PostgreSQL 18 by default with a broad preset extension package bundle (pg18-*)
  • Allows password access from both intra and world ranges in pg_hba_rules for fast testing
  • Keeps optional capabilities (Code-Server, Jupyter, JuiceFS, Claude CLI) as commented settings

Notes:

  • This template is designed for development and demos; tighten pg_hba_rules and password policy for production
  • Mount /data in the container runtime to persist PostgreSQL and component data

6.8 - pgsql

Native PostgreSQL kernel with stable support for PostgreSQL 14 to 18 and a PG19 Beta evaluation option

The pgsql configuration template uses the native PostgreSQL kernel, Pigsty’s default database kernel, with stable support for PostgreSQL 14 to 18. The current configure also accepts version 19, but PG19 remains Beta; use the dedicated pg19 template for evaluation.

Overview

  • Config Name: pgsql
  • Node Count: Single node
  • Description: Native PostgreSQL kernel configuration template
  • OS Distro: el8, el9, el10, d12, d13, u22, u24, u26
  • OS Arch: x86_64, aarch64
  • Related: meta

Usage:

./configure -c pgsql [-i <primary_ip>]

To specify a non-default PostgreSQL version (e.g., 16):

./configure -c pgsql -v 16

Content

Source: pigsty/conf/pgsql.yml

---
#==============================================================#
# File      :   pgsql.yml
# Desc      :   1-node PostgreSQL Config template
# Ctime     :   2025-02-23
# Mtime     :   2025-12-28
# Docs      :   https://pigsty.io/docs/conf/pgsql
# License   :   Apache-2.0 @ https://pigsty.io/docs/about/license/
# Copyright :   2018-2026  Ruohang Feng / Vonng ([email protected])
#==============================================================#

# This is the config template for basical PostgreSQL Kernel.
# Nothing special, just a basic setup with one node.
# tutorial: https://pigsty.io/docs/pgsql/kernel/postgres
#
# Usage:
#   curl https://repo.pigsty.io/get | bash
#   ./configure -c pgsql
#   ./deploy.yml

all:
  children:
    infra: { hosts: { 10.10.10.10: { infra_seq: 1 }} ,vars: { repo_enabled: false }}
    etcd:  { hosts: { 10.10.10.10: { etcd_seq: 1  }} ,vars: { etcd_cluster: etcd  }}
    #minio: { hosts: { 10.10.10.10: { minio_seq: 1 }} ,vars: { minio_cluster: minio }}

    #----------------------------------------------#
    # PostgreSQL Cluster
    #----------------------------------------------#
    pg-meta:
      hosts:
        10.10.10.10: { pg_seq: 1, pg_role: primary }
      vars:
        pg_cluster: pg-meta
        pg_users:
          - { name: dbuser_meta ,password: DBUser.Meta   ,pgbouncer: true ,roles: [dbrole_admin   ] ,comment: pigsty admin user }
          - { name: dbuser_view ,password: DBUser.Viewer ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer  }
        pg_databases:
          - { name: meta, baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty] ,extensions: [ postgis, timescaledb, vector ]}
        pg_extensions: [ postgis, timescaledb, pgvector, pg_wait_sampling ]
        pg_libs: 'timescaledb, pg_stat_statements, auto_explain, pg_wait_sampling'
        pg_hba_rules:   # https://pigsty.io/docs/pgsql/config/hba
          - { user: all ,db: all ,addr: intra ,auth: pwd ,title: 'everyone intranet access with password' ,order: 800 }
        pg_crontab:     # https://pigsty.io/docs/pgsql/admin/crontab
          - '00 01 * * * /pg/bin/pg-backup full'

  vars:
    #----------------------------------------------#
    # INFRA : https://pigsty.io/docs/infra/param
    #----------------------------------------------#
    version: v4.5.0                   # pigsty version string
    admin_ip: 10.10.10.10             # admin node ip address
    region: default                   # upstream mirror region: default,china,europe
    infra_portal:                     # infra services exposed via portal
      home : { domain: i.pigsty }     # default domain name

    #----------------------------------------------#
    # NODE : https://pigsty.io/docs/node/param
    #----------------------------------------------#
    nodename_overwrite: false             # do not overwrite node hostname on single node mode
    node_repo_modules: node,infra,pgsql # add these repos directly to the singleton node
    node_tune: oltp                     # node tuning specs: oltp,olap,tiny,crit

    #----------------------------------------------#
    # PGSQL : https://pigsty.io/docs/pgsql/param
    #----------------------------------------------#
    pg_version: 18                      # Default PostgreSQL Major Version is 18
    pg_conf: oltp.yml                   # pgsql tuning specs: {oltp,olap,tiny,crit}.yml
    pg_packages: [ pgsql-main, pgsql-common ]   # pg kernel and common utils
    #pg_extensions: [ pg18-time ,pg18-gis ,pg18-rag ,pg18-fts ,pg18-olap ,pg18-feat ,pg18-lang ,pg18-type ,pg18-util ,pg18-func ,pg18-admin ,pg18-stat ,pg18-sec ,pg18-fdw ,pg18-sim ,pg18-etl]
    #repo_extra_packages: [ pg18-main ,pg18-time ,pg18-gis ,pg18-rag ,pg18-fts ,pg18-olap ,pg18-feat ,pg18-lang ,pg18-type ,pg18-util ,pg18-func ,pg18-admin ,pg18-stat ,pg18-sec ,pg18-fdw ,pg18-sim ,pg18-etl]

    #----------------------------------------------#
    # PASSWORD : https://pigsty.io/docs/setup/security/
    #----------------------------------------------#
    grafana_admin_password: pigsty
    grafana_view_password: DBUser.Viewer
    pg_admin_password: DBUser.DBA
    pg_monitor_password: DBUser.Monitor
    pg_replication_password: DBUser.Replicator
    patroni_password: Patroni.API
    haproxy_admin_password: pigsty
    minio_secret_key: S3User.MinIO
    etcd_root_password: Etcd.Root
...

Explanation

The pgsql template is Pigsty’s standard kernel configuration, using community-native PostgreSQL.

Version Support:

  • PostgreSQL 18 (default)
  • PostgreSQL 17, 16, 15, 14
  • PostgreSQL 19 Beta (evaluation; use ./configure -c pg19)

Use Cases:

  • Need to use the latest PostgreSQL features
  • Need the widest extension support
  • Standard production environment deployment
  • Same functionality as meta template, explicitly declaring native kernel usage

Differences from meta:

  • pgsql template explicitly declares using native PostgreSQL kernel
  • Suitable for scenarios needing clear distinction between different kernel types

6.9 - pg19

Single-node PostgreSQL 19 Beta evaluation template with the PGDG Beta repository and default backup capabilities

pg19 is the single-node PostgreSQL 19 Beta evaluation template. It follows the meta topology, enables the beta repository, and limits the local repository’s additional cache to core PGSQL packages without preinstalling extensions.

Overview

  • Config Name: pg19
  • Node Count: Single node
  • PostgreSQL Version: 19 Beta
  • Use Cases: New-version evaluation and compatibility testing
  • Related: meta, pgsql

Usage:

./configure -c pg19 [-i <primary_ip>]

Content

Source: pigsty/conf/pg19.yml

---
#==============================================================#
# File      :   pg19.yml
# Desc      :   Pigsty 1-node PostgreSQL 19 beta config
# Ctime     :   2026-06-11
# Mtime     :   2026-06-11
# Docs      :   https://pigsty.io/docs/conf/pg19
# License   :   Apache-2.0 @ https://pigsty.io/docs/about/license/
# Copyright :   2018-2026  Ruohang Feng / Vonng ([email protected])
#==============================================================#

# This is the PostgreSQL 19 beta variant of meta.yml.
# It enables the PGDG beta repository and installs a minimal PG19 runtime.
#
# Usage:
#   curl https://repo.pigsty.io/get | bash
#   ./configure -c pg19
#   ./deploy.yml

all:

  #==============================================================#
  # Clusters, Nodes, and Modules
  #==============================================================#
  children:

    #----------------------------------------------#
    # PGSQL : https://pigsty.io/docs/pgsql
    #----------------------------------------------#
    pg-meta:
      hosts:
        10.10.10.10: { pg_seq: 1, pg_role: primary }
        #x.xx.xx.xx: { pg_seq: 2, pg_role: replica }
        #x.xx.xx.xy: { pg_seq: 3, pg_role: offline }
      vars:
        pg_cluster: pg-meta

        # PG19 is beta; extension packages are intentionally not installed here.
        pg_extensions: []

        # define business users/roles : https://pigsty.io/docs/pgsql/config/user
        pg_users:
          - { name: dbuser_meta ,password: DBUser.Meta   ,pgbouncer: true ,roles: [dbrole_admin   ] ,comment: pigsty admin user }
          - { name: dbuser_view ,password: DBUser.Viewer ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer  }

        # define business databases : https://pigsty.io/docs/pgsql/config/db
        pg_databases:
          - { name: meta, baseline: cmdb.sql, comment: "pigsty meta database", schemas: [pigsty] }

        pg_hba_rules:   # https://pigsty.io/docs/pgsql/config/hba
          - { user: all ,db: all ,addr: intra ,auth: pwd ,title: 'everyone intranet access with password' ,order: 800 }

        pg_crontab:     # make a full backup every day at 1am
          - '00 01 * * * /pg/bin/pg-backup full'

        # define (OPTIONAL) L2 VIP that bind to primary
        #pg_vip_enabled: true
        #pg_vip_address: 10.10.10.2/24


    #----------------------------------------------#
    # INFRA : https://pigsty.io/docs/infra
    #----------------------------------------------#
    infra:
      hosts:
        10.10.10.10: { infra_seq: 1 }
      vars:
        repo_enabled: false   # disable local repo in 1-node mode
        #repo_extra_packages: [ pgsql-core ]  # if local repo is enabled, mirror PG19 beta core packages

    #----------------------------------------------#
    # ETCD : https://pigsty.io/docs/etcd
    #----------------------------------------------#
    etcd:
      hosts:
        10.10.10.10: { etcd_seq: 1 }
      vars:
        etcd_cluster: etcd
        etcd_safeguard: false

    #----------------------------------------------#
    # MINIO : https://pigsty.io/docs/minio
    #----------------------------------------------#
    #minio:
    #  hosts:
    #    10.10.10.10: { minio_seq: 1 }
    #  vars:
    #    minio_cluster: minio
    #    minio_users:
    #      - { access_key: pgbackrest  ,secret_key: S3User.Backup ,policy: pgsql }
    #      - { access_key: s3user_meta ,secret_key: S3User.Meta   ,policy: meta  }
    #      - { access_key: s3user_data ,secret_key: S3User.Data   ,policy: data  }

    #----------------------------------------------#
    # DOCKER : https://pigsty.io/docs/docker
    # APP    : https://pigsty.io/docs/app
    #----------------------------------------------#
    app:
      hosts: { 10.10.10.10: {} }
      vars:
        docker_enabled: true
        #docker_registry_mirrors: ["https://docker.1panel.live","https://docker.1ms.run","https://docker.xuanyuan.me","https://registry-1.docker.io"]
        app: pgadmin
        apps:
          pgadmin:
            conf:
              PGADMIN_DEFAULT_EMAIL: [email protected]
              PGADMIN_DEFAULT_PASSWORD: pigsty


  #==============================================================#
  # Global Parameters
  #==============================================================#
  vars:

    #----------------------------------------------#
    # INFRA : https://pigsty.io/docs/infra
    #----------------------------------------------#
    version: v4.5.0                   # pigsty version string
    admin_ip: 10.10.10.10             # admin node ip address
    region: default                   # upstream mirror region: default|china|europe
    proxy_env:
      no_proxy: "localhost,127.0.0.1,10.0.0.0/8,192.168.0.0/16,*.pigsty,*.aliyun.com,mirrors.*,*.myqcloud.com,*.tsinghua.edu.cn"
      # http_proxy:  # set your proxy here: e.g http://user:[email protected]
      # https_proxy: # set your proxy here: e.g http://user:[email protected]
      # all_proxy:   # set your proxy here: e.g http://user:[email protected]
    infra_portal:
      home : { domain: i.pigsty }
      pgadmin : { domain: adm.pigsty ,endpoint: "${admin_ip}:8885" }
      #minio  : { domain: m.pigsty ,endpoint: "${admin_ip}:9001" ,scheme: https ,websocket: true }

    #----------------------------------------------#
    # NODE : https://pigsty.io/docs/node/param
    #----------------------------------------------#
    nodename_overwrite: false             # do not overwrite node hostname on single node mode
    node_tune: oltp                       # node tuning specs: oltp,olap,tiny,crit
    node_etc_hosts: [ '${admin_ip} i.pigsty sss.pigsty' ]
    node_repo_modules: 'node,infra,pgsql,beta' # PG19 beta packages come from PGDG testing repo
    #node_repo_modules: local             # use this if you want to build & use local repo
    node_repo_remove: true                # remove existing node repo for node managed by pigsty
    node_firewall_public_port: [22, 80, 443, 5432]

    #----------------------------------------------#
    # PGSQL : https://pigsty.io/docs/pgsql/param
    #----------------------------------------------#
    pg_version: 19                      # PostgreSQL 19 beta
    pg_conf: oltp.yml                   # pgsql tuning specs: {oltp,olap,tiny,crit}.yml
    pg_safeguard: false                 # prevent purging running postgres instance?
    pg_extensions: []                   # do not install extension packages during PG19 beta trial
    repo_modules: node,infra,pgsql,beta # add PGDG testing repo when building a local repo
    repo_extra_packages: [ pgsql-core ] # only mirror PG19 beta core packages

    #----------------------------------------------#
    # BACKUP : https://pigsty.io/docs/pgsql/backup
    #----------------------------------------------#
    pgbackrest_enabled: true            # pgBackRest 2.59 supports PostgreSQL 19 beta2
    pgbackrest_exporter_enabled: true   # expose pgBackRest metrics to the monitoring system
    #pgbackrest_method: minio
    #pgbackrest_repo:
    #  minio:
    #    type: s3
    #    s3_endpoint: sss.pigsty
    #    s3_region: us-east-1
    #    s3_bucket: pgsql
    #    s3_key: pgbackrest
    #    s3_key_secret: S3User.Backup
    #    s3_uri_style: path
    #    path: /pgbackrest
    #    storage_port: 9000
    #    storage_ca_file: /etc/pki/ca.crt
    #    bundle: y
    #    bundle_limit: 20MiB
    #    bundle_size: 128MiB
    #    cipher_type: aes-256-cbc
    #    cipher_pass: pgBackRest
    #    retention_full_type: time
    #    retention_full: 14

    #----------------------------------------------#
    # PASSWORD : https://pigsty.io/docs/setup/security/
    #----------------------------------------------#
    grafana_admin_password: pigsty
    grafana_view_password: DBUser.Viewer
    pg_admin_password: DBUser.DBA
    pg_monitor_password: DBUser.Monitor
    pg_replication_password: DBUser.Replicator
    patroni_password: Patroni.API
    haproxy_admin_password: pigsty
    minio_secret_key: S3User.MinIO
    etcd_root_password: Etcd.Root
...

Explanation

Important defaults and limitations:

  • node_repo_modules: node,infra,pgsql,beta obtains PG19 packages from the PGDG Beta repository
  • repo_extra_packages: [pgsql-core] limits the local repository’s additional cache to core PGSQL packages; instances still use the role’s default pgsql-main pgsql-common installation set
  • pg_extensions: [] installs no extension packages
  • pgbackrest_enabled: true and pgbackrest_exporter_enabled: true; pg-meta retains its daily 01:00 full-backup job
  • INFRA, ETCD, PGSQL, and optional pgAdmin remain available for a single-node evaluation

This is a Beta evaluation configuration, not a production template. Do not treat -v 19 on an ordinary template as a production-ready PG19 deployment; validate extension compatibility, backup and recovery, and upgrade procedures separately.

6.10 - mssql

Babelfish template pinned to a PostgreSQL 17-compatible kernel with SQL Server protocol and T-SQL support

The mssql configuration template uses a PostgreSQL 17-compatible Babelfish kernel instead of native PostgreSQL, providing Microsoft SQL Server wire protocol (TDS) and T-SQL syntax compatibility. The current template is pinned to pg_version: 17; configure does not apply -v overrides to this fixed-kernel template.

Since Pigsty v4.2, Babelfish is built directly by Pigsty, no longer using the WiltonDB repository, and is available on all supported Linux platforms.

For the complete tutorial, see: Babelfish (MSSQL) Kernel Guide

Overview

  • Config Name: mssql
  • Node Count: Single node
  • Description: Babelfish (PG17) configuration template with SQL Server protocol compatibility
  • OS Distro: el8, el9, el10, d12, d13, u22, u24, u26
  • OS Arch: x86_64, aarch64
  • Related: meta

Usage:

./configure -c mssql [-i <primary_ip>]

Content

Source: pigsty/conf/mssql.yml

---
#==============================================================#
# File      :   mssql.yml
# Desc      :   Babelfish (MSSQL Wire-Compatible) template
# Ctime     :   2020-08-01
# Mtime     :   2026-07-08
# Docs      :   https://pigsty.io/docs/conf/mssql
# License   :   Apache-2.0 @ https://pigsty.io/docs/about/license/
# Copyright :   2018-2026  Ruohang Feng / Vonng ([email protected])
#==============================================================#

# This is the config template for Babelfish Kernel made by Pigsty
# Which is a PostgreSQL 17/18 fork with SQL Server Compatibility
# tutorial: https://pigsty.io/docs/pgsql/kernel/babelfish
#
# Usage:
#   curl https://repo.pigsty.io/get | bash
#   ./configure -c mssql [-v 17/18]
#   ./deploy.yml

all:
  children:
    infra: { hosts: { 10.10.10.10: { infra_seq: 1 }} ,vars: { repo_enabled: false }}
    etcd:  { hosts: { 10.10.10.10: { etcd_seq: 1  }} ,vars: { etcd_cluster: etcd  }}
    #minio: { hosts: { 10.10.10.10: { minio_seq: 1 }} ,vars: { minio_cluster: minio }}

    #----------------------------------------------#
    # Babelfish Database Cluster
    #----------------------------------------------#
    pg-meta:
      hosts:
        10.10.10.10: { pg_seq: 1, pg_role: primary }
      vars:
        pg_cluster: pg-meta
        pg_users:
          - {name: dbuser_mssql ,password: DBUser.MSSQL ,superuser: true, pgbouncer: true ,roles: [dbrole_admin], comment: superuser & owner for babelfish  }
        pg_databases:
          - name: mssql
            baseline: mssql.sql
            extensions: [uuid-ossp, babelfishpg_common, babelfishpg_tsql, babelfishpg_tds, babelfishpg_money ]
            owner: dbuser_mssql
            parameters: { 'babelfishpg_tsql.migration_mode' : 'multi-db' }
            comment: babelfish cluster, a MSSQL compatible pg cluster
        pg_crontab:     # https://pigsty.io/docs/pgsql/admin/crontab
          - '00 01 * * * /pg/bin/pg-backup full'

        # Babelfish Ad Hoc Settings
        pg_mode: mssql                     # Microsoft SQL Server Compatible Mode
        pg_version: 17
        pg_packages: [ babelfish, pgsql-common, sqlcmd ]
        pg_libs: 'babelfishpg_tds, pg_stat_statements, auto_explain' # preload Babelfish TDS listener
        pg_hba_rules:   # https://pigsty.io/docs/pgsql/config/hba
          - { user: dbuser_mssql ,db: mssql ,addr: intra ,auth: md5 ,title: 'allow mssql dbsu intranet access'      ,order: 525 } # <--- use md5 auth method for mssql user
          - { user: all          ,db: all   ,addr: intra ,auth: md5 ,title: 'everyone intranet access with md5 pwd' ,order: 800 }
        pg_default_services: # route primary & replica service to mssql port 1433
          - { name: primary ,port: 5433 ,dest: 1433  ,check: /primary   ,selector: "[]" }
          - { name: replica ,port: 5434 ,dest: 1433  ,check: /read-only ,selector: "[]" , backup: "[? pg_role == `primary` || pg_role == `offline` ]" }
          - { name: default ,port: 5436 ,dest: postgres ,check: /primary   ,selector: "[]" }
          - { name: offline ,port: 5438 ,dest: postgres ,check: /replica   ,selector: "[? pg_role == `offline` || pg_offline_query ]" , backup: "[? pg_role == `replica` && !pg_offline_query]" }

  vars:
    #----------------------------------------------#
    # INFRA : https://pigsty.io/docs/infra/param
    #----------------------------------------------#
    version: v4.5.0                   # pigsty version string
    admin_ip: 10.10.10.10             # admin node ip address
    region: default                   # upstream mirror region: default,china,europe
    infra_portal:                     # infra services exposed via portal
      home : { domain: i.pigsty }     # default domain name

    #----------------------------------------------#
    # NODE : https://pigsty.io/docs/node/param
    #----------------------------------------------#
    nodename_overwrite: false                 # do not overwrite node hostname on single node mode
    node_repo_modules: node,infra,pgsql       # babelfish kernel is in the pgsql repo
    node_tune: oltp                           # node tuning specs: oltp,olap,tiny,crit

    #----------------------------------------------#
    # PGSQL : https://pigsty.io/docs/pgsql/param
    #----------------------------------------------#
    pg_version: 17                            # Babelfish kernel is compatible with postgres 17
    pg_conf: oltp.yml                         # pgsql tuning specs: {oltp,olap,tiny,crit}.yml

    #----------------------------------------------#
    # PASSWORD : https://pigsty.io/docs/setup/security/
    #----------------------------------------------#
    grafana_admin_password: pigsty
    grafana_view_password: DBUser.Viewer
    pg_admin_password: DBUser.DBA
    pg_monitor_password: DBUser.Monitor
    pg_replication_password: DBUser.Replicator
    patroni_password: Patroni.API
    haproxy_admin_password: pigsty
    minio_secret_key: S3User.MinIO
    etcd_root_password: Etcd.Root
...

Explanation

The mssql template allows you to use SQL Server Management Studio (SSMS) or other SQL Server client tools to connect to PostgreSQL (through Babelfish protocol compatibility).

Key Features:

  • Uses TDS protocol (port 1433), compatible with SQL Server clients
  • Supports T-SQL syntax, low migration cost
  • Retains PostgreSQL’s ACID properties and extension ecosystem (the current template uses PG17)
  • Supports multi-db and single-db migration modes
  • Default package set: babelfish + pgsql-common + sqlcmd
  • Creates uuid-ossp, babelfishpg_common, babelfishpg_tsql, babelfishpg_tds, and babelfishpg_money by default
  • v4.2.0 adds full mainstream platform coverage (EL 8/9/10, Debian 12/13, Ubuntu 22/24/26; x86_64 / aarch64)

Connection Methods:

# Using sqlcmd command line tool
sqlcmd -S 10.10.10.10,1433 -U dbuser_mssql -P DBUser.MSSQL -d mssql

# Using SSMS or Azure Data Studio
# Server: 10.10.10.10,1433
# Authentication: SQL Server Authentication
# Login: dbuser_mssql
# Password: DBUser.MSSQL

Use Cases:

  • Migrating from SQL Server to PostgreSQL
  • Applications needing to support both SQL Server and PostgreSQL clients
  • Leveraging PostgreSQL ecosystem while maintaining T-SQL compatibility

Notes:

  • The current mssql template is pinned to a PostgreSQL 17-compatible kernel; do not rely on -v to switch its major version
  • Default migration mode is multi-db (babelfishpg_tsql.migration_mode), configurable to single-db when needed
  • Some T-SQL syntax may have compatibility differences, refer to Babelfish compatibility documentation
  • Must use md5 authentication method (not scram-sha-256)

6.11 - polar

PolarDB for PostgreSQL kernel, provides Aurora-style storage-compute separation capability

The polar configuration template uses Alibaba Cloud’s PolarDB for PostgreSQL database kernel instead of native PostgreSQL, providing “cloud-native” Aurora-style storage-compute separation capability.

For the complete tutorial, see: PolarDB for PostgreSQL (POLAR) Kernel Guide. For kernel differences and version references, see the PGSQL kernel overview.

Overview

  • Config Name: polar
  • Node Count: Single node
  • Description: Uses PolarDB for PostgreSQL kernel
  • OS Distro: el8, el9, el10, d12, d13, u22, u24, u26
  • OS Arch: x86_64, aarch64
  • Related: meta

Usage:

./configure -c polar [-i <primary_ip>]

Content

Source: pigsty/conf/polar.yml

---
#==============================================================#
# File      :   polar.yml
# Desc      :   Pigsty 1-node PolarDB Kernel Config Template
# Ctime     :   2020-08-05
# Mtime     :   2026-07-08
# Docs      :   https://pigsty.io/docs/conf/polar
# License   :   Apache-2.0 @ https://pigsty.io/docs/about/license/
# Copyright :   2018-2026  Ruohang Feng / Vonng ([email protected])
#==============================================================#

# This is the config template for PolarDB PG Kernel,
# Which is a PostgreSQL 17 fork with RAC flavor features
# tutorial: https://pigsty.io/docs/pgsql/kernel/polardb
#
# Usage:
#   curl https://repo.pigsty.io/get | bash
#   ./configure -c polar
#   ./deploy.yml

all:
  children:
    infra: { hosts: { 10.10.10.10: { infra_seq: 1 }} ,vars: { repo_enabled: false }}
    etcd:  { hosts: { 10.10.10.10: { etcd_seq: 1  }} ,vars: { etcd_cluster: etcd  }}
    #minio: { hosts: { 10.10.10.10: { minio_seq: 1 }} ,vars: { minio_cluster: minio }}

    #----------------------------------------------#
    # PolarDB Database Cluster
    #----------------------------------------------#
    pg-meta:
      hosts:
        10.10.10.10: { pg_seq: 1, pg_role: primary }
      vars:
        pg_cluster: pg-meta
        pg_users:
          - {name: dbuser_meta ,password: DBUser.Meta   ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: pigsty admin user }
          - {name: dbuser_view ,password: DBUser.Viewer ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer for meta database }
        pg_databases:
          - {name: meta ,baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty]}
        pg_hba_rules:   # https://pigsty.io/docs/pgsql/config/hba
          - { user: all ,db: all ,addr: intra ,auth: pwd ,title: 'everyone intranet access with password' ,order: 800 }
        pg_crontab:     # https://pigsty.io/docs/pgsql/admin/crontab
          - '00 01 * * * /pg/bin/pg-backup full'

        # PolarDB Ad Hoc Settings
        pg_version: 17                            # PolarDB PG is based on PG 17
        pg_mode: polar                            # PolarDB PG Compatible mode
        pg_packages: [ polardb, pgsql-common ]    # Replace PG kernel with PolarDB kernel
        pg_exporter_exclude_database: 'template0,template1,postgres,polardb_admin'
        pg_default_roles:                         # PolarDB require replicator as superuser
          - { name: dbrole_readonly  ,login: false ,comment: role for global read-only access     }
          - { name: dbrole_offline   ,login: false ,comment: role for restricted read-only access }
          - { name: dbrole_readwrite ,login: false ,roles: [dbrole_readonly] ,comment: role for global read-write access }
          - { name: dbrole_admin     ,login: false ,roles: [pg_monitor, dbrole_readwrite] ,comment: role for object creation }
          - { name: postgres     ,superuser: true  ,comment: system superuser }
          - { name: replicator   ,superuser: true  ,replication: true ,roles: [pg_monitor, dbrole_readonly] ,comment: system replicator } # <- superuser is required for replication
          - { name: dbuser_dba   ,superuser: true  ,roles: [dbrole_admin]  ,pgbouncer: true ,pool_mode: session, pool_connlimit: 16 ,comment: pgsql admin user }
          - { name: dbuser_monitor ,roles: [pg_monitor] ,pgbouncer: true ,parameters: {log_min_duration_statement: 1000 } ,pool_mode: session ,pool_connlimit: 8 ,comment: pgsql monitor user }

  vars:                               # global variables
    #----------------------------------------------#
    # INFRA : https://pigsty.io/docs/infra/param
    #----------------------------------------------#
    version: v4.5.0                   # pigsty version string
    admin_ip: 10.10.10.10             # admin node ip address
    region: default                   # upstream mirror region: default,china,europe
    infra_portal:                     # infra services exposed via portal
      home : { domain: i.pigsty }     # default domain name

    #----------------------------------------------#
    # NODE : https://pigsty.io/docs/node/param
    #----------------------------------------------#
    nodename_overwrite: false           # do not overwrite node hostname on single node mode
    node_repo_modules: node,infra,pgsql # add these repos directly to the singleton node
    node_tune: oltp                     # node tuning specs: oltp,olap,tiny,crit

    #----------------------------------------------#
    # PGSQL : https://pigsty.io/docs/pgsql/param
    #----------------------------------------------#
    pg_version: 17                      # PolarDB is compatible with PG 17
    pg_conf: oltp.yml                   # pgsql tuning specs: {oltp,olap,tiny,crit}.yml

    #----------------------------------------------#
    # PASSWORD : https://pigsty.io/docs/setup/security/
    #----------------------------------------------#
    grafana_admin_password: pigsty
    grafana_view_password: DBUser.Viewer
    pg_admin_password: DBUser.DBA
    pg_monitor_password: DBUser.Monitor
    pg_replication_password: DBUser.Replicator
    patroni_password: Patroni.API
    haproxy_admin_password: pigsty
    minio_secret_key: S3User.MinIO
    etcd_root_password: Etcd.Root

...

Explanation

The polar template uses Alibaba Cloud’s open-source PolarDB for PostgreSQL kernel, providing cloud-native database capabilities.

Key Features:

  • Storage-compute separation architecture, compute and storage nodes can scale independently
  • Supports one-write-multiple-read, read replicas scale in seconds
  • Compatible with PostgreSQL ecosystem, maintains SQL compatibility
  • Supports shared storage scenarios, suitable for cloud environment deployment
  • Default PolarDB kernel path is /usr/polar-17
  • Available extensions follow the PolarDB 17 kernel catalog. Common extensions include pgaudit, pg_partman, pg_profile, pg_repack, pg_stat_kcache, pg_cron, and pg_hint_plan

Use Cases:

  • Cloud-native scenarios requiring storage-compute separation architecture
  • Read-heavy write-light workloads
  • Scenarios requiring quick scaling of read replicas
  • Test environments for evaluating PolarDB features

Notes:

  • PolarDB is now based on PostgreSQL 17
  • Replication user requires superuser privileges (different from native PostgreSQL)
  • Some PostgreSQL extensions may have compatibility issues
  • The current template provides packages for both x86_64 and aarch64

6.12 - ivory

IvorySQL kernel, provides Oracle syntax and PL/SQL compatibility

The ivory configuration template uses Highgo’s IvorySQL database kernel instead of native PostgreSQL, providing Oracle syntax and PL/SQL compatibility.

For the complete tutorial, see: IvorySQL (Oracle Compatible) Kernel Guide

Overview

  • Config Name: ivory
  • Node Count: Single node
  • Description: Uses IvorySQL Oracle-compatible kernel
  • OS Distro: el8, el9, el10, d12, d13, u22, u24, u26
  • OS Arch: x86_64, aarch64
  • Related: meta

Usage:

./configure -c ivory [-i <primary_ip>]

Content

Source: pigsty/conf/ivory.yml

---
#==============================================================#
# File      :   ivory.yml
# Desc      :   IvorySQL 5 (Oracle Compatible) template
# Ctime     :   2024-08-05
# Mtime     :   2026-07-08
# Docs      :   https://pigsty.io/docs/conf/ivory
# License   :   Apache-2.0 @ https://pigsty.io/docs/about/license/
# Copyright :   2018-2026  Ruohang Feng / Vonng ([email protected])
#==============================================================#

# This is the config template for IvorySQL 5 Kernel,
# Which is a PostgreSQL 18 fork with Oracle Compatibility
# tutorial: https://pigsty.io/docs/pgsql/kernel/ivorysql
# Oracle compatible port (PGSQL Wire) is 1521
#
# Usage:
#   curl https://repo.pigsty.io/get | bash
#   ./configure -c ivory
#   ./deploy.yml

all:
  children:
    infra: { hosts: { 10.10.10.10: { infra_seq: 1 }} ,vars: { repo_enabled: false }}
    etcd:  { hosts: { 10.10.10.10: { etcd_seq: 1  }} ,vars: { etcd_cluster: etcd  }}
    #minio: { hosts: { 10.10.10.10: { minio_seq: 1 }} ,vars: { minio_cluster: minio }}

    #----------------------------------------------#
    # IvorySQL Database Cluster
    #----------------------------------------------#
    pg-meta:
      hosts:
        10.10.10.10: { pg_seq: 1, pg_role: primary }
      vars:
        pg_cluster: pg-meta
        pg_users:
          - {name: dbuser_meta ,password: DBUser.Meta   ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: pigsty admin user }
          - {name: dbuser_view ,password: DBUser.Viewer ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer for meta database }
        pg_databases:
          - {name: meta ,baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty]}
        pg_hba_rules:   # https://pigsty.io/docs/pgsql/config/hba
          - { user: all ,db: all ,addr: intra ,auth: pwd ,title: 'everyone intranet access with password' ,order: 800 }
        pg_crontab:     # https://pigsty.io/docs/pgsql/admin/crontab
          - '00 01 * * * /pg/bin/pg-backup full'

        # IvorySQL Ad Hoc Settings
        pg_mode: ivory                                                 # Use IvorySQL Oracle Compatible Mode
        pg_packages: [ ivorysql, pgsql-common ]                        # install IvorySQL instead of postgresql kernel
        pg_libs: 'liboracle_parser, pg_stat_statements, auto_explain'  # pre-load oracle parser

  vars:                               # global variables
    #----------------------------------------------#
    # INFRA : https://pigsty.io/docs/infra/param
    #----------------------------------------------#
    version: v4.5.0                   # pigsty version string
    admin_ip: 10.10.10.10             # admin node ip address
    region: default                   # upstream mirror region: default,china,europe
    infra_portal:                     # infra services exposed via portal
      home : { domain: i.pigsty }     # default domain name

    #----------------------------------------------#
    # NODE : https://pigsty.io/docs/node/param
    #----------------------------------------------#
    nodename_overwrite: false           # do not overwrite node hostname on single node mode
    node_repo_modules: node,infra,pgsql # add these repos directly to the singleton node
    node_tune: oltp                     # node tuning specs: oltp,olap,tiny,crit

    #----------------------------------------------#
    # PGSQL : https://pigsty.io/docs/pgsql/param
    #----------------------------------------------#
    pg_version: 18                      # IvorySQL kernel is compatible with postgres 18
    pg_conf: oltp.yml                   # pgsql tuning specs: {oltp,olap,tiny,crit}.yml

    #----------------------------------------------#
    # PASSWORD : https://pigsty.io/docs/setup/security/
    #----------------------------------------------#
    grafana_admin_password: pigsty
    grafana_view_password: DBUser.Viewer
    pg_admin_password: DBUser.DBA
    pg_monitor_password: DBUser.Monitor
    pg_replication_password: DBUser.Replicator
    patroni_password: Patroni.API
    haproxy_admin_password: pigsty
    minio_secret_key: S3User.MinIO
    etcd_root_password: Etcd.Root
...

Explanation

The ivory template uses Highgo’s open-source IvorySQL kernel, providing Oracle database compatibility.

Key Features:

  • Supports Oracle PL/SQL syntax
  • Compatible with Oracle data types (NUMBER, VARCHAR2, etc.)
  • Supports Oracle-style packages
  • Retains all standard PostgreSQL functionality

Use Cases:

  • Migrating from Oracle to PostgreSQL
  • Applications needing both Oracle and PostgreSQL syntax support
  • Leveraging PostgreSQL ecosystem while maintaining PL/SQL compatibility
  • Test environments for evaluating IvorySQL features

Notes:

  • IvorySQL 5 is based on PostgreSQL 18
  • Using liboracle_parser requires loading into shared_preload_libraries
  • pgbackrest may have checksum issues in Oracle-compatible mode, PITR capability is limited
  • The current package matrix covers EL 8/9/10, Debian 12/13, Ubuntu 22/24/26, and both architectures

6.13 - agens

AgensGraph kernel template with property graph model and Cypher query support

The agens configuration template replaces native PostgreSQL with the AgensGraph kernel and enables property-graph modeling plus Cypher queries.

For the full guide, see: AgensGraph kernel guide

Overview

  • Config name: agens
  • Node count: Single node
  • Description: AgensGraph (PG17) graph database kernel template
  • Supported OS: el8, el9, el10, d12, d13, u22, u24, u26
  • Supported arch: x86_64, aarch64
  • Related templates: meta, pgsql

Enable with:

./configure -c agens [-i <primary_ip>]

Template Content

Source: pigsty/conf/agens.yml

---
#==============================================================#
# File      :   agens.yml
# Desc      :   1-node AgensGraph (Graph DB) template
# Ctime     :   2026-02-26
# Mtime     :   2026-07-06
# Docs      :   https://pigsty.io/docs/conf/agens
# License   :   Apache-2.0 @ https://pigsty.io/docs/about/license/
# Copyright :   2018-2026  Ruohang Feng / Vonng ([email protected])
#==============================================================#

# This is the config template for AgensGraph Kernel,
# Which is a PostgreSQL 17 fork with graph capabilities.
# tutorial: https://pigsty.io/docs/pgsql/kernel/agensgraph
#
# Usage:
#   curl https://repo.pigsty.io/get | bash
#   ./configure -c agens
#   ./deploy.yml

all:
  children:
    infra: { hosts: { 10.10.10.10: { infra_seq: 1 }} ,vars: { repo_enabled: false }}
    etcd:  { hosts: { 10.10.10.10: { etcd_seq: 1  }} ,vars: { etcd_cluster: etcd  }}
    #minio: { hosts: { 10.10.10.10: { minio_seq: 1 }} ,vars: { minio_cluster: minio }}

    #----------------------------------------------#
    # AgensGraph Database Cluster
    #----------------------------------------------#
    pg-meta:
      hosts:
        10.10.10.10: { pg_seq: 1, pg_role: primary }
      vars:
        pg_cluster: pg-meta
        pg_users:
          - {name: dbuser_meta ,password: DBUser.Meta   ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: pigsty admin user }
          - {name: dbuser_view ,password: DBUser.Viewer ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer for meta database }
        pg_databases:
          - {name: meta ,baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty]}
        pg_hba_rules:   # https://pigsty.io/docs/pgsql/config/hba
          - { user: all ,db: all ,addr: intra ,auth: pwd ,title: 'everyone intranet access with password' ,order: 800 }
        pg_crontab:     # https://pigsty.io/docs/pgsql/admin/crontab
          - '00 01 * * * /pg/bin/pg-backup full'

        # AgensGraph Ad Hoc Settings
        pg_mode: agens                                   # AgensGraph compatible mode
        pg_packages: [ agensgraph, pgsql-common ]        # install AgensGraph kernel package + common utils

  vars:
    #----------------------------------------------#
    # INFRA : https://pigsty.io/docs/infra/param
    #----------------------------------------------#
    version: v4.5.0                   # pigsty version string
    admin_ip: 10.10.10.10             # admin node ip address
    region: default                   # upstream mirror region: default,china,europe
    infra_portal:                     # infra services exposed via portal
      home : { domain: i.pigsty }     # default domain name

    #----------------------------------------------#
    # NODE : https://pigsty.io/docs/node/param
    #----------------------------------------------#
    nodename_overwrite: false           # do not overwrite node hostname on single node mode
    node_repo_modules: node,infra,pgsql # add these repos directly to the singleton node
    node_tune: oltp                     # node tuning specs: oltp,olap,tiny,crit

    #----------------------------------------------#
    # PGSQL : https://pigsty.io/docs/pgsql/param
    #----------------------------------------------#
    pg_version: 17                      # AgensGraph kernel is compatible with postgres 17
    pg_conf: oltp.yml                   # pgsql tuning specs: {oltp,olap,tiny,crit}.yml

    #----------------------------------------------#
    # PASSWORD : https://pigsty.io/docs/setup/security/
    #----------------------------------------------#
    grafana_admin_password: pigsty
    grafana_view_password: DBUser.Viewer
    pg_admin_password: DBUser.DBA
    pg_monitor_password: DBUser.Monitor
    pg_replication_password: DBUser.Replicator
    patroni_password: Patroni.API
    haproxy_admin_password: pigsty
    minio_secret_key: S3User.MinIO
    etcd_root_password: Etcd.Root
...

Notes

The agens template enables pg_mode: agens in the pg-meta cluster and installs the agensgraph kernel package instead of standard PostgreSQL.

Key features:

  • Property graph model support (Vertex / Edge)
  • Cypher query syntax, can be combined with SQL
  • Compatible with PostgreSQL ecosystem and standard operations
  • Based on PostgreSQL 17-compatible kernel by default

Typical use cases:

  • Graph relationship analysis and path queries
  • Social graph, risk linkage, knowledge graph scenarios
  • Workloads requiring graph queries within PostgreSQL operations

Caveats:

  • Current AgensGraph template is pinned to pg_version: 17
  • Default topology is single-node for quick validation; production should extend with HA topology planning
  • Graph schema and Cypher semantics should follow official AgensGraph docs

6.14 - pgedge

pgEdge kernel template for distributed multi-master PostgreSQL in edge scenarios

The pgedge configuration template replaces native PostgreSQL with the pgEdge kernel and provides distributed, multi-master capabilities for edge deployments.

For the full guide, see: pgEdge kernel guide. For kernel differences and version references, see the PGSQL kernel overview.

Overview

  • Config name: pgedge
  • Node count: Single node
  • Description: pgEdge (PG18) distributed kernel template
  • Supported OS: d12, d13, u22, u24, u26 for PG18 packages. For EL/RPM platforms, check current PGSQL repository availability for pgedge_18.
  • Supported arch: x86_64, aarch64
  • Related templates: meta, pgsql

Enable with:

./configure -c pgedge [-i <primary_ip>]

Template Content

Source: pigsty/conf/pgedge.yml

---
#==============================================================#
# File      :   pgedge.yml
# Desc      :   1-node pgEdge (Distributed PG) template
# Ctime     :   2026-02-26
# Mtime     :   2026-07-08
# Docs      :   https://pigsty.io/docs/conf/pgedge
# License   :   Apache-2.0 @ https://pigsty.io/docs/about/license/
# Copyright :   2018-2026  Ruohang Feng / Vonng ([email protected])
#==============================================================#

# This is the config template for pgEdge Kernel,
# Which is a PostgreSQL 15/16/17/18 compatible fork, default to 18.
# tutorial: https://pigsty.io/docs/pgsql/kernel/pgedge
#
# Usage:
#   curl https://repo.pigsty.io/get | bash
#   ./configure -c pgedge [-v 15/16/17/18]
#   ./deploy.yml

all:
  children:
    infra: { hosts: { 10.10.10.10: { infra_seq: 1 }} ,vars: { repo_enabled: false }}
    etcd:  { hosts: { 10.10.10.10: { etcd_seq: 1  }} ,vars: { etcd_cluster: etcd  }}
    #minio: { hosts: { 10.10.10.10: { minio_seq: 1 }} ,vars: { minio_cluster: minio }}

    #----------------------------------------------#
    # pgEdge Database Cluster
    #----------------------------------------------#
    pg-meta:
      hosts:
        10.10.10.10: { pg_seq: 1, pg_role: primary }
      vars:
        pg_cluster: pg-meta
        pg_users:
          - {name: dbuser_meta ,password: DBUser.Meta   ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: pigsty admin user }
          - {name: dbuser_view ,password: DBUser.Viewer ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer for meta database }
        pg_databases:
          - {name: meta ,baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty] ,extensions: [spock, snowflake, lolor]}
        pg_hba_rules:   # https://pigsty.io/docs/pgsql/config/hba
          - { user: all ,db: all ,addr: intra ,auth: pwd ,title: 'everyone intranet access with password' ,order: 800 }
        pg_crontab:     # https://pigsty.io/docs/pgsql/admin/crontab
          - '00 01 * * * /pg/bin/pg-backup full'

        # pgEdge Ad Hoc Settings
        pg_mode: pgedge                               # pgEdge compatible mode
        pg_packages: [ pgedge, pgsql-common ]         # install pgEdge kernel package + common utils
        pg_libs: 'spock, lolor, pg_stat_statements, auto_explain' # preload required libs for pgEdge logical replication

  vars:
    #----------------------------------------------#
    # INFRA : https://pigsty.io/docs/infra/param
    #----------------------------------------------#
    version: v4.5.0                   # pigsty version string
    admin_ip: 10.10.10.10             # admin node ip address
    region: default                   # upstream mirror region: default,china,europe
    infra_portal:                     # infra services exposed via portal
      home : { domain: i.pigsty }     # default domain name

    #----------------------------------------------#
    # NODE : https://pigsty.io/docs/node/param
    #----------------------------------------------#
    nodename_overwrite: false           # do not overwrite node hostname on single node mode
    node_repo_modules: node,infra,pgsql # add these repos directly to the singleton node
    node_tune: oltp                     # node tuning specs: oltp,olap,tiny,crit

    #----------------------------------------------#
    # PGSQL : https://pigsty.io/docs/pgsql/param
    #----------------------------------------------#
    pg_version: 18                      # pgEdge kernel is compatible with postgres 18
    pg_conf: oltp.yml                   # pgsql tuning specs: {oltp,olap,tiny,crit}.yml

    #----------------------------------------------#
    # PASSWORD : https://pigsty.io/docs/setup/security/
    #----------------------------------------------#
    grafana_admin_password: pigsty
    grafana_view_password: DBUser.Viewer
    pg_admin_password: DBUser.DBA
    pg_monitor_password: DBUser.Monitor
    pg_replication_password: DBUser.Replicator
    patroni_password: Patroni.API
    haproxy_admin_password: pigsty
    minio_secret_key: S3User.MinIO
    etcd_root_password: Etcd.Root
...

Notes

The pgedge template enables pg_mode: pgedge in pg-meta and pre-installs pgEdge core extensions for logical replication and edge distribution.

Key features:

  • Uses the pgedge kernel package (PG15/16/17/18 compatible, default PG18)
  • Bundles spock, snowflake, and lolor in the pgedge-$v kernel package and creates them in the meta database by default
  • Preloads spock and lolor for multi-master setup readiness
  • Keeps Pigsty standard backup, monitoring, and operations workflow

Typical use cases:

  • Multi-region edge deployment with nearby writes
  • Multi-master logical replication with conflict handling
  • Single-node validation before distributed rollout

Caveats:

  • Current template is for single-node kernel validation; production multi-master needs explicit topology and replication strategy planning
  • Default is pg_version: 18; keep consistent with target cluster versions
  • Evaluate latency and conflict policy before cross-region replication

6.15 - mysql

OpenHalo kernel, provides MySQL protocol and syntax compatibility

The mysql configuration template uses OpenHalo database kernel instead of native PostgreSQL, providing MySQL wire protocol and SQL syntax compatibility.

Overview

  • Config Name: mysql
  • Node Count: Single node
  • Description: OpenHalo MySQL-compatible kernel configuration
  • OS Distro: EL 8/9/10, Debian 12/13, Ubuntu 22/24/26
  • OS Arch: x86_64, aarch64
  • Related: meta

Usage:

./configure -c mysql [-i <primary_ip>]

Content

Source: pigsty/conf/mysql.yml

---
#==============================================================#
# File      :   mysql.yml
# Desc      :   1-node OpenHaloDB (MySQL Compatible) template
# Ctime     :   2025-04-03
# Mtime     :   2026-07-08
# Docs      :   https://pigsty.io/docs/conf/mysql
# License   :   Apache-2.0 @ https://pigsty.io/docs/about/license/
# Copyright :   2018-2026  Ruohang Feng / Vonng ([email protected])
#==============================================================#

# This is the config template for OpenHalo PG Kernel,
# Which is a PostgreSQL 14 fork with MySQL Wire Compatibility
# tutorial: https://pigsty.io/docs/pgsql/kernel/openhalo
#
# Usage:
#   curl https://repo.pigsty.io/get | bash
#   ./configure -c mysql
#   ./deploy.yml

all:
  children:
    infra: { hosts: { 10.10.10.10: { infra_seq: 1 }} ,vars: { repo_enabled: false }}
    etcd:  { hosts: { 10.10.10.10: { etcd_seq: 1  }} ,vars: { etcd_cluster: etcd  }}
    #minio: { hosts: { 10.10.10.10: { minio_seq: 1 }} ,vars: { minio_cluster: minio }}

    #----------------------------------------------#
    # OpenHalo Database Cluster
    #----------------------------------------------#
    # connect with mysql client: mysql -h 10.10.10.10 -u dbuser_meta -D mysql (the actual database is 'postgres', and 'mysql' is a schema)
    pg-meta:
      hosts:
        10.10.10.10: { pg_seq: 1, pg_role: primary }
      vars:
        pg_cluster: pg-meta
        pg_users:
          - {name: dbuser_meta ,password: DBUser.Meta   ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: pigsty admin user }
          - {name: dbuser_view ,password: DBUser.Viewer ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer for meta database }
        pg_databases:
          - {name: postgres, extensions: [aux_mysql]} # the mysql compatible database
          - {name: meta ,baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty]}
        pg_hba_rules:   # https://pigsty.io/docs/pgsql/config/hba
          - { user: all ,db: all ,addr: intra ,auth: pwd ,title: 'everyone intranet access with password' ,order: 800 }
        pg_crontab:     # https://pigsty.io/docs/pgsql/admin/crontab
          - '00 01 * * * /pg/bin/pg-backup full'

        # OpenHalo Ad Hoc Setting
        pg_mode: mysql                    # MySQL Compatible Mode by HaloDB
        pg_version: 14                    # OpenHaloDB is compatible with PG Major Version 14
        pg_packages: [ openhalo, pgsql-common ]  # install openhalodb instead of postgresql kernel

  vars:
    #----------------------------------------------#
    # INFRA : https://pigsty.io/docs/infra/param
    #----------------------------------------------#
    version: v4.5.0                   # pigsty version string
    admin_ip: 10.10.10.10             # admin node ip address
    region: default                   # upstream mirror region: default,china,europe
    infra_portal:                     # infra services exposed via portal
      home : { domain: i.pigsty }     # default domain name

    #----------------------------------------------#
    # NODE : https://pigsty.io/docs/node/param
    #----------------------------------------------#
    nodename_overwrite: false           # do not overwrite node hostname on single node mode
    node_repo_modules: node,infra,pgsql # add these repos directly to the singleton node
    node_tune: oltp                     # node tuning specs: oltp,olap,tiny,crit

    #----------------------------------------------#
    # PGSQL : https://pigsty.io/docs/pgsql/param
    #----------------------------------------------#
    pg_version: 14                      # OpenHalo is compatible with PG 14
    pg_conf: oltp.yml                   # pgsql tuning specs: {oltp,olap,tiny,crit}.yml

    #----------------------------------------------#
    # PASSWORD : https://pigsty.io/docs/setup/security/
    #----------------------------------------------#
    grafana_admin_password: pigsty
    grafana_view_password: DBUser.Viewer
    pg_admin_password: DBUser.DBA
    pg_monitor_password: DBUser.Monitor
    pg_replication_password: DBUser.Replicator
    patroni_password: Patroni.API
    haproxy_admin_password: pigsty
    minio_secret_key: S3User.MinIO
    etcd_root_password: Etcd.Root
...

Explanation

The mysql template uses the OpenHalo kernel, allowing you to connect to PostgreSQL using MySQL client tools.

Key Features:

  • Uses MySQL protocol (port 3306), compatible with MySQL clients
  • Supports a subset of MySQL SQL syntax
  • Retains PostgreSQL’s ACID properties and storage engine
  • Supports both PostgreSQL and MySQL protocol connections simultaneously

Connection Methods:

# Using MySQL client
mysql -h 10.10.10.10 -P 3306 -u dbuser_meta -pDBUser.Meta

# Also retains PostgreSQL connection capability
psql postgres://dbuser_meta:[email protected]:5432/meta

Use Cases:

  • Migrating from MySQL to PostgreSQL
  • Applications needing to support both MySQL and PostgreSQL clients
  • Leveraging PostgreSQL ecosystem while maintaining MySQL compatibility

Notes:

  • OpenHalo is based on PostgreSQL 14, does not support higher version features
  • Some MySQL syntax may have compatibility differences
  • The current openhalo package alias covers Pigsty’s supported Linux platforms on both architectures; actual installation still depends on the target platform’s repository index

6.16 - pgtde

Percona PostgreSQL kernel, provides Transparent Data Encryption (pg_tde) capability

The pgtde configuration template uses Percona PostgreSQL database kernel, providing Transparent Data Encryption (TDE) capability.

Overview

  • Config Name: pgtde
  • Node Count: Single node
  • Description: Percona PostgreSQL transparent data encryption configuration
  • OS Distro: el8, el9, el10, d12, d13, u22, u24, u26
  • OS Arch: x86_64, aarch64
  • Related: meta

Usage:

./configure -c pgtde [-i <primary_ip>]

Content

Source: pigsty/conf/pgtde.yml

---
#==============================================================#
# File      :   pgtde.yml
# Desc      :   PG TDE with Percona PostgreSQL 1-node template
# Ctime     :   2025-07-04
# Mtime     :   2026-07-23
# Docs      :   https://pigsty.io/docs/conf/pgtde
# License   :   Apache-2.0 @ https://pigsty.io/docs/about/license/
# Copyright :   2018-2026  Ruohang Feng / Vonng ([email protected])
#==============================================================#

# This is the config template for Percona PostgreSQL Distribution
# with pg_tde, currently based on PostgreSQL 18
# tutorial: https://pigsty.io/docs/pgsql/kernel/percona
#
# Usage:
#   curl https://repo.pigsty.io/get | bash
#   ./configure -c pgtde
#   ./deploy.yml

all:
  children:
    infra: { hosts: { 10.10.10.10: { infra_seq: 1 }} ,vars: { repo_enabled: false }}
    etcd:  { hosts: { 10.10.10.10: { etcd_seq: 1  }} ,vars: { etcd_cluster: etcd  }}
    #minio: { hosts: { 10.10.10.10: { minio_seq: 1 }} ,vars: { minio_cluster: minio }}

    #----------------------------------------------#
    # Percona Postgres Database Cluster
    #----------------------------------------------#
    pg-meta:
      hosts:
        10.10.10.10: { pg_seq: 1, pg_role: primary }
      vars:
        pg_mode: pgtde
        pg_cluster: pg-meta
        pg_users:
          - { name: dbuser_meta ,password: DBUser.Meta   ,pgbouncer: true ,roles: [dbrole_admin   ] ,comment: pigsty admin user }
          - { name: dbuser_view ,password: DBUser.Viewer ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer  }
        pg_databases:
          - name: meta
            baseline: cmdb.sql
            comment: pigsty tde database
            schemas: [pigsty]
            extensions: [ vector, postgis, pg_tde ,pgaudit, { name: pg_stat_monitor, schema: monitor } ]
        pg_hba_rules:   # https://pigsty.io/docs/pgsql/config/hba
          - { user: all ,db: all ,addr: intra ,auth: pwd ,title: 'everyone intranet access with password' ,order: 800 }
        pg_crontab:     # https://pigsty.io/docs/pgsql/admin/crontab
          - '00 01 * * * /pg/bin/pg-backup full'

        # Percona PostgreSQL TDE Kernel Settings
        pg_packages: [ pgtde, pgsql-common ]  # install Pigsty private-prefix Percona packages
        pg_libs: 'pg_tde, pgaudit, pg_stat_statements, pg_stat_monitor, auto_explain'

  vars:
    #----------------------------------------------#
    # INFRA : https://pigsty.io/docs/infra/param
    #----------------------------------------------#
    version: v4.5.0                   # pigsty version string
    admin_ip: 10.10.10.10             # admin node ip address
    region: default                   # upstream mirror region: default,china,europe
    infra_portal:                     # infra services exposed via portal
      home : { domain: i.pigsty }     # default domain name

    #----------------------------------------------#
    # NODE : https://pigsty.io/docs/node/param
    #----------------------------------------------#
    nodename_overwrite: false             # do not overwrite node hostname on single node mode
    node_repo_modules: node,infra,pgsql
    node_tune: oltp

    #----------------------------------------------#
    # PGSQL : https://pigsty.io/docs/pgsql/param
    #----------------------------------------------#
    pg_version: 18                      # Default Percona TDE PG Major Version is 18
    pg_conf: oltp.yml                   # pgsql tuning specs: {oltp,olap,tiny,crit}.yml

    #----------------------------------------------#
    # PASSWORD : https://pigsty.io/docs/setup/security/
    #----------------------------------------------#
    grafana_admin_password: pigsty
    grafana_view_password: DBUser.Viewer
    pg_admin_password: DBUser.DBA
    pg_monitor_password: DBUser.Monitor
    pg_replication_password: DBUser.Replicator
    patroni_password: Patroni.API
    haproxy_admin_password: pigsty
    minio_secret_key: S3User.MinIO
    etcd_root_password: Etcd.Root
...

Explanation

The pgtde template selects pg_mode: pgtde and installs the pgtde package alias. Pigsty links the private /usr/pgtde-$v prefix (currently /usr/pgtde-18) to its stable /usr/pgsql entry point.

Key Features:

  • Transparent Data Encryption: Data automatically encrypted on disk, transparent to applications
  • Key Management: Supports local keys and external Key Management Systems (KMS)
  • Table-level Encryption: Selectively encrypt sensitive tables
  • Full Compatibility: Fully compatible with native PostgreSQL

Use Cases:

  • Meeting data security compliance requirements (e.g., PCI-DSS, HIPAA)
  • Storing sensitive data (e.g., personal information, financial data)
  • Scenarios requiring data-at-rest encryption
  • Enterprise environments with strict data security requirements

Usage:

CREATE EXTENSION pg_tde;

SELECT pg_tde_add_database_key_provider_file(
    'local-file',
    '/secure/path/pg_tde_keys'
);
SELECT pg_tde_set_principal_key('app-principal-key', 'local-file');

-- Create encrypted table
CREATE TABLE sensitive_data (
    id SERIAL PRIMARY KEY,
    ssn VARCHAR(11)
) USING tde_heap;

-- Or enable encryption on existing table
ALTER TABLE existing_table SET ACCESS METHOD tde_heap;

Notes:

  • Percona PostgreSQL is based on PostgreSQL 18
  • Encryption brings some performance overhead (typically 5-15%)
  • Encryption keys must be properly managed
  • Both x86_64 and aarch64 packages are available on the listed distributions

6.17 - oriole

OrioleDB kernel, provides bloat-free OLTP enhanced storage engine

The oriole configuration template uses OrioleDB storage engine instead of PostgreSQL’s default Heap storage, providing bloat-free, high-performance OLTP capability.

Overview

  • Config Name: oriole
  • Node Count: Single node
  • Description: OrioleDB bloat-free storage engine configuration
  • PostgreSQL Major: 16, 17, or 18
  • OS Distro: el8, el9, el10, d12, d13, u22, u24, u26
  • OS Arch: x86_64, aarch64
  • Related: meta

Usage:

./configure -c oriole [-i <primary_ip>]

# Select a PostgreSQL major version explicitly
./configure -c oriole -v 16
./configure -c oriole -v 17
./configure -c oriole -v 18

Content

Source: pigsty/conf/oriole.yml

---
#==============================================================#
# File      :   oriole.yml
# Desc      :   1-node OrioleDB (OLTP Enhancement) template
# Ctime     :   2025-04-05
# Mtime     :   2026-07-08
# Docs      :   https://pigsty.io/docs/conf/oriole
# License   :   Apache-2.0 @ https://pigsty.io/docs/about/license/
# Copyright :   2018-2026  Ruohang Feng / Vonng ([email protected])
#==============================================================#

# This is the config template for OrioleDB Kernel,
# Which is a Patched PostgreSQL 16/17/18 fork
# tutorial: https://pigsty.io/docs/pgsql/kernel/orioledb
#
# Usage:
#   curl https://repo.pigsty.io/get | bash
#   ./configure -c oriole [-v 16/17/18]
#   ./deploy.yml

all:
  children:
    infra: { hosts: { 10.10.10.10: { infra_seq: 1 }} ,vars: { repo_enabled: false }}
    etcd:  { hosts: { 10.10.10.10: { etcd_seq: 1  }} ,vars: { etcd_cluster: etcd  }}
    #minio: { hosts: { 10.10.10.10: { minio_seq: 1 }} ,vars: { minio_cluster: minio }}

    #----------------------------------------------#
    # OrioleDB Database Cluster
    #----------------------------------------------#
    pg-meta:
      hosts:
        10.10.10.10: { pg_seq: 1, pg_role: primary }
      vars:
        pg_cluster: pg-meta
        pg_users:
          - {name: dbuser_meta ,password: DBUser.Meta   ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: pigsty admin user }
          - {name: dbuser_view ,password: DBUser.Viewer ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer for meta database }
        pg_databases:
          - {name: meta ,baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty], extensions: [orioledb]}
        pg_hba_rules:   # https://pigsty.io/docs/pgsql/config/hba
          - { user: all ,db: all ,addr: intra ,auth: pwd ,title: 'everyone intranet access with password' ,order: 800 }
        pg_crontab:     # https://pigsty.io/docs/pgsql/admin/crontab
          - '00 01 * * * /pg/bin/pg-backup full'

        # OrioleDB Ad Hoc Settings
        pg_mode: oriole                                         # OrioleDB compatible mode
        pg_packages: [ orioledb, pgsql-common ]                 # install OrioleDB kernel
        pg_libs: 'orioledb, pg_stat_statements, auto_explain'   # Load OrioleDB Extension

  vars:                               # global variables
    #----------------------------------------------#
    # INFRA : https://pigsty.io/docs/infra/param
    #----------------------------------------------#
    version: v4.5.0                   # pigsty version string
    admin_ip: 10.10.10.10             # admin node ip address
    region: default                   # upstream mirror region: default,china,europe
    infra_portal:                     # infra services exposed via portal
      home : { domain: i.pigsty }     # default domain name

    #----------------------------------------------#
    # NODE : https://pigsty.io/docs/node/param
    #----------------------------------------------#
    nodename_overwrite: false           # do not overwrite node hostname on single node mode
    node_repo_modules: node,infra,pgsql # add these repos directly to the singleton node
    node_tune: oltp                     # node tuning specs: oltp,olap,tiny,crit

    #----------------------------------------------#
    # PGSQL : https://pigsty.io/docs/pgsql/param
    #----------------------------------------------#
    pg_version: 18                      # OrioleDB Kernel is based on PG 16/17/18
    pg_conf: oltp.yml                   # pgsql tuning specs: {oltp,olap,tiny,crit}.yml

    #----------------------------------------------#
    # PASSWORD : https://pigsty.io/docs/setup/security/
    #----------------------------------------------#
    grafana_admin_password: pigsty
    grafana_view_password: DBUser.Viewer
    pg_admin_password: DBUser.DBA
    pg_monitor_password: DBUser.Monitor
    pg_replication_password: DBUser.Replicator
    patroni_password: Patroni.API
    haproxy_admin_password: pigsty
    minio_secret_key: S3User.MinIO
    etcd_root_password: Etcd.Root
...

Explanation

The oriole template uses OrioleDB storage engine, fundamentally solving PostgreSQL table bloat problems.

Key Features:

  • Bloat-free Design: Uses UNDO logs instead of Multi-Version Concurrency Control (MVCC)
  • No VACUUM Required: Eliminates performance jitter from autovacuum
  • Row-level WAL: More efficient logging and replication
  • Compressed Storage: Built-in data compression, reduces storage space

Use Cases:

  • High-frequency update OLTP workloads
  • Applications sensitive to write latency
  • Need for stable response times (eliminates VACUUM impact)
  • Large tables with frequent updates causing bloat

Usage:

-- Create table using OrioleDB storage
CREATE TABLE orders (
    id SERIAL PRIMARY KEY,
    customer_id INT,
    amount DECIMAL(10,2)
) USING orioledb;

-- Existing tables cannot be directly converted, need to be rebuilt

Notes:

  • OrioleDB supports PostgreSQL 16, 17, and 18; the template defaults to PG18, and you can select a major version with -v 16, -v 17, or -v 18
  • Need to add orioledb to shared_preload_libraries
  • Some PostgreSQL features may not be fully supported
  • Use the matching OrioleDB packages for the selected PostgreSQL major and OS architecture

6.18 - PostgreSQL Mongo Mode

Run PostgreSQL in Mongo-compatible mode with DocumentDB and the FerretDB Docker APP.

The mongo configuration template is a PostgreSQL deployment mode, not an independent Pigsty module. It combines:

  • PostgreSQL 18 managed by the standard PGSQL module
  • The documentdb extension and its required preload libraries
  • A stateless FerretDB proxy deployed with Pigsty’s Docker APP workflow

All data, high availability, backup, monitoring, and lifecycle management remain PostgreSQL responsibilities. FerretDB only provides the MongoDB wire-compatible endpoint.

Quick Start

The default template is a single-node deployment on 10.10.10.10. FerretDB listens on loopback by default.

Install mongosh separately if it is not already available, or use another MongoDB-compatible client.

./configure -c mongo
./deploy.yml
./docker.yml -l pg-meta
./app.yml -l pg-meta
mongosh 'mongodb://mongod:[email protected]:27017/'

The dedicated mongod PostgreSQL login is declared by the template. FerretDB authentication is enabled, but MongoDB authorization roles are not implemented; PostgreSQL remains the security boundary.

Architecture

Layer Implementation Responsibility
Data PostgreSQL + DocumentDB Durable storage, transactions, HA, PITR, ACL, monitoring
Protocol FerretDB Docker APP Stateless MongoDB wire compatibility
Access 127.0.0.1:27017 by default Local MongoDB client endpoint

The container connects to Pigsty’s local primary service on port 5436 through host.docker.internal. The default Mongo endpoint is not exposed to the network; change FERRETDB_BIND_ADDR only when remote access is required.

Configuration

Source: pigsty/conf/mongo.yml

---
#==============================================================#
# File      :   mongo.yml
# Desc      :   PostgreSQL Mongo Mode (DocumentDB + FerretDB)
# Ctime     :   2025-02-23
# Mtime     :   2026-08-05
# Docs      :   https://pigsty.io/docs/conf/mongo
# License   :   Apache-2.0 @ https://pigsty.io/docs/about/license/
# Copyright :   2018-2026  Ruohang Feng / Vonng ([email protected])
#==============================================================#

# This is the PostgreSQL Mongo mode template, powered by DocumentDB + FerretDB
# It provides a MongoDB wire-compatible endpoint backed by PostgreSQL
# This config template works with PostgreSQL 16, 17, 18
# tutorial: https://pigsty.io/docs/conf/mongo
#
# Usage:
#   curl https://repo.pigsty.io/get | bash
#   ./configure -c mongo
#   ./deploy.yml
#   ./docker.yml -l pg-meta
#   ./app.yml -l pg-meta
#   # install mongosh separately if it is not already available
#   mongosh 'mongodb://mongod:[email protected]:27017/'

all:
  children:
    infra: { hosts: { 10.10.10.10: { infra_seq: 1 }} ,vars: { repo_enabled: false }}
    etcd:
      hosts:
        10.10.10.10: { etcd_seq: 1 }
        #10.10.10.11: { etcd_seq: 2 }
        #10.10.10.12: { etcd_seq: 3 }
      vars: { etcd_cluster: etcd }
    #minio: { hosts: { 10.10.10.10: { minio_seq: 1 }} ,vars: { minio_cluster: minio }}

    #----------------------------------#
    # PGSQL Database Cluster
    #----------------------------------#
    pg-meta:
      hosts:
        10.10.10.10: { pg_seq: 1, pg_role: primary }
      vars:
        pg_cluster: pg-meta
        pg_users:
          - { name: mongod      ,password: DBUser.Mongo  ,superuser: true  ,comment: FerretDB backend user }
          - { name: dbuser_meta ,password: DBUser.Meta   ,pgbouncer: true ,roles: [dbrole_admin   ] ,comment: pigsty admin user }
          - { name: dbuser_view ,password: DBUser.Viewer ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer  }
        pg_databases:
          - { name: postgres, extensions: [ documentdb, postgis, vector, pg_cron, rum ]}  # run on the postgres database
        pg_hba_rules:
          - { user: dbuser_view , db: all ,addr: infra ,auth: pwd ,title: 'allow grafana dashboard access cmdb from infra nodes' }
          # WARNING: demo/dev only. Avoid world access for dbsu in production.
          - { user: postgres    , db: all ,addr: world ,auth: pwd ,title: 'dbsu password access everywhere' }
          - { user: all ,db: all ,addr: localhost ,order: 1  ,auth: trust ,title: 'documentdb localhost trust access' }
          - { user: all ,db: all ,addr: local     ,order: 1  ,auth: trust ,title: 'documentdb local     trust access' }
          - { user: all ,db: all ,addr: intra ,auth: pwd ,title: 'everyone intranet access with password' ,order: 800 }
        pg_parameters: { cron.database_name: postgres }
        pg_extensions: [ documentdb, postgis, pgvector, pg_cron, rum ]
        pg_libs: 'pg_documentdb, pg_documentdb_core, pg_documentdb_extended_rum, pg_cron, pg_stat_statements, auto_explain'
        pg_crontab:     # https://pigsty.io/docs/pgsql/admin/crontab
          - '00 01 * * * /pg/bin/pg-backup full'

        # FerretDB Docker APP on the same node, exposed at 127.0.0.1:27017
        docker_enabled: true
        app: ferretdb
        apps:
          ferretdb:
            conf:
              FERRETDB_IMAGE: ghcr.io/ferretdb/ferretdb:2.7.0
              FERRETDB_POSTGRESQL_URL: 'postgres://mongod:[email protected]:5436/postgres?pool_min_conns=1&pool_max_conns=20'
              FERRETDB_BIND_ADDR: 127.0.0.1
              FERRETDB_PORT: 27017
              FERRETDB_LISTEN_ADDR: ':27017'
              FERRETDB_AUTH: true
              FERRETDB_TELEMETRY: disabled

    #--------------------------------------------------------------------------#
    # OPTIONAL: Three-node PostgreSQL + DocumentDB + FerretDB HA cluster
    # Uncomment this entire block and the two additional etcd members above.
    # Then run: ./docker.yml -l pg-mongo && ./app.yml -l pg-mongo
    # Endpoint: mongodb://mongod:[email protected]:27017/
    #--------------------------------------------------------------------------#
    # pg-mongo:
    #   hosts:
    #     10.10.10.11: { pg_seq: 1, pg_role: primary, vip_role: master }
    #     10.10.10.12: { pg_seq: 2, pg_role: replica, vip_role: backup }
    #     10.10.10.13: { pg_seq: 3, pg_role: replica, vip_role: backup }
    #   vars:
    #     pg_cluster: pg-mongo
    #     node_cluster: pg-mongo
    #     pg_users:
    #       - { name: mongod, password: DBUser.Mongo, superuser: true, comment: FerretDB backend user }
    #     pg_databases:
    #       - { name: postgres, extensions: [ documentdb, postgis, vector, pg_cron, rum ] }
    #     pg_hba_rules:
    #       - { user: all, db: all, addr: localhost, order: 1, auth: trust, title: 'documentdb localhost trust access' }
    #       - { user: all, db: all, addr: local, order: 1, auth: trust, title: 'documentdb local trust access' }
    #       - { user: mongod, db: postgres, addr: intra, order: 800, auth: pwd, title: 'ferretdb intranet access with password' }
    #     pg_parameters: { cron.database_name: postgres }
    #     pg_extensions: [ documentdb, postgis, pgvector, pg_cron, rum ]
    #     pg_libs: 'pg_documentdb, pg_documentdb_core, pg_documentdb_extended_rum, pg_cron, pg_stat_statements, auto_explain'
    #     pg_crontab:
    #       - '00 01 * * 1 /pg/bin/pg-backup full'
    #       - '00 01 * * 2,3,4,5,6,7 /pg/bin/pg-backup'
    #
    #     # FerretDB Docker cluster and HAProxy service
    #     docker_enabled: true
    #     app: ferretdb
    #     apps:
    #       ferretdb:
    #         conf:
    #           FERRETDB_IMAGE: ghcr.io/ferretdb/ferretdb:2.7.0
    #           FERRETDB_POSTGRESQL_URL: 'postgres://mongod:[email protected]:5436/postgres?pool_min_conns=1&pool_max_conns=20'
    #           FERRETDB_BIND_ADDR: '{{ inventory_hostname }}'
    #           FERRETDB_PORT: 27018
    #           FERRETDB_LISTEN_ADDR: ':27017'
    #           FERRETDB_AUTH: true
    #           FERRETDB_TELEMETRY: disabled
    #
    #     # HA Mongo endpoint: mongo.pigsty / 10.10.10.4:27017
    #     vip_enabled: true
    #     vip_vrid: 27
    #     vip_address: 10.10.10.4
    #     vip_preempt: false
    #     haproxy_services:
    #       - name: mongo
    #         port: 27017
    #         protocol: tcp
    #         balance: leastconn
    #         options:
    #           - option tcp-check
    #         servers:
    #           - { name: ferretdb-1, ip: 10.10.10.11, port: 27018, options: 'check port 27018' }
    #           - { name: ferretdb-2, ip: 10.10.10.12, port: 27018, options: 'check port 27018' }
    #           - { name: ferretdb-3, ip: 10.10.10.13, port: 27018, options: 'check port 27018' }

  vars:                               # global variables
    #----------------------------------------------#
    # INFRA : https://pigsty.io/docs/infra/param
    #----------------------------------------------#
    version: v4.5.0                   # pigsty version string
    admin_ip: 10.10.10.10             # admin node ip address
    region: default                   # upstream mirror region: default,china,europe
    infra_portal:                     # infra services exposed via portal
      home : { domain: i.pigsty }     # default domain name

    #----------------------------------------------#
    # NODE : https://pigsty.io/docs/node/param
    #----------------------------------------------#
    nodename_overwrite: false           # do not overwrite node hostname
    node_repo_modules: node,infra,pgsql # install from upstream repo directly
    node_tune: oltp                     # node tuning specs: oltp,olap,tiny,crit

    #----------------------------------------------#
    # PGSQL : https://pigsty.io/docs/pgsql/param
    #----------------------------------------------#
    pg_version: 18                      # default postgres version (16,17,18)
    pg_conf: oltp.yml                   # pgsql tuning specs: {oltp,olap,tiny,crit}.yml

    #----------------------------------------------#
    # PASSWORD : https://pigsty.io/docs/setup/security/
    #----------------------------------------------#
    grafana_admin_password: pigsty
    grafana_view_password: DBUser.Viewer
    pg_admin_password: DBUser.DBA
    pg_monitor_password: DBUser.Monitor
    pg_replication_password: DBUser.Replicator
    patroni_password: Patroni.API
    haproxy_admin_password: pigsty
    minio_secret_key: S3User.MinIO
    etcd_root_password: Etcd.Root
...

FerretDB settings are ordinary APP overrides under apps.ferretdb.conf:

app: ferretdb
apps:
  ferretdb:
    conf:
      FERRETDB_IMAGE: ghcr.io/ferretdb/ferretdb:2.7.0
      FERRETDB_POSTGRESQL_URL: 'postgres://mongod:[email protected]:5436/postgres?pool_min_conns=1&pool_max_conns=20'
      FERRETDB_BIND_ADDR: 127.0.0.1
      FERRETDB_PORT: 27017
      FERRETDB_AUTH: true
      FERRETDB_TELEMETRY: disabled

Use the standard PostgreSQL parameters, playbooks, dashboards, and administration procedures for the backend cluster. There are no mongo_* inventory parameters or standalone mongo.yml playbook.

Optional HA Topology

The template contains a commented pg-mongo example for three PostgreSQL/FerretDB nodes. Uncomment that block and the two additional etcd members when needed.

In HA mode, each FerretDB container binds {{ inventory_hostname }}:27018; HAProxy exposes all three backends through the floating endpoint 10.10.10.4:27017 (mongo.pigsty). PostgreSQL failover is still handled by Patroni, while FerretDB remains stateless.

Notes

  • The template includes development-friendly HBA examples; tighten them for production.
  • Client-side MongoDB TLS is not enabled by default.
  • Monitor the backend with the standard PostgreSQL and Docker dashboards; there is no separate FERRET module or dedicated module dashboard.
  • Repeat an authenticated CRUD smoke test after upgrading FerretDB or DocumentDB.

6.19 - ha/simu

20-node production environment simulation for large-scale deployment testing

The ha/simu configuration template is a 20-node production environment simulation, requiring a powerful host machine to run.

Overview

  • Config Name: ha/simu
  • Node Count: 20 nodes, pigsty/vagrant/spec/simu.rb
  • Description: 20-node production environment simulation, requires powerful host machine
  • OS Distro: el8, el9, el10, d12, d13, u22, u24, u26
  • OS Arch: x86_64, aarch64

Usage:

./configure -c ha/simu [-i <primary_ip>]

Content

Source: pigsty/conf/ha/simu.yml

---
#==============================================================#
# File      :   simu.yml
# Desc      :   Pigsty Simubox: a 20 node prod simulation env
# Ctime     :   2023-07-20
# Mtime     :   2026-01-19
# Docs      :   https://pigsty.io/docs/conf/simu
# License   :   Apache-2.0 @ https://pigsty.io/docs/about/license
# Copyright :   2018-2025  Ruohang Feng / Vonng ([email protected])
#==============================================================#

all:

  children:

    #==========================================================#
    # infra: 3 nodes
    #==========================================================#
    # ./infra.yml -l infra
    # ./docker.yml -l infra (optional)
    infra:
      hosts:
        10.10.10.10: { infra_seq: 1 }
        10.10.10.11: { infra_seq: 2, repo_enabled: false }
        10.10.10.12: { infra_seq: 3, repo_enabled: false }
      vars:
        docker_enabled: true
        node_tune: oltp         # use oltp template for infra nodes
        pg_conf: oltp.yml       # use oltp template for infra pgsql
        pg_exporters:           # bin/pgmon-add pg-meta2/pg-src2/pg-dst2
          20001: {pg_cluster: pg-meta2   ,pg_seq: 1 ,pg_host: 10.10.10.10, pg_databases: [{ name: meta }]}
          20002: {pg_cluster: pg-meta2   ,pg_seq: 2 ,pg_host: 10.10.10.11, pg_databases: [{ name: meta }]}
          20003: {pg_cluster: pg-meta2   ,pg_seq: 3 ,pg_host: 10.10.10.12, pg_databases: [{ name: meta }]}

          20004: {pg_cluster: pg-src2    ,pg_seq: 1 ,pg_host: 10.10.10.31, pg_databases: [{ name: src }]}
          20005: {pg_cluster: pg-src2    ,pg_seq: 2 ,pg_host: 10.10.10.32, pg_databases: [{ name: src }]}
          20006: {pg_cluster: pg-src2    ,pg_seq: 3 ,pg_host: 10.10.10.33, pg_databases: [{ name: src }]}

          20007: {pg_cluster: pg-dst2    ,pg_seq: 1 ,pg_host: 10.10.10.41, pg_databases: [{ name: dst }]}
          20008: {pg_cluster: pg-dst2    ,pg_seq: 2 ,pg_host: 10.10.10.42, pg_databases: [{ name: dst }]}
          20009: {pg_cluster: pg-dst2    ,pg_seq: 3 ,pg_host: 10.10.10.43, pg_databases: [{ name: dst }]}


    #==========================================================#
    # etcd: 5 nodes dedicated etcd cluster
    #==========================================================#
    # ./etcd.yml -l etcd;
    etcd:
      hosts:
        10.10.10.25: { etcd_seq: 1 }
        10.10.10.26: { etcd_seq: 2 }
        10.10.10.27: { etcd_seq: 3 }
        10.10.10.28: { etcd_seq: 4 }
        10.10.10.29: { etcd_seq: 5 }
      vars:
        etcd_cluster: etcd

    #==========================================================#
    # minio: 4 nodes dedicated minio cluster
    #==========================================================#
    # ./minio.yml -l minio;
    minio:
      hosts:
        10.10.10.21: { minio_seq: 1 }
        10.10.10.22: { minio_seq: 2 }
        10.10.10.23: { minio_seq: 3 }
        10.10.10.24: { minio_seq: 4 }
      vars:
        minio_cluster: minio
        minio_data: '/data{1...4}' # 4 node x 4 disk
        minio_users:                      # list of minio user to be created
          - { access_key: pgbackrest  ,secret_key: S3User.Backup ,policy: pgsql }
          - { access_key: s3user_meta ,secret_key: S3User.Meta   ,policy: meta  }
          - { access_key: s3user_data ,secret_key: S3User.Data   ,policy: data  }


    #==========================================================#
    # proxy: 2 nodes used as dedicated haproxy server
    #==========================================================#
    # ./node.yml -l proxy
    proxy:
      hosts:
        10.10.10.18: { vip_role: master }
        10.10.10.19: { vip_role: backup }
      vars:
        vip_enabled: true
        vip_address: 10.10.10.20
        vip_vrid: 20
        haproxy_services:      # expose minio service : sss.pigsty:9000
          - name: minio        # [REQUIRED] service name, unique
            port: 9000         # [REQUIRED] service port, unique
            balance: leastconn # Use leastconn algorithm and minio health check
            options: [ "option httpchk", "option http-keep-alive", "http-check send meth OPTIONS uri /minio/health/live", "http-check expect status 200" ]
            servers:           # reload service with ./node.yml -t haproxy_config,haproxy_reload
              - { name: minio-1 ,ip: 10.10.10.21 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
              - { name: minio-2 ,ip: 10.10.10.22 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
              - { name: minio-3 ,ip: 10.10.10.23 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
              - { name: minio-4 ,ip: 10.10.10.24 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }

    #==========================================================#
    # pg-meta: reuse infra node as meta cmdb
    #==========================================================#
    # ./pgsql.yml -l pg-meta
    pg-meta:
      hosts:
        10.10.10.10: { pg_seq: 1 , pg_role: primary }
        10.10.10.11: { pg_seq: 2 , pg_role: replica }
        10.10.10.12: { pg_seq: 3 , pg_role: replica }
      vars:
        pg_cluster: pg-meta
        pg_vip_enabled: true
        pg_vip_address: 10.10.10.2/24
        pg_users:
          - {name: dbuser_meta     ,password: DBUser.Meta     ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: pigsty admin user }
          - {name: dbuser_view     ,password: DBUser.Viewer   ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer for meta database }
          - {name: dbuser_grafana  ,password: DBUser.Grafana  ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for grafana database    }
          - {name: dbuser_bytebase ,password: DBUser.Bytebase ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for bytebase database   }
          - {name: dbuser_kong     ,password: DBUser.Kong     ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for kong api gateway    }
          - {name: dbuser_gitea    ,password: DBUser.Gitea    ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for gitea service       }
          - {name: dbuser_wiki     ,password: DBUser.Wiki     ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for wiki.js service     }
          - {name: dbuser_noco     ,password: DBUser.Noco     ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for nocodb service      }
        pg_databases:
          - { name: meta ,baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty] ,extensions: [{name: vector}]}
          - { name: grafana  ,owner: dbuser_grafana  ,revokeconn: true ,comment: grafana primary database }
          - { name: bytebase ,owner: dbuser_bytebase ,revokeconn: true ,comment: bytebase primary database }
          - { name: kong     ,owner: dbuser_kong     ,revokeconn: true ,comment: kong the api gateway database }
          - { name: gitea    ,owner: dbuser_gitea    ,revokeconn: true ,comment: gitea meta database }
          - { name: wiki     ,owner: dbuser_wiki     ,revokeconn: true ,comment: wiki meta database }
          - { name: noco     ,owner: dbuser_noco     ,revokeconn: true ,comment: nocodb database }
        pg_libs: 'pg_stat_statements, auto_explain' # add timescaledb to shared_preload_libraries

    #==========================================================#
    # pg-src: dedicate 3 node source cluster
    #==========================================================#
    # ./pgsql.yml -l pg-src
    pg-src:
      hosts:
        10.10.10.31: { pg_seq: 1, pg_role: primary }
        10.10.10.32: { pg_seq: 2, pg_role: replica }
        10.10.10.33: { pg_seq: 3, pg_role: replica }
      vars:
        pg_cluster: pg-src
        pg_vip_enabled: true
        pg_vip_address: 10.10.10.3/24
        pg_users:  [{ name: test , password: test , pgbouncer: true , roles: [ dbrole_admin ] }]
        pg_databases: [{ name: src }]


    #==========================================================#
    # pg-dst: dedicate 3 node destination cluster
    #==========================================================#
    # ./pgsql.yml -l pg-dst
    pg-dst:
      hosts:
        10.10.10.41: { pg_seq: 1, pg_role: primary }
        10.10.10.42: { pg_seq: 2, pg_role: replica }
        10.10.10.43: { pg_seq: 3, pg_role: replica }
      vars:
        pg_cluster: pg-dst
        pg_vip_enabled: true
        pg_vip_address: 10.10.10.4/24
        pg_users: [ { name: test , password: test , pgbouncer: true , roles: [ dbrole_admin ] } ]
        pg_databases: [ { name: dst } ]


    #==========================================================#
    # redis-meta: reuse the 5 etcd nodes as redis sentinel
    #==========================================================#
    # ./redis.yml -l redis-meta
    redis-meta:
      hosts:
        10.10.10.25: { redis_node: 1 , redis_instances: { 26379: {} } }
        10.10.10.26: { redis_node: 2 , redis_instances: { 26379: {} } }
        10.10.10.27: { redis_node: 3 , redis_instances: { 26379: {} } }
        10.10.10.28: { redis_node: 4 , redis_instances: { 26379: {} } }
        10.10.10.29: { redis_node: 5 , redis_instances: { 26379: {} } }
      vars:
        redis_cluster: redis-meta
        redis_password: 'redis.meta'
        redis_mode: sentinel
        redis_max_memory: 256MB
        redis_sentinel_monitor:  # primary list for redis sentinel, use cls as name, primary ip:port
          - { name: redis-src, host: 10.10.10.31, port: 6379 ,password: redis.src, quorum: 1 }
          - { name: redis-dst, host: 10.10.10.41, port: 6379 ,password: redis.dst, quorum: 1 }

    #==========================================================#
    # redis-src: reuse pg-src 3 nodes for redis
    #==========================================================#
    # ./redis.yml -l redis-src
    redis-src:
      hosts:
        10.10.10.31: { redis_node: 1 , redis_instances: {6379: {  } }}
        10.10.10.32: { redis_node: 2 , redis_instances: {6379: { replica_of: '10.10.10.31 6379' }, 6380: { replica_of: '10.10.10.32 6379' } }}
        10.10.10.33: { redis_node: 3 , redis_instances: {6379: { replica_of: '10.10.10.31 6379' }, 6380: { replica_of: '10.10.10.33 6379' } }}
      vars:
        redis_cluster: redis-src
        redis_password: 'redis.src'
        redis_max_memory: 64MB

    #==========================================================#
    # redis-dst: reuse pg-dst 3 nodes for redis
    #==========================================================#
    # ./redis.yml -l redis-dst
    redis-dst:
      hosts:
        10.10.10.41: { redis_node: 1 , redis_instances: {6379: {  }                               }}
        10.10.10.42: { redis_node: 2 , redis_instances: {6379: { replica_of: '10.10.10.41 6379' } }}
        10.10.10.43: { redis_node: 3 , redis_instances: {6379: { replica_of: '10.10.10.41 6379' } }}
      vars:
        redis_cluster: redis-dst
        redis_password: 'redis.dst'
        redis_max_memory: 64MB

    #==========================================================#
    # pg-tmp: reuse proxy nodes as pgsql cluster
    #==========================================================#
    # ./pgsql.yml -l pg-tmp
    pg-tmp:
      hosts:
        10.10.10.18: { pg_seq: 1 ,pg_role: primary }
        10.10.10.19: { pg_seq: 2 ,pg_role: replica }
      vars:
        pg_cluster: pg-tmp
        pg_users: [ { name: test , password: test , pgbouncer: true , roles: [ dbrole_admin ] } ]
        pg_databases: [ { name: tmp } ]

    #==========================================================#
    # pg-etcd: reuse etcd nodes as pgsql cluster
    #==========================================================#
    # ./pgsql.yml -l pg-etcd
    pg-etcd:
      hosts:
        10.10.10.25: { pg_seq: 1 ,pg_role: primary }
        10.10.10.26: { pg_seq: 2 ,pg_role: replica }
        10.10.10.27: { pg_seq: 3 ,pg_role: replica }
        10.10.10.28: { pg_seq: 4 ,pg_role: replica }
        10.10.10.29: { pg_seq: 5 ,pg_role: offline }
      vars:
        pg_cluster: pg-etcd
        pg_users: [ { name: test , password: test , pgbouncer: true , roles: [ dbrole_admin ] } ]
        pg_databases: [ { name: etcd } ]

    #==========================================================#
    # pg-minio: reuse minio nodes as pgsql cluster
    #==========================================================#
    # ./pgsql.yml -l pg-minio
    pg-minio:
      hosts:
        10.10.10.21: { pg_seq: 1 ,pg_role: primary }
        10.10.10.22: { pg_seq: 2 ,pg_role: replica }
        10.10.10.23: { pg_seq: 3 ,pg_role: replica }
        10.10.10.24: { pg_seq: 4 ,pg_role: replica }
      vars:
        pg_cluster: pg-minio
        pg_users: [ { name: test , password: test , pgbouncer: true , roles: [ dbrole_admin ] } ]
        pg_databases: [ { name: minio } ]

  #============================================================#
  # Global Variables
  #============================================================#
  vars:

    #==========================================================#
    # INFRA
    #==========================================================#
    version: v4.5.0                   # pigsty version string
    admin_ip: 10.10.10.10             # admin node ip address
    region: default                   # upstream mirror region: default|china|europe
    infra_portal:                     # infra services exposed via portal
      home         : { domain: i.pigsty }     # default domain name
      minio        : { domain: m.pigsty    ,endpoint: "10.10.10.21:9001" ,scheme: https ,websocket: true }
      postgrest    : { domain: api.pigsty  ,endpoint: "127.0.0.1:8884" }
      pgadmin      : { domain: adm.pigsty  ,endpoint: "127.0.0.1:8885" }
      pgweb        : { domain: cli.pigsty  ,endpoint: "127.0.0.1:8886" }
      bytebase     : { domain: ddl.pigsty  ,endpoint: "127.0.0.1:8887" }
      jupyter      : { domain: lab.pigsty  ,endpoint: "127.0.0.1:8888"  , websocket: true }
      supa         : { domain: supa.pigsty ,endpoint: "10.10.10.10:8000", websocket: true }

    #==========================================================#
    # NODE
    #==========================================================#
    node_id_from_pg: true             # use nodename rather than pg identity as hostname
    node_tune: tiny                   # use small node template
    node_firewall_mode: zone          # default: trust intranet, expose selected public ports
    node_timezone: Asia/Hong_Kong     # use Asia/Hong_Kong Timezone
    node_dns_servers:                 # DNS servers in /etc/resolv.conf
      - 10.10.10.10
      - 10.10.10.11
    node_etc_hosts:
      - 10.10.10.10 i.pigsty
      - 10.10.10.20 sss.pigsty        # point minio service domain to the L2 VIP of proxy cluster
    node_ntp_servers:                 # NTP servers in /etc/chrony.conf
      - pool cn.pool.ntp.org iburst
      - pool 10.10.10.10 iburst
    node_admin_ssh_exchange: false    # exchange admin ssh key among node cluster

    #==========================================================#
    # PGSQL
    #==========================================================#
    pg_conf: tiny.yml
    pgbackrest_method: minio          # USE THE HA MINIO THROUGH A LOAD BALANCER
    pg_dbsu_ssh_exchange: false       # do not exchange dbsu ssh key among pgsql cluster
    pgbackrest_repo:                  # pgbackrest repo: https://pgbackrest.org/configuration.html#section-repository
      local:                          # default pgbackrest repo with local posix fs
        path: /pg/backup              # local backup directory, `/pg/backup` by default
        retention_full_type: count    # retention full backups by count
        retention_full: 2             # keep 2, at most 3 full backup when using local fs repo
      minio:                          # optional minio repo for pgbackrest
        type: s3                      # minio is s3-compatible, so s3 is used
        s3_endpoint: sss.pigsty       # minio endpoint domain name, `sss.pigsty` by default
        s3_region: us-east-1          # minio region, us-east-1 by default, useless for minio
        s3_bucket: pgsql              # minio bucket name, `pgsql` by default
        s3_key: pgbackrest            # minio user access key for pgbackrest
        s3_key_secret: S3User.Backup  # minio user secret key for pgbackrest
        s3_uri_style: path            # use path style uri for minio rather than host style
        path: /pgbackrest             # minio backup path, default is `//pgbackrest`
        storage_port: 9000            # minio port, 9000 by default
        storage_ca_file: /etc/pki/ca.crt  # minio ca file path, `/etc/pki/ca.crt` by default
        block: y                      # Enable block incremental backup
        bundle: y                     # bundle small files into a single file
        bundle_limit: 20MiB           # Limit for file bundles, 20MiB for object storage
        bundle_size: 128MiB           # Target size for file bundles, 128MiB for object storage
        cipher_type: aes-256-cbc      # enable AES encryption for remote backup repo
        cipher_pass: pgBackRest       # AES encryption password, default is 'pgBackRest'
        retention_full_type: time     # retention full backup by time on minio repo
        retention_full: 14            # keep full backup for last 14 days
    pg_crontab:  # make a full backup on monday 1am, and an incremental backup during weekdays
      - '00 01  * * * /pg/bin/pg-backup'
      - '00 05 * * *  /pg/bin/pg-vacuum'
    pg_hba_rules:   # https://pigsty.io/docs/pgsql/config/hba
      - { user: all ,db: all ,addr: intra ,auth: pwd ,title: 'everyone intranet access with password' ,order: 800 }

    #==========================================================#
    # Repo
    #==========================================================#
    repo_packages: [
      node-bootstrap, infra-package, infra-addons, node-package1, node-package2, node-package3, pgsql-utility, extra-modules,
      pg18-core ,pg18-time ,pg18-gis ,pg18-rag ,pg18-fts ,pg18-olap ,pg18-feat ,pg18-lang ,pg18-type ,pg18-util ,pg18-func ,pg18-admin ,pg18-stat ,pg18-sec ,pg18-fdw ,pg18-sim ,pg18-etl
    ]

    #----------------------------------------------#
    # PASSWORD : https://pigsty.io/docs/setup/security/
    #----------------------------------------------#
    grafana_admin_password: pigsty
    grafana_view_password: DBUser.Viewer
    pg_admin_password: DBUser.DBA
    pg_monitor_password: DBUser.Monitor
    pg_replication_password: DBUser.Replicator
    patroni_password: Patroni.API
    haproxy_admin_password: pigsty
    minio_secret_key: S3User.MinIO
    etcd_root_password: Etcd.Root
...

Explanation

The ha/simu template is a large-scale production environment simulation for testing and validating complex scenarios.

Architecture:

  • 2-node HA INFRA (monitoring/alerting/Nginx/DNS)
  • 5-node HA ETCD and MINIO (Silo, multi-disk)
  • 2-node Proxy (HAProxy + Keepalived VIP)
  • Multiple PostgreSQL clusters:
    • pg-meta: 2-node HA
    • pg-v14~v18: Single-node multi-version testing
    • pg-pitr: Single-node PITR testing
    • pg-test: 4-node HA
    • pg-src/pg-dst: 3+2 node replication testing
    • pg-citus: 10-node distributed cluster
  • Multiple Redis modes: primary-replica, sentinel, cluster

Use Cases:

  • Large-scale deployment testing and validation
  • High availability failover drills
  • Performance benchmarking
  • New feature preview and evaluation

Notes:

  • Requires powerful host machine (64GB+ RAM recommended)
  • Uses Vagrant virtual machines for simulation

6.20 - ha/octo

Compact eight-node HA simulation with three INFRA nodes, five etcd nodes, eight object-storage nodes, and two PostgreSQL clusters.

ha/octo uses the first eight nodes from vagrant/spec/deci.rb to build a compact high-availability simulation. It exercises co-located modules, VIPs, remote backup, and larger membership counts. Do not use it directly as a production blueprint without reviewing capacity, security, and failure domains.

Overview

  • Config name: ha/octo
  • Node addresses: 10.10.10.10 through 10.10.10.17
  • INFRA: 3 nodes; only the first builds and serves the local repository, while Docker can be installed separately on all three as noted in comments
  • ETCD: 5 nodes on the last five hosts
  • Object storage: one eight-node, single-drive cluster; the template does not override minio_type, so both deployment and removal roles default to Silo; verify that value, the exact target, and data paths before removal
  • pg-meta: 3-node PostgreSQL cluster with VIP 10.10.10.2/24
  • pg-test: 5-node PostgreSQL cluster whose final instance has the offline role, with VIP 10.10.10.3/24
  • Backup: uses the object-storage repository through sss.pigsty:9002 and also retains a local repository
./configure -c ha/octo
./deploy.yml

This template depends on fixed eight-node addresses and VIPs. For any other environment, update the host addresses, VIPs, interfaces, DNS, repository node, and every public example credential together.

Content

Source: pigsty/conf/ha/octo.yml

---
#==============================================================#
# File      :   octo.yml
# Desc      :   Pigsty 8-node compact HA simulation config
# Ctime     :   2026-07-29
# Mtime     :   2026-07-29
# Docs      :   https://pigsty.io/docs/conf
# License   :   Apache-2.0 @ https://pigsty.io/docs/about/license/
# Copyright :   2018-2026  Ruohang Feng / Vonng ([email protected])
#==============================================================#

# Use the first 8 nodes from `vagrant/spec/deci.rb`:
#
#  node  address      vagrant name  modules
#  1     10.10.10.10  meta-0        infra(repo,docker), minio-1, pg-meta-1
#  2     10.10.10.11  meta-1        infra(docker),      minio-2, pg-meta-2
#  3     10.10.10.12  meta-2        infra(docker),      minio-3, pg-meta-3
#  4     10.10.10.13  node-3        etcd-1, minio-4, pg-test-1
#  5     10.10.10.14  node-4        etcd-2, minio-5, pg-test-2
#  6     10.10.10.15  node-5        etcd-3, minio-6, pg-test-3
#  7     10.10.10.16  node-6        etcd-4, minio-7, pg-test-4
#  8     10.10.10.17  node-7        etcd-5, minio-8, pg-test-5 (offline)
#
# Nodes 10.10.10.18 and 10.10.10.19 from the deci template are unused.

all:

  #============================================================#
  # Clusters, Nodes, and Modules
  #============================================================#
  children:

    # 3-node infra cluster; only node 1 builds and serves the repo
    infra:
      hosts:
        10.10.10.10: { infra_seq: 1, repo_enabled: true }
        10.10.10.11: { infra_seq: 2, repo_enabled: false }
        10.10.10.12: { infra_seq: 3, repo_enabled: false }
      vars:
        docker_enabled: true          # install with ./docker.yml -l infra

    # 5-node etcd cluster, co-located with pg-test
    etcd:
      hosts:
        10.10.10.13: { etcd_seq: 1 }
        10.10.10.14: { etcd_seq: 2 }
        10.10.10.15: { etcd_seq: 3 }
        10.10.10.16: { etcd_seq: 4 }
        10.10.10.17: { etcd_seq: 5 }
      vars:
        etcd_cluster: etcd

    # 8-node single-drive MinIO cluster, spanning all nodes
    minio:
      hosts:
        10.10.10.10: { minio_seq: 1, vip_role: master }
        10.10.10.11: { minio_seq: 2 }
        10.10.10.12: { minio_seq: 3 }
        10.10.10.13: { minio_seq: 4 }
        10.10.10.14: { minio_seq: 5 }
        10.10.10.15: { minio_seq: 6 }
        10.10.10.16: { minio_seq: 7 }
        10.10.10.17: { minio_seq: 8 }
      vars:
        minio_cluster: minio
        minio_data: /data/minio       # 8 nodes x 1 disk
        minio_users:
          - { access_key: pgbackrest  ,secret_key: S3User.Backup ,policy: pgsql }
          - { access_key: s3user_meta ,secret_key: S3User.Meta   ,policy: meta  }
          - { access_key: s3user_data ,secret_key: S3User.Data   ,policy: data  }

        # HA MinIO endpoint: https://sss.pigsty:9002
        vip_enabled: true
        vip_vrid: 128
        vip_address: 10.10.10.9
        haproxy_services:
          - name: minio
            port: 9002
            balance: leastconn
            options:
              - option httpchk
              - option http-keep-alive
              - http-check send meth OPTIONS uri /minio/health/live
              - http-check expect status 200
            servers:
              - { name: minio-1 ,ip: 10.10.10.10 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
              - { name: minio-2 ,ip: 10.10.10.11 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
              - { name: minio-3 ,ip: 10.10.10.12 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
              - { name: minio-4 ,ip: 10.10.10.13 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
              - { name: minio-5 ,ip: 10.10.10.14 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
              - { name: minio-6 ,ip: 10.10.10.15 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
              - { name: minio-7 ,ip: 10.10.10.16 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
              - { name: minio-8 ,ip: 10.10.10.17 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }

    # 3-node PostgreSQL meta cluster, co-located with infra
    pg-meta:
      hosts:
        10.10.10.10: { pg_seq: 1, pg_role: primary }
        10.10.10.11: { pg_seq: 2, pg_role: replica }
        10.10.10.12: { pg_seq: 3, pg_role: replica }
      vars:
        pg_cluster: pg-meta
        pg_users:
          - { name: dbuser_meta ,password: DBUser.Meta   ,pgbouncer: true ,roles: [ dbrole_admin ]    ,comment: pigsty admin user }
          - { name: dbuser_view ,password: DBUser.Viewer ,pgbouncer: true ,roles: [ dbrole_readonly ] ,comment: read-only viewer for meta database }
        pg_databases:
          - { name: meta ,baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [ pigsty ] }
        pg_vip_enabled: true
        pg_vip_address: 10.10.10.2/24
        pg_crontab:
          - '00 01 * * * /pg/bin/pg-backup full'

    # 5-node PostgreSQL test cluster; node 8 is the offline instance
    pg-test:
      hosts:
        10.10.10.13: { pg_seq: 1, pg_role: primary }
        10.10.10.14: { pg_seq: 2, pg_role: replica }
        10.10.10.15: { pg_seq: 3, pg_role: replica }
        10.10.10.16: { pg_seq: 4, pg_role: replica }
        10.10.10.17: { pg_seq: 5, pg_role: offline }
      vars:
        pg_cluster: pg-test
        pg_users:
          - { name: test ,password: test ,pgbouncer: true ,roles: [ dbrole_admin ] }
        pg_databases:
          - { name: test }
        pg_vip_enabled: true
        pg_vip_address: 10.10.10.3/24
        pg_crontab:
          - '00 01 * * 1 /pg/bin/pg-backup full'
          - '00 01 * * 2,3,4,5,6,7 /pg/bin/pg-backup'

  #============================================================#
  # Global Parameters
  #============================================================#
  vars:
    version: v4.5.0
    admin_ip: 10.10.10.10
    region: default
    node_tune: oltp
    pg_conf: oltp.yml

    proxy_env:
      no_proxy: "localhost,127.0.0.1,10.0.0.0/8,192.168.0.0/16,*.pigsty,*.aliyun.com,mirrors.*,*.myqcloud.com,*.tsinghua.edu.cn"
      # http_proxy:
      # https_proxy:
      # all_proxy:

    infra_portal:
      home:  { domain: i.pigsty }
      minio: { domain: m.pigsty ,endpoint: "10.10.10.10:9001" ,scheme: https ,websocket: true }

    # Node 1 serves the local repository; every node installs from it
    repo_remove: true
    node_repo_remove: true
    node_repo_modules: local
    repo_extra_packages: [ pg18-main ]
    pg_version: 18

    # MinIO VIP and pgBackRest object-storage repository
    minio_endpoint: https://sss.pigsty:9002
    node_etc_hosts:
      - '${admin_ip} i.pigsty'
      - '10.10.10.9 sss.pigsty'
    pgbackrest_method: minio
    pgbackrest_repo:
      local:
        path: /pg/backup
        retention_full_type: count
        retention_full: 2
      minio:
        type: s3
        s3_endpoint: sss.pigsty
        s3_region: us-east-1
        s3_bucket: pgsql
        s3_key: pgbackrest
        s3_key_secret: S3User.Backup
        s3_uri_style: path
        path: /pgbackrest
        storage_port: 9002
        storage_ca_file: /etc/pki/ca.crt
        block: y
        bundle: y
        bundle_limit: 20MiB
        bundle_size: 128MiB
        cipher_type: aes-256-cbc
        cipher_pass: pgBackRest
        retention_full_type: time
        retention_full: 14

    # Default credentials for this disposable sample
    grafana_admin_password: pigsty
    grafana_view_password: DBUser.Viewer
    pg_admin_password: DBUser.DBA
    pg_monitor_password: DBUser.Monitor
    pg_replication_password: DBUser.Replicator
    patroni_password: Patroni.API
    haproxy_admin_password: pigsty
    minio_secret_key: S3User.MinIO
    etcd_root_password: Etcd.Root

...

Explanation

  • The three INFRA nodes and five etcd nodes are separate sets. The pg-meta and pg-test PostgreSQL clusters are co-located with those two sets respectively.
  • Object storage spans all eight nodes and exposes sss.pigsty through Keepalived VIP 10.10.10.9 and HAProxy port 9002. Silo is the current default engine, while the module and variables retain minio_* compatibility names.
  • pg-meta takes one full backup daily. pg-test takes a weekly full backup and incremental backups on the remaining days; both write to the encrypted S3 pgBackRest repository.
  • The two INFRA replicas with repo_enabled: false do not build local repositories. Every node still installs packages from the first node’s local repository.
  • The database, Grafana, Patroni, HAProxy, Silo, and etcd passwords at the end of the template are suitable only for a disposable simulation and must all be rotated in real environments.

For a conventional minimal HA deployment, prefer ha/trio. For a larger full-scenario simulation, see ha/simu.

6.21 - ha/full

Four-node complete feature demonstration environment with two PostgreSQL clusters, Silo, Redis, etc.

The ha/full configuration template is Pigsty’s recommended sandbox demonstration environment, deploying two PostgreSQL clusters across four nodes for testing and demonstrating various Pigsty capabilities.

Most Pigsty tutorials and examples are based on this template’s sandbox environment.

Overview

  • Config Name: ha/full
  • Node Count: Four nodes
  • Description: Four-node complete feature demonstration environment with two PostgreSQL clusters, Silo, Redis, etc.
  • OS Distro: el8, el9, el10, d12, d13, u22, u24, u26
  • OS Arch: x86_64, aarch64
  • Related: ha/trio, ha/safe, demo/demo

Usage:

./configure -c ha/full [-i <primary_ip>]

After configuration, modify the IP addresses of the other three nodes.

Content

Source: pigsty/conf/ha/full.yml

---
#==============================================================#
# File      :   full.yml
# Desc      :   Pigsty Local Sandbox 4-node Demo Config
# Ctime     :   2020-05-22
# Mtime     :   2026-01-16
# Docs      :   https://pigsty.io/docs/conf/full
# License   :   Apache-2.0 @ https://pigsty.io/docs/about/license/
# Copyright :   2018-2026  Ruohang Feng / Vonng ([email protected])
#==============================================================#


all:

  #==============================================================#
  # Clusters, Nodes, and Modules
  #==============================================================#
  children:

    # infra: monitor, alert, repo, etc..
    infra:
      hosts:
        10.10.10.10: { infra_seq: 1 }
      vars:
        docker_enabled: true      # enabled docker with ./docker.yml
        #docker_registry_mirrors: ["https://docker.1panel.live","https://docker.1ms.run","https://docker.xuanyuan.me","https://registry-1.docker.io"]
        #repo_extra_packages: [ pg18-main ,pg18-time ,pg18-gis ,pg18-rag ,pg18-fts ,pg18-olap ,pg18-feat ,pg18-lang ,pg18-type ,pg18-util ,pg18-func ,pg18-admin ,pg18-stat ,pg18-sec ,pg18-fdw ,pg18-sim ,pg18-etl]

    # etcd cluster for HA postgres DCS
    etcd:
      hosts:
        10.10.10.10: { etcd_seq: 1 }
      vars:
        etcd_cluster: etcd

    # minio (single node, used as backup repo)
    minio:
      hosts:
        10.10.10.10: { minio_seq: 1 }
      vars:
        minio_cluster: minio
        minio_users:                      # list of minio user to be created
          - { access_key: pgbackrest  ,secret_key: S3User.Backup ,policy: pgsql }
          - { access_key: s3user_meta ,secret_key: S3User.Meta   ,policy: meta  }
          - { access_key: s3user_data ,secret_key: S3User.Data   ,policy: data  }

    # postgres cluster: pg-meta
    pg-meta:
      hosts:
        10.10.10.10: { pg_seq: 1, pg_role: primary }
      vars:
        pg_cluster: pg-meta
        pg_users:
          - { name: dbuser_meta ,password: DBUser.Meta     ,pgbouncer: true ,roles: [ dbrole_admin ]    ,comment: pigsty admin user }
          - { name: dbuser_view ,password: DBUser.Viewer   ,pgbouncer: true ,roles: [ dbrole_readonly ] ,comment: read-only viewer for meta database }
        pg_databases:
          - { name: meta ,baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [ pigsty ] }
        pg_hba_rules:   # https://pigsty.io/docs/pgsql/config/hba
          - { user: all ,db: all ,addr: intra ,auth: pwd ,title: 'everyone intranet access with password' ,order: 800 }
        pg_crontab:     # https://pigsty.io/docs/pgsql/admin/crontab
          - '00 01 * * * /pg/bin/pg-backup full'
        pg_vip_enabled: true
        pg_vip_address: 10.10.10.2/24


    # pgsql 3 node ha cluster: pg-test
    pg-test:
      hosts:
        10.10.10.11: { pg_seq: 1, pg_role: primary }   # primary instance, leader of cluster
        10.10.10.12: { pg_seq: 2, pg_role: replica }   # replica instance, follower of leader
        10.10.10.13: { pg_seq: 3, pg_role: replica, pg_offline_query: true } # replica with offline access
      vars:
        pg_cluster: pg-test           # define pgsql cluster name
        pg_users:  [{ name: test , password: test , pgbouncer: true , roles: [ dbrole_admin ] }]
        pg_databases: [{ name: test }]
        pg_vip_enabled: true
        pg_vip_address: 10.10.10.3/24
        pg_crontab:  # make a full backup on monday 1am, and an incremental backup during weekdays
          - '00 01 * * 1 /pg/bin/pg-backup full'
          - '00 01 * * 2,3,4,5,6,7 /pg/bin/pg-backup'

    #----------------------------------#
    # redis ms, sentinel, native cluster
    #----------------------------------#
    redis-ms: # redis classic primary & replica
      hosts: { 10.10.10.10: { redis_node: 1 , redis_instances: { 6379: { }, 6380: { replica_of: '10.10.10.10 6379' } } } }
      vars: { redis_cluster: redis-ms ,redis_password: 'redis.ms' ,redis_max_memory: 64MB }

    redis-meta: # redis sentinel x 3
      hosts: { 10.10.10.11: { redis_node: 1 , redis_instances: { 26379: { } ,26380: { } ,26381: { } } } }
      vars:
        redis_cluster: redis-meta
        redis_password: 'redis.meta'
        redis_mode: sentinel
        redis_max_memory: 16MB
        redis_sentinel_monitor: # primary list for redis sentinel, use cls as name, primary ip:port
          - { name: redis-ms, host: 10.10.10.10, port: 6379 ,password: redis.ms, quorum: 2 }

    redis-test: # redis native cluster: 3m x 3s
      hosts:
        10.10.10.12: { redis_node: 1 ,redis_instances: { 6379: { } ,6380: { } ,6381: { } } }
        10.10.10.13: { redis_node: 2 ,redis_instances: { 6379: { } ,6380: { } ,6381: { } } }
      vars: { redis_cluster: redis-test ,redis_password: 'redis.test' ,redis_mode: cluster, redis_max_memory: 32MB }


  #==============================================================#
  # Global Parameters
  #==============================================================#
  vars:
    version: v4.5.0                   # pigsty version string
    admin_ip: 10.10.10.10             # admin node ip address
    region: default                   # upstream mirror region: default|china|europe
    node_tune: oltp                   # node tuning specs: oltp,olap,tiny,crit
    pg_conf: oltp.yml                 # pgsql tuning specs: {oltp,olap,tiny,crit}.yml
    proxy_env:                        # global proxy env when downloading packages
      no_proxy: "localhost,127.0.0.1,10.0.0.0/8,192.168.0.0/16,*.pigsty,*.aliyun.com,mirrors.*,*.myqcloud.com,*.tsinghua.edu.cn"
      # http_proxy:  # set your proxy here: e.g http://user:[email protected]
      # https_proxy: # set your proxy here: e.g http://user:[email protected]
      # all_proxy:   # set your proxy here: e.g http://user:[email protected]
    infra_portal:                     # infra services exposed via portal
      home : { domain: i.pigsty }     # default domain name
      #minio : { domain: m.pigsty ,endpoint: "${admin_ip}:9001" ,scheme: https ,websocket: true }

    #----------------------------------#
    # MinIO Related Options
    #----------------------------------#
    node_etc_hosts: [ '${admin_ip} i.pigsty sss.pigsty' ]
    pgbackrest_method: minio          # if you want to use minio as backup repo instead of 'local' fs, uncomment this
    pgbackrest_repo:                  # pgbackrest repo: https://pgbackrest.org/configuration.html#section-repository
      local:                          # default pgbackrest repo with local posix fs
        path: /pg/backup              # local backup directory, `/pg/backup` by default
        retention_full_type: count    # retention full backups by count
        retention_full: 2             # keep 2, at most 3 full backup when using local fs repo
      minio:                          # optional minio repo for pgbackrest
        type: s3                      # minio is s3-compatible, so s3 is used
        s3_endpoint: sss.pigsty       # minio endpoint domain name, `sss.pigsty` by default
        s3_region: us-east-1          # minio region, us-east-1 by default, useless for minio
        s3_bucket: pgsql              # minio bucket name, `pgsql` by default
        s3_key: pgbackrest            # minio user access key for pgbackrest
        s3_key_secret: S3User.Backup  # minio user secret key for pgbackrest
        s3_uri_style: path            # use path style uri for minio rather than host style
        path: /pgbackrest             # minio backup path, default is `/pgbackrest`
        storage_port: 9000            # minio port, 9000 by default
        storage_ca_file: /etc/pki/ca.crt  # minio ca file path, `/etc/pki/ca.crt` by default
        block: y                      # Enable block incremental backup
        bundle: y                     # bundle small files into a single file
        bundle_limit: 20MiB           # Limit for file bundles, 20MiB for object storage
        bundle_size: 128MiB           # Target size for file bundles, 128MiB for object storage
        cipher_type: aes-256-cbc      # enable AES encryption for remote backup repo
        cipher_pass: pgBackRest       # AES encryption password, default is 'pgBackRest'
        retention_full_type: time     # retention full backup by time on minio repo
        retention_full: 14            # keep full backup for last 14 days

    #----------------------------------#
    # Repo, Node, Packages
    #----------------------------------#
    repo_remove: true                 # remove existing repo on admin node during repo bootstrap
    node_repo_remove: true            # remove existing node repo for node managed by pigsty
    repo_extra_packages: [ pg18-main ] #,pg18-core ,pg18-time ,pg18-gis ,pg18-rag ,pg18-fts ,pg18-olap ,pg18-feat ,pg18-lang ,pg18-type ,pg18-util ,pg18-func ,pg18-admin ,pg18-stat ,pg18-sec ,pg18-fdw ,pg18-sim ,pg18-etl]
    pg_version: 18                    # default postgres version
    #pg_extensions: [pg18-time ,pg18-gis ,pg18-rag ,pg18-fts ,pg18-feat ,pg18-lang ,pg18-type ,pg18-util ,pg18-func ,pg18-admin ,pg18-stat ,pg18-sec ,pg18-fdw ,pg18-sim ,pg18-etl ,pg18-olap]

    #----------------------------------------------#
    # PASSWORD : https://pigsty.io/docs/setup/security/
    #----------------------------------------------#
    grafana_admin_password: pigsty
    grafana_view_password: DBUser.Viewer
    pg_admin_password: DBUser.DBA
    pg_monitor_password: DBUser.Monitor
    pg_replication_password: DBUser.Replicator
    patroni_password: Patroni.API
    haproxy_admin_password: pigsty
    minio_secret_key: S3User.MinIO
    etcd_root_password: Etcd.Root
...

Explanation

The ha/full template is Pigsty’s complete feature demonstration configuration, showcasing the collaboration of various components.

Components Overview:

Component Node Distribution Description
INFRA Node 1 Monitoring/Alerting/Nginx/DNS
ETCD Node 1 DCS Service
Silo Node 1 S3-compatible Storage
pg-meta Node 1 Single-node PostgreSQL
pg-test Nodes 2-4 Three-node HA PostgreSQL
redis-ms Node 1 Redis Primary-Replica Mode
redis-meta Node 2 Redis Sentinel Mode
redis-test Nodes 3-4 Redis Native Cluster Mode

Use Cases:

  • Pigsty feature demonstration and learning
  • Development testing environments
  • Evaluating HA architecture
  • Comparing different Redis modes

Differences from ha/trio:

  • Added second PostgreSQL cluster (pg-test)
  • Added three Redis cluster mode examples
  • Infrastructure uses single node (instead of three nodes)

Notes:

  • This template is mainly for demonstration and testing; for production, refer to ha/trio or ha/safe
  • MINIO object-storage backup is enabled by default. The current source defaults to Silo; comment out the related configuration if it is not needed

6.22 - ha/safe

Three-node high-availability and security-hardening configuration example.

ha/safe uses a three-node high-availability topology to demonstrate TLS, client certificates, password checks, backup encryption, the CRIT parameter template, and related security settings. It is a configuration example to customize, not a compliance-certified template.

Overview

  • Configuration: ha/safe
  • Nodes: 3 INFRA, etcd, and PostgreSQL nodes; optional delayed replica
  • Operating systems: el8, el9, el10, d12, d13, u22, u24, u26
  • Architecture: x86_64; some security extensions do not have ARM64 packages
  • Related configurations: ha/trio, ha/full

Generate the configuration:

./configure -c ha/safe -g [-i <primary_ip>]

-g randomizes only credentials recognized by the configuration wizard. You must still replace Silo users, the pgBackRest cipher_pass, and other template example values.

Hardening Controls

Setting Template Behavior Boundary and Follow-up
PostgreSQL HBA Main TCP rules use ssl; public administrator access uses cert Local ident and selected localhost pwd rules remain
PgBouncer pgbouncer_sslmode: require Clients must still verify the server certificate where required
Patroni REST API uses HTTPS and a constrained listen address Basic Auth remains; rotate the password
Password check passwordcheck is preloaded through pg_libs Affects only newly set or changed passwords
Account lifetime Built-in and example application users set expire_in: 7300 Twenty years is not a rotation policy; shorten it to organizational requirements
Listen addresses PostgreSQL is limited to ${ip},${vip},${lo} Firewalls and HBA are still required
Backup Uses Silo with AES-256-CBC pgBR.${pg_cluster} is a predictable example and must be replaced
PostgreSQL parameters pg-meta uses crit.yml Strict synchronous mode can block writes without a synchronous replica
Logging CRIT logs connection and disconnection events Fine-grained SQL auditing requires explicit pgaudit configuration
Security extensions Installs passwordcheck, credcheck, pgaudit, and related packages Installation does not preload, create, or configure an extension
Delayed replica Provides a commented one-hour delayed-cluster example Not created by default; enable it explicitly

Preflight Checklist

  • Replace every public example credential, especially minio_users, pgbackrest_repo, application users, and API passwords.
  • Confirm that the three nodes occupy independent failure domains, and update IPs, VIP, and domains for the target network.
  • Configure database clients with sslmode=verify-full and a trusted CA.
  • Confirm that the availability impact of strict synchronous mode meets application requirements.
  • Preload and configure pgaudit, credcheck, and other extensions as required.
  • Check extension package availability on ARM64.
  • Test backup recovery, failover, and certificate verification.

See Security Model, Authentication, Encrypted Communication, and Data Security for the underlying mechanisms.

Configuration

Source: pigsty/conf/ha/safe.yml

---
#==============================================================#
# File      :   safe.yml
# Desc      :   Pigsty 3-node security enhance template
# Ctime     :   2020-05-22
# Mtime     :   2025-12-12
# Docs      :   https://pigsty.io/docs/conf/safe
# License   :   Apache-2.0 @ https://pigsty.io/docs/about/license/
# Copyright :   2018-2026  Ruohang Feng / Vonng ([email protected])
#==============================================================#


#===== SECURITY ENHANCEMENT CONFIG TEMPLATE WITH 3 NODES ======#
#   * 3 infra nodes, 3 etcd nodes, single minio node
#   * 3-instance pgsql cluster with an extra delayed instance
#   * crit.yml templates, no data loss, checksum enforced
#   * enforce ssl on postgres & pgbouncer, use postgres by default
#   * enforce an expiration date for all users (20 years by default)
#   * enforce strong password policy with passwordcheck extension
#   * enforce changing default password for all users
#   * log connections and disconnections
#   * restrict listen ip address for postgres/patroni/pgbouncer


all:
  children:

    infra: # infra cluster for proxy, monitor, alert, etc
      hosts: # 1 for common usage, 3 nodes for production
        10.10.10.10: { infra_seq: 1 } # identity required
        10.10.10.11: { infra_seq: 2, repo_enabled: false }
        10.10.10.12: { infra_seq: 3, repo_enabled: false }
      vars: { patroni_watchdog_mode: 'off' }

    minio: # minio cluster, s3 compatible object storage
      hosts: { 10.10.10.10: { minio_seq: 1 } }
      vars: { minio_cluster: minio }

    etcd: # dcs service for postgres/patroni ha consensus
      hosts: # 1 node for testing, 3 or 5 for production
        10.10.10.10: { etcd_seq: 1 }  # etcd_seq required
        10.10.10.11: { etcd_seq: 2 }  # assign from 1 ~ n
        10.10.10.12: { etcd_seq: 3 }  # three-member cluster keeps an odd voter count
      vars: # cluster level parameter override roles/etcd
        etcd_cluster: etcd  # mark etcd cluster name etcd
        etcd_safeguard: false # safeguard against purging

    pg-meta: # 3 instance postgres cluster `pg-meta`
      hosts:
        10.10.10.10: { pg_seq: 1, pg_role: primary }
        10.10.10.11: { pg_seq: 2, pg_role: replica }
        10.10.10.12: { pg_seq: 3, pg_role: replica , pg_offline_query: true }
      vars:
        pg_cluster: pg-meta
        pg_conf: crit.yml
        pg_users:
          - { name: dbuser_meta , password: Pleas3-ChangeThisPwd ,expire_in: 7300 ,pgbouncer: true ,roles: [ dbrole_admin ]    ,comment: pigsty admin user }
          - { name: dbuser_view , password: Make.3ure-Compl1ance  ,expire_in: 7300 ,pgbouncer: true ,roles: [ dbrole_readonly ] ,comment: read-only viewer for meta database }
        pg_databases:
          - { name: meta ,baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [ pigsty ] ,extensions: [ { name: vector } ] }
        pg_services:
          - { name: standby , ip: "*" ,port: 5435 , dest: default ,selector: "[]" , backup: "[? pg_role == `primary`]" }
        pg_crontab:     # https://pigsty.io/docs/pgsql/admin/crontab
          - '00 01 * * * /pg/bin/pg-backup full'
        pg_listen: '${ip},${vip},${lo}'
        pg_vip_enabled: true
        pg_vip_address: 10.10.10.2/24

    # OPTIONAL delayed cluster for pg-meta
    #pg-meta-delay: # delayed instance for pg-meta (1 hour ago)
    #  hosts: { 10.10.10.13: { pg_seq: 1, pg_role: primary, pg_upstream: 10.10.10.10, pg_delay: 1h } }
    #  vars: { pg_cluster: pg-meta-delay }


  ####################################################################
  #                          Parameters                              #
  ####################################################################
  vars: # global variables
    version: v4.5.0                   # pigsty version string
    admin_ip: 10.10.10.10             # admin node ip address
    region: default                   # upstream mirror region: default|china|europe
    node_tune: oltp                   # node tuning specs: oltp,olap,tiny,crit
    pg_conf: oltp.yml                 # pgsql tuning specs: {oltp,olap,tiny,crit}.yml
    #docker_registry_mirrors: ["https://docker.1panel.live","https://docker.1ms.run","https://docker.xuanyuan.me","https://registry-1.docker.io"]
    patroni_ssl_enabled: true         # secure patroni RestAPI communications with SSL?
    pgbouncer_sslmode: require        # pgbouncer client ssl mode: disable|allow|prefer|require|verify-ca|verify-full, disable by default
    pg_default_service_dest: postgres # default service destination to postgres instead of pgbouncer
    pgbackrest_method: minio          # pgbackrest repo method: local,minio,[user-defined...]

    #----------------------------------#
    # MinIO Related Options
    #----------------------------------#
    minio_users: # and configure `pgbackrest_repo` & `minio_users` accordingly
      - { access_key: dba , secret_key: S3User.DBA.Strong.Password, policy: consoleAdmin }
      - { access_key: pgbackrest , secret_key: Min10.bAckup ,policy: readwrite }
    pgbackrest_repo: # pgbackrest repo: https://pgbackrest.org/configuration.html#section-repository
      local: # default pgbackrest repo with local posix fs
        path: /pg/backup              # local backup directory, `/pg/backup` by default
        retention_full_type: count    # retention full backups by count
        retention_full: 2             # keep 2, at most 3 full backup when using local fs repo
      minio: # optional minio repo for pgbackrest
        s3_key: pgbackrest            # <-------- CHANGE THIS, SAME AS `minio_users` access_key
        s3_key_secret: Min10.bAckup   # <-------- CHANGE THIS, SAME AS `minio_users` secret_key
        cipher_pass: 'pgBR.${pg_cluster}'  # <-------- CHANGE THIS, you can use cluster name as part of password
        type: s3                      # minio is s3-compatible, so s3 is used
        s3_endpoint: sss.pigsty       # minio endpoint domain name, `sss.pigsty` by default
        s3_region: us-east-1          # minio region, us-east-1 by default, useless for minio
        s3_bucket: pgsql              # minio bucket name, `pgsql` by default
        s3_uri_style: path            # use path style uri for minio rather than host style
        path: /pgbackrest             # minio backup path, default is `/pgbackrest`
        storage_port: 9000            # minio port, 9000 by default
        storage_ca_file: /etc/pki/ca.crt  # minio ca file path, `/etc/pki/ca.crt` by default
        block: y                      # Enable block incremental backup
        bundle: y                     # bundle small files into a single file
        bundle_limit: 20MiB           # Limit for file bundles, 20MiB for object storage
        bundle_size: 128MiB           # Target size for file bundles, 128MiB for object storage
        cipher_type: aes-256-cbc      # enable AES encryption for remote backup repo
        retention_full_type: time     # retention full backup by time on minio repo
        retention_full: 14            # keep full backup for last 14 days


    #----------------------------------#
    # Access Control
    #----------------------------------#
    # add passwordcheck_cracklib extension to enforce strong password policy
    pg_libs: '$libdir/passwordcheck_cracklib, pg_stat_statements, auto_explain'
    pg_extensions:
      - passwordcheck_cracklib, supautils, pgsodium, pg_vault, pg_session_jwt, pg_anon, pgsmcrypto, pgauditlogtofile, pgaudit #, pgaudit17, pgaudit16, pgaudit15, pgaudit14
      - pg_auth_mon, credcheck, pgcryptokey, pg_jobmon, logerrors, login_hook, set_user, pgextwlist, pg_auditor, sslutils, pg_noset #pg_tde #pg_snakeoil
    pg_default_roles: # default roles and users in postgres cluster
      - { name: dbrole_readonly  ,login: false ,comment: role for global read-only access }
      - { name: dbrole_offline   ,login: false ,comment: role for restricted read-only access }
      - { name: dbrole_readwrite ,login: false ,roles: [ dbrole_readonly ]               ,comment: role for global read-write access }
      - { name: dbrole_admin     ,login: false ,roles: [ pg_monitor, dbrole_readwrite ]  ,comment: role for object creation }
      - { name: postgres     ,superuser: true  ,expire_in: 7300                        ,comment: system superuser }
      - { name: replicator ,replication: true  ,expire_in: 7300 ,roles: [ pg_monitor, dbrole_readonly ]   ,comment: system replicator }
      - { name: dbuser_dba   ,superuser: true  ,expire_in: 7300 ,roles: [ dbrole_admin ]  ,pgbouncer: true ,pool_mode: session, pool_connlimit: 16 , comment: pgsql admin user }
      - { name: dbuser_monitor ,roles: [ pg_monitor ] ,expire_in: 7300 ,pgbouncer: true ,parameters: { log_min_duration_statement: 1000 } ,pool_mode: session ,pool_connlimit: 8 ,comment: pgsql monitor user }
    pg_default_hba_rules: # postgres host-based auth rules by default, order by `order`
      - { user: '${dbsu}'    ,db: all         ,addr: local     ,auth: ident ,title: 'dbsu access via local os user ident'   ,order: 100}
      - { user: '${dbsu}'    ,db: replication ,addr: local     ,auth: ident ,title: 'dbsu replication from local os ident'  ,order: 150}
      - { user: '${repl}'    ,db: replication ,addr: localhost ,auth: ssl   ,title: 'replicator replication from localhost' ,order: 200}
      - { user: '${repl}'    ,db: replication ,addr: intra     ,auth: ssl   ,title: 'replicator replication from intranet'  ,order: 250}
      - { user: '${repl}'    ,db: postgres    ,addr: intra     ,auth: ssl   ,title: 'replicator postgres db from intranet'  ,order: 300}
      - { user: '${monitor}' ,db: all         ,addr: localhost ,auth: pwd   ,title: 'monitor from localhost with password'  ,order: 350}
      - { user: '${monitor}' ,db: all         ,addr: infra     ,auth: ssl   ,title: 'monitor from infra host with password' ,order: 400}
      - { user: '${admin}'   ,db: all         ,addr: infra     ,auth: ssl   ,title: 'admin @ infra nodes with pwd & ssl'    ,order: 450}
      - { user: '${admin}'   ,db: all         ,addr: world     ,auth: cert  ,title: 'admin @ everywhere with ssl & cert'    ,order: 500}
      - { user: '+dbrole_readonly',db: all    ,addr: localhost ,auth: ssl   ,title: 'pgbouncer read/write via local socket' ,order: 550}
      - { user: '+dbrole_readonly',db: all    ,addr: intra     ,auth: ssl   ,title: 'read/write biz user via password'      ,order: 600}
      - { user: '+dbrole_offline' ,db: all    ,addr: intra     ,auth: ssl   ,title: 'allow etl offline tasks from intranet' ,order: 650}
    pgb_default_hba_rules: # pgbouncer host-based authentication rules, order by `order`
      - { user: '${dbsu}'    ,db: pgbouncer   ,addr: local     ,auth: peer  ,title: 'dbsu local admin access with os ident' ,order: 100}
      - { user: 'all'        ,db: all         ,addr: localhost ,auth: pwd   ,title: 'allow all user local access with pwd'  ,order: 150}
      - { user: '${monitor}' ,db: pgbouncer   ,addr: intra     ,auth: ssl   ,title: 'monitor access via intranet with pwd'  ,order: 200}
      - { user: '${monitor}' ,db: all         ,addr: world     ,auth: deny  ,title: 'reject all other monitor access addr'  ,order: 250}
      - { user: '${admin}'   ,db: all         ,addr: intra     ,auth: ssl   ,title: 'admin access via intranet with pwd'    ,order: 300}
      - { user: '${admin}'   ,db: all         ,addr: world     ,auth: deny  ,title: 'reject all other admin access addr'    ,order: 350}
      - { user: 'all'        ,db: all         ,addr: intra     ,auth: ssl   ,title: 'allow all user intra access with pwd'  ,order: 400}

    #----------------------------------#
    # Repo, Node, Packages
    #----------------------------------#
    repo_remove: true                 # remove existing repo on admin node during repo bootstrap
    node_repo_remove: true            # remove existing node repo for node managed by pigsty
    #node_selinux_mode: enforcing     # set selinux mode: enforcing,permissive,disabled
    node_firewall_mode: zone          # firewall mode: zone (default), off (disable), none (skip & self-managed)
    repo_extra_packages: [ pg18-main ] #,pg18-core ,pg18-time ,pg18-gis ,pg18-rag ,pg18-fts ,pg18-olap ,pg18-feat ,pg18-lang ,pg18-type ,pg18-util ,pg18-func ,pg18-admin ,pg18-stat ,pg18-sec ,pg18-fdw ,pg18-sim ,pg18-etl]
    pg_version: 18                    # default postgres version
    #pg_extensions: [ pg18-time ,pg18-gis ,pg18-rag ,pg18-fts ,pg18-olap ,pg18-feat ,pg18-lang ,pg18-type ,pg18-util ,pg18-func ,pg18-admin ,pg18-stat ,pg18-sec ,pg18-fdw ,pg18-sim ,pg18-etl]

    #----------------------------------------------#
    # PASSWORD : https://pigsty.io/docs/setup/security/
    #----------------------------------------------#
    #grafana_admin_username: admin
    grafana_admin_password: You.Have2Use-A_VeryStrongPassword
    grafana_view_password: DBUser.Viewer
    #pg_admin_username: dbuser_dba
    pg_admin_password: PessWorb.Should8eStrong-eNough
    #pg_monitor_username: dbuser_monitor
    pg_monitor_password: MekeSuerYour.PassWordI5secured
    #pg_replication_username: replicator
    pg_replication_password: doNotUseThis-PasswordFor.AnythingElse
    #patroni_username: postgres
    patroni_password: don.t-forget-to-change-thEs3-password
    #haproxy_admin_username: admin
    haproxy_admin_password: GneratePasswordWith-pwgen-s-16-1
    minio_secret_key: S3User.MinIO
    etcd_root_password: Etcd.Root
...

6.23 - ha/trio

Three-node standard HA configuration where PostgreSQL, ETCD, and Silo tolerate one node failure

Three nodes is the minimum scale for majority-based high availability. The ha/trio template distributes INFRA, ETCD, PGSQL, and Silo across three servers. PostgreSQL, ETCD, and object storage continue serving when one server is unavailable.

Overview

  • Config Name: ha/trio
  • Node Count: Three nodes
  • Description: Three-node standard HA architecture with a three-node, single-drive Silo cluster and one HA S3 endpoint
  • OS Distro: el8, el9, el10, d12, d13, u22, u24, u26
  • OS Arch: x86_64, aarch64
  • Related: ha/dual, ha/full, ha/safe

Usage:

./configure -c ha/trio [-i <primary_ip>]

After configuration, modify placeholder IPs 10.10.10.11 and 10.10.10.12 to actual node IP addresses.

Content

Source: pigsty/conf/ha/trio.yml

---
#==============================================================#
# File      :   trio.yml
# Desc      :   Pigsty 3-node security enhance template
# Ctime     :   2020-05-23
# Mtime     :   2026-08-14
# Docs      :   https://pigsty.io/docs/conf/trio
# License   :   Apache-2.0 @ https://pigsty.io/docs/about/license/
# Copyright :   2018-2026  Ruohang Feng / Vonng ([email protected])
#==============================================================#

# 3 infra node, 3 etcd node, 3 pgsql node, and 3 minio nodes
all:  # top level object
  #==============================================================#
  # Clusters, Nodes, and Modules
  #==============================================================#
  children:
    #----------------------------------#
    # infra: monitor, alert, repo, etc..
    #----------------------------------#
    infra: # infra cluster for proxy, monitor, alert, etc
      hosts: # 1 for common usage, 3 nodes for production
        10.10.10.10: { infra_seq: 1 } # identity required
        10.10.10.11: { infra_seq: 2, repo_enabled: false }
        10.10.10.12: { infra_seq: 3, repo_enabled: false }
      vars:
        patroni_watchdog_mode: 'off' # do not fencing infra

    etcd: # dcs service for postgres/patroni ha consensus
      hosts: # 1 node for testing, 3 or 5 for production
        10.10.10.10: { etcd_seq: 1 }  # etcd_seq required
        10.10.10.11: { etcd_seq: 2 }  # assign from 1 ~ n
        10.10.10.12: { etcd_seq: 3 }  # three-member cluster keeps an odd voter count
      vars: # cluster level parameter override roles/etcd
        etcd_cluster: etcd  # mark etcd cluster name etcd
        etcd_safeguard: false # safeguard against purging

    # compact 3-node x 1-drive Silo cluster: EC:1, tolerates one node failure
    # use a dedicated local mount for /data/minio; do not expand a 1-node cluster in place
    minio: # minio cluster, s3 compatible object storage
      hosts:
        10.10.10.10: { minio_seq: 1, vip_role: master }
        10.10.10.11: { minio_seq: 2 }
        10.10.10.12: { minio_seq: 3 }
      vars:
        minio_cluster: minio
        minio_data: /data/minio
        minio_users:
          - { access_key: pgbackrest  ,secret_key: S3User.Backup ,policy: pgsql }
          - { access_key: s3user_meta ,secret_key: S3User.Meta   ,policy: meta  }
          - { access_key: s3user_data ,secret_key: S3User.Data   ,policy: data  }
        vip_enabled: true
        vip_vrid: 128
        vip_address: 10.10.10.9
        haproxy_services:
          - name: minio
            port: 9002
            balance: leastconn
            options:
              - option httpchk
              - option http-keep-alive
              - http-check send meth OPTIONS uri /minio/health/live
              - http-check expect status 200
            servers:
              - { name: minio-1, ip: 10.10.10.10, port: 9000, options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
              - { name: minio-2, ip: 10.10.10.11, port: 9000, options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
              - { name: minio-3, ip: 10.10.10.12, port: 9000, options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }

    pg-meta:  # 3 instance postgres cluster `pg-meta`
      hosts:  # pg-meta-3 is marked as offline readable replica
        10.10.10.10: { pg_seq: 1, pg_role: primary }
        10.10.10.11: { pg_seq: 2, pg_role: replica }
        10.10.10.12: { pg_seq: 3, pg_role: replica , pg_offline_query: true }
      vars:   # cluster level parameters
        pg_cluster: pg-meta
        pg_users: # https://pigsty.io/docs/pgsql/config/user
          - { name: dbuser_meta , password: DBUser.Meta ,pgbouncer: true   ,roles: [ dbrole_admin ]    ,comment: pigsty admin user }
          - { name: dbuser_view , password: DBUser.Viewer ,pgbouncer: true ,roles: [ dbrole_readonly ] ,comment: read-only viewer for meta database }
        pg_databases:
          - { name: meta ,baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [ pigsty ] ,extensions: [ { name: vector } ] }
        pg_hba_rules:   # https://pigsty.io/docs/pgsql/config/hba
          - { user: all ,db: all ,addr: intra ,auth: pwd ,title: 'everyone intranet access with password' ,order: 800 }
        pg_crontab:     # https://pigsty.io/docs/pgsql/admin/crontab
          - '00 01 * * * /pg/bin/pg-backup full'
        pg_vip_enabled: true
        pg_vip_address: 10.10.10.2/24


  #==============================================================#
  # Global Parameters
  #==============================================================#
  vars:
    #----------------------------------#
    # Meta Data
    #----------------------------------#
    version: v4.5.0                   # pigsty version string
    admin_ip: 10.10.10.10             # admin node ip address
    region: default                   # upstream mirror region: default|china|europe
    node_tune: oltp                   # node tuning specs: oltp,olap,tiny,crit
    pg_conf: oltp.yml                 # pgsql tuning specs: {oltp,olap,tiny,crit}.yml
    #docker_registry_mirrors: ["https://docker.1panel.live","https://docker.1ms.run","https://docker.xuanyuan.me","https://registry-1.docker.io"]
    proxy_env:                        # global proxy env when downloading packages
      no_proxy: "localhost,127.0.0.1,10.0.0.0/8,192.168.0.0/16,*.pigsty,*.aliyun.com,mirrors.*,*.myqcloud.com,*.tsinghua.edu.cn"
      # http_proxy:  # set your proxy here: e.g http://user:[email protected]
      # https_proxy: # set your proxy here: e.g http://user:[email protected]
      # all_proxy:   # set your proxy here: e.g http://user:[email protected]
    infra_portal:                     # infra services exposed via portal
      home         : { domain: i.pigsty }     # default domain name
      minio        : { domain: m.pigsty ,endpoint: "${admin_ip}:9001" ,scheme: https ,websocket: true }

    #----------------------------------#
    # Repo, Node, Packages
    #----------------------------------#
    repo_remove: true                 # remove existing repo on admin node during repo bootstrap
    node_repo_remove: true            # remove existing node repo for node managed by pigsty
    repo_extra_packages: [ pg18-main ] #,pg18-core ,pg18-time ,pg18-gis ,pg18-rag ,pg18-fts ,pg18-olap ,pg18-feat ,pg18-lang ,pg18-type ,pg18-util ,pg18-func ,pg18-admin ,pg18-stat ,pg18-sec ,pg18-fdw ,pg18-sim ,pg18-etl]
    pg_version: 18                    # default postgres version
    #pg_extensions: [ pg18-time ,pg18-gis ,pg18-rag ,pg18-fts ,pg18-olap ,pg18-feat ,pg18-lang ,pg18-type ,pg18-util ,pg18-func ,pg18-admin ,pg18-stat ,pg18-sec ,pg18-fdw ,pg18-sim ,pg18-etl]

    #----------------------------------#
    # MinIO Related Options
    #----------------------------------#
    minio_endpoint: https://sss.pigsty:9002
    node_etc_hosts:
      - '${admin_ip} i.pigsty'        # static dns record that point to repo node
      - '10.10.10.9 sss.pigsty'       # static dns record that point to minio vip
    pgbackrest_method: minio          # if you want to use minio as backup repo instead of 'local' fs, uncomment this
    pgbackrest_repo:                  # pgbackrest repo: https://pgbackrest.org/configuration.html#section-repository
      local:                          # default pgbackrest repo with local posix fs
        path: /pg/backup              # local backup directory, `/pg/backup` by default
        retention_full_type: count    # retention full backups by count
        retention_full: 2             # keep 2, at most 3 full backup when using local fs repo
      minio:                          # optional minio repo for pgbackrest
        type: s3                      # minio is s3-compatible, so s3 is used
        s3_endpoint: sss.pigsty       # minio endpoint domain name, `sss.pigsty` by default
        s3_region: us-east-1          # minio region, us-east-1 by default, useless for minio
        s3_bucket: pgsql              # minio bucket name, `pgsql` by default
        s3_key: pgbackrest            # minio user access key for pgbackrest
        s3_key_secret: S3User.Backup  # minio user secret key for pgbackrest
        s3_uri_style: path            # use path style uri for minio rather than host style
        path: /pgbackrest             # minio backup path, default is `/pgbackrest`
        storage_port: 9002            # minio ha endpoint exposed by haproxy
        storage_ca_file: /etc/pki/ca.crt  # minio ca file path, `/etc/pki/ca.crt` by default
        block: y                      # Enable block incremental backup
        bundle: y                     # bundle small files into a single file
        bundle_limit: 20MiB           # Limit for file bundles, 20MiB for object storage
        bundle_size: 128MiB           # Target size for file bundles, 128MiB for object storage
        cipher_type: aes-256-cbc      # enable AES encryption for remote backup repo
        cipher_pass: pgBackRest       # AES encryption password, default is 'pgBackRest'
        retention_full_type: time     # retention full backup by time on minio repo
        retention_full: 14            # keep full backup for last 14 days

    #----------------------------------------------#
    # PASSWORD : https://pigsty.io/docs/setup/security/
    #----------------------------------------------#
    grafana_admin_password: pigsty
    grafana_view_password: DBUser.Viewer
    pg_admin_password: DBUser.DBA
    pg_monitor_password: DBUser.Monitor
    pg_replication_password: DBUser.Replicator
    patroni_password: Patroni.API
    haproxy_admin_password: pigsty
    minio_secret_key: S3User.MinIO
    etcd_root_password: Etcd.Root

...

Explanation

The ha/trio template is Pigsty’s standard HA configuration, providing true automatic failover capability.

Architecture:

  • Three-node INFRA: Distributed deployment of VictoriaMetrics/Grafana/Nginx
  • Three-node ETCD: DCS majority election, tolerates single-point failure
  • Three-node PostgreSQL: One primary, two replicas, automatic failover
  • Three-node Silo: One data path per node, using EC:1 by default (two data shards and one parity shard)
  • HA S3 endpoint: Keepalived VIP 10.10.10.9 with HAProxy listening on 9002 on all three nodes

HA Guarantees:

  • Three-node ETCD tolerates one node failure, maintains majority
  • PostgreSQL primary failure triggers automatic Patroni election for new primary
  • L2 VIP follows primary, applications don’t need to modify connection config
  • Silo retains read and write quorum while one node or one data drive is unavailable
  • sss.pigsty resolves to the object-storage VIP; pgBackRest and mcli use https://sss.pigsty:9002

Object Storage:

  • minio_data: /data/minio is a filesystem directory, not a raw device such as /dev/sdb.
  • Distributed Silo rejects data paths on the root filesystem. /data/minio must reside on a separately mounted /data filesystem or be a mount point itself.
  • The backing storage may be a local disk, cloud volume, separate partition, or LVM logical volume. For production, prefer dedicated persistent drives of similar capacity on all three nodes.
  • Use findmnt -T /data/minio to inspect the actual mount. A result that still points to / means the path is only a directory on the root drive.
  • The three-node, single-drive topology provides about two-thirds raw capacity efficiency. It is compact HA; use a multi-node, multi-drive topology for greater capacity, throughput, and drive redundancy.
  • A single-node object-storage pool cannot be converted in place by adding two members. Create a new three-node cluster and migrate the objects instead.

The template’s S3 API endpoint is highly available. The Portal administration UI still connects to port 9001 on the first node and is outside this API HA path.

Use Cases:

  • Minimum HA deployment for production environments
  • Critical business requiring automatic failover
  • Foundation architecture for larger scale deployments

Extension Suggestions:

  • For stronger data security, refer to ha/safe template
  • For more demo features, refer to ha/full template
  • Use a multi-drive Silo cluster when object-storage capacity or performance requirements are higher

6.24 - ha/dual

Two-node configuration, limited HA deployment tolerating specific server failure

The ha/dual template uses two-node deployment, implementing a “semi-HA” architecture with one primary and one standby. If you only have two servers, this is a pragmatic choice.

Overview

  • Config Name: ha/dual
  • Node Count: Two nodes
  • Description: Two-node limited HA deployment, tolerates specific server failure
  • OS Distro: el8, el9, el10, d12, d13, u22, u24, u26
  • OS Arch: x86_64, aarch64
  • Related: ha/trio, slim

Usage:

./configure -c ha/dual [-i <primary_ip>]

After configuration, modify placeholder IP 10.10.10.11 to actual standby node IP address.

Content

Source: pigsty/conf/ha/dual.yml

---
#==============================================================#
# File      :   dual.yml
# Desc      :   Pigsty deployment example for two nodes
# Ctime     :   2020-05-22
# Mtime     :   2025-12-12
# Docs      :   https://pigsty.io/docs/conf/dual
# License   :   Apache-2.0 @ https://pigsty.io/docs/about/license/
# Copyright :   2018-2026  Ruohang Feng / Vonng ([email protected])
#==============================================================#


# It is recommended to use at least three nodes in production deployment.
# But sometimes, there are only two nodes available, that's dual.yml for
#
# In this setup, we have two nodes, .10 (admin_node) and .11 (pgsql_primary):
#
# If .11 is down, .10 will take over since the dcs:etcd is still alive
# If .10 is down, .11 (pgsql primary) will still be functioning as a primary if:
#   - Only dcs:etcd is down
#   - Only pgsql is down
# if both etcd & pgsql are down (e.g. node down), the primary will still demote itself.


all:
  children:

    # infra cluster for proxy, monitor, alert, etc..
    infra: { hosts: { 10.10.10.10: { infra_seq: 1 } } }

    # etcd cluster for ha postgres
    etcd: { hosts: { 10.10.10.10: { etcd_seq: 1 } }, vars: { etcd_cluster: etcd } }

    # minio cluster, optional backup repo for pgbackrest
    #minio: { hosts: { 10.10.10.10: { minio_seq: 1 } }, vars: { minio_cluster: minio } }

    # postgres cluster 'pg-meta' with single primary instance
    pg-meta:
      hosts:
        10.10.10.10: { pg_seq: 1, pg_role: replica }
        10.10.10.11: { pg_seq: 2, pg_role: primary }  # <----- use this as primary by default
      vars:
        pg_cluster: pg-meta
        pg_databases: [ { name: meta ,baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [ pigsty ] ,extensions: [ { name: vector }] } ]
        pg_users:
          - { name: dbuser_meta ,password: DBUser.Meta   ,pgbouncer: true ,roles: [ dbrole_admin ]    ,comment: pigsty admin user }
          - { name: dbuser_view ,password: DBUser.Viewer ,pgbouncer: true ,roles: [ dbrole_readonly ] ,comment: read-only viewer for meta database }
        pg_hba_rules:   # https://pigsty.io/docs/pgsql/config/hba
          - { user: all ,db: all ,addr: intra ,auth: pwd ,title: 'everyone intranet access with password' ,order: 800 }
        pg_crontab:     # https://pigsty.io/docs/pgsql/admin/crontab
          - '00 01 * * * /pg/bin/pg-backup full'
        pg_vip_enabled: true
        pg_vip_address: 10.10.10.2/24

  vars:                               # global parameters
    version: v4.5.0                   # pigsty version string
    admin_ip: 10.10.10.10             # admin node ip address
    region: default                   # upstream mirror region: default,china,europe
    node_tune: oltp                   # node tuning specs: oltp,olap,tiny,crit
    pg_conf: oltp.yml                 # pgsql tuning specs: {oltp,olap,tiny,crit}.yml
    #docker_registry_mirrors: ["https://docker.1panel.live","https://docker.1ms.run","https://docker.xuanyuan.me","https://registry-1.docker.io"]
    infra_portal:                     # domain names and upstream servers
      home   : { domain: i.pigsty }
      #minio : { domain: m.pigsty ,endpoint: "${admin_ip}:9001" ,scheme: https ,websocket: true }

    #----------------------------------#
    # Repo, Node, Packages
    #----------------------------------#
    repo_remove: true                 # remove existing repo on admin node during repo bootstrap
    node_repo_remove: true            # remove existing node repo for node managed by pigsty
    repo_extra_packages: [ pg18-main ] #,pg18-core ,pg18-time ,pg18-gis ,pg18-rag ,pg18-fts ,pg18-olap ,pg18-feat ,pg18-lang ,pg18-type ,pg18-util ,pg18-func ,pg18-admin ,pg18-stat ,pg18-sec ,pg18-fdw ,pg18-sim ,pg18-etl]
    pg_version: 18                    # default postgres version
    #pg_extensions: [ pg18-time ,pg18-gis ,pg18-rag ,pg18-fts ,pg18-olap ,pg18-feat ,pg18-lang ,pg18-type ,pg18-util ,pg18-func ,pg18-admin ,pg18-stat ,pg18-sec ,pg18-fdw ,pg18-sim ,pg18-etl]

    #----------------------------------------------#
    # PASSWORD : https://pigsty.io/docs/setup/security/
    #----------------------------------------------#
    grafana_admin_password: pigsty
    grafana_view_password: DBUser.Viewer
    pg_admin_password: DBUser.DBA
    pg_monitor_password: DBUser.Monitor
    pg_replication_password: DBUser.Replicator
    patroni_password: Patroni.API
    haproxy_admin_password: pigsty
    minio_secret_key: S3User.MinIO
    etcd_root_password: Etcd.Root
...

Explanation

The ha/dual template is Pigsty’s two-node limited HA configuration, designed for scenarios with only two servers.

Architecture:

  • Node A (10.10.10.10): Admin node, runs Infra + etcd + PostgreSQL replica
  • Node B (10.10.10.11): Data node, runs PostgreSQL primary only

Failure Scenario Analysis:

Failed Node Impact Auto Recovery
Node B down Primary switches to Node A Auto
Node A etcd down Primary continues running (no DCS) Manual
Node A pgsql down Primary continues running Manual
Node A complete failure Primary degrades to standalone Manual

Use Cases:

  • Budget-limited environments with only two servers
  • Acceptable that some failure scenarios need manual intervention
  • Transitional solution before upgrading to three-node HA

Notes:

  • True HA requires at least three nodes (DCS needs majority)
  • Recommend upgrading to three-node architecture as soon as possible
  • L2 VIP requires network environment support (same broadcast domain)

6.25 - ha/citus

13-node Citus distributed PostgreSQL cluster, 1 coordinator + 5 worker groups with HA

The ha/citus template deploys a complete Citus distributed PostgreSQL cluster with 1 infra node, 1 coordinator group, and 5 worker groups (12 Citus nodes total), providing transparent horizontal scaling and data sharding.

Overview

  • Config Name: ha/citus
  • Node Count: 13 nodes (1 infra + 1 coordinator×2 + 5 workers×2)
  • Description: Citus distributed PostgreSQL HA cluster
  • OS Distro: el8, el9, el10, d12, d13, u22, u24, u26
  • OS Arch: x86_64
  • Related: meta, ha/trio

Usage:

./configure -c ha/citus
Note

This is a 13-node template. Modify the node addresses after generation.

Content

Source: pigsty/conf/ha/citus.yml

---
#==============================================================#
# File      :   citus.yml
# Desc      :   13-node Citus (6-group Distributive) Config Template
# Ctime     :   2020-05-22
# Mtime     :   2025-01-20
# Docs      :   https://pigsty.io/docs/conf/citus
# License   :   Apache-2.0 @ https://pigsty.io/docs/about/license/
# Copyright :   2018-2026  Ruohang Feng / Vonng ([email protected])
#==============================================================#

# This is the config template for Citus Distributive Cluster
# tutorial: https://pigsty.io/docs/pgsql/kernel/citus
# we will use the local repo for cluster bootstrapping
#
# Topology:
#   - pg-citus0: coordinator (10.10.10.10)         VIP: 10.10.10.19
#   - pg-citus1: worker group 1 (10.10.10.21, 22)  VIP: 10.10.10.29
#   - pg-citus2: worker group 2 (10.10.10.31, 32)  VIP: 10.10.10.39
#   - pg-citus3: worker group 3 (10.10.10.41, 42)  VIP: 10.10.10.49
#   - pg-citus4: worker group 4 (10.10.10.51, 52)  VIP: 10.10.10.59
#   - pg-citus5: worker group 5 (10.10.10.61, 62)  VIP: 10.10.10.69
#   - pg-citus6: worker group 6 (10.10.10.71, 72)  VIP: 10.10.10.79
#
# Usage:
#   curl https://repo.pigsty.io/get | bash
#   ./configure -c citus
#   ./deploy.yml

all:
  children:
    infra: { hosts: { 10.10.10.10: { infra_seq: 1 }}}
    etcd:  { hosts: { 10.10.10.10: { etcd_seq: 1  }}, vars: { etcd_cluster: etcd }}
    pg-meta:
      hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
      vars:
        pg_cluster: pg-meta
        pg_users:
          - { name: dbuser_meta ,password: DBUser.Meta   ,pgbouncer: true ,roles: [dbrole_admin   ] ,comment: pigsty admin user }
        pg_databases:
          - name: meta
            baseline: cmdb.sql
            comment: "pigsty meta database"
            schemas: [pigsty]
            extensions: [ postgis, vector ]
        pg_crontab: [ '00 01 * * * /pg/bin/pg-backup full' ] # make a full backup every day 1am

    #----------------------------------------------------------#
    # pg-citus: 6 cluster groups, 12 nodes total
    #----------------------------------------------------------#
    pg-citus:
      hosts:

        # coordinator (group 0) on infra node
        10.10.10.21: { pg_group: 0, pg_cluster: pg-citus1 ,pg_vip_address: 10.10.10.29/24 ,pg_seq: 1, pg_role: primary }
        10.10.10.22: { pg_group: 0, pg_cluster: pg-citus1 ,pg_vip_address: 10.10.10.29/24 ,pg_seq: 2, pg_role: replica }

        # worker group 2
        10.10.10.31: { pg_group: 1, pg_cluster: pg-citus2 ,pg_vip_address: 10.10.10.39/24 ,pg_seq: 1, pg_role: primary }
        10.10.10.32: { pg_group: 1, pg_cluster: pg-citus2 ,pg_vip_address: 10.10.10.39/24 ,pg_seq: 2, pg_role: replica }

        # worker group 3
        10.10.10.41: { pg_group: 2, pg_cluster: pg-citus3 ,pg_vip_address: 10.10.10.49/24 ,pg_seq: 1, pg_role: primary }
        10.10.10.42: { pg_group: 2, pg_cluster: pg-citus3 ,pg_vip_address: 10.10.10.49/24 ,pg_seq: 2, pg_role: replica }

        # worker group 4
        10.10.10.51: { pg_group: 3, pg_cluster: pg-citus4 ,pg_vip_address: 10.10.10.59/24 ,pg_seq: 1, pg_role: primary }
        10.10.10.52: { pg_group: 3, pg_cluster: pg-citus4 ,pg_vip_address: 10.10.10.59/24 ,pg_seq: 2, pg_role: replica }

        # worker group 5
        10.10.10.61: { pg_group: 4, pg_cluster: pg-citus5 ,pg_vip_address: 10.10.10.69/24 ,pg_seq: 1, pg_role: primary }
        10.10.10.62: { pg_group: 4, pg_cluster: pg-citus5 ,pg_vip_address: 10.10.10.69/24 ,pg_seq: 2, pg_role: replica }

        # worker group 6
        10.10.10.71: { pg_group: 5, pg_cluster: pg-citus6 ,pg_vip_address: 10.10.10.79/24 ,pg_seq: 1, pg_role: primary }
        10.10.10.72: { pg_group: 5, pg_cluster: pg-citus6 ,pg_vip_address: 10.10.10.79/24 ,pg_seq: 2, pg_role: replica }

      vars:
        pg_mode: citus                            # pgsql cluster mode: citus
        pg_shard: pg-citus                        # citus shard name: pg-citus
        pg_primary_db: citus                      # primary database used by citus
        pg_dbsu_password: DBUser.Postgres         # enable dbsu password access for citus
        pg_extensions: [ citus, postgis, pgvector, topn, pg_cron, hll ]
        pg_libs: 'citus, pg_cron, pg_stat_statements'
        pg_users: [{ name: dbuser_citus ,password: DBUser.Citus ,pgbouncer: true ,roles: [ dbrole_admin ] }]
        pg_databases: [{ name: citus ,owner: dbuser_citus ,extensions: [ citus, vector, topn, pg_cron, hll ] }]
        pg_parameters:
          cron.database_name: citus
          citus.node_conninfo: 'sslrootcert=/pg/cert/ca.crt sslmode=verify-full'
        pg_hba_rules:
          - { user: 'all' ,db: all  ,addr: 127.0.0.1/32  ,auth: ssl ,title: 'all user ssl access from localhost' }
          - { user: 'all' ,db: all  ,addr: intra         ,auth: ssl ,title: 'all user ssl access from intranet'  }
        pg_vip_enabled: true
        pg_crontab: [ '00 01 * * * /pg/bin/pg-backup full' ] # make a full backup every day 1am

  vars:
    #----------------------------------------------#
    # INFRA : https://pigsty.io/docs/infra/param
    #----------------------------------------------#
    version: v4.5.0
    admin_ip: 10.10.10.10
    region: default
    infra_portal:
      home : { domain: i.pigsty }

    #----------------------------------------------#
    # NODE : https://pigsty.io/docs/node/param
    #----------------------------------------------#
    nodename_overwrite: true
    node_repo_modules: node,infra,pgsql
    node_tune: oltp

    #----------------------------------------------#
    # PGSQL : https://pigsty.io/docs/pgsql/param
    #----------------------------------------------#
    pg_version: 18  # PostgreSQL 14-18
    pg_conf: oltp.yml
    pg_packages: [ pgsql-main, pgsql-common ]

    #----------------------------------------------#
    # PASSWORD : https://pigsty.io/docs/setup/security/
    #----------------------------------------------#
    grafana_admin_password: pigsty
    grafana_view_password: DBUser.Viewer
    pg_admin_password: DBUser.DBA
    pg_monitor_password: DBUser.Monitor
    pg_replication_password: DBUser.Replicator
    patroni_password: Patroni.API
    haproxy_admin_password: pigsty
    minio_secret_key: S3User.MinIO
    etcd_root_password: Etcd.Root
...

Topology

Cluster Nodes IP Addresses VIP Role
pg-meta 1 10.10.10.10 - Infra + CMDB
pg-citus1 2 10.10.10.21, 22 10.10.10.29 Coordinator (group 0)
pg-citus2 2 10.10.10.31, 32 10.10.10.39 Worker (group 1)
pg-citus3 2 10.10.10.41, 42 10.10.10.49 Worker (group 2)
pg-citus4 2 10.10.10.51, 52 10.10.10.59 Worker (group 3)
pg-citus5 2 10.10.10.61, 62 10.10.10.69 Worker (group 4)
pg-citus6 2 10.10.10.71, 72 10.10.10.79 Worker (group 5)

Architecture:

  • pg-meta: Infra node running Grafana, VictoriaMetrics, etcd, plus a standalone CMDB
  • pg-citus1: Coordinator (group 0), receives queries and routes to workers, 1 primary + 1 replica
  • pg-citus2~6: Workers (group 1~5), store sharded data, each with 1 primary + 1 replica via Patroni
  • VIP: Each group has L2 VIP managed by vip-manager for transparent failover

Explanation

The ha/citus template deploys production-grade Citus cluster for large-scale horizontal scaling scenarios.

Key Features:

  • Horizontal Scaling: 5 worker groups for linear storage/compute scaling
  • High Availability: Each group with 1 primary + 1 replica, auto-failover
  • L2 VIP: Virtual IP per group, transparent failover to clients
  • SSL Encryption: Inter-node communication uses SSL certificates
  • Transparent Sharding: Data auto-distributed across workers

Pre-installed Extensions:

pg_extensions: [ citus, postgis, pgvector, topn, pg_cron, hll ]
pg_libs: 'citus, pg_cron, pg_stat_statements'

Security:

  • pg_dbsu_password enabled for Citus inter-node communication
  • HBA rules require SSL authentication
  • Inter-node uses certificate verification: sslmode=verify-full

Deployment 

# 1. Download Pigsty
curl -fsSL https://repo.pigsty.io/get | bash; cd ~/pigsty

# 2. Use ha/citus template
./configure -c ha/citus

# 3. Modify IPs and passwords
vi pigsty.yml

# 4. Deploy entire cluster
./deploy.yml

Verify after deployment:

-- Connect to coordinator
psql -h 10.10.10.29 -U dbuser_citus -d citus

-- Check worker nodes
SELECT * FROM citus_get_active_worker_nodes();

-- Check shard distribution
SELECT * FROM citus_shards;

Examples

Create Distributed Table:

-- Create table
CREATE TABLE events (
    tenant_id INT,
    event_id BIGSERIAL,
    event_time TIMESTAMPTZ DEFAULT now(),
    payload JSONB,
    PRIMARY KEY (tenant_id, event_id)
);

-- Distribute by tenant_id
SELECT create_distributed_table('events', 'tenant_id');

-- Insert (auto-routed to correct shard)
INSERT INTO events (tenant_id, payload)
VALUES (1, '{"type": "click"}');

-- Query (parallel execution)
SELECT tenant_id, count(*)
FROM events
GROUP BY tenant_id;

Create Reference Table (replicated to all nodes):

CREATE TABLE tenants (
    tenant_id INT PRIMARY KEY,
    name TEXT
);

SELECT create_reference_table('tenants');

Use Cases

  • Multi-tenant SaaS: Shard by tenant_id for data isolation and parallel queries
  • Real-time Analytics: Large-scale event data aggregation
  • Timeseries Data: Combine with TimescaleDB for massive timeseries
  • Horizontal Scaling: When single-table data exceeds single-node capacity

Notes

  • PostgreSQL Version: Citus supports PG 14~18, this template defaults to PG18
  • Distribution Column: Choose wisely (typically tenant_id or timestamp), critical for performance
  • Cross-shard Limits: Foreign keys must include distribution column, some DDL restrictions
  • Network: pg_vip_interface defaults to auto; specify an interface explicitly for unusual network environments
  • Architecture: Citus extension does not support ARM64

6.26 - demo/bare

Minimal readable configuration declaring only INFRA, ETCD, and single-node PostgreSQL

demo/bare is Pigsty’s smallest configuration example. It keeps only three core groups and three global parameters to show a working inventory skeleton.

Overview

  • Config Name: demo/bare
  • Node Count: Single node
  • Modules: INFRA, ETCD, PGSQL
  • Related: meta, slim
./configure -c demo/bare [-i <primary_ip>]

Content

Source: pigsty/conf/demo/bare.yml

---
all:
  children:
    infra:   { hosts: { 10.10.10.10: { infra_seq: 1 } } }
    etcd:    { hosts: { 10.10.10.10: { etcd_seq: 1 } }, vars: { etcd_cluster: etcd } }
    pg-meta: { hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }, vars: { pg_cluster: pg-meta } }
  vars:
    version: v4.5.0
    admin_ip: 10.10.10.10
    region: default
...

Explanation

This template relies on Pigsty defaults and defines no business users, databases, extensions, backup policy, or security hardening. Use it to learn configuration hierarchy or as a minimal customization base; explicitly add passwords, HBA rules, backup, and safeguards for a real environment.

6.27 - demo/el

Configuration template optimized for Enterprise Linux (RHEL/Rocky/Alma)

The demo/el configuration template is optimized for Enterprise Linux family distributions (RHEL, Rocky Linux, Alma Linux, Oracle Linux).

Overview

  • Config Name: demo/el
  • Node Count: Single node
  • Description: Enterprise Linux optimized configuration template
  • OS Distro: el8, el9, el10
  • OS Arch: x86_64, aarch64
  • Related: meta, demo/debian

Usage:

./configure -c demo/el [-i <primary_ip>]

Content

Source: pigsty/conf/demo/el.yml

---
#==============================================================#
# File      :   el.yml
# Desc      :   Default parameters for EL System in Pigsty
# Ctime     :   2020-05-22
# Mtime     :   2026-08-02
# Docs      :   https://pigsty.io/docs/conf/el
# License   :   Apache-2.0 @ https://pigsty.io/docs/about/license/
# Copyright :   2018-2026  Ruohang Feng / Vonng ([email protected])
#==============================================================#


#==============================================================#
#                        Sandbox (4-node)                      #
#==============================================================#
# admin user : vagrant  (nopass ssh & sudo already set)        #
# 1.  meta    :    10.10.10.10     (2 Core | 4GB)    pg-meta   #
# 2.  node-1  :    10.10.10.11     (1 Core | 1GB)    pg-test-1 #
# 3.  node-2  :    10.10.10.12     (1 Core | 1GB)    pg-test-2 #
# 4.  node-3  :    10.10.10.13     (1 Core | 1GB)    pg-test-3 #
# (replace these ip if your 4-node env have different ip addr) #
# VIP 2: (l2 vip is available inside same LAN )                #
#     pg-meta --->  10.10.10.2 ---> 10.10.10.10                #
#     pg-test --->  10.10.10.3 ---> 10.10.10.1{1,2,3}          #
#==============================================================#


all:

  ##################################################################
  #                            CLUSTERS                            #
  ##################################################################
  # meta nodes, nodes, pgsql, redis, pgsql clusters are defined as
  # k:v pair inside `all.children`. Where the key is cluster name
  # and value is cluster definition consist of two parts:
  # `hosts`: cluster members ip and instance level variables
  # `vars` : cluster level variables
  ##################################################################
  children:                                 # groups definition

    # infra cluster for proxy, monitor, alert, etc..
    infra: { hosts: { 10.10.10.10: { infra_seq: 1 } } }

    # etcd cluster for ha postgres
    etcd: { hosts: { 10.10.10.10: { etcd_seq: 1 } }, vars: { etcd_cluster: etcd } }

    # minio cluster, s3 compatible object storage
    minio: { hosts: { 10.10.10.10: { minio_seq: 1 } }, vars: { minio_cluster: minio } }

    #----------------------------------#
    # pgsql cluster: pg-meta (CMDB)    #
    #----------------------------------#
    pg-meta:
      hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary , pg_offline_query: true } }
      vars:
        pg_cluster: pg-meta

        # define business databases here: https://pigsty.io/docs/pgsql/config/db
        pg_databases:                       # define business databases on this cluster, array of database definition
          - name: meta                      # REQUIRED, `name` is the only mandatory field of a database definition
            #state: create                  # optional, create|absent|recreate, create by default
            baseline: cmdb.sql              # optional, database sql baseline path, (relative path among ansible search path, e.g: files/)
            schemas: [pigsty]               # optional, additional schemas to be created, array of schema names
            extensions:                     # optional, additional extensions to be installed: array of `{name[,schema]}`
              - { name: vector }            # install pgvector extension on this database by default
            comment: pigsty meta database   # optional, comment string for this database
            #pgbouncer: true                # optional, add this database to pgbouncer database list? true by default
            #owner: postgres                # optional, database owner, current user if not specified
            #template: template1            # optional, which template to use, template1 by default
            #strategy: FILE_COPY            # optional, clone strategy: FILE_COPY or WAL_LOG (PG15+), default to PG's default
            #encoding: UTF8                 # optional, inherited from template / cluster if not defined (UTF8)
            #locale: C                      # optional, inherited from template / cluster if not defined (C)
            #lc_collate: C                  # optional, inherited from template / cluster if not defined (C)
            #lc_ctype: C                    # optional, inherited from template / cluster if not defined (C)
            #locale_provider: libc          # optional, locale provider: libc, icu, builtin (PG15+)
            #icu_locale: en-US              # optional, icu locale for icu locale provider (PG15+)
            #icu_rules: ''                  # optional, icu rules for icu locale provider (PG16+)
            #builtin_locale: C.UTF-8        # optional, builtin locale for builtin locale provider (PG17+)
            #tablespace: pg_default         # optional, default tablespace, pg_default by default
            #is_template: false             # optional, mark database as template, allowing clone by any user with CREATEDB privilege
            #allowconn: true                # optional, allow connection, true by default. false will disable connect at all
            #revokeconn: false              # optional, revoke public connection privilege. false by default. (leave connect with grant option to owner)
            #register_datasource: true      # optional, register this database to grafana datasources? true by default
            #connlimit: -1                  # optional, database connection limit, default -1 disable limit
            #pool_auth_user: dbuser_meta    # optional, all connection to this pgbouncer database will be authenticated by this user
            #pool_mode: transaction         # optional, pgbouncer pool mode at database level, default transaction
            #pool_size: 64                  # optional, pgbouncer pool size at database level, default 64
            #pool_reserve: 32               # optional, pgbouncer pool size reserve at database level, default 32
            #pool_size_min: 0               # optional, pgbouncer pool size min at database level, default 0
            #pool_connlimit: 100            # optional, max database connections at database level, default 100
          #- { name: grafana  ,owner: dbuser_grafana  ,revokeconn: true ,comment: grafana primary database }
          #- { name: bytebase ,owner: dbuser_bytebase ,revokeconn: true ,comment: bytebase primary database }
          #- { name: kong     ,owner: dbuser_kong     ,revokeconn: true ,comment: kong the api gateway database }
          #- { name: gitea    ,owner: dbuser_gitea    ,revokeconn: true ,comment: gitea meta database }
          #- { name: wiki     ,owner: dbuser_wiki     ,revokeconn: true ,comment: wiki meta database }

        # define business users here: https://pigsty.io/docs/pgsql/config/user
        pg_users:                           # define business users/roles on this cluster, array of user definition
          - name: dbuser_meta               # REQUIRED, `name` is the only mandatory field of a user definition
            password: DBUser.Meta           # optional, password, can be a scram-sha-256 hash string or plain text
            pgbouncer: true                 # optional, add this user to pgbouncer user-list? false by default (production user should be true explicitly)
            comment: pigsty admin user      # optional, comment string for this user/role
            roles: [ dbrole_admin ]         # optional, belonged roles. default roles are: dbrole_{admin,readonly,readwrite,offline}
            #login: true                     # optional, can log in, true by default  (new biz ROLE should be false)
            #superuser: false                # optional, is superuser? false by default
            #createdb: false                 # optional, can create database? false by default
            #createrole: false               # optional, can create role? false by default
            #inherit: true                   # optional, can this role use inherited privileges? true by default
            #replication: false              # optional, can this role do replication? false by default
            #bypassrls: false                # optional, can this role bypass row level security? false by default
            #connlimit: -1                   # optional, user connection limit, default -1 disable limit
            #expire_in: 3650                 # optional, now + n days when this role is expired (OVERWRITE expire_at)
            #expire_at: '2030-12-31'         # optional, YYYY-MM-DD 'timestamp' when this role is expired  (OVERWRITTEN by expire_in)
            #parameters: {}                  # optional, role level parameters with `ALTER ROLE SET`
            #pool_mode: transaction          # optional, pgbouncer pool mode at user level, transaction by default
            #pool_connlimit: -1              # optional, max database connections at user level, default -1 disable limit
          - {name: dbuser_view     ,password: DBUser.Viewer   ,pgbouncer: true ,roles: [dbrole_readonly], comment: read-only viewer for meta database}
          #- {name: dbuser_grafana  ,password: DBUser.Grafana  ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for grafana database   }
          #- {name: dbuser_bytebase ,password: DBUser.Bytebase ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for bytebase database  }
          #- {name: dbuser_gitea    ,password: DBUser.Gitea    ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for gitea service      }
          #- {name: dbuser_wiki     ,password: DBUser.Wiki     ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for wiki.js service    }

        # define business service here: https://pigsty.io/docs/pgsql/service
        pg_services:                        # extra services in addition to pg_default_services, array of service definition
          # standby service will route {ip|name}:5435 to sync replica's pgbouncer (5435->6432 standby)
          - name: standby                   # required, service name, the actual svc name will be prefixed with `pg_cluster`, e.g: pg-meta-standby
            port: 5435                      # required, service exposed port (work as kubernetes service node port mode)
            ip: "*"                         # optional, service bind ip address, `*` for all ip by default
            selector: "[]"                  # required, service member selector, use JMESPath to filter inventory
            dest: default                   # optional, destination port, default|postgres|pgbouncer|<port_number>, 'default' by default
            check: /sync                    # optional, health check url path, / by default
            backup: "[? pg_role == `primary`]"  # backup server selector
            maxconn: 3000                   # optional, max allowed front-end connection
            balance: roundrobin             # optional, haproxy load balance algorithm (roundrobin by default, other: leastconn)
            #options: 'inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100'

        # define pg extensions: https://pigsty.io/docs/pgsql/ext/
        pg_libs: 'pg_stat_statements, auto_explain' # add timescaledb to shared_preload_libraries
        #pg_extensions: [] # extensions to be installed on this cluster

        # define HBA rules here: https://pigsty.io/docs/pgsql/config/hba
        pg_hba_rules:
          - {user: dbuser_view , db: all ,addr: infra ,auth: pwd ,title: 'allow grafana dashboard access cmdb from infra nodes'}

        pg_vip_enabled: true
        pg_vip_address: 10.10.10.2/24

        pg_crontab:  # make a full backup 1 am everyday
          - '00 01 * * * /pg/bin/pg-backup full'

    #----------------------------------#
    # pgsql cluster: pg-test (3 nodes) #
    #----------------------------------#
    # pg-test --->  10.10.10.3 ---> 10.10.10.1{1,2,3}
    pg-test:                          # define the new 3-node cluster pg-test
      hosts:
        10.10.10.11: { pg_seq: 1, pg_role: primary }   # primary instance, leader of cluster
        10.10.10.12: { pg_seq: 2, pg_role: replica }   # replica instance, follower of leader
        10.10.10.13: { pg_seq: 3, pg_role: replica, pg_offline_query: true } # replica with offline access
      vars:
        pg_cluster: pg-test           # define pgsql cluster name
        pg_users:  [{ name: test , password: test , pgbouncer: true , roles: [ dbrole_admin ] }]
        pg_databases: [{ name: test }] # create a database and user named 'test'
        node_tune: tiny
        pg_conf: tiny.yml
        pg_vip_enabled: true
        pg_vip_address: 10.10.10.3/24
        pg_crontab:  # make a full backup on monday 1am, and an incremental backup during weekdays
          - '00 01 * * 1 /pg/bin/pg-backup full'
          - '00 01 * * 2,3,4,5,6,7 /pg/bin/pg-backup'

    #----------------------------------#
    # redis ms, sentinel, native cluster
    #----------------------------------#
    redis-ms: # redis classic primary & replica
      hosts: { 10.10.10.10: { redis_node: 1 , redis_instances: { 6379: { }, 6380: { replica_of: '10.10.10.10 6379' } } } }
      vars: { redis_cluster: redis-ms ,redis_password: 'redis.ms' ,redis_max_memory: 64MB }

    redis-meta: # redis sentinel x 3
      hosts: { 10.10.10.11: { redis_node: 1 , redis_instances: { 26379: { } ,26380: { } ,26381: { } } } }
      vars:
        redis_cluster: redis-meta
        redis_password: 'redis.meta'
        redis_mode: sentinel
        redis_max_memory: 16MB
        redis_sentinel_monitor: # primary list for redis sentinel, use cls as name, primary ip:port
          - { name: redis-ms, host: 10.10.10.10, port: 6379 ,password: redis.ms, quorum: 2 }

    redis-test: # redis native cluster: 3m x 3s
      hosts:
        10.10.10.12: { redis_node: 1 ,redis_instances: { 6379: { } ,6380: { } ,6381: { } } }
        10.10.10.13: { redis_node: 2 ,redis_instances: { 6379: { } ,6380: { } ,6381: { } } }
      vars: { redis_cluster: redis-test ,redis_password: 'redis.test' ,redis_mode: cluster, redis_max_memory: 32MB }


  ####################################################################
  #                             VARS                                 #
  ####################################################################
  vars:                               # global variables


    #================================================================#
    #                         VARS: INFRA                            #
    #================================================================#

    #-----------------------------------------------------------------
    # META
    #-----------------------------------------------------------------
    version: v4.5.0                   # pigsty version string
    admin_ip: 10.10.10.10             # admin node ip address
    region: default                   # upstream mirror region: default,china,europe
    language: en                      # default language: en, zh
    proxy_env:                        # global proxy env when downloading packages
      no_proxy: "localhost,127.0.0.1,10.0.0.0/8,192.168.0.0/16,*.pigsty,*.aliyun.com,mirrors.*,*.myqcloud.com,*.tsinghua.edu.cn"
      # http_proxy:  # set your proxy here: e.g http://user:[email protected]
      # https_proxy: # set your proxy here: e.g http://user:[email protected]
      # all_proxy:   # set your proxy here: e.g http://user:[email protected]

    #-----------------------------------------------------------------
    # CA
    #-----------------------------------------------------------------
    ca_create: true                   # create ca if not exists? or just abort
    ca_cn: pigsty-ca                  # ca common name, fixed as pigsty-ca
    cert_validity: 7300d              # cert validity, 20 years by default

    #-----------------------------------------------------------------
    # INFRA_IDENTITY
    #-----------------------------------------------------------------
    #infra_seq: 1                     # infra node identity, explicitly required
    infra_portal:                     # infra services exposed via portal
      home : { domain: i.pigsty }     # default domain name
    infra_data: /data/infra           # default data path for infrastructure data
    infra_services:                   # home page navigation entries
      - { name: Metrics            ,url: '/vmetrics/vmui/'         ,desc: 'VictoriaMetrics Query UI'    ,icon: metrics  ,name_cn: '指标查询' ,desc_cn: 'VictoriaMetrics 指标查询界面' }
      - { name: Logs               ,url: '/vlogs/select/vmui/'     ,desc: 'VictoriaLogs Query UI'       ,icon: logs     ,name_cn: '日志查询' ,desc_cn: 'VictoriaLogs 日志查询界面' }
      - { name: Traces             ,url: '/vtraces/select/vmui/'   ,desc: 'VictoriaTraces Query UI'     ,icon: traces   ,name_cn: '链路追踪' ,desc_cn: 'VictoriaTraces 链路查询界面' }
      - { name: Monitor Targets    ,url: '/vmetrics/targets'       ,desc: 'Prometheus Scrape Targets'   ,icon: target   ,name_cn: '监控目标' ,desc_cn: 'VictoriaMetrics 监控对象列表' }
      - { name: Alert Rules        ,url: '/vmalert/vmalert/groups' ,desc: 'VMAlert alert/record Rules'  ,icon: alert    ,name_cn: '告警规则' ,desc_cn: 'VMAlert 告警规则管理' }
      - { name: Alert Manager      ,url: '/alertmgr/#/alerts'      ,desc: 'Alert Manage & Silence'      ,icon: alertmgr ,name_cn: '告警管理' ,desc_cn: 'AlertManager 告警管理与屏蔽' }
      - { name: CA Certificate     ,url: '/ca.crt'                 ,desc: 'Self-Signed CA Certificate'  ,icon: lock     ,name_cn: 'CA 证书'  ,desc_cn: 'Pigsty 自签CA根证书' }
      - { name: Software Repo      ,url: '/pigsty'                 ,desc: 'Local YUM/APT Repository'    ,icon: package  ,name_cn: '软件仓库' ,desc_cn: '本地 YUM/APT 软件源' }
      - { name: Explain Visualizer ,url: '/pev'                    ,desc: 'Postgres EXPLAIN Visualizer' ,icon: search   ,name_cn: '执行计划' ,desc_cn: 'PG 执行计划可视化工具' }
    infra_extra_services: []          # extra services to be added on infra home page

    #-----------------------------------------------------------------
    # REPO
    #-----------------------------------------------------------------
    repo_enabled: true                # create a yum repo on this infra node?
    repo_home: /www                   # repo home dir, `/www` by default
    repo_name: pigsty                 # repo name, pigsty by default
    repo_endpoint: http://${admin_ip}:80 # access point to this repo by domain or ip:port
    repo_remove: true                 # remove existing upstream repo
    repo_modules: infra,node,pgsql    # which repo modules are installed in repo_upstream
    repo_upstream:                    # where to download
      - { name: pigsty-local   ,description: 'Pigsty Local'       ,module: local   ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'http://${admin_ip}/pigsty'  } ,meta: { module_hotfixes: 1 }} # used by intranet nodes
      - { name: pigsty-infra   ,description: 'Pigsty INFRA'       ,module: infra   ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://repo.pigsty.io/yum/infra/$basearch' ,china: 'https://repo.pigsty.cc/yum/infra/$basearch' } ,meta: { module_hotfixes: 1 }}
      - { name: pigsty-pgsql   ,description: 'Pigsty PGSQL'       ,module: pgsql   ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://repo.pigsty.io/yum/pgsql/el$releasever.$basearch' ,china: 'https://repo.pigsty.cc/yum/pgsql/el$releasever.$basearch' } ,meta: { module_hotfixes: 1 }}
      - { name: nginx          ,description: 'Nginx Repo'         ,module: infra   ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://nginx.org/packages/rhel/$releasever/$basearch/' } ,meta: { module_hotfixes: 1 }}
      - { name: docker-ce      ,description: 'Docker CE'          ,module: infra   ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://download.docker.com/linux/centos/$releasever/$basearch/stable'    ,china: 'https://mirrors.cloud.tencent.com/docker-ce/linux/centos/$releasever/$basearch/stable https://repo.huaweicloud.com/docker-ce/linux/centos/$releasever/$basearch/stable https://mirrors.aliyun.com/docker-ce/linux/centos/$releasever/$basearch/stable' ,europe: 'https://mirrors.xtom.de/docker-ce/linux/centos/$releasever/$basearch/stable' }}
      - { name: baseos         ,description: 'EL 8+ BaseOS'       ,module: node    ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://dl.rockylinux.org/pub/rocky/$releasever/BaseOS/$basearch/os/'     ,china: 'https://mirrors.cloud.tencent.com/rocky/$releasever/BaseOS/$basearch/os/ https://repo.huaweicloud.com/rockylinux/$releasever/BaseOS/$basearch/os/ https://mirrors.aliyun.com/rockylinux/$releasever/BaseOS/$basearch/os/'         ,europe: 'https://mirrors.xtom.de/rocky/$releasever/BaseOS/$basearch/os/'     }}
      - { name: appstream      ,description: 'EL 8+ AppStream'    ,module: node    ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://dl.rockylinux.org/pub/rocky/$releasever/AppStream/$basearch/os/'  ,china: 'https://mirrors.cloud.tencent.com/rocky/$releasever/AppStream/$basearch/os/ https://repo.huaweicloud.com/rockylinux/$releasever/AppStream/$basearch/os/ https://mirrors.aliyun.com/rockylinux/$releasever/AppStream/$basearch/os/'      ,europe: 'https://mirrors.xtom.de/rocky/$releasever/AppStream/$basearch/os/'  }}
      - { name: extras         ,description: 'EL 8+ Extras'       ,module: node    ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://dl.rockylinux.org/pub/rocky/$releasever/extras/$basearch/os/'     ,china: 'https://mirrors.cloud.tencent.com/rocky/$releasever/extras/$basearch/os/ https://repo.huaweicloud.com/rockylinux/$releasever/extras/$basearch/os/ https://mirrors.aliyun.com/rockylinux/$releasever/extras/$basearch/os/'         ,europe: 'https://mirrors.xtom.de/rocky/$releasever/extras/$basearch/os/'     }}
      - { name: powertools     ,description: 'EL 8 PowerTools'    ,module: node    ,releases: [8     ] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://dl.rockylinux.org/pub/rocky/$releasever/PowerTools/$basearch/os/' ,china: 'https://mirrors.cloud.tencent.com/rocky/$releasever/PowerTools/$basearch/os/ https://repo.huaweicloud.com/rockylinux/$releasever/PowerTools/$basearch/os/ https://mirrors.aliyun.com/rockylinux/$releasever/PowerTools/$basearch/os/'     ,europe: 'https://mirrors.xtom.de/rocky/$releasever/PowerTools/$basearch/os/' }}
      - { name: crb            ,description: 'EL 9 CRB'           ,module: node    ,releases: [  9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://dl.rockylinux.org/pub/rocky/$releasever/CRB/$basearch/os/'        ,china: 'https://mirrors.cloud.tencent.com/rocky/$releasever/CRB/$basearch/os/ https://repo.huaweicloud.com/rockylinux/$releasever/CRB/$basearch/os/ https://mirrors.aliyun.com/rockylinux/$releasever/CRB/$basearch/os/'            ,europe: 'https://mirrors.xtom.de/rocky/$releasever/CRB/$basearch/os/'        }}
      - { name: epel           ,description: 'EL 8+ EPEL'         ,module: node    ,releases: [8,9   ] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://mirrors.edge.kernel.org/fedora-epel/$releasever/Everything/$basearch/' ,china: 'https://mirrors.cloud.tencent.com/epel/$releasever/Everything/$basearch/ https://repo.huaweicloud.com/epel/$releasever/Everything/$basearch/ https://mirrors.aliyun.com/epel/$releasever/Everything/$basearch/'         ,europe: 'https://mirrors.xtom.de/epel/$releasever/Everything/$basearch/'     }}
      - { name: epel           ,description: 'EL 10 EPEL'         ,module: node    ,releases: [    10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://mirrors.edge.kernel.org/fedora-epel/$releasever/Everything/$basearch/'   ,china: 'https://mirrors.cloud.tencent.com/epel/$releasever/Everything/$basearch/ https://repo.huaweicloud.com/epel/$releasever/Everything/$basearch/ https://mirrors.aliyun.com/epel/$releasever/Everything/$basearch/'       ,europe: 'https://mirrors.xtom.de/epel/$releasever/Everything/$basearch/'     }}
      - { name: pgdg-common    ,description: 'PostgreSQL Common'  ,module: pgsql   ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://download.postgresql.org/pub/repos/yum/common/redhat/rhel-$releasever-$basearch'          ,china: 'https://mirrors.cloud.tencent.com/postgresql/repos/yum/common/redhat/rhel-$releasever-$basearch https://repo.pigsty.cc/yum/pgdg/common/redhat/rhel-$releasever-$basearch'          ,europe: 'https://mirrors.xtom.de/postgresql/repos/yum/common/redhat/rhel-$releasever-$basearch' } ,meta: { module_hotfixes: 1 }}
      - { name: pgdg14         ,description: 'PostgreSQL 14'      ,module: pgsql   ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://download.postgresql.org/pub/repos/yum/14/redhat/rhel-$releasever-$basearch'          ,china: 'https://mirrors.cloud.tencent.com/postgresql/repos/yum/14/redhat/rhel-$releasever-$basearch https://repo.pigsty.cc/yum/pgdg/14/redhat/rhel-$releasever-$basearch'          ,europe: 'https://mirrors.xtom.de/postgresql/repos/yum/14/redhat/rhel-$releasever-$basearch' } ,meta: { module_hotfixes: 1 }}
      - { name: pgdg15         ,description: 'PostgreSQL 15'      ,module: pgsql   ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://download.postgresql.org/pub/repos/yum/15/redhat/rhel-$releasever-$basearch'          ,china: 'https://mirrors.cloud.tencent.com/postgresql/repos/yum/15/redhat/rhel-$releasever-$basearch https://repo.pigsty.cc/yum/pgdg/15/redhat/rhel-$releasever-$basearch'          ,europe: 'https://mirrors.xtom.de/postgresql/repos/yum/15/redhat/rhel-$releasever-$basearch' } ,meta: { module_hotfixes: 1 }}
      - { name: pgdg16         ,description: 'PostgreSQL 16'      ,module: pgsql   ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://download.postgresql.org/pub/repos/yum/16/redhat/rhel-$releasever-$basearch'          ,china: 'https://mirrors.cloud.tencent.com/postgresql/repos/yum/16/redhat/rhel-$releasever-$basearch https://repo.pigsty.cc/yum/pgdg/16/redhat/rhel-$releasever-$basearch'          ,europe: 'https://mirrors.xtom.de/postgresql/repos/yum/16/redhat/rhel-$releasever-$basearch' } ,meta: { module_hotfixes: 1 }}
      - { name: pgdg17         ,description: 'PostgreSQL 17'      ,module: pgsql   ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://download.postgresql.org/pub/repos/yum/17/redhat/rhel-$releasever-$basearch'          ,china: 'https://mirrors.cloud.tencent.com/postgresql/repos/yum/17/redhat/rhel-$releasever-$basearch https://repo.pigsty.cc/yum/pgdg/17/redhat/rhel-$releasever-$basearch'          ,europe: 'https://mirrors.xtom.de/postgresql/repos/yum/17/redhat/rhel-$releasever-$basearch' } ,meta: { module_hotfixes: 1 }}
      - { name: pgdg18         ,description: 'PostgreSQL 18'      ,module: pgsql   ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://download.postgresql.org/pub/repos/yum/18/redhat/rhel-$releasever-$basearch'          ,china: 'https://mirrors.cloud.tencent.com/postgresql/repos/yum/18/redhat/rhel-$releasever-$basearch https://repo.pigsty.cc/yum/pgdg/18/redhat/rhel-$releasever-$basearch'          ,europe: 'https://mirrors.xtom.de/postgresql/repos/yum/18/redhat/rhel-$releasever-$basearch' } ,meta: { module_hotfixes: 1 }}
      - { name: pgdg-beta      ,description: 'PostgreSQL Testing' ,module: beta    ,releases: [8,9,10] ,arch: [x86_64         ] ,baseurl: { default: 'https://download.postgresql.org/pub/repos/yum/testing/19/redhat/rhel-$releasever-$basearch'  ,china: 'https://mirrors.cloud.tencent.com/postgresql/repos/yum/testing/19/redhat/rhel-$releasever-$basearch https://repo.pigsty.cc/yum/pgdg/testing/19/redhat/rhel-$releasever-$basearch'  ,europe: 'https://mirrors.xtom.de/postgresql/repos/yum/testing/19/redhat/rhel-$releasever-$basearch'  } ,meta: { module_hotfixes: 1 }}
      - { name: pgdg-beta      ,description: 'PostgreSQL Testing' ,module: beta    ,releases: [  9,10] ,arch: [        aarch64] ,baseurl: { default: 'https://download.postgresql.org/pub/repos/yum/testing/19/redhat/rhel-$releasever-$basearch'  ,china: 'https://mirrors.cloud.tencent.com/postgresql/repos/yum/testing/19/redhat/rhel-$releasever-$basearch https://repo.pigsty.cc/yum/pgdg/testing/19/redhat/rhel-$releasever-$basearch'  ,europe: 'https://mirrors.xtom.de/postgresql/repos/yum/testing/19/redhat/rhel-$releasever-$basearch'  } ,meta: { module_hotfixes: 1 }}
      - { name: pgdg-extras    ,description: 'PostgreSQL Extra'   ,module: extra   ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://download.postgresql.org/pub/repos/yum/extras/redhat/rhel-$releasever-$basearch'      ,china: 'https://mirrors.cloud.tencent.com/postgresql/repos/yum/extras/redhat/rhel-$releasever-$basearch https://repo.pigsty.cc/yum/pgdg/extras/redhat/rhel-$releasever-$basearch'      ,europe: 'https://mirrors.xtom.de/postgresql/repos/yum/extras/redhat/rhel-$releasever-$basearch'      } ,meta: { module_hotfixes: 1 }}
      - { name: pgdg14-nonfree ,description: 'PostgreSQL 14+'     ,module: extra   ,releases: [8,9,10] ,arch: [x86_64         ] ,baseurl: { default: 'https://download.postgresql.org/pub/repos/yum/non-free/14/redhat/rhel-$releasever-$basearch' ,china: 'https://mirrors.cloud.tencent.com/postgresql/repos/yum/non-free/14/redhat/rhel-$releasever-$basearch https://repo.pigsty.cc/yum/pgdg/non-free/14/redhat/rhel-$releasever-$basearch' ,europe: 'https://mirrors.xtom.de/postgresql/repos/yum/non-free/14/redhat/rhel-$releasever-$basearch' } ,meta: { module_hotfixes: 1 }}
      - { name: pgdg15-nonfree ,description: 'PostgreSQL 15+'     ,module: extra   ,releases: [8,9,10] ,arch: [x86_64         ] ,baseurl: { default: 'https://download.postgresql.org/pub/repos/yum/non-free/15/redhat/rhel-$releasever-$basearch' ,china: 'https://mirrors.cloud.tencent.com/postgresql/repos/yum/non-free/15/redhat/rhel-$releasever-$basearch https://repo.pigsty.cc/yum/pgdg/non-free/15/redhat/rhel-$releasever-$basearch' ,europe: 'https://mirrors.xtom.de/postgresql/repos/yum/non-free/15/redhat/rhel-$releasever-$basearch' } ,meta: { module_hotfixes: 1 }}
      - { name: pgdg16-nonfree ,description: 'PostgreSQL 16+'     ,module: extra   ,releases: [8,9,10] ,arch: [x86_64         ] ,baseurl: { default: 'https://download.postgresql.org/pub/repos/yum/non-free/16/redhat/rhel-$releasever-$basearch' ,china: 'https://mirrors.cloud.tencent.com/postgresql/repos/yum/non-free/16/redhat/rhel-$releasever-$basearch https://repo.pigsty.cc/yum/pgdg/non-free/16/redhat/rhel-$releasever-$basearch' ,europe: 'https://mirrors.xtom.de/postgresql/repos/yum/non-free/16/redhat/rhel-$releasever-$basearch' } ,meta: { module_hotfixes: 1 }}
      - { name: pgdg17-nonfree ,description: 'PostgreSQL 17+'     ,module: extra   ,releases: [8,9,10] ,arch: [x86_64         ] ,baseurl: { default: 'https://download.postgresql.org/pub/repos/yum/non-free/17/redhat/rhel-$releasever-$basearch' ,china: 'https://mirrors.cloud.tencent.com/postgresql/repos/yum/non-free/17/redhat/rhel-$releasever-$basearch https://repo.pigsty.cc/yum/pgdg/non-free/17/redhat/rhel-$releasever-$basearch' ,europe: 'https://mirrors.xtom.de/postgresql/repos/yum/non-free/17/redhat/rhel-$releasever-$basearch' } ,meta: { module_hotfixes: 1 }}
      - { name: pgdg18-nonfree ,description: 'PostgreSQL 18+'     ,module: extra   ,releases: [8,9,10] ,arch: [x86_64         ] ,baseurl: { default: 'https://download.postgresql.org/pub/repos/yum/non-free/18/redhat/rhel-$releasever-$basearch' ,china: 'https://mirrors.cloud.tencent.com/postgresql/repos/yum/non-free/18/redhat/rhel-$releasever-$basearch https://repo.pigsty.cc/yum/pgdg/non-free/18/redhat/rhel-$releasever-$basearch' ,europe: 'https://mirrors.xtom.de/postgresql/repos/yum/non-free/18/redhat/rhel-$releasever-$basearch' } ,meta: { module_hotfixes: 1 }}
      - { name: timescaledb    ,description: 'TimescaleDB'        ,module: extra   ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://packagecloud.io/timescale/timescaledb/el/$releasever/$basearch'  }}
      - { name: percona        ,description: 'Percona TDE'        ,module: percona ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://repo.pigsty.io/yum/percona/el$releasever.$basearch' ,china: 'https://repo.pigsty.cc/yum/percona/el$releasever.$basearch' ,origin: 'http://repo.percona.com/ppg-18.4/yum/release/$releasever/RPMS/$basearch'  } ,meta: { module_hotfixes: 1 }}
      - { name: wiltondb       ,description: 'WiltonDB'           ,module: mssql   ,releases: [8,9   ] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://repo.pigsty.io/yum/mssql/el$releasever.$basearch', china: 'https://repo.pigsty.cc/yum/mssql/el$releasever.$basearch' , origin: 'https://download.copr.fedorainfracloud.org/results/wiltondb/wiltondb/epel-$releasever-$basearch/' }}
      - { name: groonga        ,description: 'Groonga'            ,module: groonga ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://packages.groonga.org/almalinux/$releasever/$basearch/' }}
      - { name: mysql          ,description: 'MySQL 8.4 LTS'      ,module: mysql   ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://repo.mysql.com/yum/mysql-8.4-community/el/$releasever/$basearch/' } ,meta: { module_hotfixes: 1 }}
      - { name: mongo          ,description: 'MongoDB'            ,module: mongo   ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/8.0/$basearch/' }}
      - { name: redis          ,description: 'Redis'              ,module: redis   ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://rpmfind.net/linux/remi/enterprise/$releasever/redis72/$basearch/' } ,meta: { module_hotfixes: 1 }}
      - { name: grafana        ,description: 'Grafana'            ,module: grafana ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://rpm.grafana.com', china: 'https://mirrors.cloud.tencent.com/grafana/yum/rpm/' }}
      - { name: kubernetes     ,description: 'Kubernetes'         ,module: kube    ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://pkgs.k8s.io/core:/stable:/v1.36/rpm/', china: 'https://mirrors.ustc.edu.cn/kubernetes/core:/stable:/v1.36/rpm/' }}
      - { name: gitlab-ee      ,description: 'Gitlab EE'          ,module: gitlab  ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://packages.gitlab.com/gitlab/gitlab-ee/el/$releasever/$basearch' }}
      - { name: gitlab-ce      ,description: 'Gitlab CE'          ,module: gitlab  ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://packages.gitlab.com/gitlab/gitlab-ce/el/$releasever/$basearch' }}
      - { name: clickhouse     ,description: 'ClickHouse'         ,module: click   ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://packages.clickhouse.com/rpm/stable/', china: 'https://repo.huaweicloud.com/clickhouse/rpm/stable/' }}

    repo_packages: [ node-bootstrap, infra-package, infra-addons, node-package1, node-package2, node-package3, pgsql-utility, extra-modules ]
    repo_extra_packages: [ pgsql-main ]
    repo_url_packages: []

    #-----------------------------------------------------------------
    # INFRA_PACKAGE
    #-----------------------------------------------------------------
    infra_packages:                   # packages to be installed on infra nodes
      - grafana,grafana-plugins,grafana-victorialogs-ds,grafana-victoriametrics-ds,victoria-metrics,victoria-logs,victoria-traces,vmutils,vlogscli,alertmanager
      - node-exporter,blackbox-exporter,nginx-exporter,pg-exporter,pev2,nginx,dnsmasq,ansible,etcd,python3-requests,redis,mcli,restic,certbot,python3-certbot-nginx

    #-----------------------------------------------------------------
    # NGINX
    #-----------------------------------------------------------------
    nginx_enabled: true               # enable nginx on this infra node?
    nginx_clean: false                # clean existing nginx config during init?
    nginx_exporter_enabled: true      # enable nginx_exporter on this infra node?
    nginx_exporter_port: 9113         # nginx_exporter listen port, 9113 by default
    nginx_sslmode: enable             # nginx ssl mode? disable,enable,enforce
    nginx_cert_validity: 397d         # nginx self-signed cert validity, 397d by default
    nginx_home: /www                  # nginx content dir, `/www` by default (soft link to nginx_data)
    nginx_data: /data/nginx           # nginx actual data dir, /data/nginx by default
    nginx_users: { admin : pigsty }   # nginx basic auth users: name and pass dict
    nginx_port: 80                    # nginx listen port, 80 by default
    nginx_ssl_port: 443               # nginx ssl listen port, 443 by default
    certbot_sign: false               # sign nginx cert with certbot during setup?
    certbot_email: [email protected]     # certbot email address, used for free ssl
    certbot_options: ''               # certbot extra options

    #-----------------------------------------------------------------
    # DNS
    #-----------------------------------------------------------------
    dns_enabled: true                 # setup dnsmasq on this infra node?
    dns_port: 53                      # dns server listen port, 53 by default
    dns_records:                      # dynamic dns records resolved by dnsmasq
      - "${admin_ip} i.pigsty"
      - "${admin_ip} m.pigsty supa.pigsty api.pigsty adm.pigsty cli.pigsty ddl.pigsty"

    #-----------------------------------------------------------------
    # VICTORIA
    #-----------------------------------------------------------------
    vmetrics_enabled: true            # enable victoria-metrics on this infra node?
    vmetrics_clean: false             # whether clean existing victoria metrics data during init?
    vmetrics_port: 8428               # victoria-metrics listen port, 8428 by default
    vmetrics_scrape_interval: 10s     # victoria global scrape interval, 10s by default
    vmetrics_scrape_timeout: 8s       # victoria global scrape timeout, 8s by default
    vmetrics_options: >-
      -retentionPeriod=15d
      -promscrape.fileSDCheckInterval=5s
    vlogs_enabled: true               # enable victoria-logs on this infra node?
    vlogs_clean: false                # clean victoria-logs data during init?
    vlogs_port: 9428                  # victoria-logs listen port, 9428 by default
    vlogs_options: >-
      -retentionPeriod=15d
      -retention.maxDiskSpaceUsageBytes=50GiB
      -insert.maxLineSizeBytes=1MB
      -search.maxQueryDuration=120s
    vtraces_enabled: true             # enable victoria-traces on this infra node?
    vtraces_clean: false                # clean victoria-trace data during inti?
    vtraces_port: 10428               # victoria-traces listen port, 10428 by default
    vtraces_options: >-
      -retentionPeriod=15d
      -retention.maxDiskSpaceUsageBytes=50GiB
    vmalert_enabled: true             # enable vmalert on this infra node?
    vmalert_port: 8880                # vmalert listen port, 8880 by default
    vmalert_options: ''              # vmalert extra server options

    #-----------------------------------------------------------------
    # PROMETHEUS
    #-----------------------------------------------------------------
    blackbox_enabled: true            # setup blackbox_exporter on this infra node?
    blackbox_port: 9115               # blackbox_exporter listen port, 9115 by default
    blackbox_options: ''              # blackbox_exporter extra server options
    alertmanager_enabled: true        # setup alertmanager on this infra node?
    alertmanager_port: 9059           # alertmanager listen port, 9059 by default
    alertmanager_options: ''          # alertmanager extra server options
    exporter_metrics_path: /metrics   # exporter metric path, `/metrics` by default

    #-----------------------------------------------------------------
    # GRAFANA
    #-----------------------------------------------------------------
    grafana_enabled: true             # enable grafana on this infra node?
    grafana_port: 3000                # default listen port for grafana
    grafana_clean: false              # clean grafana data during init?
    grafana_admin_username: admin     # grafana admin username, `admin` by default
    grafana_admin_password: pigsty    # grafana admin password, `pigsty` by default
    grafana_auth_proxy: false         # enable grafana auth proxy?
    grafana_pgurl: ''                 # external postgres database url for grafana if given
    grafana_view_password: DBUser.Viewer # password for grafana meta pg datasource


    #================================================================#
    #                         VARS: NODE                             #
    #================================================================#

    #-----------------------------------------------------------------
    # NODE_IDENTITY
    #-----------------------------------------------------------------
    #nodename:           # [INSTANCE] # node instance identity, use hostname if missing, optional
    node_cluster: nodes   # [CLUSTER] # node cluster identity, use 'nodes' if missing, optional
    nodename_overwrite: true          # overwrite node's hostname with nodename?
    nodename_exchange: false          # exchange nodename among play hosts?
    node_id_from_pg: true             # use postgres identity as node identity if applicable?

    #-----------------------------------------------------------------
    # NODE_DNS
    #-----------------------------------------------------------------
    node_write_etc_hosts: true        # modify `/etc/hosts` on target node?
    node_default_etc_hosts:           # static dns records in `/etc/hosts`
      - "${admin_ip} i.pigsty"
    node_etc_hosts: []                # extra static dns records in `/etc/hosts`
    node_dns_method: add              # how to handle dns servers: add,none,overwrite
    node_dns_servers: ['${admin_ip}'] # dynamic nameserver in `/etc/resolv.conf`
    node_dns_options:                 # dns resolv options in `/etc/resolv.conf`
      - options single-request-reopen timeout:1

    #-----------------------------------------------------------------
    # NODE_PACKAGE
    #-----------------------------------------------------------------
    node_repo_modules: local          # upstream repo to be added on node, local by default
    node_repo_remove: true            # remove existing repo on node?
    node_packages: [openssh-server]   # packages to be installed current nodes with latest version
    node_default_packages:            # default packages to be installed on all nodes
      - lz4,unzip,bzip2,pv,jq,git,ncdu,make,patch,bash,lsof,wget,uuid,tuned,nvme-cli,numactl,sysstat,iotop,htop,rsync,tcpdump
      - python3,python3-pip,socat,lrzsz,net-tools,ipvsadm,telnet,ca-certificates,openssl,keepalived,etcd,haproxy,chrony,pig
      - zlib,yum,audit,bind-utils,readline,vim-minimal,node-exporter,grubby,openssh-server,openssh-clients,chkconfig,vector
    node_uv_env: /data/venv           # uv venv path, empty string to skip
    node_pip_packages: ''             # pip packages to install in uv venv

    #-----------------------------------------------------------------
    # NODE_SEC
    #-----------------------------------------------------------------
    node_selinux_mode: permissive     # set selinux mode: enforcing,permissive,disabled
    node_firewall_mode: zone          # firewall mode: zone (default), off (disable), none (skip & self-managed)
    node_firewall_intranet:           # which intranet cidr considered as internal network
      - 10.0.0.0/8
      - 192.168.0.0/16
      - 172.16.0.0/12
    node_firewall_public_port:        # expose these ports to public network in (zone, strict) mode
      - 22                            # enable ssh access
      - 80                            # enable http access
      - 443                           # enable https access
      - 5432                          # enable postgres access

    #-----------------------------------------------------------------
    # NODE_TUNE
    #-----------------------------------------------------------------
    node_disable_numa: false          # disable node numa, reboot required
    node_disable_swap: false          # disable node swap, use with caution
    node_static_network: true         # preserve dns resolver settings after reboot
    node_disk_prefetch: false         # setup disk prefetch on HDD to increase performance
    node_kernel_modules: [ softdog, ip_vs, ip_vs_rr, ip_vs_wrr, ip_vs_sh ]
    node_hugepage_count: 0            # number of 2MB hugepage, take precedence over ratio
    node_hugepage_ratio: 0            # node mem hugepage ratio, 0 disable it by default
    node_overcommit_ratio: 0          # node mem overcommit ratio, 0 disable it by default
    node_tune: oltp                   # node tuned profile: none,oltp,olap,crit,tiny
    node_sysctl_params:              # sysctl parameters in k:v format in addition to tuned
      fs.nr_open: 8388608

    #-----------------------------------------------------------------
    # NODE_ADMIN
    #-----------------------------------------------------------------
    node_data: /data                  # node main data directory, `/data` by default
    node_admin_enabled: true          # create a admin user on target node?
    node_admin_uid: 88                # uid and gid for node admin user
    node_admin_username: dba          # name of node admin user, `dba` by default
    node_admin_sudo: nopass           # admin sudo privilege, all,nopass. nopass by default
    node_admin_ssh_exchange: true     # exchange admin ssh key among node cluster
    node_admin_pk_current: true       # add current user's ssh pk to admin authorized_keys
    node_admin_pk_list: []            # ssh public keys to be added to admin user
    node_aliases: {}                  # extra shell aliases to be added, k:v dict

    #-----------------------------------------------------------------
    # NODE_TIME
    #-----------------------------------------------------------------
    node_timezone: ''                 # setup node timezone, empty string to skip
    node_ntp_enabled: true            # enable chronyd time sync service?
    node_ntp_servers:                 # ntp servers in `/etc/chrony.conf`
      - pool pool.ntp.org iburst
    node_crontab_overwrite: true      # overwrite or append to `/etc/crontab`?
    node_crontab: [ ]                 # crontab entries in `/etc/crontab`

    #-----------------------------------------------------------------
    # NODE_VIP
    #-----------------------------------------------------------------
    vip_enabled: false                # enable vip on this node cluster?
    # vip_address:         [IDENTITY] # node vip address in ipv4 format, required if vip is enabled
    # vip_vrid:            [IDENTITY] # required, integer, 1-254, should be unique among same VLAN
    vip_role: backup                  # optional, `master|backup`, backup by default, use as init role
    vip_preempt: false                # optional, `true/false`, false by default, enable vip preemption
    vip_interface: auto               # node vip network interface to listen, `auto` by default
    vip_dns_suffix: ''                # node vip dns name suffix, empty string by default
    vip_auth_pass: ''                 # empty to use '<cls>-<vrid>' as the default
    vip_exporter_port: 9650           # keepalived exporter listen port, 9650 by default

    #-----------------------------------------------------------------
    # HAPROXY
    #-----------------------------------------------------------------
    haproxy_enabled: true             # enable haproxy on this node?
    haproxy_clean: false              # cleanup all existing haproxy config?
    haproxy_reload: true              # reload haproxy after config?
    haproxy_auth_enabled: true        # enable authentication for haproxy admin page
    haproxy_admin_username: admin     # haproxy admin username, `admin` by default
    haproxy_admin_password: pigsty    # haproxy admin password, `pigsty` by default
    haproxy_exporter_port: 9101       # haproxy admin/exporter port, 9101 by default
    haproxy_client_timeout: 24h       # client side connection timeout, 24h by default
    haproxy_server_timeout: 24h       # server side connection timeout, 24h by default
    haproxy_services: []              # list of haproxy service to be exposed on node

    #-----------------------------------------------------------------
    # NODE_EXPORTER
    #-----------------------------------------------------------------
    node_exporter_enabled: true       # setup node_exporter on this node?
    node_exporter_port: 9100          # node exporter listen port, 9100 by default
    node_exporter_options: '--no-collector.softnet --no-collector.nvme --collector.tcpstat --collector.processes'

    #-----------------------------------------------------------------
    # VECTOR
    #-----------------------------------------------------------------
    vector_enabled: true              # enable vector log collector?
    vector_clean: false               # purge vector data dir during init?
    vector_data: /data/vector         # vector data dir, /data/vector by default
    vector_port: 9598                 # vector metrics port, 9598 by default
    vector_read_from: beginning       # vector read from beginning or end
    vector_log_endpoint: [ infra ]    # if defined, sending vector log to this endpoint.


    #================================================================#
    #                        VARS: DOCKER                            #
    #================================================================#
    docker_enabled: false             # enable docker on this node?
    docker_data: /data/docker         # docker data directory, /data/docker by default
    docker_storage_driver: overlay2   # docker storage driver, can be zfs, btrfs
    docker_cgroups_driver: systemd    # docker cgroup fs driver: cgroupfs,systemd
    docker_registry_mirrors: []       # docker registry mirror list
    docker_exporter_port: 9323        # docker metrics exporter port, 9323 by default
    docker_image: []                  # docker image to be pulled after bootstrap
    docker_image_cache: /tmp/docker/*.tgz # docker image cache glob pattern

    #================================================================#
    #                         VARS: ETCD                             #
    #================================================================#
    #etcd_seq: 1                      # etcd instance identifier, explicitly required
    etcd_cluster: etcd                # etcd cluster & group name, etcd by default
    etcd_safeguard: false             # prevent purging running etcd instance?
    etcd_data: /data/etcd             # etcd data directory, /data/etcd by default
    etcd_port: 2379                   # etcd client port, 2379 by default
    etcd_peer_port: 2380              # etcd peer port, 2380 by default
    etcd_init: new                    # etcd initial cluster state, new or existing
    etcd_election_timeout: 1000       # etcd election timeout, 1000ms by default
    etcd_heartbeat_interval: 100      # etcd heartbeat interval, 100ms by default
    etcd_root_password: Etcd.Root     # etcd root password for RBAC, change it!


    #================================================================#
    #                         VARS: MINIO                            #
    #================================================================#
    #minio_seq: 1                     # minio instance identifier, REQUIRED
    #minio_cluster:                   # minio cluster identifier, REQUIRED (define in cluster vars)
    minio_user: minio                 # minio os user, `minio` by default
    minio_https: true                 # use https for minio, true by default
    minio_node: '${minio_cluster}-${minio_seq}.pigsty' # minio node name pattern
    minio_data: '/data/minio'         # minio data dir(s), use {x...y} to specify multi drivers
    #minio_volumes:                   # minio data volumes, override defaults if specified
    minio_domain: sss.pigsty          # minio external domain name, `sss.pigsty` by default
    minio_port: 9000                  # minio service port, 9000 by default
    minio_admin_port: 9001            # minio console port, 9001 by default
    minio_access_key: minioadmin      # root access key, `minioadmin` by default
    minio_secret_key: S3User.MinIO    # root secret key, `S3User.MinIO` by default
    minio_extra_vars: ''              # extra environment variables
    minio_provision: true             # run minio provisioning tasks?
    minio_alias: sss                  # alias name for local minio deployment
    #minio_endpoint: https://sss.pigsty:9000 # if not specified, overwritten by defaults
    minio_buckets:                    # list of minio bucket to be created
      - { name: pgsql }
      - { name: meta ,versioning: true }
      - { name: data }
    minio_users:                      # list of minio user to be created
      - { access_key: pgbackrest  ,secret_key: S3User.Backup ,policy: pgsql }
      - { access_key: s3user_meta ,secret_key: S3User.Meta   ,policy: meta  }
      - { access_key: s3user_data ,secret_key: S3User.Data   ,policy: data  }
    minio_safeguard: false            # prevent purging running minio instance?
    minio_rm_data: true               # purging minio data and config?
    minio_rm_pkg: false               # uninstall minio packages?


    #================================================================#
    #                         VARS: REDIS                            #
    #================================================================#
    #redis_cluster:        <CLUSTER> # redis cluster name, required identity parameter
    #redis_node: 1            <NODE> # redis node sequence number, node int id required
    #redis_instances: {}      <NODE> # redis instances definition on this redis node
    redis_fs_main: /data/redis        # redis main data directory, `/data/redis` by default
    redis_exporter_enabled: true      # install redis exporter on redis nodes?
    redis_exporter_port: 9121         # redis exporter listen port, 9121 by default
    redis_exporter_options: ''        # cli args and extra options for redis exporter
    redis_type: redis                 # redis implementation: redis or valkey
    redis_mode: standalone            # redis mode: standalone,cluster,sentinel
    redis_conf: redis.conf            # redis config template path, except sentinel
    redis_bind_address: '0.0.0.0'     # redis bind address, empty string will use host ip
    redis_max_memory: 1GB             # max memory used by each redis instance
    redis_mem_policy: allkeys-lru     # redis memory eviction policy
    redis_password: ''                # redis password, empty string will disable password
    redis_rdb_save: ['1200 1']        # redis rdb save directives, disable with empty list
    redis_aof_enabled: false          # enable redis append only file?
    redis_rename_commands: {}         # rename redis dangerous commands
    redis_cluster_replicas: 1         # replica number for one master in redis cluster
    redis_sentinel_monitor: []        # sentinel master list, works on sentinel cluster only
    redis_safeguard: false            # prevent purging running redis instance?
    redis_rm_data: true               # remove redis data dir?
    redis_rm_pkg: false               # uninstall selected engine & redis-exporter packages?


    #================================================================#
    #                         VARS: PGSQL                            #
    #================================================================#

    #-----------------------------------------------------------------
    # PG_IDENTITY
    #-----------------------------------------------------------------
    pg_mode: pgsql          #CLUSTER  # pgsql cluster mode: pgsql,citus,mssql,mysql,ivory,pgtde,polar,gpsql,agens,oriole,pgedge
    # pg_cluster:           #CLUSTER  # pgsql cluster name, required identity parameter
    # pg_seq: 0             #INSTANCE # pgsql instance seq number, required identity parameter
    # pg_role: replica      #INSTANCE # pgsql role, required, could be primary,replica,offline
    # pg_instances: {}      #INSTANCE # define multiple pg instances on node in `{port:ins_vars}` format
    # pg_upstream:          #INSTANCE # repl upstream ip addr for standby cluster or cascade replica
    # pg_shard:             #CLUSTER  # pgsql shard name, optional identity for sharding clusters
    # pg_group: 0           #CLUSTER  # pgsql shard index number, optional identity for sharding clusters
    # gp_role: master       #CLUSTER  # greenplum role of this cluster, could be master or segment
    pg_offline_query: false #INSTANCE # set to true to enable offline queries on this instance

    #-----------------------------------------------------------------
    # PG_BUSINESS
    #-----------------------------------------------------------------
    # postgres business object definition, overwrite in group vars
    pg_users: []                      # postgres business users
    pg_databases: []                  # postgres business databases
    pg_services: []                   # postgres business services
    pg_hba_rules: []                  # business hba rules for postgres
    pgb_hba_rules: []                 # business hba rules for pgbouncer
    pg_crontab: []                    # postgres crontab entries for dbsu
    # global credentials, overwrite in global vars
    pg_dbsu_password: ''              # dbsu password, empty string means no dbsu password by default
    pg_replication_username: replicator
    pg_replication_password: DBUser.Replicator
    pg_admin_username: dbuser_dba
    pg_admin_password: DBUser.DBA
    pg_monitor_username: dbuser_monitor
    pg_monitor_password: DBUser.Monitor

    #-----------------------------------------------------------------
    # PG_INSTALL
    #-----------------------------------------------------------------
    pg_dbsu: postgres                 # os dbsu name, postgres by default, better not change it
    pg_dbsu_uid: 26                   # os dbsu uid and gid, 26 for default postgres users and groups
    pg_dbsu_sudo: limit               # dbsu sudo privilege, none,limit,all,nopass. limit by default
    pg_dbsu_home: /var/lib/pgsql      # postgresql home directory, `/var/lib/pgsql` by default
    pg_dbsu_ssh_exchange: true        # exchange postgres dbsu ssh key among same pgsql cluster
    pg_version: 18                    # postgres major version to be installed, 18 by default
    pg_bin_dir: /usr/pgsql/bin        # postgres binary dir, `/usr/pgsql/bin` by default
    pg_log_dir: /pg/log/postgres      # postgres log dir, `/pg/log/postgres` by default
    pg_packages:                      # pg packages to be installed, alias can be used
      - pgsql-main pgsql-common
    pg_extensions: []                 # pg extensions to be installed, alias can be used

    #-----------------------------------------------------------------
    # PG_BOOTSTRAP
    #-----------------------------------------------------------------
    pg_data: /pg/data                 # postgres data directory, `/pg/data` by default
    pg_fs_main: /data/postgres        # postgres main data directory, `/data/postgres` by default
    pg_fs_backup: /data/backups       # postgres backup data directory, `/data/backups` by default
    pg_storage_type: SSD              # storage type for pg main data, SSD,HDD, SSD by default
    pg_dummy_filesize: 64MiB          # size of `/pg/dummy`, hold 64MB disk space for emergency use
    pg_listen: '0.0.0.0'              # postgres/pgbouncer listen addresses, comma separated list
    pg_port: 5432                     # postgres listen port, 5432 by default
    pg_localhost: /var/run/postgresql # postgres unix socket dir for localhost connection
    patroni_enabled: true             # if disabled, no postgres cluster will be created during init
    patroni_mode: default             # patroni working mode: default,pause,remove
    pg_namespace: /pg                 # top level key namespace in etcd, used by patroni & vip
    patroni_port: 8008                # patroni listen port, 8008 by default
    patroni_log_dir: /pg/log/patroni  # patroni log dir, `/pg/log/patroni` by default
    patroni_ssl_enabled: false        # secure patroni RestAPI communications with SSL?
    patroni_watchdog_mode: 'off'      # patroni watchdog mode: automatic,required,off. off by default
    patroni_username: postgres        # patroni restapi username, `postgres` by default
    patroni_password: Patroni.API     # patroni restapi password, `Patroni.API` by default
    pg_etcd_password: ''              # etcd password for this pg cluster, '' to use pg_cluster
    pg_primary_db: postgres           # primary database name, used by citus,etc... ,postgres by default
    pg_parameters: {}                 # extra parameters in postgresql.auto.conf
    pg_files: []                      # extra files to be copied to postgres data directory (e.g. license)
    pg_conf: oltp.yml                 # config template: oltp,olap,crit,tiny. `oltp.yml` by default
    pg_max_conn: auto                 # postgres max connections, `auto` will use recommended value
    pg_shared_buffer_ratio: 0.25      # postgres shared buffers ratio, 0.25 by default, 0.1~0.4
    pg_io_method: worker              # io method for postgres, auto,fsync,worker,io_uring, worker by default
    pg_rto: norm                      # shared rto mode for patroni & haproxy: fast,norm,safe,wide
    pg_rto_plan:  # [ttl, loop, retry, start, margin, inter, fastinter, downinter, rise, fall]
      fast: [ 20  ,5  ,5  ,15 ,5  ,'1s' ,'0.5s' ,'1s' ,3 ,3 ]
      norm: [ 30  ,5  ,10 ,25 ,5  ,'2s' ,'1s'   ,'2s' ,3 ,3 ]
      safe: [ 60  ,10 ,20 ,45 ,10 ,'3s' ,'1.5s' ,'3s' ,3 ,3 ]
      wide: [ 120 ,20 ,30 ,95 ,15 ,'4s' ,'2s'   ,'4s' ,3 ,3 ]
    pg_rpo: 1048576                   # recovery point objective in bytes, `1MiB` at most by default
    pg_libs: 'pg_stat_statements, auto_explain'  # preloaded libraries, `pg_stat_statements,auto_explain` by default
    pg_delay: 0                       # replication apply delay for standby cluster leader
    pg_checksum: true                 # enable data checksum for postgres cluster?
    pg_pwd_enc: scram-sha-256         # password encryption algorithm
    pg_encoding: UTF8                 # database cluster encoding, `UTF8` by default
    pg_locale: C                      # database cluster local, `C` by default
    pg_lc_collate: C                  # database cluster collate, `C` by default
    pg_lc_ctype: C                    # database character type, `C` by default
    #pgsodium_key: ""                 # pgsodium key, 64 hex digit, default to sha256(pg_cluster)
    #pgsodium_getkey_script: ""       # pgsodium getkey script path, pgsodium_getkey by default

    #-----------------------------------------------------------------
    # PG_PROVISION
    #-----------------------------------------------------------------
    pg_provision: true                # provision postgres cluster after bootstrap
    pg_init: pg-init                  # provision init script for cluster template, `pg-init` by default
    pg_default_roles:                 # default roles and users in postgres cluster
      - { name: dbrole_readonly  ,login: false ,comment: role for global read-only access     }
      - { name: dbrole_offline   ,login: false ,comment: role for restricted read-only access }
      - { name: dbrole_readwrite ,login: false ,roles: [dbrole_readonly] ,comment: role for global read-write access }
      - { name: dbrole_admin     ,login: false ,roles: [pg_monitor, dbrole_readwrite] ,comment: role for object creation }
      - { name: postgres     ,superuser: true  ,comment: system superuser }
      - { name: replicator ,replication: true  ,roles: [pg_monitor, dbrole_readonly] ,comment: system replicator }
      - { name: dbuser_dba   ,superuser: true  ,roles: [dbrole_admin]  ,pgbouncer: true ,pool_mode: session, pool_connlimit: 16 ,comment: pgsql admin user }
      - { name: dbuser_monitor ,roles: [pg_monitor, dbrole_readonly] ,pgbouncer: true ,parameters: {log_min_duration_statement: 1000 } ,pool_mode: session ,pool_connlimit: 8 ,comment: pgsql monitor user }
    pg_default_privileges:            # default privileges when created by admin user
      - GRANT USAGE      ON SCHEMAS    TO  dbrole_readonly
      - GRANT SELECT     ON TABLES     TO  dbrole_readonly
      - GRANT SELECT     ON SEQUENCES  TO  dbrole_readonly
      - GRANT EXECUTE    ON FUNCTIONS  TO  dbrole_readonly
      - GRANT USAGE      ON SCHEMAS    TO  dbrole_offline
      - GRANT SELECT     ON TABLES     TO  dbrole_offline
      - GRANT SELECT     ON SEQUENCES  TO  dbrole_offline
      - GRANT EXECUTE    ON FUNCTIONS  TO  dbrole_offline
      - GRANT INSERT     ON TABLES     TO  dbrole_readwrite
      - GRANT UPDATE     ON TABLES     TO  dbrole_readwrite
      - GRANT DELETE     ON TABLES     TO  dbrole_readwrite
      - GRANT USAGE      ON SEQUENCES  TO  dbrole_readwrite
      - GRANT UPDATE     ON SEQUENCES  TO  dbrole_readwrite
      - GRANT TRUNCATE   ON TABLES     TO  dbrole_admin
      - GRANT REFERENCES ON TABLES     TO  dbrole_admin
      - GRANT TRIGGER    ON TABLES     TO  dbrole_admin
      - GRANT CREATE     ON SCHEMAS    TO  dbrole_admin
    pg_default_schemas: [ monitor ]   # default schemas to be created
    pg_default_extensions:            # default extensions to be created
      - { name: pg_stat_statements ,schema: monitor }
      - { name: pgstattuple        ,schema: monitor }
      - { name: pg_buffercache     ,schema: monitor }
      - { name: pageinspect        ,schema: monitor }
      - { name: pg_prewarm         ,schema: monitor }
      - { name: pg_visibility      ,schema: monitor }
      - { name: pg_freespacemap    ,schema: monitor }
      - { name: postgres_fdw       ,schema: public  }
      - { name: file_fdw           ,schema: public  }
      - { name: btree_gist         ,schema: public  }
      - { name: btree_gin          ,schema: public  }
      - { name: pg_trgm            ,schema: public  }
      - { name: intagg             ,schema: public  }
      - { name: intarray           ,schema: public  }
      - { name: pg_repack }
    pg_reload: true                   # reload postgres after hba changes
    pg_default_hba_rules:             # postgres default host-based authentication rules, order by `order`
      - {user: '${dbsu}'    ,db: all         ,addr: local     ,auth: ident ,title: 'dbsu access via local os user ident'  ,order: 100}
      - {user: '${dbsu}'    ,db: replication ,addr: local     ,auth: ident ,title: 'dbsu replication from local os ident' ,order: 150}
      - {user: '${repl}'    ,db: replication ,addr: localhost ,auth: pwd   ,title: 'replicator replication from localhost',order: 200}
      - {user: '${repl}'    ,db: replication ,addr: intra     ,auth: pwd   ,title: 'replicator replication from intranet' ,order: 250}
      - {user: '${repl}'    ,db: postgres    ,addr: intra     ,auth: pwd   ,title: 'replicator postgres db from intranet' ,order: 300}
      - {user: '${monitor}' ,db: all         ,addr: localhost ,auth: pwd   ,title: 'monitor from localhost with password' ,order: 350}
      - {user: '${monitor}' ,db: all         ,addr: infra     ,auth: pwd   ,title: 'monitor from infra host with password',order: 400}
      - {user: '${admin}'   ,db: all         ,addr: intra     ,auth: pwd   ,title: 'admin @ intranet nodes with pwd'      ,order: 450}
      - {user: '${admin}'   ,db: all         ,addr: world     ,auth: ssl   ,title: 'admin @ everywhere with ssl & pwd'    ,order: 500}
      - {user: '+dbrole_readonly',db: all    ,addr: localhost ,auth: pwd   ,title: 'pgbouncer read/write via local socket',order: 550}
      - {user: '+dbrole_readonly',db: all    ,addr: intra     ,auth: pwd   ,title: 'read/write biz user via password'     ,order: 600}
      - {user: '+dbrole_offline' ,db: all    ,addr: intra     ,auth: pwd   ,title: 'allow etl offline tasks from intranet',order: 650}
    pgb_default_hba_rules:            # pgbouncer default host-based authentication rules, order by `order`
      - {user: '${dbsu}'    ,db: pgbouncer   ,addr: local     ,auth: peer  ,title: 'dbsu local admin access with os ident',order: 100}
      - {user: 'all'        ,db: all         ,addr: localhost ,auth: pwd   ,title: 'allow all user local access with pwd' ,order: 150}
      - {user: '${monitor}' ,db: pgbouncer   ,addr: intra     ,auth: pwd   ,title: 'monitor access via intranet with pwd' ,order: 200}
      - {user: '${monitor}' ,db: all         ,addr: world     ,auth: deny  ,title: 'reject all other monitor access addr' ,order: 250}
      - {user: '${admin}'   ,db: all         ,addr: intra     ,auth: pwd   ,title: 'admin access via intranet with pwd'   ,order: 300}
      - {user: '${admin}'   ,db: all         ,addr: world     ,auth: deny  ,title: 'reject all other admin access addr'   ,order: 350}
      - {user: 'all'        ,db: all         ,addr: intra     ,auth: pwd   ,title: 'allow all user intra access with pwd' ,order: 400}

    #-----------------------------------------------------------------
    # PG_BACKUP
    #-----------------------------------------------------------------
    pgbackrest_enabled: true          # enable pgbackrest on pgsql host?
    pgbackrest_log_dir: /pg/log/pgbackrest # pgbackrest log dir, `/pg/log/pgbackrest` by default
    pgbackrest_method: local          # pgbackrest repo method: local,minio,[user-defined...]
    pgbackrest_init_backup: true      # take a full backup after pgbackrest is initialized?
    pgbackrest_repo:                  # pgbackrest repo: https://pgbackrest.org/configuration.html#section-repository
      local:                          # default pgbackrest repo with local posix fs
        path: /pg/backup              # local backup directory, `/pg/backup` by default
        retention_full_type: count    # retention full backups by count
        retention_full: 2             # keep 2, at most 3 full backups when using local fs repo
      minio:                          # optional minio repo for pgbackrest
        type: s3                      # minio is s3-compatible, so s3 is used
        s3_endpoint: sss.pigsty       # minio endpoint domain name, `sss.pigsty` by default
        s3_region: us-east-1          # minio region, us-east-1 by default, useless for minio
        s3_bucket: pgsql              # minio bucket name, `pgsql` by default
        s3_key: pgbackrest            # minio user access key for pgbackrest
        s3_key_secret: S3User.Backup  # minio user secret key for pgbackrest
        s3_uri_style: path            # use path style uri for minio rather than host style
        path: /pgbackrest             # minio backup path, default is `/pgbackrest`
        storage_port: 9000            # minio port, 9000 by default
        storage_ca_file: /etc/pki/ca.crt  # minio ca file path, `/etc/pki/ca.crt` by default
        block: y                      # Enable block incremental backup
        bundle: y                     # bundle small files into a single file
        bundle_limit: 20MiB           # Limit for file bundles, 20MiB for object storage
        bundle_size: 128MiB           # Target size for file bundles, 128MiB for object storage
        cipher_type: aes-256-cbc      # enable AES encryption for remote backup repo
        cipher_pass: pgBackRest       # AES encryption password, default is 'pgBackRest'
        retention_full_type: time     # retention full backup by time on minio repo
        retention_full: 14            # keep full backup for the the last 14 days

    #-----------------------------------------------------------------
    # PG_ACCESS
    #-----------------------------------------------------------------
    pgbouncer_enabled: true           # if disabled, pgbouncer will not be launched on pgsql host
    pgbouncer_port: 6432              # pgbouncer listen port, 6432 by default
    pgbouncer_log_dir: /pg/log/pgbouncer  # pgbouncer log dir, `/pg/log/pgbouncer` by default
    pgbouncer_auth_query: false       # query postgres to retrieve unlisted business users?
    pgbouncer_poolmode: transaction   # pooling mode: transaction,session,statement, transaction by default
    pgbouncer_sslmode: disable        # pgbouncer client ssl mode, disable by default
    pgbouncer_ignore_param: [ extra_float_digits, application_name, TimeZone, DateStyle, IntervalStyle, search_path ]
    pg_weight: 100          #INSTANCE # relative load balance weight in service, 100 by default, 0-255
    pg_service_provider: ''           # dedicate haproxy node group name, or empty string for local nodes by default
    pg_default_service_dest: pgbouncer # default service destination if svc.dest='default'
    pg_default_services:              # postgres default service definitions
      - { name: primary ,port: 5433 ,dest: default  ,check: /primary   ,selector: "[]" }
      - { name: replica ,port: 5434 ,dest: default  ,check: /read-only ,selector: "[]" , backup: "[? pg_role == `primary` || pg_role == `offline` ]" }
      - { name: default ,port: 5436 ,dest: postgres ,check: /primary   ,selector: "[]" }
      - { name: offline ,port: 5438 ,dest: postgres ,check: /replica   ,selector: "[? pg_role == `offline` || pg_offline_query ]" , backup: "[? pg_role == `replica` && !pg_offline_query]"}
    pg_vip_enabled: false             # enable a l2 vip for pgsql primary? false by default
    pg_vip_address: 127.0.0.1/24      # vip address in `<ipv4>/<mask>` format, require if vip is enabled
    pg_vip_interface: auto            # vip network interface to listen, auto by default
    pg_dns_suffix: ''                 # pgsql dns suffix, '' by default
    pg_dns_target: auto               # auto, primary, vip, none, or ad hoc ip

    #-----------------------------------------------------------------
    # PG_MONITOR
    #-----------------------------------------------------------------
    pg_exporter_enabled: true              # enable pg_exporter on pgsql hosts?
    pg_exporter_config: pg_exporter.yml    # pg_exporter configuration file name
    pg_exporter_cache_ttls: '1,10,60,300'  # pg_exporter collector ttl stage in seconds, '1,10,60,300' by default
    pg_exporter_port: 9630                 # pg_exporter listen port, 9630 by default
    pg_exporter_params: 'sslmode=disable'  # extra url parameters for pg_exporter dsn
    pg_exporter_url: ''                    # overwrite auto-generate pg dsn if specified
    pg_exporter_auto_discovery: true       # enable auto database discovery? enabled by default
    pg_exporter_exclude_database: 'template0,template1,postgres' # csv of database that WILL NOT be monitored during auto-discovery
    pg_exporter_include_database: ''       # csv of database that WILL BE monitored during auto-discovery
    pg_exporter_connect_timeout: 200       # pg_exporter connect timeout in ms, 200 by default
    pg_exporter_options: ''                # overwrite extra options for pg_exporter
    pgbouncer_exporter_enabled: true       # enable pgbouncer_exporter on pgsql hosts?
    pgbouncer_exporter_port: 9631          # pgbouncer_exporter listen port, 9631 by default
    pgbouncer_exporter_url: ''             # overwrite auto-generate pgbouncer dsn if specified
    pgbouncer_exporter_options: ''         # overwrite extra options for pgbouncer_exporter
    pgbackrest_exporter_enabled: true      # enable pgbackrest_exporter on pgsql hosts?
    pgbackrest_exporter_port: 9854         # pgbackrest_exporter listen port, 9854 by default
    pgbackrest_exporter_options: >-
      --collect.interval=120
      --log.level=info

    #-----------------------------------------------------------------
    # PG_REMOVE
    #-----------------------------------------------------------------
    pg_safeguard: false               # stop pg_remove running if pg_safeguard is enabled, false by default
    pg_rm_data: true                  # remove postgres data during remove? true by default
    pg_rm_backup: true                # remove pgbackrest backup during primary remove? true by default
    pg_rm_pkg: true                   # uninstall postgres packages during remove? true by default

...

Explanation

The demo/el template is optimized for Enterprise Linux family distributions.

Supported Distributions:

  • RHEL 8/9/10
  • Rocky Linux 8/9/10
  • Alma Linux 8/9/10
  • Oracle Linux 8/9

Key Features:

  • Uses EPEL and PGDG repositories
  • Optimized for YUM/DNF package manager
  • Supports EL-specific package names

Use Cases:

  • Enterprise production environments (RHEL/Rocky/Alma recommended)
  • Long-term support and stability requirements
  • Environments using Red Hat ecosystem

6.28 - demo/debian

Configuration template optimized for Debian/Ubuntu

The demo/debian configuration template is optimized for Debian and Ubuntu distributions.

Overview

  • Config Name: demo/debian
  • Node Count: Single node
  • Description: Debian/Ubuntu optimized configuration template
  • OS Distro: d12, d13, u22, u24, u26
  • OS Arch: x86_64, aarch64
  • Related: meta, demo/el

Usage:

./configure -c demo/debian [-i <primary_ip>]

Content

Source: pigsty/conf/demo/debian.yml

---
#==============================================================#
# File      :   debian.yml
# Desc      :   Default parameters for Debian/Ubuntu in Pigsty
# Ctime     :   2020-05-22
# Mtime     :   2026-08-02
# Docs      :   https://pigsty.io/docs/conf/debian
# License   :   Apache-2.0 @ https://pigsty.io/docs/about/license/
# Copyright :   2018-2026  Ruohang Feng / Vonng ([email protected])
#==============================================================#


#==============================================================#
#                        Sandbox (4-node)                      #
#==============================================================#
# admin user : vagrant  (nopass ssh & sudo already set)        #
# 1.  meta    :    10.10.10.10     (2 Core | 4GB)    pg-meta   #
# 2.  node-1  :    10.10.10.11     (1 Core | 1GB)    pg-test-1 #
# 3.  node-2  :    10.10.10.12     (1 Core | 1GB)    pg-test-2 #
# 4.  node-3  :    10.10.10.13     (1 Core | 1GB)    pg-test-3 #
# (replace these ip if your 4-node env have different ip addr) #
# VIP 2: (l2 vip is available inside same LAN )                #
#     pg-meta --->  10.10.10.2 ---> 10.10.10.10                #
#     pg-test --->  10.10.10.3 ---> 10.10.10.1{1,2,3}          #
#==============================================================#


all:

  ##################################################################
  #                            CLUSTERS                            #
  ##################################################################
  # meta nodes, nodes, pgsql, redis, pgsql clusters are defined as
  # k:v pair inside `all.children`. Where the key is cluster name
  # and value is cluster definition consist of two parts:
  # `hosts`: cluster members ip and instance level variables
  # `vars` : cluster level variables
  ##################################################################
  children:                                 # groups definition

    # infra cluster for proxy, monitor, alert, etc..
    infra: { hosts: { 10.10.10.10: { infra_seq: 1 } } }

    # etcd cluster for ha postgres
    etcd: { hosts: { 10.10.10.10: { etcd_seq: 1 } }, vars: { etcd_cluster: etcd } }

    # minio cluster, s3 compatible object storage
    minio: { hosts: { 10.10.10.10: { minio_seq: 1 } }, vars: { minio_cluster: minio } }

    #----------------------------------#
    # pgsql cluster: pg-meta (CMDB)    #
    #----------------------------------#
    pg-meta:
      hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary , pg_offline_query: true } }
      vars:
        pg_cluster: pg-meta

        # define business databases here: https://pigsty.io/docs/pgsql/config/db
        pg_databases:                       # define business databases on this cluster, array of database definition
          - name: meta                      # REQUIRED, `name` is the only mandatory field of a database definition
            #state: create                  # optional, create|absent|recreate, create by default
            baseline: cmdb.sql              # optional, database sql baseline path, (relative path among ansible search path, e.g: files/)
            schemas: [pigsty]               # optional, additional schemas to be created, array of schema names
            extensions:                     # optional, additional extensions to be installed: array of `{name[,schema]}`
              - { name: vector }            # install pgvector extension on this database by default
            comment: pigsty meta database   # optional, comment string for this database
            #pgbouncer: true                # optional, add this database to pgbouncer database list? true by default
            #owner: postgres                # optional, database owner, current user if not specified
            #template: template1            # optional, which template to use, template1 by default
            #strategy: FILE_COPY            # optional, clone strategy: FILE_COPY or WAL_LOG (PG15+), default to PG's default
            #encoding: UTF8                 # optional, inherited from template / cluster if not defined (UTF8)
            #locale: C                      # optional, inherited from template / cluster if not defined (C)
            #lc_collate: C                  # optional, inherited from template / cluster if not defined (C)
            #lc_ctype: C                    # optional, inherited from template / cluster if not defined (C)
            #locale_provider: libc          # optional, locale provider: libc, icu, builtin (PG15+)
            #icu_locale: en-US              # optional, icu locale for icu locale provider (PG15+)
            #icu_rules: ''                  # optional, icu rules for icu locale provider (PG16+)
            #builtin_locale: C.UTF-8        # optional, builtin locale for builtin locale provider (PG17+)
            #tablespace: pg_default         # optional, default tablespace, pg_default by default
            #is_template: false             # optional, mark database as template, allowing clone by any user with CREATEDB privilege
            #allowconn: true                # optional, allow connection, true by default. false will disable connect at all
            #revokeconn: false              # optional, revoke public connection privilege. false by default. (leave connect with grant option to owner)
            #register_datasource: true      # optional, register this database to grafana datasources? true by default
            #connlimit: -1                  # optional, database connection limit, default -1 disable limit
            #pool_auth_user: dbuser_meta    # optional, all connection to this pgbouncer database will be authenticated by this user
            #pool_mode: transaction         # optional, pgbouncer pool mode at database level, default transaction
            #pool_size: 64                  # optional, pgbouncer pool size at database level, default 64
            #pool_reserve: 32               # optional, pgbouncer pool size reserve at database level, default 32
            #pool_size_min: 0               # optional, pgbouncer pool size min at database level, default 0
            #pool_connlimit: 100            # optional, max database connections at database level, default 100
          #- { name: grafana  ,owner: dbuser_grafana  ,revokeconn: true ,comment: grafana primary database }
          #- { name: bytebase ,owner: dbuser_bytebase ,revokeconn: true ,comment: bytebase primary database }
          #- { name: kong     ,owner: dbuser_kong     ,revokeconn: true ,comment: kong the api gateway database }
          #- { name: gitea    ,owner: dbuser_gitea    ,revokeconn: true ,comment: gitea meta database }
          #- { name: wiki     ,owner: dbuser_wiki     ,revokeconn: true ,comment: wiki meta database }

        # define business users here: https://pigsty.io/docs/pgsql/config/user
        pg_users:                           # define business users/roles on this cluster, array of user definition
          - name: dbuser_meta               # REQUIRED, `name` is the only mandatory field of a user definition
            password: DBUser.Meta           # optional, password, can be a scram-sha-256 hash string or plain text
            pgbouncer: true                 # optional, add this user to pgbouncer user-list? false by default (production user should be true explicitly)
            comment: pigsty admin user      # optional, comment string for this user/role
            roles: [ dbrole_admin ]         # optional, belonged roles. default roles are: dbrole_{admin,readonly,readwrite,offline}
            #login: true                     # optional, can log in, true by default  (new biz ROLE should be false)
            #superuser: false                # optional, is superuser? false by default
            #createdb: false                 # optional, can create database? false by default
            #createrole: false               # optional, can create role? false by default
            #inherit: true                   # optional, can this role use inherited privileges? true by default
            #replication: false              # optional, can this role do replication? false by default
            #bypassrls: false                # optional, can this role bypass row level security? false by default
            #connlimit: -1                   # optional, user connection limit, default -1 disable limit
            #expire_in: 3650                 # optional, now + n days when this role is expired (OVERWRITE expire_at)
            #expire_at: '2030-12-31'         # optional, YYYY-MM-DD 'timestamp' when this role is expired  (OVERWRITTEN by expire_in)
            #parameters: {}                  # optional, role level parameters with `ALTER ROLE SET`
            #pool_mode: transaction          # optional, pgbouncer pool mode at user level, transaction by default
            #pool_connlimit: -1              # optional, max database connections at user level, default -1 disable limit
          - {name: dbuser_view     ,password: DBUser.Viewer   ,pgbouncer: true ,roles: [dbrole_readonly], comment: read-only viewer for meta database}
          #- {name: dbuser_grafana  ,password: DBUser.Grafana  ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for grafana database   }
          #- {name: dbuser_bytebase ,password: DBUser.Bytebase ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for bytebase database  }
          #- {name: dbuser_gitea    ,password: DBUser.Gitea    ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for gitea service      }
          #- {name: dbuser_wiki     ,password: DBUser.Wiki     ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for wiki.js service    }

        # define business service here: https://pigsty.io/docs/pgsql/service
        pg_services:                        # extra services in addition to pg_default_services, array of service definition
          # standby service will route {ip|name}:5435 to sync replica's pgbouncer (5435->6432 standby)
          - name: standby                   # required, service name, the actual svc name will be prefixed with `pg_cluster`, e.g: pg-meta-standby
            port: 5435                      # required, service exposed port (work as kubernetes service node port mode)
            ip: "*"                         # optional, service bind ip address, `*` for all ip by default
            selector: "[]"                  # required, service member selector, use JMESPath to filter inventory
            dest: default                   # optional, destination port, default|postgres|pgbouncer|<port_number>, 'default' by default
            check: /sync                    # optional, health check url path, / by default
            backup: "[? pg_role == `primary`]"  # backup server selector
            maxconn: 3000                   # optional, max allowed front-end connection
            balance: roundrobin             # optional, haproxy load balance algorithm (roundrobin by default, other: leastconn)
            #options: 'inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100'

        # define pg extensions: https://pigsty.io/docs/pgsql/ext/
        pg_libs: 'pg_stat_statements, auto_explain' # add timescaledb to shared_preload_libraries
        #pg_extensions: [] # extensions to be installed on this cluster

        # define HBA rules here: https://pigsty.io/docs/pgsql/config/hba
        pg_hba_rules:
          - {user: dbuser_view , db: all ,addr: infra ,auth: pwd ,title: 'allow grafana dashboard access cmdb from infra nodes'}

        pg_vip_enabled: true
        pg_vip_address: 10.10.10.2/24

        pg_crontab:  # make a full backup 1 am everyday
          - '00 01 * * * /pg/bin/pg-backup full'

    #----------------------------------#
    # pgsql cluster: pg-test (3 nodes) #
    #----------------------------------#
    # pg-test --->  10.10.10.3 ---> 10.10.10.1{1,2,3}
    pg-test:                          # define the new 3-node cluster pg-test
      hosts:
        10.10.10.11: { pg_seq: 1, pg_role: primary }   # primary instance, leader of cluster
        10.10.10.12: { pg_seq: 2, pg_role: replica }   # replica instance, follower of leader
        10.10.10.13: { pg_seq: 3, pg_role: replica, pg_offline_query: true } # replica with offline access
      vars:
        pg_cluster: pg-test           # define pgsql cluster name
        pg_users:  [{ name: test , password: test , pgbouncer: true , roles: [ dbrole_admin ] }]
        pg_databases: [{ name: test }] # create a database and user named 'test'
        node_tune: tiny
        pg_conf: tiny.yml
        pg_vip_enabled: true
        pg_vip_address: 10.10.10.3/24
        pg_crontab:  # make a full backup on monday 1am, and an incremental backup during weekdays
          - '00 01 * * 1 /pg/bin/pg-backup full'
          - '00 01 * * 2,3,4,5,6,7 /pg/bin/pg-backup'

    #----------------------------------#
    # redis ms, sentinel, native cluster
    #----------------------------------#
    redis-ms: # redis classic primary & replica
      hosts: { 10.10.10.10: { redis_node: 1 , redis_instances: { 6379: { }, 6380: { replica_of: '10.10.10.10 6379' } } } }
      vars: { redis_cluster: redis-ms ,redis_password: 'redis.ms' ,redis_max_memory: 64MB }

    redis-meta: # redis sentinel x 3
      hosts: { 10.10.10.11: { redis_node: 1 , redis_instances: { 26379: { } ,26380: { } ,26381: { } } } }
      vars:
        redis_cluster: redis-meta
        redis_password: 'redis.meta'
        redis_mode: sentinel
        redis_max_memory: 16MB
        redis_sentinel_monitor: # primary list for redis sentinel, use cls as name, primary ip:port
          - { name: redis-ms, host: 10.10.10.10, port: 6379 ,password: redis.ms, quorum: 2 }

    redis-test: # redis native cluster: 3m x 3s
      hosts:
        10.10.10.12: { redis_node: 1 ,redis_instances: { 6379: { } ,6380: { } ,6381: { } } }
        10.10.10.13: { redis_node: 2 ,redis_instances: { 6379: { } ,6380: { } ,6381: { } } }
      vars: { redis_cluster: redis-test ,redis_password: 'redis.test' ,redis_mode: cluster, redis_max_memory: 32MB }


  ####################################################################
  #                             VARS                                 #
  ####################################################################
  vars:                               # global variables


    #================================================================#
    #                         VARS: INFRA                            #
    #================================================================#

    #-----------------------------------------------------------------
    # META
    #-----------------------------------------------------------------
    version: v4.5.0                   # pigsty version string
    admin_ip: 10.10.10.10             # admin node ip address
    region: default                   # upstream mirror region: default,china,europe
    language: en                      # default language: en, zh
    proxy_env:                        # global proxy env when downloading packages
      no_proxy: "localhost,127.0.0.1,10.0.0.0/8,192.168.0.0/16,*.pigsty,*.aliyun.com,mirrors.*,*.myqcloud.com,*.tsinghua.edu.cn"
      # http_proxy:  # set your proxy here: e.g http://user:[email protected]
      # https_proxy: # set your proxy here: e.g http://user:[email protected]
      # all_proxy:   # set your proxy here: e.g http://user:[email protected]

    #-----------------------------------------------------------------
    # CA
    #-----------------------------------------------------------------
    ca_create: true                   # create ca if not exists? or just abort
    ca_cn: pigsty-ca                  # ca common name, fixed as pigsty-ca
    cert_validity: 7300d              # cert validity, 20 years by default

    #-----------------------------------------------------------------
    # INFRA_IDENTITY
    #-----------------------------------------------------------------
    #infra_seq: 1                     # infra node identity, explicitly required
    infra_portal:                     # infra services exposed via portal
      home : { domain: i.pigsty }     # default domain name
    infra_data: /data/infra           # default data path for infrastructure data
    infra_services:                   # home page navigation entries
      - { name: Metrics            ,url: '/vmetrics/vmui/'         ,desc: 'VictoriaMetrics Query UI'    ,icon: metrics  ,name_cn: '指标查询' ,desc_cn: 'VictoriaMetrics 指标查询界面' }
      - { name: Logs               ,url: '/vlogs/select/vmui/'     ,desc: 'VictoriaLogs Query UI'       ,icon: logs     ,name_cn: '日志查询' ,desc_cn: 'VictoriaLogs 日志查询界面' }
      - { name: Traces             ,url: '/vtraces/select/vmui/'   ,desc: 'VictoriaTraces Query UI'     ,icon: traces   ,name_cn: '链路追踪' ,desc_cn: 'VictoriaTraces 链路查询界面' }
      - { name: Monitor Targets    ,url: '/vmetrics/targets'       ,desc: 'Prometheus Scrape Targets'   ,icon: target   ,name_cn: '监控目标' ,desc_cn: 'VictoriaMetrics 监控对象列表' }
      - { name: Alert Rules        ,url: '/vmalert/vmalert/groups' ,desc: 'VMAlert alert/record Rules'  ,icon: alert    ,name_cn: '告警规则' ,desc_cn: 'VMAlert 告警规则管理' }
      - { name: Alert Manager      ,url: '/alertmgr/#/alerts'      ,desc: 'Alert Manage & Silence'      ,icon: alertmgr ,name_cn: '告警管理' ,desc_cn: 'AlertManager 告警管理与屏蔽' }
      - { name: CA Certificate     ,url: '/ca.crt'                 ,desc: 'Self-Signed CA Certificate'  ,icon: lock     ,name_cn: 'CA 证书'  ,desc_cn: 'Pigsty 自签CA根证书' }
      - { name: Software Repo      ,url: '/pigsty'                 ,desc: 'Local YUM/APT Repository'    ,icon: package  ,name_cn: '软件仓库' ,desc_cn: '本地 YUM/APT 软件源' }
      - { name: Explain Visualizer ,url: '/pev'                    ,desc: 'Postgres EXPLAIN Visualizer' ,icon: search   ,name_cn: '执行计划' ,desc_cn: 'PG 执行计划可视化工具' }
    infra_extra_services: []          # extra services to be added on infra home page

    #-----------------------------------------------------------------
    # REPO
    #-----------------------------------------------------------------
    repo_enabled: true                # create a yum repo on this infra node?
    repo_home: /www                   # repo home dir, `/www` by default
    repo_name: pigsty                 # repo name, pigsty by default
    repo_endpoint: http://${admin_ip}:80 # access point to this repo by domain or ip:port
    repo_remove: true                 # remove existing upstream repo
    repo_modules: infra,node,pgsql    # which repo modules are installed in repo_upstream
    repo_upstream:                    # where to download
      - { name: pigsty-local   ,description: 'Pigsty Local'       ,module: local   ,releases: [11,12,13,22,24,26] ,arch: [x86_64, aarch64] ,baseurl: { default: 'http://${admin_ip}/pigsty ./' }}
      - { name: pigsty-pgsql   ,description: 'Pigsty PgSQL'       ,module: pgsql   ,releases: [11,12,13,22,24,26] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://repo.pigsty.io/apt/pgsql/${distro_codename} ${distro_codename} main', china: 'https://repo.pigsty.cc/apt/pgsql/${distro_codename} ${distro_codename} main' }}
      - { name: pigsty-infra   ,description: 'Pigsty Infra'       ,module: infra   ,releases: [11,12,13,22,24,26] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://repo.pigsty.io/apt/infra/ generic main' ,china: 'https://repo.pigsty.cc/apt/infra/ generic main' }}
      - { name: nginx          ,description: 'Nginx'              ,module: infra   ,releases: [11,12,13,22,24,26] ,arch: [x86_64, aarch64] ,baseurl: { default: 'http://nginx.org/packages/${distro_name} ${distro_codename} nginx' }}
      - { name: docker-ce      ,description: 'Docker'             ,module: infra   ,releases: [11,12,13,22,24,26] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://download.docker.com/linux/${distro_name} ${distro_codename} stable'                               ,china: 'https://mirrors.cloud.tencent.com/docker-ce/linux/${distro_name} ${distro_codename} stable' }}
      - { name: base           ,description: 'Debian Basic'       ,module: node    ,releases: [11,12,13         ] ,arch: [x86_64, aarch64] ,baseurl: { default: 'http://deb.debian.org/debian/ ${distro_codename} main non-free-firmware'                                  ,china: 'https://mirrors.cloud.tencent.com/debian/ ${distro_codename} main non-free-firmware' }}
      - { name: updates        ,description: 'Debian Updates'     ,module: node    ,releases: [11,12,13         ] ,arch: [x86_64, aarch64] ,baseurl: { default: 'http://deb.debian.org/debian/ ${distro_codename}-updates main non-free-firmware'                          ,china: 'https://mirrors.cloud.tencent.com/debian/ ${distro_codename}-updates main non-free-firmware' }}
      - { name: security       ,description: 'Debian Security'    ,module: node    ,releases: [11,12,13         ] ,arch: [x86_64, aarch64] ,baseurl: { default: 'http://security.debian.org/debian-security ${distro_codename}-security main non-free-firmware'            ,china: 'https://mirrors.cloud.tencent.com/debian-security/ ${distro_codename}-security main non-free-firmware' }}
      - { name: base           ,description: 'Ubuntu Basic'       ,module: node    ,releases: [         22,24,26] ,arch: [x86_64         ] ,baseurl: { default: 'https://mirrors.edge.kernel.org/ubuntu/ ${distro_codename}           main universe multiverse restricted' ,china: 'https://mirrors.cloud.tencent.com/ubuntu/ ${distro_codename}           main restricted universe multiverse' }}
      - { name: updates        ,description: 'Ubuntu Updates'     ,module: node    ,releases: [         22,24,26] ,arch: [x86_64         ] ,baseurl: { default: 'https://mirrors.edge.kernel.org/ubuntu/ ${distro_codename}-updates   main restricted universe multiverse' ,china: 'https://mirrors.cloud.tencent.com/ubuntu/ ${distro_codename}-updates   main restricted universe multiverse' }}
      - { name: backports      ,description: 'Ubuntu Backports'   ,module: node    ,releases: [         22,24,26] ,arch: [x86_64         ] ,baseurl: { default: 'https://mirrors.edge.kernel.org/ubuntu/ ${distro_codename}-backports main restricted universe multiverse' ,china: 'https://mirrors.cloud.tencent.com/ubuntu/ ${distro_codename}-backports main restricted universe multiverse' }}
      - { name: security       ,description: 'Ubuntu Security'    ,module: node    ,releases: [         22,24,26] ,arch: [x86_64         ] ,baseurl: { default: 'https://mirrors.edge.kernel.org/ubuntu/ ${distro_codename}-security  main restricted universe multiverse' ,china: 'https://mirrors.cloud.tencent.com/ubuntu/ ${distro_codename}-security  main restricted universe multiverse' }}
      - { name: base           ,description: 'Ubuntu Basic'       ,module: node    ,releases: [         22,24,26] ,arch: [        aarch64] ,baseurl: { default: 'http://ports.ubuntu.com/ubuntu-ports/ ${distro_codename}             main universe multiverse restricted' ,china: 'https://mirrors.cloud.tencent.com/ubuntu-ports/ ${distro_codename}           main restricted universe multiverse' }}
      - { name: updates        ,description: 'Ubuntu Updates'     ,module: node    ,releases: [         22,24,26] ,arch: [        aarch64] ,baseurl: { default: 'http://ports.ubuntu.com/ubuntu-ports/ ${distro_codename}-updates     main restricted universe multiverse' ,china: 'https://mirrors.cloud.tencent.com/ubuntu-ports/ ${distro_codename}-updates   main restricted universe multiverse' }}
      - { name: backports      ,description: 'Ubuntu Backports'   ,module: node    ,releases: [         22,24,26] ,arch: [        aarch64] ,baseurl: { default: 'http://ports.ubuntu.com/ubuntu-ports/ ${distro_codename}-backports   main restricted universe multiverse' ,china: 'https://mirrors.cloud.tencent.com/ubuntu-ports/ ${distro_codename}-backports main restricted universe multiverse' }}
      - { name: security       ,description: 'Ubuntu Security'    ,module: node    ,releases: [         22,24,26] ,arch: [        aarch64] ,baseurl: { default: 'http://ports.ubuntu.com/ubuntu-ports/ ${distro_codename}-security    main restricted universe multiverse' ,china: 'https://mirrors.cloud.tencent.com/ubuntu-ports/ ${distro_codename}-security  main restricted universe multiverse' }}
      - { name: pgdg           ,description: 'PGDG'               ,module: pgsql   ,releases: [11,12,13,22,24,26] ,arch: [x86_64, aarch64] ,baseurl: { default: 'http://apt.postgresql.org/pub/repos/apt/ ${distro_codename}-pgdg main' ,china: 'https://mirrors.cloud.tencent.com/postgresql/repos/apt/ ${distro_codename}-pgdg main' }}
      - { name: pgdg-beta      ,description: 'PGDG Beta'          ,module: beta    ,releases: [11,12,13,22,24,26] ,arch: [x86_64, aarch64] ,baseurl: { default: 'http://apt.postgresql.org/pub/repos/apt/ ${distro_codename}-pgdg-testing main 19' ,china: 'https://mirrors.cloud.tencent.com/postgresql/repos/apt/ ${distro_codename}-pgdg-testing main 19' }}
      - { name: timescaledb    ,description: 'TimescaleDB'        ,module: extra   ,releases: [11,12,13,22,24,26] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://packagecloud.io/timescale/timescaledb/${distro_name}/ ${distro_codename} main' }}
      - { name: citus          ,description: 'Citus'              ,module: extra   ,releases: [11,12,   22      ] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://packagecloud.io/citusdata/community/${distro_name}/ ${distro_codename} main' } }
      - { name: percona        ,description: 'Percona TDE'        ,module: percona ,releases: [   12,13,22,24,26] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://repo.pigsty.io/apt/percona ${distro_codename} main' ,china: 'https://repo.pigsty.cc/apt/percona ${distro_codename} main' ,origin: 'http://repo.percona.com/ppg-18.4/apt ${distro_codename} main' }}
      - { name: groonga        ,description: 'Groonga Debian'     ,module: groonga ,releases: [11,12,13         ] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://packages.groonga.org/debian/ ${distro_codename} main' }}
      - { name: groonga        ,description: 'Groonga Ubuntu'     ,module: groonga ,releases: [         22,24   ] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://ppa.launchpadcontent.net/groonga/ppa/ubuntu/ ${distro_codename} main' }}
      - { name: mysql          ,description: 'MySQL 8.4 LTS'      ,module: mysql   ,releases: [   12,13,22,24   ] ,arch: [x86_64         ] ,baseurl: { default: 'https://repo.mysql.com/apt/${distro_name} ${distro_codename} mysql-8.4-lts', china: 'https://mirrors.ustc.edu.cn/mysql-repo/apt/${distro_name} ${distro_codename} mysql-8.4-lts' }}
      - { name: mongo          ,description: 'MongoDB'            ,module: mongo   ,releases: [   12,   22,24   ] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://repo.mongodb.org/apt/${distro_name} ${distro_codename}/mongodb-org/8.0 multiverse', china: 'https://mirrors.cloud.tencent.com/mongodb/apt/${distro_name} ${distro_codename}/mongodb-org/8.0 multiverse' }}
      - { name: redis          ,description: 'Redis'              ,module: redis   ,releases: [11,12,   22,24,26] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://packages.redis.io/deb ${distro_codename} main' }}
      - { name: llvm           ,description: 'LLVM'               ,module: llvm    ,releases: [11,12,13,22,24,26] ,arch: [x86_64, aarch64] ,baseurl: { default: 'http://apt.llvm.org/${distro_codename}/ llvm-toolchain-${distro_codename} main' ,china: 'https://mirrors.tuna.tsinghua.edu.cn/llvm-apt/${distro_codename}/ llvm-toolchain-${distro_codename} main' }}
      - { name: haproxyd       ,description: 'Haproxy Debian'     ,module: haproxy ,releases: [   12            ] ,arch: [x86_64, aarch64] ,baseurl: { default: 'http://haproxy.debian.net/ ${distro_codename}-backports-3.2 main' }}
      - { name: haproxyu       ,description: 'Haproxy Ubuntu'     ,module: haproxy ,releases: [            24,26] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://ppa.launchpadcontent.net/vbernat/haproxy-3.2/ubuntu/ ${distro_codename} main' }}
      - { name: grafana        ,description: 'Grafana'            ,module: grafana ,releases: [11,12,13,22,24,26] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://apt.grafana.com stable main' ,china: 'https://mirrors.cloud.tencent.com/grafana/apt/ stable main' }}
      - { name: kubernetes     ,description: 'Kubernetes'         ,module: kube    ,releases: [11,12,13,22,24,26] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://pkgs.k8s.io/core:/stable:/v1.36/deb/ /', china: 'https://mirrors.ustc.edu.cn/kubernetes/core:/stable:/v1.36/deb/ /' }}
      - { name: gitlab-ee      ,description: 'Gitlab EE'          ,module: gitlab  ,releases: [11,12,13,22,24   ] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://packages.gitlab.com/gitlab/gitlab-ee/${distro_name}/ ${distro_codename} main' }}
      - { name: gitlab-ce      ,description: 'Gitlab CE'          ,module: gitlab  ,releases: [11,12,13,22,24   ] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://packages.gitlab.com/gitlab/gitlab-ce/${distro_name}/ ${distro_codename} main' }}
      - { name: clickhouse     ,description: 'ClickHouse'         ,module: click   ,releases: [11,12,13,22,24,26] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://packages.clickhouse.com/deb/ stable main', china: 'https://repo.huaweicloud.com/clickhouse/deb/ stable main' }}

    repo_packages: [ node-bootstrap, infra-package, infra-addons, node-package1, node-package2, node-package3, pgsql-utility, extra-modules ]
    repo_extra_packages: [ pgsql-main ]
    repo_url_packages: []

    #-----------------------------------------------------------------
    # INFRA_PACKAGE
    #-----------------------------------------------------------------
    infra_packages:                   # packages to be installed on infra nodes
      - grafana,grafana-plugins,grafana-victorialogs-ds,grafana-victoriametrics-ds,victoria-metrics,victoria-logs,victoria-traces,vmutils,vlogscli,alertmanager
      - node-exporter,blackbox-exporter,nginx-exporter,pg-exporter,pev2,nginx,dnsmasq,ansible,etcd,python3-requests,redis,mcli,restic,certbot,python3-certbot-nginx

    #-----------------------------------------------------------------
    # NGINX
    #-----------------------------------------------------------------
    nginx_enabled: true               # enable nginx on this infra node?
    nginx_clean: false                # clean existing nginx config during init?
    nginx_exporter_enabled: true      # enable nginx_exporter on this infra node?
    nginx_exporter_port: 9113         # nginx_exporter listen port, 9113 by default
    nginx_sslmode: enable             # nginx ssl mode? disable,enable,enforce
    nginx_cert_validity: 397d         # nginx self-signed cert validity, 397d by default
    nginx_home: /www                  # nginx content dir, `/www` by default (soft link to nginx_data)
    nginx_data: /data/nginx           # nginx actual data dir, /data/nginx by default
    nginx_users: { admin : pigsty }   # nginx basic auth users: name and pass dict
    nginx_port: 80                    # nginx listen port, 80 by default
    nginx_ssl_port: 443               # nginx ssl listen port, 443 by default
    certbot_sign: false               # sign nginx cert with certbot during setup?
    certbot_email: [email protected]     # certbot email address, used for free ssl
    certbot_options: ''               # certbot extra options

    #-----------------------------------------------------------------
    # DNS
    #-----------------------------------------------------------------
    dns_enabled: true                 # setup dnsmasq on this infra node?
    dns_port: 53                      # dns server listen port, 53 by default
    dns_records:                      # dynamic dns records resolved by dnsmasq
      - "${admin_ip} i.pigsty"
      - "${admin_ip} m.pigsty supa.pigsty api.pigsty adm.pigsty cli.pigsty ddl.pigsty"

    #-----------------------------------------------------------------
    # VICTORIA
    #-----------------------------------------------------------------
    vmetrics_enabled: true            # enable victoria-metrics on this infra node?
    vmetrics_clean: false             # whether clean existing victoria metrics data during init?
    vmetrics_port: 8428               # victoria-metrics listen port, 8428 by default
    vmetrics_scrape_interval: 10s     # victoria global scrape interval, 10s by default
    vmetrics_scrape_timeout: 8s       # victoria global scrape timeout, 8s by default
    vmetrics_options: >-
      -retentionPeriod=15d
      -promscrape.fileSDCheckInterval=5s
    vlogs_enabled: true               # enable victoria-logs on this infra node?
    vlogs_clean: false                # clean victoria-logs data during init?
    vlogs_port: 9428                  # victoria-logs listen port, 9428 by default
    vlogs_options: >-
      -retentionPeriod=15d
      -retention.maxDiskSpaceUsageBytes=50GiB
      -insert.maxLineSizeBytes=1MB
      -search.maxQueryDuration=120s
    vtraces_enabled: true             # enable victoria-traces on this infra node?
    vtraces_clean: false                # clean victoria-trace data during inti?
    vtraces_port: 10428               # victoria-traces listen port, 10428 by default
    vtraces_options: >-
      -retentionPeriod=15d
      -retention.maxDiskSpaceUsageBytes=50GiB
    vmalert_enabled: true             # enable vmalert on this infra node?
    vmalert_port: 8880                # vmalert listen port, 8880 by default
    vmalert_options: ''              # vmalert extra server options

    #-----------------------------------------------------------------
    # PROMETHEUS
    #-----------------------------------------------------------------
    blackbox_enabled: true            # setup blackbox_exporter on this infra node?
    blackbox_port: 9115               # blackbox_exporter listen port, 9115 by default
    blackbox_options: ''              # blackbox_exporter extra server options
    alertmanager_enabled: true        # setup alertmanager on this infra node?
    alertmanager_port: 9059           # alertmanager listen port, 9059 by default
    alertmanager_options: ''          # alertmanager extra server options
    exporter_metrics_path: /metrics   # exporter metric path, `/metrics` by default

    #-----------------------------------------------------------------
    # GRAFANA
    #-----------------------------------------------------------------
    grafana_enabled: true             # enable grafana on this infra node?
    grafana_port: 3000                # default listen port for grafana
    grafana_clean: false              # clean grafana data during init?
    grafana_admin_username: admin     # grafana admin username, `admin` by default
    grafana_admin_password: pigsty    # grafana admin password, `pigsty` by default
    grafana_auth_proxy: false         # enable grafana auth proxy?
    grafana_pgurl: ''                 # external postgres database url for grafana if given
    grafana_view_password: DBUser.Viewer # password for grafana meta pg datasource


    #================================================================#
    #                         VARS: NODE                             #
    #================================================================#

    #-----------------------------------------------------------------
    # NODE_IDENTITY
    #-----------------------------------------------------------------
    #nodename:           # [INSTANCE] # node instance identity, use hostname if missing, optional
    node_cluster: nodes   # [CLUSTER] # node cluster identity, use 'nodes' if missing, optional
    nodename_overwrite: true          # overwrite node's hostname with nodename?
    nodename_exchange: false          # exchange nodename among play hosts?
    node_id_from_pg: true             # use postgres identity as node identity if applicable?

    #-----------------------------------------------------------------
    # NODE_DNS
    #-----------------------------------------------------------------
    node_write_etc_hosts: true        # modify `/etc/hosts` on target node?
    node_default_etc_hosts:           # static dns records in `/etc/hosts`
      - "${admin_ip} i.pigsty"
    node_etc_hosts: []                # extra static dns records in `/etc/hosts`
    node_dns_method: add              # how to handle dns servers: add,none,overwrite
    node_dns_servers: ['${admin_ip}'] # dynamic nameserver in `/etc/resolv.conf`
    node_dns_options:                 # dns resolv options in `/etc/resolv.conf`
      - options single-request-reopen timeout:1

    #-----------------------------------------------------------------
    # NODE_PACKAGE
    #-----------------------------------------------------------------
    node_repo_modules: local          # upstream repo to be added on node, local by default
    node_repo_remove: true            # remove existing repo on node?
    node_packages: [openssh-server]   # packages to be installed current nodes with latest version
    node_default_packages:            # default packages to be installed on all nodes
      - lz4,unzip,bzip2,pv,jq,git,ncdu,make,patch,bash,lsof,wget,uuid,tuned,nvme-cli,numactl,sysstat,iotop,htop,rsync,tcpdump
      - python3,python3-pip,socat,lrzsz,net-tools,ipvsadm,telnet,ca-certificates,openssl,keepalived,etcd,haproxy,chrony,pig
      - zlib1g,acl,dnsutils,libreadline-dev,vim-tiny,node-exporter,openssh-server,openssh-client,vector
    node_uv_env: /data/venv           # uv venv path, empty string to skip
    node_pip_packages: ''             # pip packages to install in uv venv

    #-----------------------------------------------------------------
    # NODE_SEC
    #-----------------------------------------------------------------
    node_selinux_mode: permissive     # set selinux mode: enforcing,permissive,disabled
    node_firewall_mode: zone          # firewall mode: zone (default), off (disable), none (skip & self-managed)
    node_firewall_intranet:           # which intranet cidr considered as internal network
      - 10.0.0.0/8
      - 192.168.0.0/16
      - 172.16.0.0/12
    node_firewall_public_port:        # expose these ports to public network in (zone, strict) mode
      - 22                            # enable ssh access
      - 80                            # enable http access
      - 443                           # enable https access
      - 5432                          # enable postgres access

    #-----------------------------------------------------------------
    # NODE_TUNE
    #-----------------------------------------------------------------
    node_disable_numa: false          # disable node numa, reboot required
    node_disable_swap: false          # disable node swap, use with caution
    node_static_network: true         # preserve dns resolver settings after reboot
    node_disk_prefetch: false         # setup disk prefetch on HDD to increase performance
    node_kernel_modules: [ softdog, ip_vs, ip_vs_rr, ip_vs_wrr, ip_vs_sh ]
    node_hugepage_count: 0            # number of 2MB hugepage, take precedence over ratio
    node_hugepage_ratio: 0            # node mem hugepage ratio, 0 disable it by default
    node_overcommit_ratio: 0          # node mem overcommit ratio, 0 disable it by default
    node_tune: oltp                   # node tuned profile: none,oltp,olap,crit,tiny
    node_sysctl_params:              # sysctl parameters in k:v format in addition to tuned
      fs.nr_open: 8388608

    #-----------------------------------------------------------------
    # NODE_ADMIN
    #-----------------------------------------------------------------
    node_data: /data                  # node main data directory, `/data` by default
    node_admin_enabled: true          # create a admin user on target node?
    node_admin_uid: 88                # uid and gid for node admin user
    node_admin_username: dba          # name of node admin user, `dba` by default
    node_admin_sudo: nopass           # admin sudo privilege, all,nopass. nopass by default
    node_admin_ssh_exchange: true     # exchange admin ssh key among node cluster
    node_admin_pk_current: true       # add current user's ssh pk to admin authorized_keys
    node_admin_pk_list: []            # ssh public keys to be added to admin user
    node_aliases: {}                  # extra shell aliases to be added, k:v dict

    #-----------------------------------------------------------------
    # NODE_TIME
    #-----------------------------------------------------------------
    node_timezone: ''                 # setup node timezone, empty string to skip
    node_ntp_enabled: true            # enable chronyd time sync service?
    node_ntp_servers:                 # ntp servers in `/etc/chrony.conf`
      - pool pool.ntp.org iburst
    node_crontab_overwrite: true      # overwrite or append to `/etc/crontab`?
    node_crontab: [ ]                 # crontab entries in `/etc/crontab`

    #-----------------------------------------------------------------
    # NODE_VIP
    #-----------------------------------------------------------------
    vip_enabled: false                # enable vip on this node cluster?
    # vip_address:         [IDENTITY] # node vip address in ipv4 format, required if vip is enabled
    # vip_vrid:            [IDENTITY] # required, integer, 1-254, should be unique among same VLAN
    vip_role: backup                  # optional, `master|backup`, backup by default, use as init role
    vip_preempt: false                # optional, `true/false`, false by default, enable vip preemption
    vip_interface: auto               # node vip network interface to listen, `auto` by default
    vip_dns_suffix: ''                # node vip dns name suffix, empty string by default
    vip_auth_pass: ''                 # empty to use '<cls>-<vrid>' as the default
    vip_exporter_port: 9650           # keepalived exporter listen port, 9650 by default

    #-----------------------------------------------------------------
    # HAPROXY
    #-----------------------------------------------------------------
    haproxy_enabled: true             # enable haproxy on this node?
    haproxy_clean: false              # cleanup all existing haproxy config?
    haproxy_reload: true              # reload haproxy after config?
    haproxy_auth_enabled: true        # enable authentication for haproxy admin page
    haproxy_admin_username: admin     # haproxy admin username, `admin` by default
    haproxy_admin_password: pigsty    # haproxy admin password, `pigsty` by default
    haproxy_exporter_port: 9101       # haproxy admin/exporter port, 9101 by default
    haproxy_client_timeout: 24h       # client side connection timeout, 24h by default
    haproxy_server_timeout: 24h       # server side connection timeout, 24h by default
    haproxy_services: []              # list of haproxy service to be exposed on node

    #-----------------------------------------------------------------
    # NODE_EXPORTER
    #-----------------------------------------------------------------
    node_exporter_enabled: true       # setup node_exporter on this node?
    node_exporter_port: 9100          # node exporter listen port, 9100 by default
    node_exporter_options: '--no-collector.softnet --no-collector.nvme --collector.tcpstat --collector.processes'

    #-----------------------------------------------------------------
    # VECTOR
    #-----------------------------------------------------------------
    vector_enabled: true              # enable vector log collector?
    vector_clean: false               # purge vector data dir during init?
    vector_data: /data/vector         # vector data dir, /data/vector by default
    vector_port: 9598                 # vector metrics port, 9598 by default
    vector_read_from: beginning       # vector read from beginning or end
    vector_log_endpoint: [ infra ]    # if defined, sending vector log to this endpoint.


    #================================================================#
    #                        VARS: DOCKER                            #
    #================================================================#
    docker_enabled: false             # enable docker on this node?
    docker_data: /data/docker         # docker data directory, /data/docker by default
    docker_storage_driver: overlay2   # docker storage driver, can be zfs, btrfs
    docker_cgroups_driver: systemd    # docker cgroup fs driver: cgroupfs,systemd
    docker_registry_mirrors: []       # docker registry mirror list
    docker_exporter_port: 9323        # docker metrics exporter port, 9323 by default
    docker_image: []                  # docker image to be pulled after bootstrap
    docker_image_cache: /tmp/docker/*.tgz # docker image cache glob pattern

    #================================================================#
    #                         VARS: ETCD                             #
    #================================================================#
    #etcd_seq: 1                      # etcd instance identifier, explicitly required
    etcd_cluster: etcd                # etcd cluster & group name, etcd by default
    etcd_safeguard: false             # prevent purging running etcd instance?
    etcd_data: /data/etcd             # etcd data directory, /data/etcd by default
    etcd_port: 2379                   # etcd client port, 2379 by default
    etcd_peer_port: 2380              # etcd peer port, 2380 by default
    etcd_init: new                    # etcd initial cluster state, new or existing
    etcd_election_timeout: 1000       # etcd election timeout, 1000ms by default
    etcd_heartbeat_interval: 100      # etcd heartbeat interval, 100ms by default
    etcd_root_password: Etcd.Root     # etcd root password for RBAC, change it!


    #================================================================#
    #                         VARS: MINIO                            #
    #================================================================#
    #minio_seq: 1                     # minio instance identifier, REQUIRED
    #minio_cluster:                   # minio cluster identifier, REQUIRED (define in cluster vars)
    minio_user: minio                 # minio os user, `minio` by default
    minio_https: true                 # use https for minio, true by default
    minio_node: '${minio_cluster}-${minio_seq}.pigsty' # minio node name pattern
    minio_data: '/data/minio'         # minio data dir(s), use {x...y} to specify multi drivers
    #minio_volumes:                   # minio data volumes, override defaults if specified
    minio_domain: sss.pigsty          # minio external domain name, `sss.pigsty` by default
    minio_port: 9000                  # minio service port, 9000 by default
    minio_admin_port: 9001            # minio console port, 9001 by default
    minio_access_key: minioadmin      # root access key, `minioadmin` by default
    minio_secret_key: S3User.MinIO    # root secret key, `S3User.MinIO` by default
    minio_extra_vars: ''              # extra environment variables
    minio_provision: true             # run minio provisioning tasks?
    minio_alias: sss                  # alias name for local minio deployment
    #minio_endpoint: https://sss.pigsty:9000 # if not specified, overwritten by defaults
    minio_buckets:                    # list of minio bucket to be created
      - { name: pgsql }
      - { name: meta ,versioning: true }
      - { name: data }
    minio_users:                      # list of minio user to be created
      - { access_key: pgbackrest  ,secret_key: S3User.Backup ,policy: pgsql }
      - { access_key: s3user_meta ,secret_key: S3User.Meta   ,policy: meta  }
      - { access_key: s3user_data ,secret_key: S3User.Data   ,policy: data  }
    minio_safeguard: false            # prevent purging running minio instance?
    minio_rm_data: true               # purging minio data and config?
    minio_rm_pkg: false               # uninstall minio packages?


    #================================================================#
    #                         VARS: REDIS                            #
    #================================================================#
    #redis_cluster:        <CLUSTER> # redis cluster name, required identity parameter
    #redis_node: 1            <NODE> # redis node sequence number, node int id required
    #redis_instances: {}      <NODE> # redis instances definition on this redis node
    redis_fs_main: /data/redis        # redis main data directory, `/data/redis` by default
    redis_exporter_enabled: true      # install redis exporter on redis nodes?
    redis_exporter_port: 9121         # redis exporter listen port, 9121 by default
    redis_exporter_options: ''        # cli args and extra options for redis exporter
    redis_type: redis                 # redis implementation: redis or valkey
    redis_mode: standalone            # redis mode: standalone,cluster,sentinel
    redis_conf: redis.conf            # redis config template path, except sentinel
    redis_bind_address: '0.0.0.0'     # redis bind address, empty string will use host ip
    redis_max_memory: 1GB             # max memory used by each redis instance
    redis_mem_policy: allkeys-lru     # redis memory eviction policy
    redis_password: ''                # redis password, empty string will disable password
    redis_rdb_save: ['1200 1']        # redis rdb save directives, disable with empty list
    redis_aof_enabled: false          # enable redis append only file?
    redis_rename_commands: {}         # rename redis dangerous commands
    redis_cluster_replicas: 1         # replica number for one master in redis cluster
    redis_sentinel_monitor: []        # sentinel master list, works on sentinel cluster only
    redis_safeguard: false            # prevent purging running redis instance?
    redis_rm_data: true               # remove redis data dir?
    redis_rm_pkg: false               # uninstall selected engine & redis-exporter packages?


    #================================================================#
    #                         VARS: PGSQL                            #
    #================================================================#

    #-----------------------------------------------------------------
    # PG_IDENTITY
    #-----------------------------------------------------------------
    pg_mode: pgsql          #CLUSTER  # pgsql cluster mode: pgsql,citus,mssql,mysql,ivory,pgtde,polar,gpsql,agens,oriole,pgedge
    # pg_cluster:           #CLUSTER  # pgsql cluster name, required identity parameter
    # pg_seq: 0             #INSTANCE # pgsql instance seq number, required identity parameter
    # pg_role: replica      #INSTANCE # pgsql role, required, could be primary,replica,offline
    # pg_instances: {}      #INSTANCE # define multiple pg instances on node in `{port:ins_vars}` format
    # pg_upstream:          #INSTANCE # repl upstream ip addr for standby cluster or cascade replica
    # pg_shard:             #CLUSTER  # pgsql shard name, optional identity for sharding clusters
    # pg_group: 0           #CLUSTER  # pgsql shard index number, optional identity for sharding clusters
    # gp_role: master       #CLUSTER  # greenplum role of this cluster, could be master or segment
    pg_offline_query: false #INSTANCE # set to true to enable offline queries on this instance

    #-----------------------------------------------------------------
    # PG_BUSINESS
    #-----------------------------------------------------------------
    # postgres business object definition, overwrite in group vars
    pg_users: []                      # postgres business users
    pg_databases: []                  # postgres business databases
    pg_services: []                   # postgres business services
    pg_hba_rules: []                  # business hba rules for postgres
    pgb_hba_rules: []                 # business hba rules for pgbouncer
    pg_crontab: []                    # postgres crontab entries for dbsu
    # global credentials, overwrite in global vars
    pg_dbsu_password: ''              # dbsu password, empty string means no dbsu password by default
    pg_replication_username: replicator
    pg_replication_password: DBUser.Replicator
    pg_admin_username: dbuser_dba
    pg_admin_password: DBUser.DBA
    pg_monitor_username: dbuser_monitor
    pg_monitor_password: DBUser.Monitor

    #-----------------------------------------------------------------
    # PG_INSTALL
    #-----------------------------------------------------------------
    pg_dbsu: postgres                 # os dbsu name, postgres by default, better not change it
    pg_dbsu_uid: 543                  # os dbsu uid and gid, 26 for default postgres users and groups
    pg_dbsu_sudo: limit               # dbsu sudo privilege, none,limit,all,nopass. limit by default
    pg_dbsu_home: /var/lib/pgsql      # postgresql home directory, `/var/lib/pgsql` by default
    pg_dbsu_ssh_exchange: true        # exchange postgres dbsu ssh key among same pgsql cluster
    pg_version: 18                    # postgres major version to be installed, 18 by default
    pg_bin_dir: /usr/pgsql/bin        # postgres binary dir, `/usr/pgsql/bin` by default
    pg_log_dir: /pg/log/postgres      # postgres log dir, `/pg/log/postgres` by default
    pg_packages:                      # pg packages to be installed, alias can be used
      - pgsql-main pgsql-common
    pg_extensions: []                 # pg extensions to be installed, alias can be used

    #-----------------------------------------------------------------
    # PG_BOOTSTRAP
    #-----------------------------------------------------------------
    pg_data: /pg/data                 # postgres data directory, `/pg/data` by default
    pg_fs_main: /data/postgres        # postgres main data directory, `/data/postgres` by default
    pg_fs_backup: /data/backups       # postgres backup data directory, `/data/backups` by default
    pg_storage_type: SSD              # storage type for pg main data, SSD,HDD, SSD by default
    pg_dummy_filesize: 64MiB          # size of `/pg/dummy`, hold 64MB disk space for emergency use
    pg_listen: '0.0.0.0'              # postgres/pgbouncer listen addresses, comma separated list
    pg_port: 5432                     # postgres listen port, 5432 by default
    pg_localhost: /var/run/postgresql # postgres unix socket dir for localhost connection
    patroni_enabled: true             # if disabled, no postgres cluster will be created during init
    patroni_mode: default             # patroni working mode: default,pause,remove
    pg_namespace: /pg                 # top level key namespace in etcd, used by patroni & vip
    patroni_port: 8008                # patroni listen port, 8008 by default
    patroni_log_dir: /pg/log/patroni  # patroni log dir, `/pg/log/patroni` by default
    patroni_ssl_enabled: false        # secure patroni RestAPI communications with SSL?
    patroni_watchdog_mode: 'off'      # patroni watchdog mode: automatic,required,off. off by default
    patroni_username: postgres        # patroni restapi username, `postgres` by default
    patroni_password: Patroni.API     # patroni restapi password, `Patroni.API` by default
    pg_etcd_password: ''              # etcd password for this pg cluster, '' to use pg_cluster
    pg_primary_db: postgres           # primary database name, used by citus,etc... ,postgres by default
    pg_parameters: {}                 # extra parameters in postgresql.auto.conf
    pg_files: []                      # extra files to be copied to postgres data directory (e.g. license)
    pg_conf: oltp.yml                 # config template: oltp,olap,crit,tiny. `oltp.yml` by default
    pg_max_conn: auto                 # postgres max connections, `auto` will use recommended value
    pg_shared_buffer_ratio: 0.25      # postgres shared buffers ratio, 0.25 by default, 0.1~0.4
    pg_io_method: worker              # io method for postgres, auto,fsync,worker,io_uring, worker by default
    pg_rto: norm                      # shared rto mode for patroni & haproxy: fast,norm,safe,wide
    pg_rto_plan:  # [ttl, loop, retry, start, margin, inter, fastinter, downinter, rise, fall]
      fast: [ 20  ,5  ,5  ,15 ,5  ,'1s' ,'0.5s' ,'1s' ,3 ,3 ]
      norm: [ 30  ,5  ,10 ,25 ,5  ,'2s' ,'1s'   ,'2s' ,3 ,3 ]
      safe: [ 60  ,10 ,20 ,45 ,10 ,'3s' ,'1.5s' ,'3s' ,3 ,3 ]
      wide: [ 120 ,20 ,30 ,95 ,15 ,'4s' ,'2s'   ,'4s' ,3 ,3 ]
    pg_rpo: 1048576                   # recovery point objective in bytes, `1MiB` at most by default
    pg_libs: 'pg_stat_statements, auto_explain'  # preloaded libraries, `pg_stat_statements,auto_explain` by default
    pg_delay: 0                       # replication apply delay for standby cluster leader
    pg_checksum: true                 # enable data checksum for postgres cluster?
    pg_pwd_enc: scram-sha-256         # password encryption algorithm
    pg_encoding: UTF8                 # database cluster encoding, `UTF8` by default
    pg_locale: C                      # database cluster local, `C` by default
    pg_lc_collate: C                  # database cluster collate, `C` by default
    pg_lc_ctype: C                    # database character type, `C` by default
    #pgsodium_key: ""                 # pgsodium key, 64 hex digit, default to sha256(pg_cluster)
    #pgsodium_getkey_script: ""       # pgsodium getkey script path, pgsodium_getkey by default

    #-----------------------------------------------------------------
    # PG_PROVISION
    #-----------------------------------------------------------------
    pg_provision: true                # provision postgres cluster after bootstrap
    pg_init: pg-init                  # provision init script for cluster template, `pg-init` by default
    pg_default_roles:                 # default roles and users in postgres cluster
      - { name: dbrole_readonly  ,login: false ,comment: role for global read-only access     }
      - { name: dbrole_offline   ,login: false ,comment: role for restricted read-only access }
      - { name: dbrole_readwrite ,login: false ,roles: [dbrole_readonly] ,comment: role for global read-write access }
      - { name: dbrole_admin     ,login: false ,roles: [pg_monitor, dbrole_readwrite] ,comment: role for object creation }
      - { name: postgres     ,superuser: true  ,comment: system superuser }
      - { name: replicator ,replication: true  ,roles: [pg_monitor, dbrole_readonly] ,comment: system replicator }
      - { name: dbuser_dba   ,superuser: true  ,roles: [dbrole_admin]  ,pgbouncer: true ,pool_mode: session, pool_connlimit: 16 ,comment: pgsql admin user }
      - { name: dbuser_monitor ,roles: [pg_monitor, dbrole_readonly] ,pgbouncer: true ,parameters: {log_min_duration_statement: 1000 } ,pool_mode: session ,pool_connlimit: 8 ,comment: pgsql monitor user }
    pg_default_privileges:            # default privileges when created by admin user
      - GRANT USAGE      ON SCHEMAS    TO  dbrole_readonly
      - GRANT SELECT     ON TABLES     TO  dbrole_readonly
      - GRANT SELECT     ON SEQUENCES  TO  dbrole_readonly
      - GRANT EXECUTE    ON FUNCTIONS  TO  dbrole_readonly
      - GRANT USAGE      ON SCHEMAS    TO  dbrole_offline
      - GRANT SELECT     ON TABLES     TO  dbrole_offline
      - GRANT SELECT     ON SEQUENCES  TO  dbrole_offline
      - GRANT EXECUTE    ON FUNCTIONS  TO  dbrole_offline
      - GRANT INSERT     ON TABLES     TO  dbrole_readwrite
      - GRANT UPDATE     ON TABLES     TO  dbrole_readwrite
      - GRANT DELETE     ON TABLES     TO  dbrole_readwrite
      - GRANT USAGE      ON SEQUENCES  TO  dbrole_readwrite
      - GRANT UPDATE     ON SEQUENCES  TO  dbrole_readwrite
      - GRANT TRUNCATE   ON TABLES     TO  dbrole_admin
      - GRANT REFERENCES ON TABLES     TO  dbrole_admin
      - GRANT TRIGGER    ON TABLES     TO  dbrole_admin
      - GRANT CREATE     ON SCHEMAS    TO  dbrole_admin
    pg_default_schemas: [ monitor ]   # default schemas to be created
    pg_default_extensions:            # default extensions to be created
      - { name: pg_stat_statements ,schema: monitor }
      - { name: pgstattuple        ,schema: monitor }
      - { name: pg_buffercache     ,schema: monitor }
      - { name: pageinspect        ,schema: monitor }
      - { name: pg_prewarm         ,schema: monitor }
      - { name: pg_visibility      ,schema: monitor }
      - { name: pg_freespacemap    ,schema: monitor }
      - { name: postgres_fdw       ,schema: public  }
      - { name: file_fdw           ,schema: public  }
      - { name: btree_gist         ,schema: public  }
      - { name: btree_gin          ,schema: public  }
      - { name: pg_trgm            ,schema: public  }
      - { name: intagg             ,schema: public  }
      - { name: intarray           ,schema: public  }
      - { name: pg_repack }
    pg_reload: true                   # reload postgres after hba changes
    pg_default_hba_rules:             # postgres default host-based authentication rules, order by `order`
      - {user: '${dbsu}'    ,db: all         ,addr: local     ,auth: ident ,title: 'dbsu access via local os user ident'  ,order: 100}
      - {user: '${dbsu}'    ,db: replication ,addr: local     ,auth: ident ,title: 'dbsu replication from local os ident' ,order: 150}
      - {user: '${repl}'    ,db: replication ,addr: localhost ,auth: pwd   ,title: 'replicator replication from localhost',order: 200}
      - {user: '${repl}'    ,db: replication ,addr: intra     ,auth: pwd   ,title: 'replicator replication from intranet' ,order: 250}
      - {user: '${repl}'    ,db: postgres    ,addr: intra     ,auth: pwd   ,title: 'replicator postgres db from intranet' ,order: 300}
      - {user: '${monitor}' ,db: all         ,addr: localhost ,auth: pwd   ,title: 'monitor from localhost with password' ,order: 350}
      - {user: '${monitor}' ,db: all         ,addr: infra     ,auth: pwd   ,title: 'monitor from infra host with password',order: 400}
      - {user: '${admin}'   ,db: all         ,addr: intra     ,auth: pwd   ,title: 'admin @ intranet nodes with pwd'      ,order: 450}
      - {user: '${admin}'   ,db: all         ,addr: world     ,auth: ssl   ,title: 'admin @ everywhere with ssl & pwd'    ,order: 500}
      - {user: '+dbrole_readonly',db: all    ,addr: localhost ,auth: pwd   ,title: 'pgbouncer read/write via local socket',order: 550}
      - {user: '+dbrole_readonly',db: all    ,addr: intra     ,auth: pwd   ,title: 'read/write biz user via password'     ,order: 600}
      - {user: '+dbrole_offline' ,db: all    ,addr: intra     ,auth: pwd   ,title: 'allow etl offline tasks from intranet',order: 650}
    pgb_default_hba_rules:            # pgbouncer default host-based authentication rules, order by `order`
      - {user: '${dbsu}'    ,db: pgbouncer   ,addr: local     ,auth: peer  ,title: 'dbsu local admin access with os ident',order: 100}
      - {user: 'all'        ,db: all         ,addr: localhost ,auth: pwd   ,title: 'allow all user local access with pwd' ,order: 150}
      - {user: '${monitor}' ,db: pgbouncer   ,addr: intra     ,auth: pwd   ,title: 'monitor access via intranet with pwd' ,order: 200}
      - {user: '${monitor}' ,db: all         ,addr: world     ,auth: deny  ,title: 'reject all other monitor access addr' ,order: 250}
      - {user: '${admin}'   ,db: all         ,addr: intra     ,auth: pwd   ,title: 'admin access via intranet with pwd'   ,order: 300}
      - {user: '${admin}'   ,db: all         ,addr: world     ,auth: deny  ,title: 'reject all other admin access addr'   ,order: 350}
      - {user: 'all'        ,db: all         ,addr: intra     ,auth: pwd   ,title: 'allow all user intra access with pwd' ,order: 400}

    #-----------------------------------------------------------------
    # PG_BACKUP
    #-----------------------------------------------------------------
    pgbackrest_enabled: true          # enable pgbackrest on pgsql host?
    pgbackrest_log_dir: /pg/log/pgbackrest # pgbackrest log dir, `/pg/log/pgbackrest` by default
    pgbackrest_method: local          # pgbackrest repo method: local,minio,[user-defined...]
    pgbackrest_init_backup: true      # take a full backup after pgbackrest is initialized?
    pgbackrest_repo:                  # pgbackrest repo: https://pgbackrest.org/configuration.html#section-repository
      local:                          # default pgbackrest repo with local posix fs
        path: /pg/backup              # local backup directory, `/pg/backup` by default
        retention_full_type: count    # retention full backups by count
        retention_full: 2             # keep 2, at most 3 full backups when using local fs repo
      minio:                          # optional minio repo for pgbackrest
        type: s3                      # minio is s3-compatible, so s3 is used
        s3_endpoint: sss.pigsty       # minio endpoint domain name, `sss.pigsty` by default
        s3_region: us-east-1          # minio region, us-east-1 by default, useless for minio
        s3_bucket: pgsql              # minio bucket name, `pgsql` by default
        s3_key: pgbackrest            # minio user access key for pgbackrest
        s3_key_secret: S3User.Backup  # minio user secret key for pgbackrest
        s3_uri_style: path            # use path style uri for minio rather than host style
        path: /pgbackrest             # minio backup path, default is `/pgbackrest`
        storage_port: 9000            # minio port, 9000 by default
        storage_ca_file: /etc/pki/ca.crt  # minio ca file path, `/etc/pki/ca.crt` by default
        block: y                      # Enable block incremental backup
        bundle: y                     # bundle small files into a single file
        bundle_limit: 20MiB           # Limit for file bundles, 20MiB for object storage
        bundle_size: 128MiB           # Target size for file bundles, 128MiB for object storage
        cipher_type: aes-256-cbc      # enable AES encryption for remote backup repo
        cipher_pass: pgBackRest       # AES encryption password, default is 'pgBackRest'
        retention_full_type: time     # retention full backup by time on minio repo
        retention_full: 14            # keep full backup for the the last 14 days

    #-----------------------------------------------------------------
    # PG_ACCESS
    #-----------------------------------------------------------------
    pgbouncer_enabled: true           # if disabled, pgbouncer will not be launched on pgsql host
    pgbouncer_port: 6432              # pgbouncer listen port, 6432 by default
    pgbouncer_log_dir: /pg/log/pgbouncer  # pgbouncer log dir, `/pg/log/pgbouncer` by default
    pgbouncer_auth_query: false       # query postgres to retrieve unlisted business users?
    pgbouncer_poolmode: transaction   # pooling mode: transaction,session,statement, transaction by default
    pgbouncer_sslmode: disable        # pgbouncer client ssl mode, disable by default
    pgbouncer_ignore_param: [ extra_float_digits, application_name, TimeZone, DateStyle, IntervalStyle, search_path ]
    pg_weight: 100          #INSTANCE # relative load balance weight in service, 100 by default, 0-255
    pg_service_provider: ''           # dedicate haproxy node group name, or empty string for local nodes by default
    pg_default_service_dest: pgbouncer # default service destination if svc.dest='default'
    pg_default_services:              # postgres default service definitions
      - { name: primary ,port: 5433 ,dest: default  ,check: /primary   ,selector: "[]" }
      - { name: replica ,port: 5434 ,dest: default  ,check: /read-only ,selector: "[]" , backup: "[? pg_role == `primary` || pg_role == `offline` ]" }
      - { name: default ,port: 5436 ,dest: postgres ,check: /primary   ,selector: "[]" }
      - { name: offline ,port: 5438 ,dest: postgres ,check: /replica   ,selector: "[? pg_role == `offline` || pg_offline_query ]" , backup: "[? pg_role == `replica` && !pg_offline_query]"}
    pg_vip_enabled: false             # enable a l2 vip for pgsql primary? false by default
    pg_vip_address: 127.0.0.1/24      # vip address in `<ipv4>/<mask>` format, require if vip is enabled
    pg_vip_interface: auto            # vip network interface to listen, auto by default
    pg_dns_suffix: ''                 # pgsql dns suffix, '' by default
    pg_dns_target: auto               # auto, primary, vip, none, or ad hoc ip

    #-----------------------------------------------------------------
    # PG_MONITOR
    #-----------------------------------------------------------------
    pg_exporter_enabled: true              # enable pg_exporter on pgsql hosts?
    pg_exporter_config: pg_exporter.yml    # pg_exporter configuration file name
    pg_exporter_cache_ttls: '1,10,60,300'  # pg_exporter collector ttl stage in seconds, '1,10,60,300' by default
    pg_exporter_port: 9630                 # pg_exporter listen port, 9630 by default
    pg_exporter_params: 'sslmode=disable'  # extra url parameters for pg_exporter dsn
    pg_exporter_url: ''                    # overwrite auto-generate pg dsn if specified
    pg_exporter_auto_discovery: true       # enable auto database discovery? enabled by default
    pg_exporter_exclude_database: 'template0,template1,postgres' # csv of database that WILL NOT be monitored during auto-discovery
    pg_exporter_include_database: ''       # csv of database that WILL BE monitored during auto-discovery
    pg_exporter_connect_timeout: 200       # pg_exporter connect timeout in ms, 200 by default
    pg_exporter_options: ''                # overwrite extra options for pg_exporter
    pgbouncer_exporter_enabled: true       # enable pgbouncer_exporter on pgsql hosts?
    pgbouncer_exporter_port: 9631          # pgbouncer_exporter listen port, 9631 by default
    pgbouncer_exporter_url: ''             # overwrite auto-generate pgbouncer dsn if specified
    pgbouncer_exporter_options: ''         # overwrite extra options for pgbouncer_exporter
    pgbackrest_exporter_enabled: true      # enable pgbackrest_exporter on pgsql hosts?
    pgbackrest_exporter_port: 9854         # pgbackrest_exporter listen port, 9854 by default
    pgbackrest_exporter_options: >-
      --collect.interval=120
      --log.level=info

    #-----------------------------------------------------------------
    # PG_REMOVE
    #-----------------------------------------------------------------
    pg_safeguard: false               # stop pg_remove running if pg_safeguard is enabled, false by default
    pg_rm_data: true                  # remove postgres data during remove? true by default
    pg_rm_backup: true                # remove pgbackrest backup during primary remove? true by default
    pg_rm_pkg: true                   # uninstall postgres packages during remove? true by default

...

Explanation

The demo/debian template is optimized for Debian and Ubuntu distributions.

Supported Distributions:

  • Debian 12 (Bookworm)
  • Debian 13 (Trixie)
  • Ubuntu 22.04 LTS (Jammy)
  • Ubuntu 24.04 LTS (Noble)
  • Ubuntu 26.04 LTS (Resolute)

Key Features:

  • Uses PGDG APT repositories
  • Optimized for APT package manager
  • Supports Debian/Ubuntu-specific package names

Use Cases:

  • Cloud servers (Ubuntu widely used)
  • Container environments (Debian commonly used as base image)
  • Development and testing environments

6.29 - demo/demo

Pigsty public demo site configuration, showcasing SSL certificates, domain exposure, and full extension installation

The demo/demo configuration template is used by Pigsty’s public demo site, demonstrating how to expose services publicly, configure SSL certificates, and install all available extensions.

If you want to set up your own public service on a cloud server, you can use this template as a reference.

Overview

  • Config Name: demo/demo
  • Node Count: Single node
  • Description: Pigsty public demo site configuration
  • OS Distro: el8, el9, el10, d12, d13, u22, u24, u26
  • OS Arch: x86_64
  • Related: meta, rich

Usage:

./configure -c demo/demo [-i <primary_ip>]

Key Features

This template enhances the meta template with:

  • SSL certificate and custom domain configuration (e.g., pigsty.cc)
  • Downloads and installs all available PostgreSQL 18 extensions
  • Enables Docker with image acceleration
  • Deploys Silo object storage
  • Pre-configures multiple business databases and users
  • Adds Redis primary-replica instance examples
  • Adds Kafka sample cluster

Content

Source: pigsty/conf/demo/demo.yml

---
#==============================================================#
# File      :   demo.yml
# Desc      :   Pigsty Public Demo Configuration
# Ctime     :   2020-05-22
# Mtime     :   2025-12-12
# Docs      :   https://pigsty.io/docs/conf/demo
# License   :   Apache-2.0 @ https://pigsty.io/docs/about/license/
# Copyright :   2018-2026  Ruohang Feng / Vonng ([email protected])
#==============================================================#


all:
  children:

    # infra cluster for proxy, monitor, alert, etc..
    infra:
      hosts: { 10.10.10.10: { infra_seq: 1 } }
      vars:
        nodename: pigsty.cc       # overwrite the default hostname
        node_id_from_pg: false    # do not use the pg identity as hostname
        docker_enabled: true      # enable docker on this node
        docker_registry_mirrors: ["https://mirror.ccs.tencentyun.com", "https://docker.1ms.run"]
        # ./pgsql-monitor.yml -l infra     # monitor 'external' PostgreSQL instance
        pg_exporters:             # treat local postgres as RDS for demonstration purpose
          20001: { pg_cluster: pg-foo, pg_seq: 1, pg_host: 10.10.10.10 }
          #20002: { pg_cluster: pg-bar, pg_seq: 1, pg_host: 10.10.10.11 , pg_port: 5432 }
          #20003: { pg_cluster: pg-bar, pg_seq: 2, pg_host: 10.10.10.12 , pg_exporter_url: 'postgres://dbuser_monitor:[email protected]:5432/postgres?sslmode=disable' }
          #20004: { pg_cluster: pg-bar, pg_seq: 3, pg_host: 10.10.10.13 , pg_monitor_username: dbuser_monitor, pg_monitor_password: DBUser.Monitor }

    # etcd cluster for ha postgres
    etcd: { hosts: { 10.10.10.10: { etcd_seq: 1 } }, vars: { etcd_cluster: etcd } }

    # minio cluster, s3 compatible object storage
    minio: { hosts: { 10.10.10.10: { minio_seq: 1 } }, vars: { minio_cluster: minio } }

    # postgres example cluster: pg-meta
    pg-meta:
      hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
      vars:
        pg_cluster: pg-meta
        pg_users:
          - {name: dbuser_meta       ,password: DBUser.Meta       ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: pigsty admin user }
          - {name: dbuser_view       ,password: DBUser.Viewer     ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer for meta database }
          - {name: dbuser_grafana    ,password: DBUser.Grafana    ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for grafana database    }
          - {name: dbuser_bytebase   ,password: DBUser.Bytebase   ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for bytebase database   }
          - {name: dbuser_kong       ,password: DBUser.Kong       ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for kong api gateway    }
          - {name: dbuser_gitea      ,password: DBUser.Gitea      ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for gitea service       }
          - {name: dbuser_wiki       ,password: DBUser.Wiki       ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for wiki.js service     }
          - {name: dbuser_noco       ,password: DBUser.Noco       ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for nocodb service      }
          - {name: dbuser_odoo       ,password: DBUser.Odoo       ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for odoo service ,createdb: true } #,superuser: true}
          - {name: dbuser_mattermost ,password: DBUser.MatterMost ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for mattermost ,createdb: true }
        pg_databases:
          - {name: meta ,baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty] ,extensions: [{name: vector},{name: postgis},{name: timescaledb}]}
          - {name: grafana  ,owner: dbuser_grafana  ,revokeconn: true ,comment: grafana primary database  }
          - {name: bytebase ,owner: dbuser_bytebase ,revokeconn: true ,comment: bytebase primary database }
          - {name: kong     ,owner: dbuser_kong     ,revokeconn: true ,comment: kong api gateway database }
          - {name: gitea    ,owner: dbuser_gitea    ,revokeconn: true ,comment: gitea meta database }
          - {name: wiki     ,owner: dbuser_wiki     ,revokeconn: true ,comment: wiki meta database  }
          - {name: noco     ,owner: dbuser_noco     ,revokeconn: true ,comment: nocodb database     }
          #- {name: odoo     ,owner: dbuser_odoo     ,revokeconn: true ,comment: odoo main database  }
          - {name: mattermost ,owner: dbuser_mattermost ,revokeconn: true ,comment: mattermost main database }
        pg_hba_rules:
          - {user: dbuser_view , db: all ,addr: infra ,auth: pwd ,title: 'allow grafana dashboard access cmdb from infra nodes'}
        pg_libs: 'timescaledb,pg_stat_statements, auto_explain'  # add timescaledb to shared_preload_libraries
        pg_extensions: # extensions to be installed on this cluster
          - timescaledb timescaledb_toolkit pg_timeseries periods temporal_tables emaj table_version pg_cron pg_task pg_later pg_background
          - postgis pgrouting pointcloud pg_h3 q3c ogr_fdw geoip pg_polyline pg_geohash #mobilitydb
          - pgvector vchord pgvectorscale pg_vectorize pg_similarity smlar pg_summarize pg_tiktoken pg4ml #pgml
          - pg_search pgroonga pg_bigm zhparser pg_bestmatch vchord_bm25 hunspell
          - citus pg_duckdb pg_mooncake duckdb_fdw pg_parquet pg_fkpart pg_partman plproxy #pg_strom #hydra
          - age hll rum pg_graphql pg_jsonschema jsquery pg_hint_plan hypopg index_advisor pg_plan_filter imgsmlr pg_ivm pg_incremental pgmq pgq pg_cardano omnigres #rdkit
          - pg_tle plv8 pllua plprql pldebugger plpgsql_check plprofiler plsh pljava #plr #pgtap #faker #dbt2
          - pg_prefix pg_semver pgunit pgpdf pglite_fusion md5hash asn1oid pg_roaringbitmap pgfaceting pgsphere pg_country pg_xenophile pg_currency pgcollection pgmp numeral pg_rational pguint pg_uint128 hashtypes ip4r pg_uri pg_emailaddr pg_acl timestamp9 chkpass #pg_duration #debversion #pg_rrule
          - pg_gzip pg_bzip pg_zstd pg_http pg_net pg_curl pgjq pgjwt pg_smtp_client pg_html5_email_address url_encode pgsql_tweaks pg_extra_time pgpcre icu_ext pgqr pg_protobuf pg_envvar floatfile pg_readme ddl_historization data_historization pg_schedoc pg_hashlib pg_xxhash shacrypt cryptint pg_ecdsa pgsparql
          - pg_idkit pg_uuidv7 permuteseq pg_hashids sequential_uuids topn quantile lower_quantile count_distinct omnisketch ddsketch vasco pgxicor tdigest first_last_agg extra_window_functions floatvec aggs_for_vecs aggs_for_arrays pg_arraymath pg_math pg_random pg_base36 pg_base62 pg_base58 pg_financial
          - pg_repack pg_squeeze pg_dirtyread pgfincore pg_cooldown pg_ddlx pg_prioritize pg_checksums pg_readonly pg_upless pg_permissions pgautofailover pg_catcheck preprepare pgcozy pg_orphaned pg_crash pg_cheat_funcs pg_fio pg_savior safeupdate pg_drop_events table_log #pgagent #pgpool
          - pg_profile pg_tracing pg_show_plans pg_stat_kcache pg_stat_monitor pg_qualstats pg_store_plans pg_track_settings pg_wait_sampling system_stats pg_meta pgnodemx pg_sqlog bgw_replstatus pgmeminfo toastinfo pg_explain_ui pg_relusage pagevis powa
          - passwordcheck_cracklib supautils pgsodium pg_vault pg_session_jwt pg_anon pgsmcrypto pgaudit pgauditlogtofile pg_auth_mon credcheck pgcryptokey pg_jobmon logerrors login_hook set_user pg_snakeoil pgextwlist pg_auditor sslutils pg_noset #pg_tde
          - wrappers multicorn odbc_fdw jdbc_fdw mysql_fdw tds_fdw sqlite_fdw pgbouncer_fdw mongo_fdw redis_fdw pg_redis_pubsub kafka_fdw hdfs_fdw firebird_fdw aws_s3 log_fdw #oracle_fdw #db2_fdw
          - documentdb orafce pgtt session_variable pg_statement_rollback pg_dbms_metadata pg_dbms_lock pgmemcache #pg_dbms_job
          - pglogical pglogical_ticker pgl_ddl_deploy pg_failover_slots db_migrator wal2json wal2mongo decoderbufs decoder_raw mimeo pg_fact_loader pg_bulkload #repmgr

    redis-ms: # redis classic primary & replica
      hosts: { 10.10.10.10: { redis_node: 1 , redis_instances: { 6379: { }, 6380: { replica_of: '10.10.10.10 6379' }, 6381: { replica_of: '10.10.10.10 6379' } } } }
      vars: { redis_cluster: redis-ms ,redis_password: 'redis.ms' ,redis_max_memory: 64MB }

    # Kafka 4.x dynamic KRaft: combined broker/controller on the demo node
    kf-main:
      hosts: { 10.10.10.10: { kafka_seq: 1 } }
      vars:
        kafka_cluster: kf-main


  vars:                               # global variables
    version: v4.5.0                   # pigsty version string
    admin_ip: 10.10.10.10             # admin node ip address
    region: china                     # upstream mirror region: default|china|europe

    infra_portal:                     # infra services exposed via portal
      home         : { domain: i.pigsty }     # default domain name
      cc           : { domain: pigsty.cc      ,path:     "/www/pigsty.cc"   ,cert: /etc/cert/pigsty.cc.crt ,key: /etc/cert/pigsty.cc.key }
      minio        : { domain: m.pigsty.cc    ,endpoint: "${admin_ip}:9001" ,scheme: https ,websocket: true }
      postgrest    : { domain: api.pigsty.cc  ,endpoint: "127.0.0.1:8884"   }
      pgadmin      : { domain: adm.pigsty.cc  ,endpoint: "127.0.0.1:8885"   }
      pgweb        : { domain: cli.pigsty.cc  ,endpoint: "127.0.0.1:8886"   }
      bytebase     : { domain: ddl.pigsty.cc  ,endpoint: "127.0.0.1:8887"   }
      jupyter      : { domain: lab.pigsty.cc  ,endpoint: "127.0.0.1:8888", websocket: true }
      gitea        : { domain: git.pigsty.cc  ,endpoint: "127.0.0.1:8889" }
      wiki         : { domain: wiki.pigsty.cc ,endpoint: "127.0.0.1:9002" }
      noco         : { domain: noco.pigsty.cc ,endpoint: "127.0.0.1:9003" }
      supa         : { domain: supa.pigsty.cc ,endpoint: "10.10.10.10:8000" ,websocket: true }
      dify         : { domain: dify.pigsty.cc ,endpoint: "10.10.10.10:8001" ,websocket: true }
      odoo         : { domain: odoo.pigsty.cc ,endpoint: "127.0.0.1:8069"   ,websocket: true }
      mm           : { domain: mm.pigsty.cc   ,endpoint: "10.10.10.10:8065" ,websocket: true }
    # scp -r ~/pgsty/cc/cert/*       pj:/etc/cert/       # copy https certs
    # scp -r ~/dev/pigsty.cc/public  pj:/www/pigsty.cc   # copy pigsty.cc website


    node_etc_hosts: [ "${admin_ip} i.pigsty sss.pigsty" ]
    node_timezone: Asia/Hong_Kong
    node_ntp_servers:
      - pool cn.pool.ntp.org iburst
      - pool ${admin_ip} iburst       # assume non-admin nodes does not have internet access
    pgbackrest_enabled: false         # do not take backups since this is disposable demo env
    # keep 3GiB metrics data at most on demo env
    vmetrics_options: >-
      -retentionPeriod=15d
      -retention.maxDiskSpaceUsageBytes=3GiB

    # install all postgresql18 extensions
    pg_version: 18                    # default postgres version
    repo_extra_packages: [ pg18-core ,pg18-time ,pg18-gis ,pg18-rag ,pg18-fts ,pg18-olap ,pg18-feat ,pg18-lang ,pg18-type ,pg18-util ,pg18-func ,pg18-admin ,pg18-stat ,pg18-sec ,pg18-fdw ,pg18-sim ,pg18-etl ,kafka-stack ,java-runtime]
    pg_extensions: [pg18-time ,pg18-gis ,pg18-rag ,pg18-fts ,pg18-feat ,pg18-lang ,pg18-type ,pg18-util ,pg18-func ,pg18-admin ,pg18-stat ,pg18-sec ,pg18-fdw ,pg18-sim ,pg18-etl ] #,pg18-olap]

    #----------------------------------------------#
    # PASSWORD : https://pigsty.io/docs/setup/security/
    #----------------------------------------------#
    grafana_admin_password: pigsty
    grafana_view_password: DBUser.Viewer
    pg_admin_password: DBUser.DBA
    pg_monitor_password: DBUser.Monitor
    pg_replication_password: DBUser.Replicator
    patroni_password: Patroni.API
    haproxy_admin_password: pigsty
    minio_secret_key: S3User.MinIO
    etcd_root_password: Etcd.Root
...

Explanation

The demo/demo template is Pigsty’s public demo configuration, showcasing a complete production-grade deployment example.

Key Features:

  • HTTPS certificate and custom domain configuration
  • All available PostgreSQL extensions installed
  • Integration with Redis, Kafka, and other components
  • Docker image acceleration configured

Use Cases:

  • Setting up public demo sites
  • Scenarios requiring complete feature demonstration
  • Learning Pigsty advanced configuration

Notes:

  • SSL certificate files must be prepared
  • DNS resolution must be configured
  • Some extensions are not available on ARM64 architecture

6.30 - demo/kernel

Ten-node PostgreSQL kernel matrix demo configuration

The demo/kernel configuration template demonstrates the major PostgreSQL kernels and compatible branches supported by Pigsty in a single configuration. It is intended for feature validation and kernel difference testing, not production use.

Overview

  • Config Name: demo/kernel
  • Node Count: 10 nodes, with one node also hosting INFRA/ETCD and pg-citus
  • Description: PostgreSQL kernel matrix demo covering Citus, IvorySQL, Babelfish, PolarDB, Percona TDE, OrioleDB, OpenHalo, DocumentDB, AgensGraph, and pgEdge
  • OS Distro: depends on actual package support for each kernel
  • OS Arch: depends on actual package support for each kernel
  • Related: pgsql, mssql, mongo

Usage:

./configure -c demo/kernel
Note

This is a fixed-IP demo template. Adjust node addresses for your actual environment after generation.

Content

Source: pigsty/conf/demo/kernel.yml

---
#==============================================================#
# File      :   kernel.yml
# Desc      :   Pigsty 10-node kernel matrix demo
# Ctime     :   2025-03-25
# Mtime     :   2026-07-23
# Docs      :   https://pigsty.io/docs/conf
# License   :   Apache-2.0 @ https://pigsty.io/docs/about/license/
# Copyright :   2018-2026  Ruohang Feng / Vonng ([email protected])
#==============================================================#


all:
  children:
    infra: { hosts: { 10.10.10.10: { infra_seq: 1 } }, vars: { repo_enabled: false } }
    etcd:  { hosts: { 10.10.10.10: { etcd_seq: 1 } }, vars: { etcd_cluster: etcd } }

    # 1. Vanilla PostgreSQL + Citus in one kernel template
    pg-citus:
      hosts:
        10.10.10.10: { pg_seq: 1, pg_role: primary }
      vars:
        pg_cluster: pg-citus
        pg_version: 18
        pg_packages: [ pgsql-main, pgsql-common, citus ]
        pg_users:
          - { name: dbuser_meta ,password: DBUser.Meta   ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: pigsty admin user }
          - { name: dbuser_view ,password: DBUser.Viewer ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer  }
        pg_databases:
          - { name: meta ,baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty] ,extensions: [citus, postgis, vector] }
        pg_extensions: [ citus, postgis, timescaledb, pgvector ]
        pg_libs: 'citus, pg_stat_statements, auto_explain'

    # 2. IvorySQL kernel
    pg-ivory:
      hosts:
        10.10.10.11: { pg_seq: 1, pg_role: primary }
      vars:
        pg_mode: ivory
        pg_cluster: pg-ivory
        pg_version: 18
        pg_packages: [ ivorysql, pgsql-common ]
        pg_libs: 'liboracle_parser, pg_stat_statements, auto_explain'
        pg_users:
          - { name: dbuser_meta ,password: DBUser.Meta   ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: pigsty admin user }
          - { name: dbuser_view ,password: DBUser.Viewer ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer  }
        pg_databases:
          - { name: meta ,baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty] }

    # 3. Babelfish (MSSQL compatible) kernel
    pg-mssql:
      hosts:
        10.10.10.12: { pg_seq: 1, pg_role: primary }
      vars:
        pg_mode: mssql
        pg_cluster: pg-mssql
        pg_version: 17
        pg_packages: [ babelfish, pgsql-common, sqlcmd ]
        pg_users:
          - { name: dbuser_mssql ,password: DBUser.MSSQL ,superuser: true ,pgbouncer: true ,roles: [dbrole_admin] ,comment: superuser & owner for babelfish }
        pg_databases:
          - name: mssql
            baseline: mssql.sql
            extensions: [ uuid-ossp, babelfishpg_common, babelfishpg_tsql, babelfishpg_tds, babelfishpg_money ]
            owner: dbuser_mssql
            parameters: { 'babelfishpg_tsql.migration_mode' : 'multi-db' }
            comment: babelfish cluster, a MSSQL compatible pg cluster
        pg_libs: 'babelfishpg_tds, pg_stat_statements, auto_explain'
        pg_hba_rules:
          - { user: dbuser_mssql ,db: mssql ,addr: intra ,auth: md5 ,title: 'allow mssql dbsu intranet access'      ,order: 525 }
          - { user: all          ,db: all   ,addr: intra ,auth: md5 ,title: 'everyone intranet access with md5 pwd' ,order: 800 }
        pg_default_services:
          - { name: primary ,port: 5433 ,dest: 1433     ,check: /primary   ,selector: "[]" }
          - { name: replica ,port: 5434 ,dest: 1433     ,check: /read-only ,selector: "[]" ,backup: "[? pg_role == `primary` || pg_role == `offline` ]" }
          - { name: default ,port: 5436 ,dest: postgres ,check: /primary   ,selector: "[]" }
          - { name: offline ,port: 5438 ,dest: postgres ,check: /replica   ,selector: "[? pg_role == `offline` || pg_offline_query ]" ,backup: "[? pg_role == `replica` && !pg_offline_query]" }

    # 4. PolarDB kernel
    pg-polar:
      hosts:
        10.10.10.13: { pg_seq: 1, pg_role: primary }
      vars:
        pg_mode: polar
        pg_cluster: pg-polar
        pg_version: 17
        pg_packages: [ polardb, pgsql-common ]
        pg_exporter_exclude_database: 'template0,template1,postgres,polardb_admin'
        pg_default_roles:
          - { name: dbrole_readonly  ,login: false ,comment: role for global read-only access     }
          - { name: dbrole_offline   ,login: false ,comment: role for restricted read-only access }
          - { name: dbrole_readwrite ,login: false ,roles: [dbrole_readonly] ,comment: role for global read-write access }
          - { name: dbrole_admin     ,login: false ,roles: [pg_monitor, dbrole_readwrite] ,comment: role for object creation }
          - { name: postgres     ,superuser: true  ,comment: system superuser }
          - { name: replicator   ,superuser: true  ,replication: true ,roles: [pg_monitor, dbrole_readonly] ,comment: system replicator }
          - { name: dbuser_dba   ,superuser: true  ,roles: [dbrole_admin]  ,pgbouncer: true ,pool_mode: session ,pool_connlimit: 16 ,comment: pgsql admin user }
          - { name: dbuser_monitor ,roles: [pg_monitor] ,pgbouncer: true ,parameters: { log_min_duration_statement: 1000 } ,pool_mode: session ,pool_connlimit: 8 ,comment: pgsql monitor user }

    # 5. Percona pg_tde kernel
    pg-tde:
      hosts:
        10.10.10.14: { pg_seq: 1, pg_role: primary }
      vars:
        pg_mode: pgtde
        pg_cluster: pg-tde
        pg_version: 18
        pg_packages: [ pgtde, pgsql-common ]
        pg_libs: 'pg_tde, pgaudit, pg_stat_statements, pg_stat_monitor, auto_explain'
        pg_users:
          - { name: dbuser_meta ,password: DBUser.Meta   ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: pigsty admin user }
          - { name: dbuser_view ,password: DBUser.Viewer ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer  }
        pg_databases:
          - name: meta
            baseline: cmdb.sql
            comment: pigsty tde database
            schemas: [pigsty]
            extensions: [ vector, postgis, pg_tde ,pgaudit, { name: pg_stat_monitor, schema: monitor } ]

    # 6. OrioleDB kernel
    pg-oriole:
      hosts:
        10.10.10.15: { pg_seq: 1, pg_role: primary }
      vars:
        pg_mode: oriole
        pg_cluster: pg-oriole
        pg_version: 18
        pg_packages: [ orioledb, pgsql-common ]
        pg_libs: 'orioledb, pg_stat_statements, auto_explain'
        pg_users:
          - { name: dbuser_meta ,password: DBUser.Meta   ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: pigsty admin user }
          - { name: dbuser_view ,password: DBUser.Viewer ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer  }
        pg_databases:
          - { name: meta ,baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty] ,extensions: [orioledb] }

    # 7. OpenHaloDB (MySQL compatible) kernel
    pg-mysql:
      hosts:
        10.10.10.16: { pg_seq: 1, pg_role: primary }
      vars:
        pg_mode: mysql
        pg_cluster: pg-mysql
        pg_version: 14
        pg_packages: [ openhalo, pgsql-common ]
        pg_users:
          - { name: dbuser_meta ,password: DBUser.Meta   ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: pigsty admin user }
          - { name: dbuser_view ,password: DBUser.Viewer ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer  }
        pg_databases:
          - { name: postgres ,extensions: [aux_mysql] }
          - { name: meta ,baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty] }

    # 8. PostgreSQL Mongo mode with DocumentDB
    pg-mongo:
      hosts:
        10.10.10.17: { pg_seq: 1, pg_role: primary }
      vars:
        pg_cluster: pg-mongo
        pg_version: 18
        pg_users:
          - { name: dbuser_meta ,password: DBUser.Meta   ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: pigsty admin user }
          - { name: dbuser_view ,password: DBUser.Viewer ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer  }
        pg_databases:
          - { name: postgres ,extensions: [documentdb, postgis, vector, pg_cron, rum] }
        pg_hba_rules:
          - { user: dbuser_view ,db: all ,addr: infra     ,auth: pwd   ,title: 'allow grafana dashboard access cmdb from infra nodes' }
          - { user: postgres    ,db: all ,addr: world     ,auth: pwd   ,title: 'dbsu password access everywhere (demo only)' }
          - { user: all         ,db: all ,addr: localhost ,order: 1    ,auth: trust ,title: 'documentdb localhost trust access' }
          - { user: all         ,db: all ,addr: local     ,order: 1    ,auth: trust ,title: 'documentdb local trust access' }
          - { user: all         ,db: all ,addr: intra     ,auth: pwd   ,order: 800  ,title: 'everyone intranet access with password' }
        pg_parameters: { cron.database_name: postgres }
        pg_extensions: [ documentdb, postgis, pgvector, pg_cron, rum ]
        pg_libs: 'pg_documentdb, pg_documentdb_core, pg_documentdb_extended_rum, pg_cron, pg_stat_statements, auto_explain'

    # 9. AgensGraph kernel
    pg-agens:
      hosts:
        10.10.10.18: { pg_seq: 1, pg_role: primary }
      vars:
        pg_mode: agens
        pg_cluster: pg-agens
        pg_version: 17
        pg_packages: [ agensgraph, pgsql-common ]
        pg_users:
          - { name: dbuser_meta ,password: DBUser.Meta   ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: pigsty admin user }
          - { name: dbuser_view ,password: DBUser.Viewer ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer  }
        pg_databases:
          - { name: meta ,baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty] }

    # 10. pgedge kernel (stock pigsty pgsql repo path)
    pg-edge:
      hosts:
        10.10.10.19: { pg_seq: 1, pg_role: primary }
      vars:
        pg_mode: pgedge
        pg_cluster: pg-edge
        pg_version: 18
        pg_packages: [ pgedge, pgsql-common ]
        pg_libs: 'spock, lolor, pg_stat_statements, auto_explain'
        pg_users:
          - { name: dbuser_meta ,password: DBUser.Meta   ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: pigsty admin user }
          - { name: dbuser_view ,password: DBUser.Viewer ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer  }
        pg_databases:
          - { name: meta ,baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty] ,extensions: [spock, snowflake, lolor] }

  vars:
    version: v4.5.0                   # pigsty version string
    admin_ip: 10.10.10.10             # admin node ip address
    region: default                   # upstream mirror region: default|china|europe
    node_tune: oltp                   # node tuning specs: oltp,olap,tiny,crit
    pg_conf: oltp.yml                 # pgsql tuning specs: {oltp,olap,tiny,crit}.yml
    node_repo_modules: node,infra,pgsql
    proxy_env:
      no_proxy: "localhost,127.0.0.1,10.0.0.0/8,192.168.0.0/16,*.pigsty,*.aliyun.com,mirrors.*,*.myqcloud.com,*.tsinghua.edu.cn"
    infra_portal:
      home : { domain: i.pigsty }

    grafana_admin_password: pigsty
    grafana_view_password: DBUser.Viewer
    pg_admin_password: DBUser.DBA
    pg_monitor_password: DBUser.Monitor
    pg_replication_password: DBUser.Replicator
    patroni_password: Patroni.API
    haproxy_admin_password: pigsty
    minio_secret_key: S3User.MinIO
    etcd_root_password: Etcd.Root
...

Explanation

This template uses single-node clusters to show the minimum viable configuration for different kernels:

  • pg-citus: PostgreSQL 18 + Citus
  • pg-ivory: IvorySQL, compatible with PostgreSQL 18
  • pg-mssql: Babelfish, compatible with PostgreSQL 17
  • pg-polar: PolarDB for PostgreSQL, compatible with PostgreSQL 17
  • pg-tde: Percona PostgreSQL 18 + pg_tde
  • pg-oriole: OrioleDB, supports PostgreSQL 16, 17, and 18; the current demo config defaults to PG18
  • pg-mysql: OpenHalo, compatible with PostgreSQL 14
  • pg-mongo: DocumentDB backend for PostgreSQL Mongo mode, default PostgreSQL 18
  • pg-agens: AgensGraph, compatible with PostgreSQL 17
  • pg-edge: pgEdge, compatible with PostgreSQL 18

Notes:

  • Package support varies by kernel, OS, and architecture. Confirm the target repository is available before deployment.
  • This template includes permissive access rules for demo use. For production, use a dedicated kernel template and tighten HBA and password policies.

6.31 - demo/minio

Four-node x four-drive HA S3 object-storage cluster demo; current source defaults to Silo.

demo/minio demonstrates a highly available S3 object-storage cluster with four nodes and four drives per node, for 16 drives total. The template retains MINIO module compatibility naming and explicitly sets minio_type: silo; the current v4.5.0 source accepts only this value, and both deployment and removal roles default to silo. Still verify it together with the exact target, cluster identity, and data paths before removal.

For more tutorials, see the MINIO module documentation.

Overview

  • Config Name: demo/minio
  • Node Count: Four nodes
  • Description: High-availability multi-node multi-drive S3 object-storage demo (currently defaults to Silo)
  • OS Distro: el8, el9, el10, d12, d13, u22, u24, u26
  • OS Arch: x86_64, aarch64
  • Related: meta

Usage:

./configure -c demo/minio
Note

This is a four-node template. You need to modify the IP addresses of the other three nodes after generating the configuration.

Content

Source: pigsty/conf/demo/minio.yml

---
#==============================================================#
# File      :   minio.yml
# Desc      :   pigsty: 4 node x 4 disk MNMD minio clusters
# Ctime     :   2023-01-07
# Mtime     :   2026-08-09
# Docs      :   https://pigsty.io/docs/minio
# License   :   Apache-2.0 @ https://pigsty.io/docs/about/license/
# Copyright :   2018-2026  Ruohang Feng / Vonng ([email protected])
#==============================================================#

# One pass installation with:
# ./deploy.yml
#==============================================================#
# 1.  minio-1 @ 10.10.10.10:9000 -  - (9002) svc <-x  10.10.10.9:9002
# 2.  minio-2 @ 10.10.10.11:9000 -xx- (9002) svc <-x <----------------
# 3.  minio-3 @ 10.10.10.12:9000 -xx- (9002) svc <-x  sss.pigsty:9002
# 4.  minio-4 @ 10.10.10.13:9000 -  - (9002) svc <-x  (intranet dns)
#==============================================================#
# use minio load balancer service (9002) instead of direct access (9000)
# mcli alias set sss https://sss.pigsty:9002 minioadmin S3User.MinIO
#==============================================================#
# https://min.io/docs/minio/linux/operations/install-deploy-manage/deploy-minio-multi-node-multi-drive.html
# MINIO_VOLUMES="https://minio-{1...4}.pigsty:9000/data{1...4}/minio"


all:
  children:

    # infra cluster for proxy, monitor, alert, etc...
    infra: { hosts: { 10.10.10.10: { infra_seq: 1 } } }

    # minio cluster with 4 nodes and 4 drivers per node
    minio:
      hosts:
        10.10.10.10: { minio_seq: 1 , nodename: minio-1 }
        10.10.10.11: { minio_seq: 2 , nodename: minio-2 }
        10.10.10.12: { minio_seq: 3 , nodename: minio-3 }
        10.10.10.13: { minio_seq: 4 , nodename: minio-4 }
      vars:
        minio_type: silo
        minio_cluster: minio
        minio_data: '/data{1...4}'
        minio_buckets:                    # list of minio bucket to be created
          - { name: pgsql }
          - { name: meta ,versioning: true }
          - { name: data }
        minio_users:                      # list of minio user to be created
          - { access_key: pgbackrest  ,secret_key: S3User.Backup ,policy: pgsql }
          - { access_key: s3user_meta ,secret_key: S3User.Meta   ,policy: meta  }
          - { access_key: s3user_data ,secret_key: S3User.Data   ,policy: data  }

        # bind a node l2 vip (10.10.10.9) to minio cluster (optional)
        node_cluster: minio
        vip_enabled: true
        vip_vrid: 128
        vip_address: 10.10.10.9

        # expose minio service with haproxy on all nodes
        haproxy_services:
          - name: minio                    # [REQUIRED] service name, unique
            port: 9002                     # [REQUIRED] service port, unique
            balance: leastconn             # [OPTIONAL] load balancer algorithm
            options:                       # [OPTIONAL] minio health check
              - option httpchk
              - option http-keep-alive
              - http-check send meth OPTIONS uri /minio/health/live
              - http-check expect status 200
            servers:
              - { name: minio-1 ,ip: 10.10.10.10 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
              - { name: minio-2 ,ip: 10.10.10.11 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
              - { name: minio-3 ,ip: 10.10.10.12 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
              - { name: minio-4 ,ip: 10.10.10.13 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }

    #etcd:  { hosts: { 10.10.10.10: { etcd_seq: 1 } } }
    #pgsql:
    #  hosts:
    #    10.10.10.10: { pg_seq: 1 , pg_role: primary }
    #    10.10.10.11: { pg_seq: 2 , pg_role: replica }
    #    10.10.10.12: { pg_seq: 3 , pg_role: replica }
    #    10.10.10.13: { pg_seq: 4 , pg_role: replica }
    #  vars:
    #    pg_cluster: pgsql
    #    pgbackrest_method: minio

  vars:
    version: v4.5.0                   # pigsty version string
    admin_ip: 10.10.10.10             # admin node ip address
    region: default                   # upstream mirror region: default|china|europe

    # build a local repo without PostgreSQL packages
    repo_modules: infra,node
    repo_packages: "{{ repo_packages_default | reject('equalto', 'pgsql-utility') | list }}"
    repo_extra_packages: []

    infra_portal:                     # infra services exposed via portal
      home : { domain: i.pigsty }     # default domain name

      # domain names to access minio web console via nginx web portal (optional)
      minio        : { domain: m.pigsty     ,endpoint: "10.10.10.10:9001" ,scheme: https ,websocket: true }
      minio10      : { domain: m10.pigsty   ,endpoint: "10.10.10.10:9001" ,scheme: https ,websocket: true }
      minio11      : { domain: m11.pigsty   ,endpoint: "10.10.10.11:9001" ,scheme: https ,websocket: true }
      minio12      : { domain: m12.pigsty   ,endpoint: "10.10.10.12:9001" ,scheme: https ,websocket: true }
      minio13      : { domain: m13.pigsty   ,endpoint: "10.10.10.13:9001" ,scheme: https ,websocket: true }

    minio_endpoint: https://sss.pigsty:9002   # explicit overwrite minio endpoint with haproxy port
    node_etc_hosts: ["10.10.10.9 sss.pigsty"] # domain name to access minio from all nodes (required)

    #----------------------------------------------#
    # PASSWORD : https://pigsty.io/docs/setup/security/
    #----------------------------------------------#
    grafana_admin_password: pigsty
    haproxy_admin_password: pigsty
    minio_secret_key: S3User.MinIO
...

Explanation

demo/minio is a reference configuration for production object storage using the Multi-Node Multi-Drive (MNMD) architecture. Its volume layout, HAProxy health checks, and clients retain MinIO-compatible interfaces.

Key Features:

  • Multi-Node Multi-Drive Architecture: 4 nodes × 4 drives = 16-drive erasure coding group
  • L2 VIP High Availability: Virtual IP binding via Keepalived
  • HAProxy Load Balancing: Unified access endpoint on port 9002
  • Fine-grained Permissions: Separate users and buckets for different applications

Access:

# Configure the S3 alias with mcli (via HAProxy load balancing)
mcli alias set sss https://sss.pigsty:9002 minioadmin S3User.MinIO

# List buckets
mcli ls sss/

# Use console
# Visit https://m.pigsty or https://m10-m13.pigsty

Use Cases:

  • Environments requiring S3-compatible object storage
  • PostgreSQL backup storage (pgBackRest remote repository)
  • Data lake for big data and AI workloads
  • Production environments requiring high-availability object storage

Notes:

  • Each node requires 4 independent disks mounted at /data1 - /data4
  • Production environments recommend at least 4 nodes for erasure coding redundancy
  • VIP requires proper network interface configuration (vip_interface)

6.32 - demo/redis

Four-node demo of Redis replica, Sentinel, and native Cluster modes

demo/redis demonstrates standalone/replica, Sentinel, and native Cluster modes supported by Pigsty’s Redis module in one configuration.

Overview

  • Config Name: demo/redis
  • Node Count: 4
  • Clusters: redis-ms, redis-meta, redis-test
  • Related: demo/demo
./configure -c demo/redis -s

Content

Source: pigsty/conf/demo/redis.yml

---
#==============================================================#
# File      :   redis.yml
# Desc      :   pigsty config for redis clusters
# Ctime     :   2022-11-09
# Mtime     :   2026-08-02
# Docs      :   https://pigsty.io/docs/redis
# License   :   Apache-2.0 @ https://pigsty.io/docs/about/license/
# Copyright :   2018-2026  Ruohang Feng / Vonng ([email protected])
#==============================================================#


all:
  children:

    # infra cluster for proxy, monitor, alert, etc..
    infra: { hosts: { 10.10.10.10: { infra_seq: 1 } } }

    redis-ms: # redis classic primary & replica
      hosts: { 10.10.10.10: { redis_node: 1 , redis_instances: { 6379: { }, 6380: { replica_of: '10.10.10.10 6379' } } } }
      vars: { redis_cluster: redis-ms ,redis_password: 'redis.ms' ,redis_max_memory: 64MB }

    redis-meta: # redis sentinel x 3
      hosts: { 10.10.10.11: { redis_node: 1 , redis_instances: { 26379: { } ,26380: { } ,26381: { } } } }
      vars:
        redis_cluster: redis-meta
        redis_password: 'redis.meta'
        redis_mode: sentinel
        redis_max_memory: 16MB
        redis_sentinel_monitor: # primary list for redis sentinel, use cls as name, primary ip:port
          - { name: redis-ms, host: 10.10.10.10, port: 6379 ,password: redis.ms, quorum: 2 }

    redis-test: # redis native cluster: 3m x 3s
      hosts:
        10.10.10.12: { redis_node: 1 ,redis_instances: { 6379: { } ,6380: { } ,6381: { } } }
        10.10.10.13: { redis_node: 2 ,redis_instances: { 6379: { } ,6380: { } ,6381: { } } }
      vars: { redis_cluster: redis-test ,redis_password: 'redis.test' ,redis_mode: cluster, redis_max_memory: 32MB }


  vars:
    version: v4.5.0                   # pigsty version string
    admin_ip: 10.10.10.10             # admin node ip address
    region: default                   # upstream mirror region: default|china|europe

    #================================================================#
    #                         VARS: REDIS                            #
    #================================================================#
    # redis identity
    #redis_cluster:         <CLUSTER> # redis cluster name, required identity parameter
    #redis_node: 1             <NODE> # redis node sequence number, node int id required
    #redis_instances: {}       <NODE> # redis instances definition on this redis node

    # redis node
    redis_fs_main: /data/redis        # redis main data directory, `/data/redis` by default
    redis_exporter_enabled: true      # install redis exporter on redis nodes?
    redis_exporter_port: 9121         # redis exporter listen port, 9121 by default
    redis_exporter_options: ''        # cli args and extra options for redis exporter
    redis_type: redis                 # redis implementation: redis or valkey

    # redis instance
    redis_mode: standalone            # redis mode: standalone,cluster,sentinel
    redis_conf: redis.conf            # redis config template path, except sentinel
    redis_bind_address: '0.0.0.0'     # redis bind address, empty string will use host ip
    redis_max_memory: 32MB            # max memory used by each redis instance
    redis_mem_policy: allkeys-lru     # redis memory eviction policy
    redis_password: ''                # redis password, empty string will disable password
    redis_rdb_save: [ '1200 1' ]      # redis rdb save directives, disable with empty list
    redis_aof_enabled: false          # enable redis append only file?
    redis_rename_commands: { }        # rename redis dangerous commands
    redis_cluster_replicas: 1         # replica number for one master in redis cluster
    redis_sentinel_monitor: []        # sentinel master list, works on sentinel cluster only


    #----------------------------------------------#
    # PASSWORD : https://pigsty.io/docs/setup/security/
    #----------------------------------------------#
    grafana_admin_password: pigsty
    haproxy_admin_password: pigsty
...

Explanation

  • redis-ms: a 6379 primary and 6380 replica on one node
  • redis-meta: three Sentinel instances monitoring the redis-ms primary
  • redis-test: a native Redis Cluster across two nodes with three instances per node
  • Small per-instance memory limits keep the topology suitable for demonstrations

The IP addresses, passwords, and memory limits are demonstration values. Adjust them to the real topology, then install the Redis module with the redis.yml playbook.

6.33 - demo/kafka

Four-node dynamic KRaft example with a plaintext single-node dev cluster and a three-node TLS/SCRAM HA baseline

demo/kafka declares two Kafka 4.x dynamic KRaft clusters across four nodes: the plaintext single-node development cluster kf-meta, and the three-node TLS/SCRAM/ACL demonstration cluster kf-test.

Overview

  • Config Name: demo/kafka
  • Node Count: 4
  • kf-meta: Single combined Broker/Controller node in plaintext mode
  • kf-test: Three combined nodes with TLS/SCRAM/ACL, topic replication factor 3, and min.insync.replicas=2
  • Module Status: KAFKA BETA
./configure -c demo/kafka -s
./deploy.yml
./kafka.yml -l kf-meta
./kafka.yml -l kf-test

deploy.yml only deploys the core path and does not run the KAFKA playbook automatically. Each kafka.yml run must select one complete Kafka cluster; the role rejects convergence against only part of a cluster.

Content

Source: pigsty/conf/demo/kafka.yml

---
#==============================================================#
# File      :   kafka.yml
# Desc      :   pigsty: 4 node kafka demo (dynamic KRaft)
# Ctime     :   2026-07-17
# Mtime     :   2026-07-17
# Docs      :   https://pigsty.io/docs/kafka
# License   :   Apache-2.0 @ https://pigsty.io/docs/about/license/
# Copyright :   2018-2026  Ruohang Feng / Vonng ([email protected])
#==============================================================#

# One pass installation with:
# ./deploy.yml
# ./kafka.yml -l kf-main
# ./kafka.yml -l kf-test
#==============================================================#
# 1.  kf-meta-1 @ 10.10.10.10:9092   single-node dev cluster (plaintext)
# 2.  kf-test-1 @ 10.10.10.11:9092 \
# 3.  kf-test-2 @ 10.10.10.12:9092 --- 3-node secure HA demo baseline (scram)
# 4.  kf-test-3 @ 10.10.10.13:9092 /   dynamic KRaft, TLS/SCRAM/ACL, RF=3/minISR=2
#==============================================================#
# kafka clients are cluster-aware and connect to every broker directly:
# bootstrap with e.g. 10.10.10.11:9092,10.10.10.12:9092,10.10.10.13:9092


all:
  children:

    # infra cluster for proxy, monitor, alert, etc..
    infra: { hosts: { 10.10.10.10: { infra_seq: 1 } } }

    # single-node kafka dev cluster: combined broker/controller, plaintext
    kf-meta:
      hosts:
        10.10.10.10: { kafka_seq: 1 }
      vars:
        kafka_cluster: kf-meta
        kafka_topics:
          - { name: quickstart.events ,partitions: 1 ,replication_factor: 1 ,config: { retention.ms: 86400000 } }

    # 3-node secure HA demo baseline: dynamic KRaft, TLS/SCRAM/ACL, RF=3/minISR=2
    kf-test:
      hosts:
        10.10.10.11: { kafka_seq: 1 }
        10.10.10.12: { kafka_seq: 2 }
        10.10.10.13: { kafka_seq: 3 }
      vars:
        kafka_cluster: kf-test
        kafka_security: scram
        kafka_heap_opts: '-Xms512M -Xmx512M' # 2GiB demo nodes cannot safely spare the 1GiB production default
        kafka_users:               # app principal with prefixed topic/group acls
          - name: test-app
            password: KafkaApp.Test
            acls:
              - { resource: topic   ,name: 'test.'       ,pattern: prefixed ,operations: [ Read, Write, Describe ] }
              - { resource: group   ,name: 'test.'       ,pattern: prefixed ,operations: [ Read ] }
              - { resource: cluster ,name: kafka-cluster ,operations: [ Describe, IdempotentWrite ] }
        kafka_topics:
          - name: test.events
            partitions: 3
            replication_factor: 3
            config: { min.insync.replicas: 2 ,cleanup.policy: delete ,retention.ms: 604800000 }

  vars:
    version: v4.5.0                   # pigsty version string
    admin_ip: 10.10.10.10             # admin node ip address
    region: default                   # upstream mirror region: default|china|europe
    infra_portal:                     # infra services exposed via portal
      home : { domain: i.pigsty }     # default domain name

    # kafka & java packages are required in the local repo for the kafka module (if using local repo)
    repo_extra_packages: [ kafka-stack ,java-runtime ]

    #----------------------------------------------#
    # PASSWORD : https://pigsty.io/docs/setup/security/
    #----------------------------------------------#
    grafana_admin_password: pigsty
    grafana_view_password: DBUser.Viewer
...

Explanation

  • kf-meta creates quickstart.events for single-node development and connectivity tests.
  • kf-test creates the test-app SCRAM user, prefix ACLs, and the three-replica test.events topic.
  • Online installation maps the platform packages kafka-stack and java-runtime; when using only a local repository, cache both complete package groups first.
  • Addresses and passwords in the template are demonstration values. Replace them for your topology and security requirements before deployment.

See the KAFKA module for operations, security, and scaling constraints.

6.34 - demo/mysql

Native MySQL 8.4 pilot template with a standalone instance and a three-node InnoDB Cluster

demo/mysql is the four-node example for the native MySQL 8.4 LTS pilot module. It is distinct from conf/mysql.yml, which provides MySQL protocol compatibility through the OpenHalo PostgreSQL kernel.

Overview

  • Config Name: demo/mysql
  • Node Count: 4
  • my-meta: Standalone MySQL 8.4 instance
  • my-test: Three-node, single-primary InnoDB Cluster with MySQL Router on every member
  • Module Status: MYSQL PILOT; not included in the stable module count
  • Platform Boundary: Supported declared x86_64 RPM/DEB platforms and EL9/EL10 aarch64. Oracle APT currently has no arm64 component, so preflight rejects Debian/Ubuntu ARM.

Replace every CHANGE_ME value in the template. Real deployment also requires explicit approval. Start with read-only preflight checks:

ansible-playbook -i conf/demo/mysql.yml mysql.yml -l my-meta --check
ansible-playbook -i conf/demo/mysql.yml mysql.yml -l my-test --check

After explicitly approving an active-inventory update, run ./configure -c demo/mysql, then run both node.yml and mysql.yml with --check and real convergence against the same complete cluster scope. The three-node cluster does not accept a partial-member scope.

Content

Source: pigsty/conf/demo/mysql.yml

---
#==============================================================#
# File      :   mysql.yml
# Desc      :   MySQL 8.4 LTS standalone and three-node HA template
# Ctime     :   2026-07-16
# Mtime     :   2026-07-19
# Docs      :   https://pigsty.io/docs/mysql
# License   :   Apache-2.0 @ https://pigsty.io/docs/about/license/
# Copyright :   2018-2026  Ruohang Feng / Vonng ([email protected])
#==============================================================#

# Canonical four-node MySQL platform example:
#   my-meta: one standalone member
#   my-test: three-member InnoDB Cluster in single-primary mode
#
# This file is a template, not the active deployment inventory. Replace every
# CHANGE_ME value and review the target addresses before configure or playbooks;
# MySQL preflight rejects any canonical CHANGE_ME credential left in place.
# Native Oracle 8.4 packages are admitted on x86_64 and EL9/EL10 aarch64. Oracle's
# APT repository currently has no arm64 component, so Ubuntu/Debian ARM is rejected.
#
# Read-only template preview (does not rewrite active pigsty.yml):
#   ansible-playbook -i conf/demo/mysql.yml mysql.yml -l my-meta --check
#   ansible-playbook -i conf/demo/mysql.yml mysql.yml -l my-test --check
# Each limit must include every declared member of the selected cluster group;
# partial HA member limits are rejected before any package or service change.
# After explicit approval to update active inventory, run ./configure -c demo/mysql,
# then repeat node.yml/mysql.yml --check with the same explicit limits.
# node.yml owns the shared trusted CA at /etc/pki/ca.crt; mysql.yml installs
# only MySQL/Router leaf certificates and requires node_ca to be complete.
#
# Real playbooks install/start services and require explicit approval.

all:
  children:
    infra:
      hosts:
        10.10.10.10: { infra_seq: 1 }

    my-meta:
      hosts:
        10.10.10.10: { mysql_seq: 1 }
      vars: { mysql_cluster: my-meta, node_cluster: my-meta }

    my-test:
      hosts:
        10.10.10.11: { mysql_seq: 1 }
        10.10.10.12: { mysql_seq: 2 }
        10.10.10.13: { mysql_seq: 3 }
      vars: { mysql_cluster: my-test, node_cluster: my-test }

  vars:
    version: v4.5.0
    admin_ip: 10.10.10.10
    region: default                    # default | china; china uses USTC for MySQL

    repo_enabled: false               # use signed upstream repositories directly
    node_repo_modules: node,infra,mysql
    # For a Pigsty local/offline repository, enable repo and cache the complete
    # atomic platform set before provisioning targets:
    # repo_enabled: true
    # repo_extra_packages: [mysql]

    nodename_overwrite: false
    node_tune: oltp

    mysql_root_password: CHANGE_ME_MYSQL_ROOT
    mysql_monitor_password: CHANGE_ME_MYSQL_MONITOR
    mysql_cluster_password: CHANGE_ME_MYSQL_CLUSTER
    # Fixed MySQL 8.4, auto-tuned memory, daily local backup, and exporter are defaults.

    # Pigsty infrastructure credentials; replace before deployment.
    grafana_admin_password: CHANGE_ME_GRAFANA_ADMIN
    grafana_view_password: CHANGE_ME_GRAFANA_VIEW
    haproxy_admin_password: CHANGE_ME_HAPROXY
...

Explanation

  • MySQL server, client, Shell, Router, and XtraBackup are fixed to the 8.4 platform; this is not an arbitrary-version installer.
  • The standalone instance uses 3306. The three-node cluster also uses Group Replication on 33061, with Router RW on 6446 and RO on 6447 on each member.
  • A daily local full XtraBackup and mysqld_exporter are enabled by default. The current pilot does not provide continuous binlog archiving, PITR, or automatic recovery.
  • node.yml installs the shared trust anchor at /etc/pki/ca.crt; the MySQL role only issues and installs leaf certificates.

See the native MySQL pilot documentation for complete constraints and the confirmed removal workflow.

6.35 - build/oss

Pigsty open-source edition offline package build environment configuration

The build/oss configuration template is the build environment configuration for Pigsty open-source edition offline packages, used to batch-build offline installation packages across multiple operating systems.

This configuration is intended for developers and contributors only.

Overview

  • Config Name: build/oss
  • Node Count: Seven nodes (el9, el10, d12, d13, u22, u24, u26)
  • Description: Pigsty open-source edition offline package build environment
  • OS Distro: el9, el10, d12, d13, u22, u24, u26
  • OS Arch: x86_64

Usage:

cp conf/build/oss.yml pigsty.yml
Note

This is a build template with fixed IP addresses, intended for internal use only.

Content

Source: pigsty/conf/build/oss.yml

---
#==============================================================#
# File      :   oss.yml
# Desc      :   Pigsty 3-node building env (PG18)
# Ctime     :   2024-10-22
# Mtime     :   2026-05-01
# License   :   Apache-2.0 @ https://pigsty.io/docs/about/license/
# Copyright :   2018-2026  Ruohang Feng / Vonng ([email protected])
#==============================================================#

all:
  vars:
    version: v4.5.0
    admin_ip: 10.10.10.26
    region: china
    proxy_env:
      no_proxy: "localhost,127.0.0.1,10.0.0.0/8,192.168.0.0/16,*.pigsty,*.aliyun.com,mirrors.*,*.myqcloud.com,*.tsinghua.edu.cn,*.pigsty.cc"

    # building spec
    pg_version: 18
    repo_modules: infra,node,pgsql
    repo_packages: [ node-bootstrap, infra-package, infra-addons, node-package1, node-package2, node-package3, pgsql-utility, extra-modules ]
    pg_extensions: [ pg18-time ,pg18-gis ,pg18-rag ,pg18-fts ,pg18-olap, pg18-feat ,pg18-lang ,pg18-type ,pg18-util ,pg18-func ,pg18-admin ,pg18-stat ,pg18-sec ,pg18-fdw ,pg18-sim ,pg18-etl]

    # OSS
    cache_pkg_dir: 'dist/${version}'
    repo_extra_packages: [pg18-core ,pg18-time ,pg18-gis ,pg18-rag ,pg18-fts ,pg18-olap ,pg18-feat ,pg18-lang ,pg18-type ,pg18-util ,pg18-func ,pg18-admin ,pg18-stat ,pg18-sec ,pg18-fdw ,pg18-sim ,pg18-etl]

    # PRO
    #cache_pkg_dir: 'dist/${version}/pro'
    #repo_extra_packages: [
    #  pg18-main,pg18-time,pg18-gis,pg18-rag,pg18-fts,pg18-olap,pg18-feat,pg18-lang,pg18-type,pg18-util,pg18-func,pg18-admin,pg18-stat,pg18-sec,pg18-fdw,pg18-sim,pg18-etl,
    #  pg17-main,pg17-time,pg17-gis,pg17-rag,pg17-fts,pg17-olap,pg17-feat,pg17-lang,pg17-type,pg17-util,pg17-func,pg17-admin,pg17-stat,pg17-sec,pg17-fdw,pg17-sim,pg17-etl,
    #  pg16-main,pg16-time,pg16-gis,pg16-rag,pg16-fts,pg16-olap,pg16-feat,pg16-lang,pg16-type,pg16-util,pg16-func,pg16-admin,pg16-stat,pg16-sec,pg16-fdw,pg16-sim,pg16-etl,
    #  pg15-main,pg15-time,pg15-gis,pg15-rag,pg15-fts,pg15-olap,pg15-feat,pg15-lang,pg15-type,pg15-util,pg15-func,pg15-admin,pg15-stat,pg15-sec,pg15-fdw,pg15-sim,pg15-etl,
    #  pg14-main,pg14-time,pg14-gis,pg14-rag,pg14-fts,pg14-olap,pg14-feat,pg14-lang,pg14-type,pg14-util,pg14-func,pg14-admin,pg14-stat,pg14-sec,pg14-fdw,pg14-sim,pg14-etl,
    #  infra-extra, kafka-stack, java-runtime
    #]

  children:
    el9:  { hosts: { 10.10.10.9:  { pg_cluster: el9  ,pg_seq: 1 ,pg_role: primary }}}
    el10: { hosts: { 10.10.10.10: { pg_cluster: el10 ,pg_seq: 1 ,pg_role: primary }}}
    d12:  { hosts: { 10.10.10.12: { pg_cluster: d12  ,pg_seq: 1 ,pg_role: primary }}}
    d13:  { hosts: { 10.10.10.13: { pg_cluster: d13  ,pg_seq: 1 ,pg_role: primary }}}
    u22:  { hosts: { 10.10.10.22: { pg_cluster: u22  ,pg_seq: 1 ,pg_role: primary }}}
    u24:  { hosts: { 10.10.10.24: { pg_cluster: u24  ,pg_seq: 1 ,pg_role: primary }}}
    u26:  { hosts: { 10.10.10.26: { pg_cluster: u26  ,pg_seq: 1 ,pg_role: primary }}}
    etcd: { hosts: { 10.10.10.26:  { etcd_seq: 1 }}, vars: { etcd_cluster: etcd    }}
    infra:
      hosts:
        10.10.10.9:  { infra_seq: 1, admin_ip: 10.10.10.9  ,ansible_host: el9  }
        10.10.10.10: { infra_seq: 2, admin_ip: 10.10.10.10 ,ansible_host: el10 }
        10.10.10.12: { infra_seq: 3, admin_ip: 10.10.10.12 ,ansible_host: d12  }
        10.10.10.13: { infra_seq: 4, admin_ip: 10.10.10.13 ,ansible_host: d13  }
        10.10.10.22: { infra_seq: 5, admin_ip: 10.10.10.22 ,ansible_host: u22  }
        10.10.10.24: { infra_seq: 6, admin_ip: 10.10.10.24 ,ansible_host: u24  }
        10.10.10.26: { infra_seq: 7, admin_ip: 10.10.10.26 ,ansible_host: u26  }
      vars: { node_tune: oltp }

...

Explanation

The build/oss template is the build configuration for Pigsty open-source edition offline packages.

Build Contents:

  • PostgreSQL 18 and all categorized extension packages
  • Infrastructure packages (Prometheus, Grafana, Nginx, etc.)
  • Node packages (monitoring agents, tools, etc.)
  • Extra modules

Supported Operating Systems:

  • EL9 (Rocky/Alma/RHEL 9)
  • EL10 (Rocky 10 / RHEL 10)
  • Debian 12 (Bookworm)
  • Debian 13 (Trixie)
  • Ubuntu 22.04 (Jammy)
  • Ubuntu 24.04 (Noble)
  • Ubuntu 26.04 (Resolute)

Build Process:

# 1. Prepare build environment
cp conf/build/oss.yml pigsty.yml

# 2. Download packages on each node
./infra.yml -t repo_build

# 3. Package offline installation files
make cache

Use Cases:

  • Pigsty developers building new versions
  • Contributors testing new extensions
  • Enterprise users customizing offline packages

6.36 - build/dev

Pigsty three-node local build and development configuration

The build/dev configuration template is Pigsty’s three-node local build and development environment. It is used to validate repository build and package download workflows across EL9, Debian 12, and Ubuntu 24 nodes.

This template is intended only for developers and contributors.

Overview

  • Config Name: build/dev
  • Node Count: Three nodes (el9, d12, u24)
  • Description: Local build and development environment, default PostgreSQL 18, builds the infra,node,pgsql modules
  • OS Distro: el9, d12, u24
  • OS Arch: x86_64, aarch64
  • Related: build/oss

Usage:

cp conf/build/dev.yml pigsty.yml
Note

This is a fixed-IP development build template. Adjust host addresses for your local environment before use.

Content

Source: pigsty/conf/build/dev.yml

---
#==============================================================#
# File      :   dev.yml
# Desc      :   Pigsty 3-node local build dev config
# Ctime     :   2025-07-17
# Mtime     :   2026-07-05
# License   :   Apache-2.0 @ https://pigsty.io/docs/about/license/
# Copyright :   2018-2026  Ruohang Feng / Vonng ([email protected])
#==============================================================#

all:
  children:
    el9:  { hosts: { 10.10.10.9:  { pg_cluster: el9  ,pg_seq: 1 ,pg_role: primary }}}
    el10: { hosts: { 10.10.10.10: { pg_cluster: el10 ,pg_seq: 1 ,pg_role: primary }}}
    d12:  { hosts: { 10.10.10.12: { pg_cluster: d12  ,pg_seq: 1 ,pg_role: primary }}}
    d13:  { hosts: { 10.10.10.13: { pg_cluster: d13  ,pg_seq: 1 ,pg_role: primary }}}
    u22:  { hosts: { 10.10.10.22: { pg_cluster: u22  ,pg_seq: 1 ,pg_role: primary }}}
    u24:  { hosts: { 10.10.10.24: { pg_cluster: u24  ,pg_seq: 1 ,pg_role: primary }}}
    u26:  { hosts: { 10.10.10.26: { pg_cluster: u26  ,pg_seq: 1 ,pg_role: primary }}}
    etcd: { hosts: { 10.10.10.26:  { etcd_seq: 1 }}, vars: { etcd_cluster: etcd    }}
    infra:
      hosts:
        10.10.10.9:  { infra_seq: 1, admin_ip: 10.10.10.9  ,ansible_host: el9  }
        10.10.10.10: { infra_seq: 2, admin_ip: 10.10.10.10 ,ansible_host: el10 }
        10.10.10.12: { infra_seq: 3, admin_ip: 10.10.10.12 ,ansible_host: d12  }
        10.10.10.13: { infra_seq: 4, admin_ip: 10.10.10.13 ,ansible_host: d13  }
        10.10.10.22: { infra_seq: 5, admin_ip: 10.10.10.22 ,ansible_host: u22  }
        10.10.10.24: { infra_seq: 6, admin_ip: 10.10.10.24 ,ansible_host: u24  }
        10.10.10.26: { infra_seq: 7, admin_ip: 10.10.10.26 ,ansible_host: u26  }
      vars: { node_tune: oltp }

  vars:
    version: v4.5.0
    admin_ip: 10.10.10.26
    region: china
    proxy_env:
      no_proxy: "localhost,127.0.0.1,10.0.0.0/8,192.168.0.0/16,*.pigsty,*.aliyun.com,mirrors.*,*.myqcloud.com,*.tsinghua.edu.cn,*.pigsty.cc"

    # PRO
    #cache_pkg_dir: 'dist/${version}/pro'
    #repo_extra_packages: [
    #  pg18-main,pg18-time,pg18-gis,pg18-rag,pg18-fts,pg18-olap,pg18-feat,pg18-lang,pg18-type,pg18-util,pg18-func,pg18-admin,pg18-stat,pg18-sec,pg18-fdw,pg18-sim,pg18-etl,
    #  pg17-main,pg17-time,pg17-gis,pg17-rag,pg17-fts,pg17-olap,pg17-feat,pg17-lang,pg17-type,pg17-util,pg17-func,pg17-admin,pg17-stat,pg17-sec,pg17-fdw,pg17-sim,pg17-etl,
    #  pg16-main,pg16-time,pg16-gis,pg16-rag,pg16-fts,pg16-olap,pg16-feat,pg16-lang,pg16-type,pg16-util,pg16-func,pg16-admin,pg16-stat,pg16-sec,pg16-fdw,pg16-sim,pg16-etl,
    #  pg15-main,pg15-time,pg15-gis,pg15-rag,pg15-fts,pg15-olap,pg15-feat,pg15-lang,pg15-type,pg15-util,pg15-func,pg15-admin,pg15-stat,pg15-sec,pg15-fdw,pg15-sim,pg15-etl,
    #  pg14-main,pg14-time,pg14-gis,pg14-rag,pg14-fts,pg14-olap,pg14-feat,pg14-lang,pg14-type,pg14-util,pg14-func,pg14-admin,pg14-stat,pg14-sec,pg14-fdw,pg14-sim,pg14-etl,
    #  infra-extra, kafka-stack, java-runtime
    #]

    # building spec
    pg_version: 18
    cache_pkg_dir: 'dist/${version}'
    repo_modules: infra,node,pgsql
    repo_packages: [ node-bootstrap, infra-package, infra-addons, node-package1, node-package2, node-package3, pgsql-utility, extra-modules ]
    repo_extra_packages: [pg18-core ,pg18-time ,pg18-gis ,pg18-rag ,pg18-fts ,pg18-olap ,pg18-feat ,pg18-lang ,pg18-type ,pg18-util ,pg18-func ,pg18-admin ,pg18-stat ,pg18-sec ,pg18-fdw ,pg18-sim ,pg18-etl]
    pg_extensions:                 [ pg18-time ,pg18-gis ,pg18-rag ,pg18-fts ,pg18-olap, pg18-feat ,pg18-lang ,pg18-type ,pg18-util ,pg18-func ,pg18-admin ,pg18-stat ,pg18-sec ,pg18-fdw ,pg18-sim ,pg18-etl]
    repo_upstream:
      # EL 7/8/9/10 REPOS
      - { name: pigsty-local   ,description: 'Pigsty Local'       ,module: local   ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'http://${admin_ip}/pigsty' } ,meta: { skip_if_unavailable: 1 ,priority: 1 ,module_hotfixes: 1 }} # used by intranet nodes
      - { name: pigsty-infra   ,description: 'Pigsty INFRA'       ,module: infra   ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://repo.pigsty.io/yum/infra/$basearch'               ,china: 'http://beta.pigsty.cc/yum/infra/$basearch' } ,meta: { priority: 12 ,module_hotfixes: 1 }}
      - { name: pigsty-pgsql   ,description: 'Pigsty PGSQL'       ,module: pgsql   ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://repo.pigsty.io/yum/pgsql/el$releasever.$basearch' ,china: 'http://beta.pigsty.cc/yum/pgsql/el$releasever.$basearch' } ,meta: { priority: 11 ,module_hotfixes: 1 }}
      - { name: nginx          ,description: 'Nginx Repo'         ,module: infra   ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://nginx.org/packages/rhel/$releasever/$basearch/' } ,meta: { module_hotfixes: 1 }}
      - { name: docker-ce      ,description: 'Docker CE'          ,module: infra   ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://download.docker.com/linux/centos/$releasever/$basearch/stable'                       ,china: 'https://mirrors.cloud.tencent.com/docker-ce/linux/centos/$releasever/$basearch/stable https://repo.huaweicloud.com/docker-ce/linux/centos/$releasever/$basearch/stable https://mirrors.aliyun.com/docker-ce/linux/centos/$releasever/$basearch/stable'   ,europe: 'https://mirrors.xtom.de/docker-ce/linux/centos/$releasever/$basearch/stable' } ,meta: { skip_if_unavailable: 1 }}
      - { name: baseos         ,description: 'EL 8+ BaseOS'       ,module: node    ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://dl.rockylinux.org/pub/rocky/$releasever/BaseOS/$basearch/os/'                        ,china: 'https://mirrors.cloud.tencent.com/rocky/$releasever/BaseOS/$basearch/os/ https://repo.huaweicloud.com/rockylinux/$releasever/BaseOS/$basearch/os/ https://mirrors.aliyun.com/rockylinux/$releasever/BaseOS/$basearch/os/'             ,europe: 'https://mirrors.xtom.de/rocky/$releasever/BaseOS/$basearch/os/'     }}
      - { name: appstream      ,description: 'EL 8+ AppStream'    ,module: node    ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://dl.rockylinux.org/pub/rocky/$releasever/AppStream/$basearch/os/'                     ,china: 'https://mirrors.cloud.tencent.com/rocky/$releasever/AppStream/$basearch/os/ https://repo.huaweicloud.com/rockylinux/$releasever/AppStream/$basearch/os/ https://mirrors.aliyun.com/rockylinux/$releasever/AppStream/$basearch/os/'    ,europe: 'https://mirrors.xtom.de/rocky/$releasever/AppStream/$basearch/os/'  }}
      - { name: extras         ,description: 'EL 8+ Extras'       ,module: node    ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://dl.rockylinux.org/pub/rocky/$releasever/extras/$basearch/os/'                        ,china: 'https://mirrors.cloud.tencent.com/rocky/$releasever/extras/$basearch/os/ https://repo.huaweicloud.com/rockylinux/$releasever/extras/$basearch/os/ https://mirrors.aliyun.com/rockylinux/$releasever/extras/$basearch/os/'             ,europe: 'https://mirrors.xtom.de/rocky/$releasever/extras/$basearch/os/'     }}
      - { name: powertools     ,description: 'EL 8 PowerTools'    ,module: node    ,releases: [8     ] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://dl.rockylinux.org/pub/rocky/$releasever/PowerTools/$basearch/os/'                    ,china: 'https://mirrors.cloud.tencent.com/rocky/$releasever/PowerTools/$basearch/os/ https://repo.huaweicloud.com/rockylinux/$releasever/PowerTools/$basearch/os/ https://mirrors.aliyun.com/rockylinux/$releasever/PowerTools/$basearch/os/' ,europe: 'https://mirrors.xtom.de/rocky/$releasever/PowerTools/$basearch/os/' }}
      - { name: crb            ,description: 'EL 9 CRB'           ,module: node    ,releases: [  9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://dl.rockylinux.org/pub/rocky/$releasever/CRB/$basearch/os/'                           ,china: 'https://mirrors.cloud.tencent.com/rocky/$releasever/CRB/$basearch/os/ https://repo.huaweicloud.com/rockylinux/$releasever/CRB/$basearch/os/ https://mirrors.aliyun.com/rockylinux/$releasever/CRB/$basearch/os/'                      ,europe: 'https://mirrors.xtom.de/rocky/$releasever/CRB/$basearch/os/'        }}
      - { name: epel           ,description: 'EL 8+ EPEL'         ,module: node    ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://mirrors.edge.kernel.org/fedora-epel/$releasever/Everything/$basearch/'               ,china: 'https://mirrors.cloud.tencent.com/epel/$releasever/Everything/$basearch/ https://repo.huaweicloud.com/epel/$releasever/Everything/$basearch/ https://mirrors.aliyun.com/epel/$releasever/Everything/$basearch/'                       ,europe: 'https://mirrors.xtom.de/epel/$releasever/Everything/$basearch/'     }}
      - { name: pgdg-common    ,description: 'PostgreSQL Common'  ,module: pgsql   ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://download.postgresql.org/pub/repos/yum/common/redhat/rhel-$releasever-$basearch'      ,china: 'http://beta.pigsty.cc/yum/pgdg/common/redhat/rhel-$releasever-$basearch'      ,europe: 'https://mirrors.xtom.de/postgresql/repos/yum/common/redhat/rhel-$releasever-$basearch' } ,meta: { module_hotfixes: 1 }}
      - { name: pgdg14         ,description: 'PostgreSQL 14'      ,module: pgsql   ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://download.postgresql.org/pub/repos/yum/14/redhat/rhel-$releasever-$basearch'          ,china: 'http://beta.pigsty.cc/yum/pgdg/14/redhat/rhel-$releasever-$basearch'          ,europe: 'https://mirrors.xtom.de/postgresql/repos/yum/14/redhat/rhel-$releasever-$basearch' } ,meta: { module_hotfixes: 1 }}
      - { name: pgdg15         ,description: 'PostgreSQL 15'      ,module: pgsql   ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://download.postgresql.org/pub/repos/yum/15/redhat/rhel-$releasever-$basearch'          ,china: 'http://beta.pigsty.cc/yum/pgdg/15/redhat/rhel-$releasever-$basearch'          ,europe: 'https://mirrors.xtom.de/postgresql/repos/yum/15/redhat/rhel-$releasever-$basearch' } ,meta: { module_hotfixes: 1 }}
      - { name: pgdg16         ,description: 'PostgreSQL 16'      ,module: pgsql   ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://download.postgresql.org/pub/repos/yum/16/redhat/rhel-$releasever-$basearch'          ,china: 'http://beta.pigsty.cc/yum/pgdg/16/redhat/rhel-$releasever-$basearch'          ,europe: 'https://mirrors.xtom.de/postgresql/repos/yum/16/redhat/rhel-$releasever-$basearch' } ,meta: { module_hotfixes: 1 }}
      - { name: pgdg17         ,description: 'PostgreSQL 17'      ,module: pgsql   ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://download.postgresql.org/pub/repos/yum/17/redhat/rhel-$releasever-$basearch'          ,china: 'http://beta.pigsty.cc/yum/pgdg/17/redhat/rhel-$releasever-$basearch'          ,europe: 'https://mirrors.xtom.de/postgresql/repos/yum/17/redhat/rhel-$releasever-$basearch' } ,meta: { module_hotfixes: 1 }}
      - { name: pgdg18         ,description: 'PostgreSQL 18'      ,module: pgsql   ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://download.postgresql.org/pub/repos/yum/18/redhat/rhel-$releasever-$basearch'          ,china: 'http://beta.pigsty.cc/yum/pgdg/18/redhat/rhel-$releasever-$basearch'          ,europe: 'https://mirrors.xtom.de/postgresql/repos/yum/18/redhat/rhel-$releasever-$basearch' } ,meta: { module_hotfixes: 1 }}
      - { name: pgdg-beta      ,description: 'PostgreSQL Testing' ,module: beta    ,releases: [8,9,10] ,arch: [x86_64         ] ,baseurl: { default: 'https://download.postgresql.org/pub/repos/yum/testing/19/redhat/rhel-$releasever-$basearch'  ,china: 'http://beta.pigsty.cc/yum/pgdg/testing/19/redhat/rhel-$releasever-$basearch'  ,europe: 'https://mirrors.xtom.de/postgresql/repos/yum/testing/19/redhat/rhel-$releasever-$basearch'  } ,meta: { module_hotfixes: 1 }}
      - { name: pgdg-beta      ,description: 'PostgreSQL Testing' ,module: beta    ,releases: [  9,10] ,arch: [        aarch64] ,baseurl: { default: 'https://download.postgresql.org/pub/repos/yum/testing/19/redhat/rhel-$releasever-$basearch'  ,china: 'http://beta.pigsty.cc/yum/pgdg/testing/19/redhat/rhel-$releasever-$basearch'  ,europe: 'https://mirrors.xtom.de/postgresql/repos/yum/testing/19/redhat/rhel-$releasever-$basearch'  } ,meta: { module_hotfixes: 1 }}
      - { name: pgdg-extras    ,description: 'PostgreSQL Extra'   ,module: extra   ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://download.postgresql.org/pub/repos/yum/extras/redhat/rhel-$releasever-$basearch'      ,china: 'http://beta.pigsty.cc/yum/pgdg/extras/redhat/rhel-$releasever-$basearch'      ,europe: 'https://mirrors.xtom.de/postgresql/repos/yum/extras/redhat/rhel-$releasever-$basearch'      } ,meta: { module_hotfixes: 1 }}
      - { name: pgdg14-nonfree ,description: 'PostgreSQL 14+'     ,module: extra   ,releases: [8,9,10] ,arch: [x86_64         ] ,baseurl: { default: 'https://download.postgresql.org/pub/repos/yum/non-free/14/redhat/rhel-$releasever-$basearch' ,china: 'http://beta.pigsty.cc/yum/pgdg/non-free/14/redhat/rhel-$releasever-$basearch' ,europe: 'https://mirrors.xtom.de/postgresql/repos/yum/non-free/14/redhat/rhel-$releasever-$basearch' } ,meta: { module_hotfixes: 1 ,skip_if_unavailable: 1 }}
      - { name: pgdg15-nonfree ,description: 'PostgreSQL 15+'     ,module: extra   ,releases: [8,9,10] ,arch: [x86_64         ] ,baseurl: { default: 'https://download.postgresql.org/pub/repos/yum/non-free/15/redhat/rhel-$releasever-$basearch' ,china: 'http://beta.pigsty.cc/yum/pgdg/non-free/15/redhat/rhel-$releasever-$basearch' ,europe: 'https://mirrors.xtom.de/postgresql/repos/yum/non-free/15/redhat/rhel-$releasever-$basearch' } ,meta: { module_hotfixes: 1 ,skip_if_unavailable: 1 }}
      - { name: pgdg16-nonfree ,description: 'PostgreSQL 16+'     ,module: extra   ,releases: [8,9,10] ,arch: [x86_64         ] ,baseurl: { default: 'https://download.postgresql.org/pub/repos/yum/non-free/16/redhat/rhel-$releasever-$basearch' ,china: 'http://beta.pigsty.cc/yum/pgdg/non-free/16/redhat/rhel-$releasever-$basearch' ,europe: 'https://mirrors.xtom.de/postgresql/repos/yum/non-free/16/redhat/rhel-$releasever-$basearch' } ,meta: { module_hotfixes: 1 ,skip_if_unavailable: 1 }}
      - { name: pgdg17-nonfree ,description: 'PostgreSQL 17+'     ,module: extra   ,releases: [8,9,10] ,arch: [x86_64         ] ,baseurl: { default: 'https://download.postgresql.org/pub/repos/yum/non-free/17/redhat/rhel-$releasever-$basearch' ,china: 'http://beta.pigsty.cc/yum/pgdg/non-free/17/redhat/rhel-$releasever-$basearch' ,europe: 'https://mirrors.xtom.de/postgresql/repos/yum/non-free/17/redhat/rhel-$releasever-$basearch' } ,meta: { module_hotfixes: 1 ,skip_if_unavailable: 1 }}
      - { name: pgdg18-nonfree ,description: 'PostgreSQL 18+'     ,module: extra   ,releases: [8,9,10] ,arch: [x86_64         ] ,baseurl: { default: 'https://download.postgresql.org/pub/repos/yum/non-free/18/redhat/rhel-$releasever-$basearch' ,china: 'http://beta.pigsty.cc/yum/pgdg/non-free/18/redhat/rhel-$releasever-$basearch' ,europe: 'https://mirrors.xtom.de/postgresql/repos/yum/non-free/18/redhat/rhel-$releasever-$basearch' } ,meta: { module_hotfixes: 1 ,skip_if_unavailable: 1 }}
      - { name: timescaledb    ,description: 'TimescaleDB'        ,module: extra   ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://packagecloud.io/timescale/timescaledb/el/$releasever/$basearch'  }}
      - { name: percona        ,description: 'Percona TDE'        ,module: percona ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://repo.pigsty.io/yum/percona/el$releasever.$basearch' ,china: 'http://beta.pigsty.cc/yum/percona/el$releasever.$basearch' ,origin: 'http://repo.percona.com/ppg-18.4/yum/release/$releasever/RPMS/$basearch'  } ,meta: { module_hotfixes: 1 }}
      - { name: groonga        ,description: 'Groonga'            ,module: groonga ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://packages.groonga.org/almalinux/$releasever/$basearch/' }}
      - { name: mysql          ,description: 'MySQL 8.4 LTS'      ,module: mysql   ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://repo.mysql.com/yum/mysql-8.4-community/el/$releasever/$basearch/' } ,meta: { module_hotfixes: 1 }}
      - { name: mongo          ,description: 'MongoDB'            ,module: mongo   ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/8.0/$basearch/' }}
      - { name: redis          ,description: 'Redis'              ,module: redis   ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://rpmfind.net/linux/remi/enterprise/$releasever/redis72/$basearch/' } ,meta: { module_hotfixes: 1 }}
      - { name: grafana        ,description: 'Grafana'            ,module: grafana ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://rpm.grafana.com', china: 'https://mirrors.cloud.tencent.com/grafana/yum/rpm/' }}
      - { name: kubernetes     ,description: 'Kubernetes'         ,module: kube    ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://pkgs.k8s.io/core:/stable:/v1.36/rpm/', china: 'https://mirrors.ustc.edu.cn/kubernetes/core:/stable:/v1.36/rpm/' }}
      - { name: gitlab-ee      ,description: 'Gitlab EE'          ,module: gitlab  ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://packages.gitlab.com/gitlab/gitlab-ee/el/$releasever/$basearch' }}
      - { name: gitlab-ce      ,description: 'Gitlab CE'          ,module: gitlab  ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://packages.gitlab.com/gitlab/gitlab-ce/el/$releasever/$basearch' }}
      - { name: clickhouse     ,description: 'ClickHouse'         ,module: click   ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://packages.clickhouse.com/rpm/stable/', china: 'https://repo.huaweicloud.com/clickhouse/rpm/stable/' }}

      # DEB 12/13 Ubuntu 22/24/26 REPOS
      - { name: pigsty-local   ,description: 'Pigsty Local'       ,module: local   ,releases: [11,12,13,22,24,26] ,arch: [x86_64, aarch64] ,baseurl: { default: 'http://${admin_ip}/pigsty ./' }}
      - { name: pigsty-pgsql   ,description: 'Pigsty PgSQL'       ,module: pgsql   ,releases: [11,12,13,22,24,26] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://repo.pigsty.io/apt/pgsql/${distro_codename} ${distro_codename} main' ,china: 'http://beta.pigsty.cc/apt/pgsql/${distro_codename} ${distro_codename} main' }}
      - { name: pigsty-infra   ,description: 'Pigsty Infra'       ,module: infra   ,releases: [11,12,13,22,24,26] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://repo.pigsty.io/apt/infra/ generic main'                              ,china: 'http://beta.pigsty.cc/apt/infra/ generic main' }}
      - { name: nginx          ,description: 'Nginx'              ,module: nginx   ,releases: [11,12,13,22,24,26] ,arch: [x86_64, aarch64] ,baseurl: { default: 'http://nginx.org/packages/${distro_name} ${distro_codename} nginx' }}
      - { name: docker-ce      ,description: 'Docker'             ,module: infra   ,releases: [11,12,13,22,24,26] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://download.docker.com/linux/${distro_name} ${distro_codename} stable'                               ,china: 'https://mirrors.cloud.tencent.com/docker-ce/linux/${distro_name} ${distro_codename} stable' }}
      - { name: base           ,description: 'Debian Basic'       ,module: node    ,releases: [11,12,13         ] ,arch: [x86_64, aarch64] ,baseurl: { default: 'http://deb.debian.org/debian/ ${distro_codename} main non-free-firmware'                                  ,china: 'https://mirrors.cloud.tencent.com/debian/ ${distro_codename} main non-free-firmware' }}
      - { name: updates        ,description: 'Debian Updates'     ,module: node    ,releases: [11,12,13         ] ,arch: [x86_64, aarch64] ,baseurl: { default: 'http://deb.debian.org/debian/ ${distro_codename}-updates main non-free-firmware'                          ,china: 'https://mirrors.cloud.tencent.com/debian/ ${distro_codename}-updates main non-free-firmware' }}
      - { name: security       ,description: 'Debian Security'    ,module: node    ,releases: [11,12,13         ] ,arch: [x86_64, aarch64] ,baseurl: { default: 'http://security.debian.org/debian-security ${distro_codename}-security main non-free-firmware'            ,china: 'https://mirrors.cloud.tencent.com/debian-security/ ${distro_codename}-security main non-free-firmware' }}
      - { name: base           ,description: 'Ubuntu Basic'       ,module: node    ,releases: [         22,24,26] ,arch: [x86_64         ] ,baseurl: { default: 'https://mirrors.edge.kernel.org/ubuntu/ ${distro_codename}           main universe multiverse restricted' ,china: 'https://mirrors.cloud.tencent.com/ubuntu/ ${distro_codename}           main restricted universe multiverse' }}
      - { name: updates        ,description: 'Ubuntu Updates'     ,module: node    ,releases: [         22,24,26] ,arch: [x86_64         ] ,baseurl: { default: 'https://mirrors.edge.kernel.org/ubuntu/ ${distro_codename}-updates   main restricted universe multiverse' ,china: 'https://mirrors.cloud.tencent.com/ubuntu/ ${distro_codename}-updates   main restricted universe multiverse' }}
      - { name: backports      ,description: 'Ubuntu Backports'   ,module: node    ,releases: [         22,24,26] ,arch: [x86_64         ] ,baseurl: { default: 'https://mirrors.edge.kernel.org/ubuntu/ ${distro_codename}-backports main restricted universe multiverse' ,china: 'https://mirrors.cloud.tencent.com/ubuntu/ ${distro_codename}-backports main restricted universe multiverse' }}
      - { name: security       ,description: 'Ubuntu Security'    ,module: node    ,releases: [         22,24,26] ,arch: [x86_64         ] ,baseurl: { default: 'https://mirrors.edge.kernel.org/ubuntu/ ${distro_codename}-security  main restricted universe multiverse' ,china: 'https://mirrors.cloud.tencent.com/ubuntu/ ${distro_codename}-security  main restricted universe multiverse' }}
      - { name: base           ,description: 'Ubuntu Basic'       ,module: node    ,releases: [         22,24,26] ,arch: [        aarch64] ,baseurl: { default: 'http://ports.ubuntu.com/ubuntu-ports/ ${distro_codename}             main universe multiverse restricted' ,china: 'https://mirrors.cloud.tencent.com/ubuntu-ports/ ${distro_codename}           main restricted universe multiverse' }}
      - { name: updates        ,description: 'Ubuntu Updates'     ,module: node    ,releases: [         22,24,26] ,arch: [        aarch64] ,baseurl: { default: 'http://ports.ubuntu.com/ubuntu-ports/ ${distro_codename}-updates     main restricted universe multiverse' ,china: 'https://mirrors.cloud.tencent.com/ubuntu-ports/ ${distro_codename}-updates   main restricted universe multiverse' }}
      - { name: backports      ,description: 'Ubuntu Backports'   ,module: node    ,releases: [         22,24,26] ,arch: [        aarch64] ,baseurl: { default: 'http://ports.ubuntu.com/ubuntu-ports/ ${distro_codename}-backports   main restricted universe multiverse' ,china: 'https://mirrors.cloud.tencent.com/ubuntu-ports/ ${distro_codename}-backports main restricted universe multiverse' }}
      - { name: security       ,description: 'Ubuntu Security'    ,module: node    ,releases: [         22,24,26] ,arch: [        aarch64] ,baseurl: { default: 'http://ports.ubuntu.com/ubuntu-ports/ ${distro_codename}-security    main restricted universe multiverse' ,china: 'https://mirrors.cloud.tencent.com/ubuntu-ports/ ${distro_codename}-security  main restricted universe multiverse' }}
      - { name: pgdg           ,description: 'PGDG'               ,module: pgsql   ,releases: [11,12,13,22,24,26] ,arch: [x86_64, aarch64] ,baseurl: { default: 'http://beta.pigsty.cc/apt/pgdg/ ${distro_codename}-pgdg main' ,china: 'https://mirrors.cloud.tencent.com/postgresql/repos/apt/ ${distro_codename}-pgdg main' }}
      - { name: pgdg-beta      ,description: 'PGDG Beta'          ,module: beta    ,releases: [11,12,13,22,24,26] ,arch: [x86_64, aarch64] ,baseurl: { default: 'http://beta.pigsty.cc/apt/pgdg/ ${distro_codename}-pgdg-testing main 19' ,china: 'https://mirrors.cloud.tencent.com/postgresql/repos/apt/ ${distro_codename}-pgdg-testing main 19' }}
      - { name: timescaledb    ,description: 'TimescaleDB'        ,module: extra   ,releases: [11,12,13,22,24,26] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://packagecloud.io/timescale/timescaledb/${distro_name}/ ${distro_codename} main' }}
      - { name: citus          ,description: 'Citus'              ,module: extra   ,releases: [11,12,   22      ] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://packagecloud.io/citusdata/community/${distro_name}/ ${distro_codename} main' } }
      - { name: percona        ,description: 'Percona TDE'        ,module: percona ,releases: [   12,13,22,24,26] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://repo.pigsty.io/apt/percona ${distro_codename} main' ,china: 'http://beta.pigsty.cc/apt/percona ${distro_codename} main' }}
      - { name: groonga        ,description: 'Groonga Debian'     ,module: groonga ,releases: [11,12,13         ] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://packages.groonga.org/debian/ ${distro_codename} main' }}
      - { name: groonga        ,description: 'Groonga Ubuntu'     ,module: groonga ,releases: [         22,24   ] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://ppa.launchpadcontent.net/groonga/ppa/ubuntu/ ${distro_codename} main' }}
      - { name: mysql          ,description: 'MySQL 8.4 LTS'      ,module: mysql   ,releases: [   12,13,22,24   ] ,arch: [x86_64         ] ,baseurl: { default: 'https://repo.mysql.com/apt/${distro_name} ${distro_codename} mysql-8.4-lts' ,china: 'https://mirrors.ustc.edu.cn/mysql-repo/apt/${distro_name} ${distro_codename} mysql-8.4-lts' }}
      - { name: mongo          ,description: 'MongoDB'            ,module: mongo   ,releases: [   12,   22,24   ] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://repo.mongodb.org/apt/${distro_name} ${distro_codename}/mongodb-org/8.0 multiverse' ,china: 'https://mirrors.cloud.tencent.com/mongodb/apt/${distro_name} ${distro_codename}/mongodb-org/8.0 multiverse' }}
      - { name: redis          ,description: 'Redis'              ,module: redis   ,releases: [11,12,   22,24,26] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://packages.redis.io/deb ${distro_codename} main' }}
      - { name: llvm           ,description: 'LLVM'               ,module: llvm    ,releases: [11,12,13,22,24,26] ,arch: [x86_64, aarch64] ,baseurl: { default: 'http://apt.llvm.org/${distro_codename}/ llvm-toolchain-${distro_codename} main' ,china: 'https://mirrors.tuna.tsinghua.edu.cn/llvm-apt/${distro_codename}/ llvm-toolchain-${distro_codename} main' }}
      - { name: haproxyd       ,description: 'Haproxy Debian'     ,module: haproxy ,releases: [   12            ] ,arch: [x86_64, aarch64] ,baseurl: { default: 'http://haproxy.debian.net/ ${distro_codename}-backports-3.2 main' }}
      - { name: haproxyu       ,description: 'Haproxy Ubuntu'     ,module: haproxy ,releases: [            24,26] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://ppa.launchpadcontent.net/vbernat/haproxy-3.2/ubuntu/ ${distro_codename} main' }}
      - { name: grafana        ,description: 'Grafana'            ,module: grafana ,releases: [11,12,13,22,24,26] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://apt.grafana.com stable main' ,china: 'https://mirrors.cloud.tencent.com/grafana/apt/ stable main' }}
      - { name: kubernetes     ,description: 'Kubernetes'         ,module: kube    ,releases: [11,12,13,22,24,26] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://pkgs.k8s.io/core:/stable:/v1.36/deb/ /', china: 'https://mirrors.ustc.edu.cn/kubernetes/core:/stable:/v1.36/deb/ /' }}
      - { name: gitlab-ee      ,description: 'Gitlab EE'          ,module: gitlab  ,releases: [11,12,13,22,24   ] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://packages.gitlab.com/gitlab/gitlab-ee/${distro_name}/ ${distro_codename} main' }}
      - { name: gitlab-ce      ,description: 'Gitlab CE'          ,module: gitlab  ,releases: [11,12,13,22,24   ] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://packages.gitlab.com/gitlab/gitlab-ce/${distro_name}/ ${distro_codename} main' }}
      - { name: clickhouse     ,description: 'ClickHouse'         ,module: click   ,releases: [11,12,13,22,24,26] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://packages.clickhouse.com/deb/ stable main', china: 'https://repo.huaweicloud.com/clickhouse/deb/ stable main' }}

...

Explanation

build/dev is mainly used to validate the Pigsty software repository build pipeline, not for ordinary production installation.

Key Features:

  • Default pg_version: 18
  • Local cache directory is dist/${version}
  • Builds infra,node,pgsql modules by default
  • Preloads PostgreSQL 18 full-category extension package groups
  • Covers both RPM and DEB build paths through three distro nodes

Use Cases:

  • Pigsty new version build validation
  • Software repository and mirror source debugging
  • Extension package download and cache testing

6.37 - demo/remote

Monitor remote PostgreSQL and cloud RDS with pg_exporter instances on an INFRA node

demo/remote deploys no local PostgreSQL cluster. Instead, it declares multiple pg_exporters on an INFRA node to monitor remote PostgreSQL, PolarDB, or cloud RDS instances.

Overview

  • Config Name: demo/remote
  • Local Node Count: One INFRA node
  • Example Exporter Ports: 20001-20016
  • Related: PG Exporter
./configure -c demo/remote [-i <infra_ip>]

Content

Source: pigsty/conf/demo/remote.yml

---
#==============================================================#
# File      :   remote.yml
# Desc      :   Monitoring Remote RDS with pigsty
# Ctime     :   2020-05-22
# Mtime     :   2025-12-12
# Docs      :   https://pigsty.io/docs/conf
# License   :   Apache-2.0 @ https://pigsty.io/docs/about/license/
# Copyright :   2018-2026  Ruohang Feng / Vonng ([email protected])
#==============================================================#

all:
  children:

    infra:            # infra cluster for proxy, monitor, alert, etc..
      hosts: { 10.10.10.10: { infra_seq: 1 } }
      vars:           # install pg_exporter for remote postgres RDS on a group 'infra'
        pg_exporters: # list all remote instances here, alloc a unique unused local port as k
          20001: { pg_cluster: pg-foo, pg_seq: 1, pg_host: 10.10.10.10 }
          20002: { pg_cluster: pg-bar, pg_seq: 1, pg_host: 10.10.10.11 , pg_port: 5432 }
          20003: { pg_cluster: pg-bar, pg_seq: 2, pg_host: 10.10.10.12 , pg_exporter_url: 'postgres://dbuser_monitor:[email protected]:5432/postgres?sslmode=disable'}
          20004: { pg_cluster: pg-bar, pg_seq: 3, pg_host: 10.10.10.13 , pg_monitor_username: dbuser_monitor, pg_monitor_password: DBUser.Monitor }

          20011:
            pg_cluster: pg-polar                        # RDS Cluster Name (Identity, Explicitly Assigned, used as 'cls')
            pg_seq: 1                                   # RDS Instance Seq (Identity, Explicitly Assigned, used as part of 'ins')
            pg_host: pxx.polardbpg.rds.aliyuncs.com     # RDS Host Address
            pg_port: 1921                               # RDS Port
            pg_exporter_include_database: 'test'        # Only monitoring database in this list
            pg_monitor_username: dbuser_monitor         # monitor username, overwrite default
            pg_monitor_password: DBUser_Monitor         # monitor password, overwrite default
            pg_databases: [{ name: test }]              # database to be added to grafana datasource

          20012:
            pg_cluster: pg-polar                        # RDS Cluster Name (Identity, Explicitly Assigned, used as 'cls')
            pg_seq: 2                                   # RDS Instance Seq (Identity, Explicitly Assigned, used as part of 'ins')
            pg_host: pe-xx.polarpgmxs.rds.aliyuncs.com  # RDS Host Address
            pg_port: 1521                               # RDS Port
            pg_databases: [{ name: test }]              # database to be added to grafana datasource

          20014:
            pg_cluster: pg-rds
            pg_seq: 1
            pg_host: pgm-xx.pg.rds.aliyuncs.com
            pg_port: 5432
            pg_exporter_auto_discovery: true
            pg_exporter_include_database: 'rds'
            pg_monitor_username: dbuser_monitor
            pg_monitor_password: DBUser_Monitor
            pg_databases: [ { name: rds } ]

          20015:
            pg_cluster: pg-rdsha
            pg_seq: 1
            pg_host: pgm-2xx8wu.pg.rds.aliyuncs.com
            pg_port: 5432
            pg_exporter_auto_discovery: true
            pg_exporter_include_database: 'rds'
            pg_databases: [{ name: test }, {name: rds}]

          20016:
            pg_cluster: pg-rdsha
            pg_seq: 2
            pg_host: pgr-xx.pg.rds.aliyuncs.com
            pg_exporter_auto_discovery: true
            pg_exporter_include_database: 'rds'
            pg_databases: [{ name: test }, {name: rds}]
  
  
  vars:
    version: v4.5.0                   # pigsty version string
    admin_ip: 10.10.10.10             # admin node ip address
    region: default                   # upstream mirror region: default,china,europe

    #----------------------------------------------#
    # PASSWORD : https://pigsty.io/docs/setup/security/
    #----------------------------------------------#
    grafana_admin_password: pigsty
    grafana_view_password: DBUser.Viewer
    pg_admin_password: DBUser.DBA
    pg_monitor_password: DBUser.Monitor
    pg_replication_password: DBUser.Replicator
    patroni_password: Patroni.API
    haproxy_admin_password: pigsty
    minio_secret_key: S3User.MinIO
    etcd_root_password: Etcd.Root
...

Explanation

Each pg_exporters entry uses a unique local listen port and declares the remote instance’s pg_cluster, pg_seq, pg_host, and optional connection settings. The template demonstrates complete URLs, split credentials, database allowlists, and auto-discovery.

All hostnames and credentials are placeholders. Keep only the entries you need, use a least-privilege monitoring account, and never commit real RDS passwords.

6.38 - demo/saas

Legacy single-node SaaS bundle with PostgreSQL, Silo, Redis, and multiple application entrypoints

demo/saas is a legacy feature-rich single-node example with predefined business users, databases, and application entrypoints. It demonstrates how PostgreSQL, Silo, Redis, Docker, and the portal can be combined.

Overview

  • Config Name: demo/saas
  • Node Count: Single node
  • Modules: INFRA, ETCD, MINIO, PGSQL, REDIS, DOCKER
  • Related: rich, supabase
./configure -c demo/saas [-i <primary_ip>]

Content

Source: pigsty/conf/demo/saas.yml

---
#==============================================================#
# File      :   saas.yml (1-node)
# Desc      :   Feature rich 1-node template with all extensions
# Ctime     :   2020-05-22
# Mtime     :   2025-12-12
# Docs      :   https://pigsty.io/docs/conf
# License   :   Apache-2.0 @ https://pigsty.io/docs/about/license/
# Copyright :   2018-2026  Ruohang Feng / Vonng ([email protected])
#==============================================================#


all:

  #==============================================================#
  # Clusters, Nodes, and Modules
  #==============================================================#
  children:

    #----------------------------------#
    # infra: monitor, alert, repo, etc..
    #----------------------------------#
    infra:
      hosts:
        10.10.10.10: { infra_seq: 1 }
      vars:
        docker_enabled: true      # enabled docker with ./docker.yml
        #docker_registry_mirrors: ["https://docker.1panel.live","https://docker.1ms.run","https://docker.xuanyuan.me","https://registry-1.docker.io"]

    #----------------------------------#
    # etcd cluster for HA postgres DCS
    #----------------------------------#
    etcd:
      hosts:
        10.10.10.10: { etcd_seq: 1 }
      vars:
        etcd_cluster: etcd

    #----------------------------------#
    # minio (OPTIONAL backup repo)
    #----------------------------------#
    minio:
      hosts:
        10.10.10.10: { minio_seq: 1 }
      vars:
        minio_cluster: minio
        minio_users:                      # list of minio user to be created
          - { access_key: pgbackrest  ,secret_key: S3User.Backup ,policy: pgsql }
          - { access_key: s3user_meta ,secret_key: S3User.Meta   ,policy: meta  }
          - { access_key: s3user_data ,secret_key: S3User.Data   ,policy: data  }

    #----------------------------------#
    # pgsql (singleton on current node)
    #----------------------------------#
    # postgres cluster: pg-meta
    pg-meta:
      hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
      vars:
        pg_cluster: pg-meta
        pg_users:
          - {name: dbuser_meta     ,password: DBUser.Meta     ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: pigsty admin user }
          - {name: dbuser_view     ,password: DBUser.Viewer   ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer for meta database }
          - {name: dbuser_grafana  ,password: DBUser.Grafana  ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for grafana database    }
          - {name: dbuser_bytebase ,password: DBUser.Bytebase ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for bytebase database   }
          - {name: dbuser_kong     ,password: DBUser.Kong     ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for kong api gateway    }
          - {name: dbuser_gitea    ,password: DBUser.Gitea    ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for gitea service       }
          - {name: dbuser_wiki     ,password: DBUser.Wiki     ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for wiki.js service     }
          - {name: dbuser_noco     ,password: DBUser.Noco     ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for nocodb service      }
          - {name: dbuser_odoo     ,password: DBUser.Odoo     ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for odoo service ,createdb: true} #,superuser: true}
        pg_databases:
          - {name: meta ,baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty] ,extensions: [{name: vector},{name: postgis},{name: timescaledb}]}
          - {name: grafana  ,owner: dbuser_grafana  ,revokeconn: true ,comment: grafana primary database  }
          - {name: bytebase ,owner: dbuser_bytebase ,revokeconn: true ,comment: bytebase primary database }
          - {name: kong     ,owner: dbuser_kong     ,revokeconn: true ,comment: kong api gateway database }
          - {name: gitea    ,owner: dbuser_gitea    ,revokeconn: true ,comment: gitea meta database }
          - {name: wiki     ,owner: dbuser_wiki     ,revokeconn: true ,comment: wiki meta database  }
          - {name: noco     ,owner: dbuser_noco     ,revokeconn: true ,comment: nocodb database     }
          #- {name: odoo     ,owner: dbuser_odoo     ,revokeconn: true ,comment: odoo main database  }
        pg_hba_rules:
          - {user: dbuser_view , db: all ,addr: infra ,auth: pwd ,title: 'allow grafana dashboard access cmdb from infra nodes'}
        pg_libs: 'timescaledb,pg_stat_statements, auto_explain'  # add timescaledb to shared_preload_libraries
        node_crontab:  # make one full backup 1 am everyday
          - '00 01 * * * /pg/bin/pg-backup full'

    redis-ms: # redis classic primary & replica
      hosts: { 10.10.10.10: { redis_node: 1 , redis_instances: { 6379: { }, 6380: { replica_of: '10.10.10.10 6379' } } } }
      vars: { redis_cluster: redis-ms ,redis_password: 'redis.ms' ,redis_max_memory: 64MB }


  vars:                               # global variables
    version: v4.5.0                   # pigsty version string
    admin_ip: 10.10.10.10             # admin node ip address
    region: default                   # upstream mirror region: default|china|europe
    node_tune: oltp                   # node tuning specs: oltp,olap,tiny,crit
    pg_conf: oltp.yml                 # pgsql tuning specs: {oltp,olap,tiny,crit}.yml
    proxy_env:                        # global proxy env when downloading packages
      no_proxy: "localhost,127.0.0.1,10.0.0.0/8,192.168.0.0/16,*.pigsty,*.aliyun.com,mirrors.*,*.myqcloud.com,*.tsinghua.edu.cn"
      # http_proxy:  # set your proxy here: e.g http://user:[email protected]
      # https_proxy: # set your proxy here: e.g http://user:[email protected]
      # all_proxy:   # set your proxy here: e.g http://user:[email protected]
    infra_portal:                     # infra services exposed via portal
      home         : { domain: i.pigsty }     # default domain name
      minio        : { domain: m.pigsty    ,endpoint: "${admin_ip}:9001" ,scheme: https ,websocket: true }
      postgrest    : { domain: api.pigsty  ,endpoint: "127.0.0.1:8884" }
      pgadmin      : { domain: adm.pigsty  ,endpoint: "127.0.0.1:8885" }
      pgweb        : { domain: cli.pigsty  ,endpoint: "127.0.0.1:8886" }
      bytebase     : { domain: ddl.pigsty  ,endpoint: "127.0.0.1:8887" }
      jupyter      : { domain: lab.pigsty  ,endpoint: "127.0.0.1:8888", websocket: true }
      gitea        : { domain: git.pigsty  ,endpoint: "127.0.0.1:8889" }
      wiki         : { domain: wiki.pigsty ,endpoint: "127.0.0.1:9002" }
      noco         : { domain: noco.pigsty ,endpoint: "127.0.0.1:9003" }
      supa         : { domain: supa.pigsty ,endpoint: "10.10.10.10:8000", websocket: true }
      dify         : { domain: dify.pigsty ,endpoint: "10.10.10.10:8001", websocket: true }
      odoo         : { domain: odoo.pigsty, endpoint: "127.0.0.1:8069"  , websocket: true }

    #----------------------------------#
    # MinIO Related Options
    #----------------------------------#
    pgbackrest_method: minio          # use minio as backup repo instead of 'local'
    pgbackrest_repo:                  # pgbackrest repo: https://pgbackrest.org/configuration.html#section-repository
      local:                          # default pgbackrest repo with local posix fs
        path: /pg/backup              # local backup directory, `/pg/backup` by default
        retention_full_type: count    # retention full backups by count
        retention_full: 2             # keep 2, at most 3 full backup when using local fs repo
      minio:                          # optional minio repo for pgbackrest
        type: s3                      # minio is s3-compatible, so s3 is used
        s3_endpoint: sss.pigsty       # minio endpoint domain name, `sss.pigsty` by default
        s3_region: us-east-1          # minio region, us-east-1 by default, useless for minio
        s3_bucket: pgsql              # minio bucket name, `pgsql` by default
        s3_key: pgbackrest            # minio user access key for pgbackrest
        s3_key_secret: S3User.Backup  # minio user secret key for pgbackrest
        s3_uri_style: path            # use path style uri for minio rather than host style
        path: /pgbackrest             # minio backup path, default is `/pgbackrest`
        storage_port: 9000            # minio port, 9000 by default
        storage_ca_file: /etc/pki/ca.crt  # minio ca file path, `/etc/pki/ca.crt` by default
        block: y                      # Enable block incremental backup
        bundle: y                     # bundle small files into a single file
        bundle_limit: 20MiB           # Limit for file bundles, 20MiB for object storage
        bundle_size: 128MiB           # Target size for file bundles, 128MiB for object storage
        cipher_type: aes-256-cbc      # enable AES encryption for remote backup repo
        cipher_pass: pgBackRest       # AES encryption password, default is 'pgBackRest'
        retention_full_type: time     # retention full backup by time on minio repo
        retention_full: 14            # keep full backup for last 14 days
    node_etc_hosts: [ "${admin_ip} i.pigsty sss.pigsty" ]
    dns_records: [ "${admin_ip} api.pigsty adm.pigsty cli.pigsty ddl.pigsty lab.pigsty git.pigsty wiki.pigsty noco.pigsty supa.pigsty dify.pigsty odoo.pigsty" ]

    #----------------------------------#
    # Safe Guard
    #----------------------------------#
    # you can enable these flags after bootstrap, to prevent purging running etcd / pgsql instances
    etcd_safeguard: false             # prevent purging running etcd instance?
    pg_safeguard: false               # prevent purging running postgres instance? false by default

    #----------------------------------#
    # Repo, Node, Packages
    #----------------------------------#
    repo_remove: true                 # remove existing repo on admin node during repo bootstrap
    node_repo_remove: true            # remove existing node repo for node managed by pigsty
    repo_extra_packages: [ pg17-core ,pg18-time ,pg18-gis ,pg18-rag ,pg18-fts ,pg18-olap ,pg18-feat ,pg18-lang ,pg18-type ,pg18-util ,pg18-func ,pg18-admin ,pg18-stat ,pg18-sec ,pg18-fdw ,pg18-sim ,pg18-etl]
    pg_version: 18                    # default postgres version
    #pg_extensions: [ pg18-time ,pg18-gis ,pg18-rag ,pg18-fts ,pg18-olap ,pg18-feat ,pg18-lang ,pg18-type ,pg18-util ,pg18-func ,pg18-admin ,pg18-stat ,pg18-sec ,pg18-fdw ,pg18-sim ,pg18-etl]

    #----------------------------------------------#
    # PASSWORD : https://pigsty.io/docs/setup/security/
    #----------------------------------------------#
    grafana_admin_password: pigsty
    grafana_view_password: DBUser.Viewer
    pg_admin_password: DBUser.DBA
    pg_monitor_password: DBUser.Monitor
    pg_replication_password: DBUser.Replicator
    patroni_password: Patroni.API
    haproxy_admin_password: pigsty
    minio_secret_key: S3User.MinIO
    etcd_root_password: Etcd.Root
...

Explanation

The template contains placeholder database users and databases for Grafana, Bytebase, Kong, Gitea, Wiki, NocoDB, and Odoo. It uses Silo as the pgBackRest repository and includes a Redis replica example and multiple portal domains.

This compatibility/reference bundle does not install every listed application automatically. For new deployments, prefer rich plus the relevant app/* template. Remove unused users, databases, and entrypoints and replace all passwords first.

6.39 - demo/wool

Single-node tiny-tuning example for small cloud instances in China

demo/wool targets small cloud instances in China and defaults to region: china, PostgreSQL 18, and the tiny tuning profiles.

Overview

  • Config Name: demo/wool
  • Node Count: Single node
  • Suggested Size: Approximately 2 vCPU / 2 GB for testing
  • Related: meta, slim
./configure -c demo/wool [-i <private_ip>]

Content

Source: pigsty/conf/demo/wool.yml

---
#==============================================================#
# File      :   wool.yml
# Desc      :   Pigsty Aliyun ECS 羊毛机配置文件
# Ctime     :   2020-11-09
# Mtime     :   2025-12-12
# Docs      :   https://pigsty.io/docs/conf
# License   :   Apache-2.0 @ https://pigsty.io/docs/about/license/
# Copyright :   2018-2026  Ruohang Feng / Vonng ([email protected])
#==============================================================#

all:
  children:

    # 建议使用操作系统： RockyLinux 9.4
    # 这里的 10.10.10.10 都应该是你 ECS 的内网 IP 地址，用于安装 Infra/Etcd 模块
    infra: { hosts: { 10.10.10.10: { infra_seq: 1 } } }
    etcd:  { hosts: { 10.10.10.10: { etcd_seq: 1 } }, vars: { etcd_cluster: etcd } }

    # 定义一个单节点的 PostgreSQL 数据库实例
    pg-meta:
      hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
      vars:
        pg_cluster: pg-meta
        pg_databases:
          - { name: meta ,baseline: cmdb.sql ,schemas: [ pigsty ] }
        pg_users: # 最好把这里的两个样例用户的密码也修改一下
          - { name: dbuser_meta ,password: DBUser.Meta   ,roles: [ dbrole_admin ] }
          - { name: dbuser_view ,password: DBUser.Viewer ,roles: [ dbrole_readonly ] }
        pg_conf: tiny.yml   # 2C/2G 的云服务器，使用微型数据库配置模板
        node_tune: tiny     # 2C/2G 的云服务器，使用微型主机节点参数优化模板
        pgbackrest_enabled: false # 这么点磁盘空间，就别搞数据库物理备份了
        pg_version: 18           # 用 PostgreSQL 18

  vars:
    version: v4.5.0                   # pigsty version string
    region: china
    admin_ip: 10.10.10.10  # 这个 IP 地址应该是你 ECS 的内网IP地址
    infra_portal: # 如果你有自己的 DNS 域名，这里面的域名后缀 pigsty 换成你自己的 DNS 域名
      home : { domain: i.pigsty }     # default domain name
      minio: { domain: m.pigsty  ,endpoint: "${admin_ip}:9001" ,scheme: https ,websocket: true }
      postgrest: { domain: api.pigsty  ,endpoint: "127.0.0.1:8884" }
      pgadmin: { domain: adm.pigsty  ,endpoint: "127.0.0.1:8885" }
      pgweb: { domain: cli.pigsty  ,endpoint: "127.0.0.1:8886" }
      bytebase: { domain: ddl.pigsty  ,endpoint: "127.0.0.1:8887" ,websocket: true }
      jupyter: { domain: lab.pigsty  ,endpoint: "127.0.0.1:8888", websocket: true }
      gitea: { domain: git.pigsty  ,endpoint: "127.0.0.1:8889" }
      wiki: { domain: wiki.pigsty ,endpoint: "127.0.0.1:9002" }
      noco: { domain: noco.pigsty ,endpoint: "127.0.0.1:9003" }
      supa: { domain: supa.pigsty ,endpoint: "10.10.10.10:8000", websocket: true }

    # 把这里的密码都改掉！你也不想别人随便来串门对吧！
    #----------------------------------------------#
    # PASSWORD : https://pigsty.io/docs/setup/security/
    #----------------------------------------------#
    grafana_admin_password: pigsty
    grafana_view_password: DBUser.Viewer
    pg_admin_password: DBUser.DBA
    pg_monitor_password: DBUser.Monitor
    pg_replication_password: DBUser.Replicator
    patroni_password: Patroni.API
    haproxy_admin_password: pigsty
    minio_secret_key: S3User.MinIO
    etcd_root_password: Etcd.Root
...

Explanation

  • Explicitly sets pg_conf: tiny.yml and node_tune: tiny on pg-meta
  • Expects 10.10.10.10 to be replaced by the cloud instance’s private IP
  • Disables pgBackRest by default to reduce disk usage on a small test host
  • Includes several example portal domains

This template trades backup capability for lower resource use and is suitable only for temporary testing. Production deployments must enable and verify backups, tighten network rules, replace default passwords, and remove unused portal entries.

7 - Operations SOP Index

Pigsty and PostgreSQL operations documentation index for new users: find the right page by common task.

Getting Started Path

Order Question Entry
1 What modules does Pigsty include? Modular Architecture, PGSQL Architecture, PGSQL Cluster Model
2 How do I get it running first? Quick Start, Web UI, PostgreSQL Quick Start
3 How should I read the config file? Declarative Configuration, Configuration Guide, Configuration Parameters
4 What should I prepare for production? Planning, Preparation, Administration Model
5 How do I deploy a multi-node cluster? Production Deployment, Playbooks, PGSQL Playbooks
6 How do I operate databases daily? PGSQL Administration, Cluster Management, User Management, Database Management
7 How do I validate reliability? PostgreSQL HA, Patroni Management, Backup & Restore, Restore Operations

Task Index

Task Read First Operation Entry
Prepare servers, disks, networks, VIPs Preparation, Planning, Linux Compatibility Production Deployment
Prepare SSH, sudo, and admin users Administration Model Production Deployment
Build a local or cloud sandbox Sandbox Vagrant, Terraform
Single-node trial Quick Start ./configure -g, ./deploy.yml
Multi-node production deployment Deployment, Production Deployment ./deploy.yml, ./pgsql.yml
Deploy in an offline environment Offline Installation Repository Management
Choose a config template Config Templates, Template List ./configure -c <template>
Plan cluster, database, and user names PGSQL Cluster Model pg_cluster, pg_databases, pg_users
Create a PostgreSQL cluster Cluster Instance Config Cluster Management, ./pgsql.yml -l <cluster>
Add business users User/Role Config User Management, ./pgsql-user.yml -l <cluster>
Add business databases Database Config Database Management, ./pgsql-db.yml -l <cluster>
Configure access endpoints Service/Access pg_services, pg_default_services
Modify HBA HBA Config HBA Management
Switchover Patroni Management patronictl switchover
HA drills PostgreSQL HA, RPO, RTO 3-of-2 Failure Drill
Configure VIP HA Service Access Configure PG VIP
Configure backup policy Backup Policy Backup Admin Commands
Perform PITR Point-in-Time Recovery Restore Operations
Recover dropped data, tables, or databases Drop Recovery Manual Recovery
Clone or fork a cluster Clone Database Cluster Fork Instance
Use Silo for backups MINIO Module Silo Config, Backup Repository
View monitoring and alerts Monitoring System PGSQL Monitoring, PGSQL Dashboards
Troubleshoot database failures PGSQL FAQ Troubleshooting, Component Management
Scale PostgreSQL clusters Cluster Instance Config Cluster Management
Upgrade PostgreSQL Version Upgrade Kernel Versions
Install or enable extensions Extensions Extension Management
Migrate existing databases Data Migration Migration Playbook
Harden security Security Considerations Access Control, CA & Certificates
Manage domains and web entrypoints Domain Management Nginx Management
Maintain infrastructure INFRA Administration infra.yml, infra-rm.yml
Maintain Etcd ETCD Config ETCD Management, ETCD FAQ
Deploy app templates Applications Docker Module, ./app.yml

Preparation And Deployment

For production deployment, start with Planning and Preparation. These two pages cover node count, disks, filesystems, networks, VIPs, domains, and software sources.

After machines are ready, read Administration Model: admin users, passwordless SSH, sudo, reachability, and firewall handling are covered there. Check system versions and architectures in Linux Compatibility.

Use Quick Start for the first installation. Use Production Deployment for multi-node production environments. If there is no internet access, read Offline Installation and Repository Management.

Do not overthink template choice at the beginning: use meta for single-node default deployment; ha/trio for three-node HA; ha/full for more complete HA; ha/safe when consistency matters; and ha/dual or ha/simu when resources are tight.

Naming And Configuration

First distinguish three names: cluster name, database name, and service name.

pg_cluster is the top-level name Pigsty uses to manage a PostgreSQL cluster. It affects instance names, service names, backup stanzas, monitoring labels, and many file paths. It is not a display name that can be casually changed. See PGSQL Cluster Model for naming rules, Cluster Instance Config for instance roles, and Service/Access for service names and connection entrypoints.

Database names and user names are logical objects inside PostgreSQL. See Database Config and Database Management for databases; User/Role Config and User Management for users and roles; and Access Control plus ACL Config for the privilege model.

In practice, use lowercase letters, digits, and hyphens for cluster names, such as pg-meta, pg-test, and pg-user-prod. Use snake_case for database object names. Avoid non-ASCII names, spaces, mixed case, and SQL keywords. For more background, read Database Cluster Management Concepts and Entity Naming Rules and PostgreSQL Convention (2024 Edition).

Follow one habit for config changes: edit pigsty.yml first, then run the corresponding playbook. See Declarative Configuration and Configuration Guide for config structure; Configuration Parameters and Parameter List for parameter meanings; Playbooks and Playbook List for playbook entrypoints.

Daily Operations

The main entrypoint for database operations is PGSQL Administration.

Operation Documentation
Create, scale, shrink, retire, and clone clusters Cluster Management
Create, modify, and remove business users User Management
Create, modify, drop, and rebuild databases Database Management
Refresh and troubleshoot HBA HBA Management
View HA status, switchover, restart, and reinitialize replicas Patroni Management
Manage connection pools Pgbouncer Management
Start and stop PostgreSQL, Patroni, Pgbouncer, and Exporter Component Management
Manage backups, checks, cleanup, and restore Backup & Restore
Configure scheduled backup, vacuum, analyze, and other tasks Crontab
Upgrade versions and extensions Version Upgrade, Extension Management

For background reading, see Routine PostgreSQL Maintenance.

High Availability Drills

To understand HA, start with PostgreSQL HA. Do not only ask “can it fail over automatically”; also read RPO and RTO: the former is the maximum acceptable data loss, and the latter is the time to restore service.

For access-layer behavior, see HA Service Access and Service/Access. For component relationships, see PGSQL Architecture. For Etcd’s role, see ETCD Config.

Drill entrypoints are concentrated in three places: Patroni Management for planned switchover; Component Management for service status; and 3-of-2 Failure Drill for extreme failures. If you need VIP, read Configure PG VIP.

Background article: How Should PostgreSQL High Availability Be Done?.

Backup And Recovery

For PITR, read Point-in-Time Recovery first, then Mechanism, Architecture, Tradeoffs, and Scenarios.

For configuration and maintenance, see Backup & Restore, Backup Policy, Backup Mechanism, Backup Repository, and Backup Admin Commands.

When actually restoring, use Restore Operations for the automatic path and Manual Recovery for drills. For dropped data, tables, or databases, see Drop Recovery. If you do not want to touch the original cluster directly, start with Clone Database Cluster or Fork Instance.

Before recovery, confirm at least four things: the target timestamp or restore point is clear; backup and WAL are continuous; business writes have stopped; and you know whether you are restoring in place or first pulling up a new validation cluster.

Background articles: Overview of Backup and Recovery Methods and PgBackRest2 Documentation.

Monitoring And Troubleshooting

For monitoring overview, see Monitoring System. For entrypoints and domains, see Web UI. For database metrics, logs, and alerts, see PGSQL Monitoring and PGSQL Dashboards.

Monitoring for non-database modules is documented separately: INFRA Monitoring, NODE Monitoring, ETCD Monitoring, and MINIO Monitoring.

Start troubleshooting with PGSQL FAQ, then Troubleshooting. For connection authentication issues, see HBA Management; for HA status issues, see Patroni Management; for process state issues, see Component Management.

General PostgreSQL troubleshooting articles: Routine PG Server Logging Configuration, Macro Query Optimization with pg_stat_statements, Incident File: PostgreSQL Transaction ID Wraparound, Finding Fake Indexes, and Table Bloat Cleanup.

Scaling, Upgrades, And Migration

For capacity and topology design, see Planning, Preparation, and PGSQL Cluster Model.

When scaling by module, see Cluster Management for PGSQL; NODE Management for NODE; ETCD Management for ETCD; MINIO Management for MINIO; INFRA Administration for INFRA; and REDIS Management for REDIS.

For PostgreSQL upgrades, see Version Upgrade and Kernel Versions. For extensions, see Extensions, Extension Management, Extension Repository, and Package Aliases.

For migrating existing PostgreSQL databases, see Data Migration and PGSQL Migration Playbook. For low-downtime migration ideas, see Migration without Downtime.

If you need horizontal scaling, then read Citus Cluster Deployment and Citus Kernel Branch.

Security And Entrypoints

For deployment security, start with Security Considerations. For the security model, see Security and Compliance. For PostgreSQL privileges, see Access Control and ACL Config; for authentication rules, see Authentication, HBA Config, and HBA Management.

For certificates, see CA & Certificates. For domains, Nginx, and web entrypoints, see Domain Management and Nginx Management.

For production, at minimum change default passwords, tighten HBA, clearly separate business users from admin users, and confirm backup repository retention, encryption, and access permissions.

Application Access

Before applications connect to databases, read Service/Access and PostgreSQL Quick Start. For connection pool behavior, see Pgbouncer Management.

When using Pigsty-managed databases and deploying stateless applications, see Application Templates and Docker Module.

Common Mistakes

Mistake Where To Read
Treating pg_cluster as a casually changeable display name PGSQL Cluster Model
Confusing database names, cluster names, and service names Naming And Configuration, Service/Access
Deploying only the primary without restore drills Manual Recovery, Restore Operations
Assuming HA always means zero data loss RPO, RTO
Doing the first failover drill directly in production Sandbox, 3-of-2 Failure Drill
Ignoring Etcd ETCD Module, ETCD FAQ
Checking backup success without verifying restore Backup & Restore, Clone Database Cluster
Changing HBA, certificates, or service entrypoints without a rollback path Security Compliance, HBA Management, Nginx Management

Further Reading

Topic Article
Naming and entity model Database Cluster Management Concepts and Entity Naming Rules
PostgreSQL conventions PostgreSQL Convention (2024 Edition)
High availability How Should PostgreSQL High Availability Be Done?
Backup and recovery Overview of Backup and Recovery Methods, PgBackRest2 Documentation
Routine maintenance Routine PostgreSQL Maintenance
Connection pooling Pgbouncer Quick Start
Query and workload Macro Query Optimization with pg_stat_statements, PostgreSQL KPIs
Logging and incidents Routine PG Server Logging Configuration, Incident File: PostgreSQL Transaction ID Wraparound
Ecosystem and extensions PostgreSQL Is Eating the Database World, Pig: Package Manager for PostgreSQL Kernels and Extensions

8 - Module: PGSQL

Declare, deploy, expose, observe, back up, and manage PostgreSQL clusters with Pigsty v4.5.

PGSQL is Pigsty’s core module. Ansible inventory declares PostgreSQL clusters; Patroni and etcd provide HA orchestration; pgBackRest provides backup and PITR; HAProxy, VIP, DNS, PgBouncer, and the observability stack expose and monitor database services.

This page follows the Pigsty v4.5.0 source layout. Exact defaults live only in the parameter reference so the module landing page does not become a second stale parameter snapshot.

Modeling and Configuration

  • Cluster model: clusters, instances, identity, and roles.
  • Architecture: relationships among Patroni, etcd, service access, and observability.
  • Cluster configuration: primary, replica, offline instance, synchronous commit, standby cluster, delayed cluster, and Citus.
  • Kernel: PostgreSQL major version, distribution, and package selection.
  • Users, databases, HBA, and ACL: business objects and access control.
  • Service access: read-write/read-only services, HAProxy, VIP, DNS, and pooling.
  • Extension catalog: the current 575 packaged extensions and platform coverage.

Deployment and Administration

Task Entry point
Initialize a cluster or add an instance Cluster administration · pgsql.yml
Create or change users User administration · pgsql-user.yml
Create or change databases Database administration · pgsql-db.yml
Change HBA or parameters HBA administration · Component administration
Patroni switchovers, maintenance, and incidents Patroni administration
Install, create, update, or remove extensions Extension administration
Monitor an external instance pgsql-monitor.yml
Prepare a migration Migration · pgsql-migration.yml
Remove an instance or cluster Safe removal workflow · pgsql-rm.yml

Real runs of pgsql.yml, pgsql-user.yml, pgsql-db.yml, and related playbooks change the target environment; pgsql-rm.yml can delete data and backups by default. Resolve the exact cluster/node and recent backup first. Removal also requires the operator to type and confirm the exact target.

Backup and Recovery

Restore is destructive. Production recovery requires an independent recent tested backup and separate gates for shutdown, restore, data validation, timeline promotion, DCS rebuild, replica rebuild, and a fresh full backup.

Monitoring

The current source contains 29 PostgreSQL/PGCAT dashboards under files/grafana/pgsql, covering fleet, cluster, instance, database, table, query, session, transaction, replication, service, PgBouncer, PITR, and alerts.

Parameter Groups

The PGSQL parameter reference is the single documentation source for v4.5.0 defaults and semantics:

  • PG_ID: cluster and instance identity.
  • PG_BUSINESS: users, databases, services, and other business objects.
  • PG_INSTALL: kernel, packages, and extensions.
  • PG_BOOTSTRAP: Patroni bootstrap, replication, and database initialization.
  • PG_PROVISION: in-database objects and privileges.
  • PG_BACKUP: pgBackRest and backup repositories.
  • PG_ACCESS: PgBouncer, services, VIP, and DNS.
  • PG_MONITOR: exporters, monitoring registration, and metrics.
  • PG_REMOVE: removal safeguards and cleanup scope.

Further Reading

8.1 - Configuration

Choose the appropriate instance and cluster types based on your requirements to configure PostgreSQL database clusters that meet your needs.

Pigsty is a “configuration-driven” PostgreSQL platform: all behaviors come from the combination of inventory files in ~/pigsty/conf/*.yml and PGSQL parameters. Once you’ve written the configuration, you can replicate a customized cluster with instances, users, databases, access control, extensions, and tuning policies in just a few minutes.

Configuration Entry

  1. Prepare Inventory: Copy a pigsty/conf/*.yml template or write an Ansible Inventory from scratch, placing cluster groups (all.children.<cls>.hosts) and global variables (all.vars) in the same file.
  2. Define Parameters: Override the required PGSQL parameters in the vars block. The override order from global → cluster → host determines the final value.
  3. Apply Configuration: Run ./configure -c <conf> or bin/pgsql-add <cls> and other playbooks to apply the configuration. Pigsty will generate the configuration files needed for Patroni/pgbouncer/pgbackrest based on the parameters.

Pigsty’s default demo inventory conf/pgsql.yml is a minimal example: one pg-meta cluster, global pg_version: 18, and a few business user and database definitions. You can expand with more clusters from this base.

Focus Areas & Documentation Index

Pigsty’s PostgreSQL configuration can be organized from the following dimensions. Subsequent documentation will explain “how to configure” each:

  • Cluster & Instances: Define instance topology (standalone, primary-replica, standby cluster, delayed cluster, Citus, etc.) through pg_cluster / pg_role / pg_seq / pg_upstream.
  • Kernel Version: Select the core version, flavor, and tuning templates using pg_version, pg_mode, pg_packages, pg_extensions, pg_conf, and other parameters.
  • Users/Roles: Declare system roles, business accounts, password policies, and connection pool attributes in pg_default_roles and pg_users.
  • Database Objects: Create databases as needed using pg_databases, baseline, schemas, extensions, pool_* fields and automatically integrate with pgbouncer/Grafana.
  • Access Control (HBA): Maintain host-based authentication policies using pg_default_hba_rules and pg_hba_rules to ensure access boundaries for different roles/networks.
  • Privilege Model (ACL): Converge object privileges through pg_default_privileges, pg_default_roles, pg_revoke_public parameters, providing an out-of-the-box layered role system.

After understanding these parameters, you can write declarative inventory manifests as “configuration as infrastructure” for any business requirement. Pigsty will handle execution and ensure idempotency.

A Typical Example

The following snippet shows how to control instance topology, kernel version, extensions, users, and databases in the same configuration file:

all:
  children:
    pg-analytics:
      hosts:
        10.10.10.11: { pg_seq: 1, pg_role: primary }
        10.10.10.12: { pg_seq: 2, pg_role: replica, pg_offline_query: true }
      vars:
        pg_cluster: pg-analytics
        pg_conf: olap.yml
        pg_extensions: [ postgis, timescaledb, pgvector ]
        pg_databases:
          - { name: bi, owner: dbuser_bi, schemas: [mart], extensions: [timescaledb], pool_mode: session }
        pg_users:
          - { name: dbuser_bi, password: DBUser.BI, roles: [dbrole_admin], pgbouncer: true }
  vars:
    pg_version: 18
    pg_packages: [ pgsql-main, pgsql-common ]
    pg_hba_rules:
      - { user: dbuser_bi, db: bi, addr: intra, auth: ssl, title: 'BI only allows intranet SSL access' }
  • The pg-analytics cluster contains one primary and one offline replica.
  • Global settings specify pg_version: 18 with a set of extension examples and load olap.yml tuning.
  • Declare business objects in pg_databases and pg_users, automatically generating schema/extension and connection pool entries.
  • Additional pg_hba_rules restrict access sources and authentication methods.

Modify and apply this inventory to get a customized PostgreSQL cluster without manual configuration.

8.1.1 - Cluster / Instance

Choose the appropriate instance and cluster types based on your requirements to configure PostgreSQL database clusters that meet your needs.

Choose the appropriate instance and cluster types based on your requirements to configure PostgreSQL database clusters that meet your needs.

You can define different types of instances and clusters. Here are several common PostgreSQL instance/cluster types in Pigsty:

  • Primary: Define a single instance cluster.
  • Replica: Define a basic HA cluster with one primary and one replica.
  • Offline: Define an instance dedicated to OLAP/ETL/interactive queries
  • Sync Standby: Enable synchronous commit to ensure no data loss.
  • Quorum Commit: Use quorum sync commit for a higher consistency level.
  • Standby Cluster: Clone an existing cluster and follow it
  • Delayed Cluster: Clone an existing cluster for emergency data recovery
  • Citus Cluster: Define a Citus distributed database cluster

Primary

We start with the simplest case: a single instance cluster consisting of one primary:

pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
  vars:
    pg_cluster: pg-test

This configuration is concise and self-describing, consisting only of identity parameters. Matching the Ansible group name to pg_cluster remains convenient for -l pg-test, but it is not a hard membership constraint. Current code discovers actual members from each host’s pg_cluster identity, so one PostgreSQL cluster may span multiple inventory groups.

Use the following command to create this cluster:

bin/pgsql-add pg-test

For demos, development testing, hosting temporary requirements, or performing non-critical analytical tasks, a single database instance may not be a big problem. However, such a single-node cluster has no high availability. When hardware failures occur, you’ll need to use PITR or other recovery methods to ensure the cluster’s RTO/RPO. For this reason, you may consider adding several read-only replicas to the cluster.

Replica

To add a read-only replica instance, you can add a new node to pg-test and set its pg_role to replica.

pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica }  # <--- newly added replica
  vars:
    pg_cluster: pg-test

If the entire cluster doesn’t exist, you can directly create the complete cluster. If the cluster primary has already been initialized, you can add a replica to the existing cluster:

bin/pgsql-add pg-test               # initialize the entire cluster at once
bin/pgsql-add pg-test 10.10.10.12   # add replica to existing cluster

When the cluster primary fails, the read-only instance (Replica) can take over the primary’s work with the help of the high availability system. Additionally, read-only instances can be used to execute read-only queries: many businesses have far more read requests than write requests, and most read-only query loads can be handled by replica instances.

Offline

Offline instances are dedicated read-only replicas specifically for serving slow queries, ETL, OLAP traffic, and interactive queries. Slow queries/long transactions have adverse effects on the performance and stability of online business, so it’s best to isolate them from online business.

To add an offline instance, assign it a new instance and set pg_role to offline.

pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica }
    10.10.10.13: { pg_seq: 3, pg_role: offline }  # <--- newly added offline replica
  vars:
    pg_cluster: pg-test

Dedicated offline instances work similarly to common replica instances, but they serve as backup servers in the pg-test-replica service. That is, only when all replica instances are down will the offline and primary instances provide this read-only service.

In many cases, database resources are limited, and using a separate server as an offline instance is not economical. As a compromise, you can select an existing replica instance and mark it with the pg_offline_query flag to indicate it can handle “offline queries”. In this case, this read-only replica will handle both online read-only requests and offline queries. You can use pg_default_hba_rules and pg_hba_rules for additional access control on offline instances.

Sync Standby

When Sync Standby is enabled, PostgreSQL will select one replica as the sync standby, with all other replicas as candidates. The primary database will wait for the standby instance to flush to disk before confirming commits. The standby instance always has the latest data with no replication lag, and primary-standby switchover to the sync standby will have no data loss.

PostgreSQL uses asynchronous streaming replication by default. If the primary fails, WAL that has not yet replicated may be lost. pg_rpo is Patroni’s sampled lag threshold for failover candidates, not a hard upper bound on actual loss; the real window also depends on write rate, replication state, and Patroni sampling timing.

However, in some critical scenarios (e.g., financial transactions), data loss is completely unacceptable, or read replication lag is unacceptable. In such cases, you can use synchronous commit to solve this problem. To enable sync standby mode, you can simply use the crit.yml template in pg_conf.

pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica }
    10.10.10.13: { pg_seq: 3, pg_role: replica }
  vars:
    pg_cluster: pg-test
    pg_conf: crit.yml   # <--- use crit template

To enable sync standby on an existing cluster, configure the cluster and enable synchronous_mode:

$ pg edit-config pg-test    # run as admin user on admin node
+++
-synchronous_mode: false    # <--- old value
+synchronous_mode: true     # <--- new value
 synchronous_mode_strict: false

Apply these changes? [y/N]: y

In this case, the PostgreSQL configuration parameter synchronous_standby_names is automatically managed by Patroni. One replica will be elected as the sync standby, and its application_name will be written to the PostgreSQL primary configuration file and applied.

Quorum Commit

Quorum Commit provides more powerful control than sync standby: especially when you have multiple replicas, you can set criteria for successful commits, achieving higher/lower consistency levels (and trade-offs with availability).

If you want at least two replicas to confirm commits, you can adjust the synchronous_node_count parameter through Patroni cluster configuration and apply it:

synchronous_mode: true          # ensure synchronous commit is enabled
synchronous_node_count: 2       # specify "at least" how many replicas must successfully commit

If you want to use more sync replicas, modify the synchronous_node_count value. When the cluster size changes, you should ensure this configuration is still valid to avoid service unavailability.

In this case, the PostgreSQL configuration parameter synchronous_standby_names is automatically managed by Patroni.

synchronous_standby_names = '2 ("pg-test-3","pg-test-2")'
Example: Using multiple sync standbys
$ pg edit-config pg-test
---
+synchronous_node_count: 2

Apply these changes? [y/N]: y

After applying the configuration, two sync standbys appear.

+ Cluster: pg-test (7080814403632534854) +---------+----+-----------+-----------------+
| Member    | Host        | Role         | State   | TL | Lag in MB | Tags            |
+-----------+-------------+--------------+---------+----+-----------+-----------------+
| pg-test-1 | 10.10.10.10 | Leader       | running |  1 |           | clonefrom: true |
| pg-test-2 | 10.10.10.11 | Sync Standby | running |  1 |         0 | clonefrom: true |
| pg-test-3 | 10.10.10.12 | Sync Standby | running |  1 |         0 | clonefrom: true |
+-----------+-------------+--------------+---------+----+-----------+-----------------+

Another scenario is using any n replicas to confirm commits. In this case, the configuration is slightly different. For example, if we only need any one replica to confirm commits:

synchronous_mode: quorum        # use quorum commit
postgresql:
  parameters:                   # modify PostgreSQL's configuration parameter synchronous_standby_names, using `ANY n ()` syntax
    synchronous_standby_names: 'ANY 1 (*)'  # you can specify a specific replica list or use * to wildcard all replicas.
Example: Enable ANY quorum commit
$ pg edit-config pg-test

+    synchronous_standby_names: 'ANY 1 (*)' # in ANY mode, this parameter is needed
- synchronous_node_count: 2  # in ANY mode, this parameter is not needed

Apply these changes? [y/N]: y

After applying, the configuration takes effect, and all standbys become regular replicas in Patroni. However, in pg_stat_replication, you can see sync_state becomes quorum.

Standby Cluster

You can clone an existing cluster and create a standby cluster for data migration, horizontal splitting, multi-region deployment, or disaster recovery.

Under normal circumstances, the standby cluster will follow the upstream cluster and keep content synchronized. You can promote the standby cluster to become a truly independent cluster.

The standby cluster definition is basically the same as a normal cluster definition, except that the pg_upstream parameter is additionally defined on the primary. The primary of the standby cluster is called the Standby Leader.

For example, below defines a pg-test cluster and its standby cluster pg-test2. The configuration inventory might look like this:

# pg-test is the original cluster
pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
  vars: { pg_cluster: pg-test }

# pg-test2 is the standby cluster of pg-test
pg-test2:
  hosts:
    10.10.10.12: { pg_seq: 1, pg_role: primary , pg_upstream: 10.10.10.11 } # <--- pg_upstream defined here
    10.10.10.13: { pg_seq: 2, pg_role: replica }
  vars: { pg_cluster: pg-test2 }

The primary node pg-test2-1 of the pg-test2 cluster will be a downstream replica of pg-test and serve as the Standby Leader in the pg-test2 cluster.

Just ensure the pg_upstream parameter is configured on the standby cluster’s primary node to automatically pull backups from the original upstream.

bin/pgsql-add pg-test     # create original cluster
bin/pgsql-add pg-test2    # create standby cluster
Example: Change replication upstream

If necessary (e.g., upstream primary-standby switchover/failover), you can change the standby cluster’s replication upstream through cluster configuration.

To do this, simply change standby_cluster.host to the new upstream IP address and apply.

$ pg edit-config pg-test2

 standby_cluster:
   create_replica_methods:
   - basebackup
-  host: 10.10.10.13     # <--- old upstream
+  host: 10.10.10.12     # <--- new upstream
   port: 5432

 Apply these changes? [y/N]: y
Example: Promote standby cluster

You can promote the standby cluster to an independent cluster at any time, so the cluster can independently handle write requests and diverge from the original cluster.

To do this, you must configure the cluster and completely erase the standby_cluster section, then apply.

$ pg edit-config pg-test2
-standby_cluster:
-  create_replica_methods:
-  - basebackup
-  host: 10.10.10.11
-  port: 5432

Apply these changes? [y/N]: y
Example: Cascade replication

If you specify pg_upstream on a replica instead of the primary, you can configure cascade replication for the cluster.

When configuring cascade replication, you must use the IP address of an instance in the cluster as the parameter value, otherwise initialization will fail. The replica performs streaming replication from a specific instance rather than the primary.

The instance acting as a WAL relay is called a Bridge Instance. Using a bridge instance can share the burden of sending WAL from the primary. When you have dozens of replicas, using bridge instance cascade replication is a good idea.

pg-test:
 hosts: # pg-test-1 ---> pg-test-2 ---> pg-test-3
   10.10.10.11: { pg_seq: 1, pg_role: primary }
   10.10.10.12: { pg_seq: 2, pg_role: replica } # <--- bridge instance
   10.10.10.13: { pg_seq: 3, pg_role: replica, pg_upstream: 10.10.10.12 }
   # ^--- replicate from pg-test-2 (bridge) instead of pg-test-1 (primary)
 vars: { pg_cluster: pg-test }

Delayed Cluster

A Delayed Cluster is a special type of standby cluster used to quickly recover “accidentally deleted” data.

For example, if you want a cluster named pg-testdelay whose data content is the same as the pg-test cluster from one hour ago:

# pg-test is the original cluster
pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
  vars: { pg_cluster: pg-test }

# pg-testdelay is the delayed cluster of pg-test
pg-testdelay:
  hosts:
    10.10.10.12: { pg_seq: 1, pg_role: primary , pg_upstream: 10.10.10.11, pg_delay: 1d }
    10.10.10.13: { pg_seq: 2, pg_role: replica }
  vars: { pg_cluster: pg-testdelay }

You can also configure a “replication delay” on an existing standby cluster.

$ pg edit-config pg-testdelay
 standby_cluster:
   create_replica_methods:
   - basebackup
   host: 10.10.10.11
   port: 5432
+  recovery_min_apply_delay: 1h    # <--- add delay duration here, e.g. 1 hour

Apply these changes? [y/N]: y

When some tuples and tables are accidentally deleted, you can modify this parameter to advance this delayed cluster to an appropriate point in time, read data from it, and quickly fix the original cluster.

Delayed clusters require additional resources, but are much faster than PITR and have much less impact on the system. For very critical clusters, consider setting up delayed clusters.

Citus Cluster

Pigsty natively supports Citus. You can refer to conf/ha/citus.yml as a complete example.

To define a Citus cluster, you need to specify the following parameters:

  • pg_mode must be set to citus, not the default pgsql
  • The shard name pg_shard and shard number pg_group must be defined on each shard cluster
  • pg_primary_db must be defined to specify the database managed by Patroni.
  • If you want to use pg_dbsu postgres instead of the default pg_admin_username to execute admin commands, then pg_dbsu_password must be set to a non-empty plaintext password

Additionally, extra hba rules are needed to allow SSL access from localhost and other data nodes. As shown below:

all:
  children:
    pg-citus0: # citus shard 0
      hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
      vars: { pg_cluster: pg-citus0 , pg_group: 0 }
    pg-citus1: # citus shard 1
      hosts: { 10.10.10.11: { pg_seq: 1, pg_role: primary } }
      vars: { pg_cluster: pg-citus1 , pg_group: 1 }
    pg-citus2: # citus shard 2
      hosts: { 10.10.10.12: { pg_seq: 1, pg_role: primary } }
      vars: { pg_cluster: pg-citus2 , pg_group: 2 }
    pg-citus3: # citus shard 3
      hosts:
        10.10.10.13: { pg_seq: 1, pg_role: primary }
        10.10.10.14: { pg_seq: 2, pg_role: replica }
      vars: { pg_cluster: pg-citus3 , pg_group: 3 }
  vars:                               # global parameters for all Citus clusters
    pg_mode: citus                    # pgsql cluster mode must be set to: citus
    pg_shard: pg-citus                # citus horizontal shard name: pg-citus
    pg_primary_db: meta               # citus database name: meta
    pg_dbsu_password: DBUser.Postgres # if using dbsu, need to configure a password for it
    pg_users: [ { name: dbuser_meta ,password: DBUser.Meta ,pgbouncer: true ,roles: [ dbrole_admin ] } ]
    pg_databases: [ { name: meta ,extensions: [ { name: citus }, { name: postgis }, { name: timescaledb } ] } ]
    pg_hba_rules:
      - { user: 'all' ,db: all  ,addr: 127.0.0.1/32 ,auth: ssl ,title: 'all user ssl access from localhost' }
      - { user: 'all' ,db: all  ,addr: intra        ,auth: ssl ,title: 'all user ssl access from intranet'  }

On the coordinator node, you can create distributed tables and reference tables and query them from any data node. Starting from 11.2, any Citus database node can act as a coordinator.

SELECT create_distributed_table('pgbench_accounts', 'aid'); SELECT truncate_local_data_after_distributing_table($$public.pgbench_accounts$$);
SELECT create_reference_table('pgbench_branches')         ; SELECT truncate_local_data_after_distributing_table($$public.pgbench_branches$$);
SELECT create_reference_table('pgbench_history')          ; SELECT truncate_local_data_after_distributing_table($$public.pgbench_history$$);
SELECT create_reference_table('pgbench_tellers')          ; SELECT truncate_local_data_after_distributing_table($$public.pgbench_tellers$$);

8.1.2 - Kernel Version

How to choose the appropriate PostgreSQL kernel and major version.

Choosing a “kernel” in Pigsty means determining the PostgreSQL major version, mode/distribution, packages to install, and tuning templates to load.

The Pigsty v4.5 source currently supports PostgreSQL 14-18 and uses 18 by default. The following content shows how to make these choices through configuration files.

Major Version and Packages

  • pg_version: Specify the PostgreSQL major version (default 18). Pigsty will automatically map to the correct package name prefix based on the version.
  • pg_packages: Define the core package set to install, supports using package aliases (default pgsql-main pgsql-common, includes kernel + patroni/pgbouncer/pgbackrest and other common tools).
  • pg_extensions: List of additional extension packages to install, also supports aliases; defaults to empty meaning only core dependencies are installed.
all:
  vars:
    pg_version: 18
    pg_packages: [ pgsql-main, pgsql-common ]
    pg_extensions: [ postgis, timescaledb, pgvector, pgml ]

Effect: Ansible will pull packages corresponding to pg_version=18 during installation, pre-install extensions to the system, and database initialization scripts can then directly CREATE EXTENSION.

Extension support varies across versions in Pigsty’s offline repository: 14 has relatively fewer available extensions, while 17/18 have the broadest coverage. If an extension is not pre-packaged, it can be added via repo_extra_packages.

Kernel Mode (pg_mode)

pg_mode controls the kernel “flavor” to deploy. Default pgsql indicates standard PostgreSQL. Pigsty currently supports the following modes:

Mode Scenario
pgsql Standard PostgreSQL, HA + replication
citus Citus distributed cluster, requires additional pg_shard / pg_group
gpsql Cloudberry / Greenplum / MatrixDB
mssql Babelfish
mysql OpenGauss/HaloDB compatible with MySQL protocol
polar Alibaba PolarDB (based on pg polar distribution)
ivory IvorySQL (Oracle-compatible syntax)
pgtde Percona PostgreSQL with pg_tde under /usr/pgtde-$v
oriole OrioleDB storage engine
agens AgensGraph graph database kernel
pgedge pgEdge distributed replication kernel

pg_mode determines binary paths, Patroni integration, and some kernel-specific logic; it does not automatically add every required package, extension, and business database. Use the matching conf/*.yml template in real deployments, or explicitly configure pg_packages, pg_extensions, pg_libs, and pg_databases. Here is a minimal Citus example:

all:
  children:
    pg-citus1:
      hosts: { 10.10.10.11: { pg_seq: 1, pg_role: primary } }
      vars: { pg_cluster: pg-citus1, pg_group: 0 }
    pg-citus2:
      hosts: { 10.10.10.12: { pg_seq: 1, pg_role: primary } }
      vars: { pg_cluster: pg-citus2, pg_group: 1 }
  vars:
    pg_mode: citus
    pg_shard: pg-citus
    pg_primary_db: citus
    pg_extensions: [ citus ]
    pg_libs: 'citus, pg_stat_statements'
    pg_databases:
      - { name: citus, extensions: [ citus ] }

conf/ha/citus.yml provides the current complete example. The minimal configuration above explicitly installs Citus packages and creates the extension in the citus database.

Extensions and Pre-installed Objects

Besides system packages, you can control components automatically loaded after database startup through the following parameters:

  • pg_libs: List to write to shared_preload_libraries. For example: pg_libs: 'timescaledb, pg_stat_statements, auto_explain'.
  • pg_default_extensions / pg_default_schemas: Control schemas and extensions pre-created in template1 and postgres by initialization scripts.
  • pg_parameters: Rendered by Pigsty into postgresql.auto.conf during configuration. Do not also manage the same settings manually with ALTER SYSTEM.

Example: Enable TimescaleDB, pgvector and customize some system parameters.

pg-analytics:
  vars:
    pg_cluster: pg-analytics
    pg_libs: 'timescaledb, pg_stat_statements, auto_explain'
    pg_default_extensions:
      - { name: timescaledb }
      - { name: vector }
    pg_parameters:
      timescaledb.max_background_workers: 8

Effect: During initialization, default extensions are created in template1 and postgres; newly created databases based on template1 inherit those objects. pg_parameters is written directly to postgresql.auto.conf.

Tuning Template (pg_conf)

pg_conf points to Patroni templates in roles/pgsql/templates/*.yml. Pigsty includes four built-in general templates:

Template Applicable Scenario
oltp.yml Default template, for 4–128 core TP workload
olap.yml Optimized for analytical scenarios
crit.yml Emphasizes sync commit/minimal latency, suitable for zero-loss scenarios like finance
tiny.yml Lightweight machines / edge scenarios / resource-constrained environments

You can directly replace the template or customize a YAML file in templates/, then specify it in cluster vars.

pg-ledger:
  hosts: { 10.10.10.21: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-ledger
    pg_conf: crit.yml
    pg_parameters:
      synchronous_commit: 'remote_apply'
      max_wal_senders: 16
      wal_keep_size: '2GB'

Effect: Copy crit.yml as Patroni configuration, overlay pg_parameters written to postgresql.auto.conf, making instances run immediately in synchronous commit mode.

Combined Instance: A Complete Example 

pg-rag:
  hosts:
    10.10.10.31: { pg_seq: 1, pg_role: primary }
    10.10.10.32: { pg_seq: 2, pg_role: replica }
  vars:
    pg_cluster: pg-rag
    pg_version: 18
    pg_mode: pgsql
    pg_conf: olap.yml
    pg_packages: [ pgsql-main, pgsql-common ]
    pg_extensions: [ pgvector, pgml, postgis ]
    pg_libs: 'pg_stat_statements, auto_explain'
    pg_parameters:
      max_parallel_workers: 8
      shared_buffers: '32GB'
  • First primary + one replica, using olap.yml tuning.
  • Install PG18 plus common RAG extensions; only libraries that actually require preloading belong in pg_libs.
  • Patroni/pgbouncer/pgbackrest generated by Pigsty, no manual intervention needed.

Replace the above parameters according to business needs to complete all kernel-level customization.

8.1.3 - Package Alias

Pigsty provides a package alias translation mechanism that shields the differences in binary package details across operating systems, making installation easier.

PostgreSQL package naming conventions vary significantly across different operating systems:

  • EL systems (RHEL/Rocky/Alma/…) use formats like pgvector_18, postgis36_18*
  • Debian/Ubuntu systems use formats like postgresql-18-pgvector, postgresql-18-postgis-3

This difference adds cognitive burden to users: you need to remember different package name rules for different systems, and handle the embedding of PostgreSQL version numbers.

Package Alias

Pigsty solves this problem through the Package Alias mechanism: you only need to use unified aliases, and Pigsty will handle all the details:

# Using aliases - simple, unified, cross-platform
pg_extensions: [ postgis, pgvector, timescaledb ]

# Equivalent to actual package names on EL9 + PG18
pg_extensions: [ postgis36_18*, pgvector_18*, timescaledb-tsl_18* ]

# Equivalent to actual package names on Ubuntu 24 + PG18
pg_extensions: [ postgresql-18-postgis-3, postgresql-18-pgvector, postgresql-18-timescaledb-tsl ]

Alias Translation

Aliases can also group a set of packages as a whole. For example, Pigsty’s default installed packages - the default value of pg_packages is:

pg_packages:                      # pg packages to be installed, alias can be used
  - pgsql-main pgsql-common

Pigsty will query the current operating system alias list (assuming el10.x86_64) and translate it to PGSQL kernel, extensions, and toolkits:

pgsql-main:    "postgresql$v postgresql$v-server postgresql$v-libs postgresql$v-contrib postgresql$v-plperl postgresql$v-plpython3 postgresql$v-pltcl postgresql$v-llvmjit pg_repack_$v* wal2json_$v* pgvector_$v*"
pgsql-common:  "patroni patroni-etcd pgbouncer pgbackrest pg_exporter pgbackrest_exporter vip-manager"

Next, Pigsty further translates pgsql-main using the currently specified PG major version (assuming pg_version = 18):

pg18-main:   "postgresql18 postgresql18-server postgresql18-libs postgresql18-contrib postgresql18-plperl postgresql18-plpython3 postgresql18-pltcl postgresql18-llvmjit pg_repack_18* wal2json_18* pgvector_18*"

Through this approach, Pigsty shields the complexity of packages, allowing users to simply specify the functional components they want.

Which Variables Can Use Aliases?

You can use package aliases in the following four parameters, and the aliases will be automatically converted to actual package names according to the translation process:

Alias List

You can find the alias mapping files for each operating system and architecture in the roles/node_id/vars/ directory of the Pigsty project source code:

How It Works

Alias Translation Process 

User config alias --> Detect OS -->  Find alias mapping table ---> Replace $v placeholder ---> Install actual packages
     ↓                 ↓                   ↓                                   ↓
  postgis          el9.x86_64         postgis36_$v*                   postgis36_18*
  postgis          u24.x86_64         postgresql-$v-postgis-3         postgresql-18-postgis-3

Version Placeholder

Pigsty’s alias system uses $v as a placeholder for the PostgreSQL version number. When you specify a PostgreSQL version using pg_version, all $v in aliases will be replaced with the actual version number.

For example, when pg_version: 18:

Alias Definition (EL) Expanded Result
postgresql$v* postgresql18*
pgvector_$v* pgvector_18*
timescaledb-tsl_$v* timescaledb-tsl_18*
Alias Definition (Debian/Ubuntu) Expanded Result
postgresql-$v postgresql-18
postgresql-$v-pgvector postgresql-18-pgvector
postgresql-$v-timescaledb-tsl postgresql-18-timescaledb-tsl

Wildcard Matching

On EL systems, many aliases use the * wildcard to match related subpackages. For example:

  • postgis36_18* will match postgis36_18, postgis36_18-client, postgis36_18-utils, etc.
  • postgresql18* will match postgresql18, postgresql18-server, postgresql18-libs, postgresql18-contrib, etc.

This design ensures you don’t need to list each subpackage individually - one alias can install the complete extension.

8.1.4 - User/Role

How to define and customize PostgreSQL users and roles through configuration?

In this document, “user” refers to a logical object within a database cluster created with CREATE USER/ROLE.

In PostgreSQL, users belong directly to the database cluster rather than a specific database. Therefore, when creating business databases and users, follow the principle of “users first, databases later”.

Pigsty defines roles and users through two config parameters:

The former defines roles/users shared across the entire environment; the latter defines business roles/users specific to a single cluster. Both have the same format as arrays of user definition objects. Users/roles are created sequentially in array order, so later users can belong to roles defined earlier.

By default, all users marked with pgbouncer: true are added to the Pgbouncer connection pool user list.

Define Users

Example from Pigsty demo pg-meta cluster:

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_users:
      - {name: dbuser_meta     ,password: DBUser.Meta     ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: pigsty admin user }
      - {name: dbuser_view     ,password: DBUser.Viewer   ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer for meta database }
      - {name: dbuser_grafana  ,password: DBUser.Grafana  ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for grafana database    }
      - {name: dbuser_bytebase ,password: DBUser.Bytebase ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for bytebase database   }
      - {name: dbuser_kong     ,password: DBUser.Kong     ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for kong api gateway    }
      - {name: dbuser_gitea    ,password: DBUser.Gitea    ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for gitea service       }
      - {name: dbuser_wiki     ,password: DBUser.Wiki     ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for wiki.js service     }
      - {name: dbuser_noco     ,password: DBUser.Noco     ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for nocodb service      }
      - {name: dbuser_remove   ,state: absent }  # use state: absent to delete user

Each user/role definition is a complex object. Only name is required:

- name: dbuser_meta               # REQUIRED, `name` is the only mandatory field
  state: create                   # Optional, user state: create (default), absent
  password: DBUser.Meta           # Optional, password, can be scram-sha-256 hash or plaintext
  login: true                     # Optional, can login, default true
  superuser: false                # Optional, is superuser, default false
  createdb: false                 # Optional, can create databases, default false
  createrole: false               # Optional, can create roles, default false
  inherit: true                   # Optional, inherit role privileges, default true
  replication: false              # Optional, can replicate, default false
  bypassrls: false                # Optional, bypass row-level security, default false
  connlimit: -1                   # Optional, connection limit, default -1 (unlimited)
  expire_in: 3650                 # Optional, expire N days from creation (priority over expire_at)
  expire_at: '2030-12-31'         # Optional, expiration date in YYYY-MM-DD format
  comment: pigsty admin user      # Optional, user comment
  roles: [dbrole_admin]           # Optional, roles array
  parameters:                     # Optional, role-level config params
    search_path: public
  pgbouncer: true                 # Optional, add to connection pool user list, default false
  pool_mode: transaction          # Optional, pgbouncer pool mode, default transaction
  pool_connlimit: 100             # Optional, user-level max pool connections; omitted inherits global default 100

User-level pool quota is consistently defined by pool_connlimit (mapped to Pgbouncer max_user_connections).

Parameter Overview

The only required field is name - a valid, unique username within the cluster. All other params have sensible defaults.

Field Category Type Attr Description
name Basic string Required Username, must be valid and unique
state Basic enum Optional State: create (default), absent
password Basic string Mutable User password, plaintext or hash
comment Basic string Mutable User comment
login Privilege bool Mutable Can login, default true
superuser Privilege bool Mutable Is superuser, default false
createdb Privilege bool Mutable Can create databases, default false
createrole Privilege bool Mutable Can create roles, default false
inherit Privilege bool Mutable Inherit role privileges, default true
replication Privilege bool Mutable Can replicate, default false
bypassrls Privilege bool Mutable Bypass RLS, default false
connlimit Privilege int Mutable Connection limit, -1 unlimited
expire_in Validity int Mutable Expire N days from now (priority)
expire_at Validity string Mutable Expiration date, YYYY-MM-DD format
roles Role array Additive Roles array, string or object format
parameters Params object Mutable Role-level parameters
pgbouncer Pool bool Mutable Add to connection pool, default false
pool_mode Pool enum Mutable Pool mode: transaction (default)
pool_connlimit Pool int Mutable Pool user max connections

Parameter Details

name

String, required. Username - must be unique within the cluster.

Must be a valid PostgreSQL identifier matching ^[a-z_][a-z0-9_]{0,62}$: starts with lowercase letter or underscore, contains only lowercase letters, digits, underscores, max 63 chars.

- name: dbuser_app         # Standard naming
- name: app_readonly       # Underscore separated
- name: _internal          # Underscore prefix (for internal roles)

state

Enum for user operation: create or absent. Default create.

State Description
create Default, create user, update if exists
absent Delete user with DROP ROLE 
- name: dbuser_app             # state defaults to create
- name: dbuser_old
  state: absent                # Delete user

These system users cannot be deleted via state: absent (to prevent cluster failure):

password

String, mutable. User password - users without password can’t login via password auth.

Password can be:

Format Example Description
Plaintext DBUser.Meta Not recommended, logged to config
SCRAM-SHA-256 SCRAM-SHA-256$4096:xxx$yyy:zzz Recommended, PG10+ default
MD5 hash md5... Legacy compatibility 
# Plaintext (not recommended, logged to config)
- name: dbuser_app
  password: MySecretPassword

# SCRAM-SHA-256 hash (recommended)
- name: dbuser_app
  password: 'SCRAM-SHA-256$4096:xxx$yyy:zzz'

When setting password, Pigsty temporarily disables logging to prevent leakage:

SET log_statement TO 'none';
ALTER USER "dbuser_app" PASSWORD 'xxx';
SET log_statement TO DEFAULT;

To generate SCRAM-SHA-256 hash:

# Using PostgreSQL (requires pgcrypto extension)
psql -c "SELECT encode(digest('password' || 'username', 'sha256'), 'hex')"

comment

String, mutable. User comment, defaults to business user {name}.

Set via COMMENT ON ROLE, supports special chars (quotes auto-escaped).

- name: dbuser_app
  comment: 'Main business application account'
COMMENT ON ROLE "dbuser_app" IS 'Main business application account';

login

Boolean, mutable. Can login, default true.

Setting false creates a Role rather than User - typically for permission grouping.

In PostgreSQL, CREATE USER equals CREATE ROLE ... LOGIN.

# Create login-able user
- name: dbuser_app
  login: true

# Create role (no login, for permission grouping)
- name: dbrole_custom
  login: false
  comment: custom permission role
CREATE USER "dbuser_app" LOGIN;
CREATE USER "dbrole_custom" NOLOGIN;

superuser

Boolean, mutable. Is superuser, default false.

Superusers have full database privileges, bypassing all permission checks.

- name: dbuser_admin
  superuser: true            # Dangerous: full privileges
ALTER USER "dbuser_admin" SUPERUSER;

Pigsty provides default superuser via pg_admin_username (dbuser_dba). Don’t create additional superusers unless necessary.

createdb

Boolean, mutable. Can create databases, default false.

- name: dbuser_dev
  createdb: true             # Allow create database
ALTER USER "dbuser_dev" CREATEDB;

Some applications (Gitea, Odoo, etc.) may require CREATEDB privilege for their admin users.

createrole

Boolean, mutable. Can create other roles, default false.

Users with CREATEROLE can create, modify, delete other non-superuser roles.

- name: dbuser_admin
  createrole: true           # Allow manage other roles
ALTER USER "dbuser_admin" CREATEROLE;

inherit

Boolean, mutable. Auto-inherit privileges from member roles, default true.

Setting false requires explicit SET ROLE to use member role privileges.

# Auto-inherit role privileges (default)
- name: dbuser_app
  inherit: true
  roles: [dbrole_readwrite]

# Requires explicit SET ROLE
- name: dbuser_special
  inherit: false
  roles: [dbrole_admin]
ALTER USER "dbuser_special" NOINHERIT;
-- User must execute SET ROLE dbrole_admin to get privileges

replication

Boolean, mutable. Can initiate streaming replication, default false.

Usually only replication users (replicator) need this. Normal users shouldn’t have it unless for logical decoding subscriptions.

- name: replicator
  replication: true          # Allow streaming replication
  roles: [pg_monitor, dbrole_readonly]
ALTER USER "replicator" REPLICATION;

bypassrls

Boolean, mutable. Bypass row-level security (RLS) policies, default false.

When enabled, user can access all rows even with RLS policies. Usually only for admins.

- name: dbuser_myappadmin
  bypassrls: true            # Bypass RLS policies
ALTER USER "dbuser_myappadmin" BYPASSRLS;

connlimit

Integer, mutable. Max concurrent connections, default -1 (unlimited).

Positive integer limits max simultaneous sessions for this user. Doesn’t affect superusers.

- name: dbuser_app
  connlimit: 100             # Max 100 concurrent connections

- name: dbuser_batch
  connlimit: 10              # Limit batch user connections
ALTER USER "dbuser_app" CONNECTION LIMIT 100;

expire_in

Integer, mutable. Expire N days from current date.

This param has higher priority than expire_at. Expiration recalculated on each playbook run - good for temp users needing periodic renewal.

- name: temp_user
  expire_in: 30              # Expire in 30 days

- name: contractor_user
  expire_in: 90              # Expire in 90 days

Generates SQL:

-- expire_in: 30, assuming current date is 2025-01-01
ALTER USER "temp_user" VALID UNTIL '2025-01-31';

expire_at

String, mutable. Expiration date in YYYY-MM-DD format, or special value infinity.

Lower priority than expire_in. Use infinity for never-expiring users.

- name: contractor_user
  expire_at: '2024-12-31'    # Expire on specific date

- name: permanent_user
  expire_at: 'infinity'      # Never expires
ALTER USER "contractor_user" VALID UNTIL '2024-12-31';
ALTER USER "permanent_user" VALID UNTIL 'infinity';

roles

Array, additive. Roles this user belongs to. Elements can be strings or objects.

Simple format - strings for role names:

- name: dbuser_app
  roles:
    - dbrole_readwrite
    - pg_read_all_data
GRANT "dbrole_readwrite" TO "dbuser_app";
GRANT "pg_read_all_data" TO "dbuser_app";

Full format - objects for fine-grained control:

- name: dbuser_app
  roles:
    - dbrole_readwrite                            # Simple string: GRANT role
    - { name: dbrole_admin, admin: true }         # WITH ADMIN OPTION
    - { name: pg_monitor, set: false }            # PG16+: disallow SET ROLE
    - { name: pg_signal_backend, inherit: false } # PG16+: don't auto-inherit
    - { name: old_role, state: absent }           # Revoke role membership

Object Format Parameters:

Param Type Description
name string Role name (required)
state enum grant (default) or absent/revoke: control membership
admin bool true: WITH ADMIN OPTION, false: REVOKE ADMIN
set bool PG16+: true: WITH SET TRUE, false: REVOKE SET
inherit bool PG16+: true: WITH INHERIT TRUE, false: REVOKE INHERIT

PostgreSQL 16+ New Features:

PostgreSQL 16 introduced finer-grained role membership control:

  • ADMIN OPTION: Allow granting role to other users
  • SET OPTION: Allow using SET ROLE to switch to this role
  • INHERIT OPTION: Auto-inherit this role’s privileges
# PostgreSQL 16+ complete example
- name: dbuser_app
  roles:
    # Normal membership
    - dbrole_readwrite

    # Can grant dbrole_admin to other users
    - { name: dbrole_admin, admin: true }

    # Cannot SET ROLE to pg_monitor (only inherit privileges)
    - { name: pg_monitor, set: false }

    # Don't auto-inherit pg_execute_server_program (need explicit SET ROLE)
    - { name: pg_execute_server_program, inherit: false }

    # Revoke old_role membership
    - { name: old_role, state: absent }

set and inherit options only work in PG16+. On earlier versions they’re ignored with warning comments.

parameters

Object, mutable. Role-level config params via ALTER ROLE ... SET. Applies to all sessions for this user.

- name: dbuser_analyst
  parameters:
    work_mem: '256MB'
    statement_timeout: '5min'
    search_path: 'analytics,public'
    log_statement: 'all'
ALTER USER "dbuser_analyst" SET "work_mem" = '256MB';
ALTER USER "dbuser_analyst" SET "statement_timeout" = '5min';
ALTER USER "dbuser_analyst" SET "search_path" = 'analytics,public';
ALTER USER "dbuser_analyst" SET "log_statement" = 'all';

Use special value DEFAULT (case-insensitive) to reset to PostgreSQL default:

- name: dbuser_app
  parameters:
    work_mem: DEFAULT          # Reset to default
    statement_timeout: '30s'   # Set new value
ALTER USER "dbuser_app" SET "work_mem" = DEFAULT;
ALTER USER "dbuser_app" SET "statement_timeout" = '30s';

Common role-level params:

Parameter Description Example
work_mem Query work memory '64MB'
statement_timeout Statement timeout '30s'
lock_timeout Lock wait timeout '10s'
idle_in_transaction_session_timeout Idle transaction timeout '10min'
search_path Schema search path 'app,public'
log_statement Log level 'ddl'
temp_file_limit Temp file size limit '10GB'

Query user-level params via pg_db_role_setting system view.

pgbouncer

Boolean, mutable. Add user to Pgbouncer user list, default false.

For prod users needing connection pool access, must explicitly set pgbouncer: true. Default false prevents accidentally exposing internal users to the pool.

# Prod user: needs connection pool
- name: dbuser_app
  password: DBUser.App
  pgbouncer: true

# Internal user: no connection pool needed
- name: dbuser_internal
  password: DBUser.Internal
  pgbouncer: false           # Default, can be omitted

Users with pgbouncer: true are added to /etc/pgbouncer/userlist.txt.

pool_mode

Enum, mutable. User-level pool mode: transaction, session, or statement. Default transaction.

Mode Description Use Case
transaction Return connection after txn Most OLTP apps, default
session Return connection after session Apps needing session state
statement Return after each statement Simple stateless queries 
# DBA user: session mode (may need SET commands etc.)
- name: dbuser_dba
  pgbouncer: true
  pool_mode: session

# Normal business user: transaction mode
- name: dbuser_app
  pgbouncer: true
  pool_mode: transaction

User-level pool params are configured via /etc/pgbouncer/useropts.txt:

dbuser_dba      = pool_mode=session max_user_connections=16
dbuser_monitor  = pool_mode=session max_user_connections=8

pool_connlimit

Integer, mutable. User-level maximum pool connections. If omitted, no user-level override is generated and Pigsty’s global pgbouncer.ini default of 100 applies. PgBouncer uses 0 to mean unlimited.

- name: dbuser_app
  pgbouncer: true
  pool_connlimit: 50         # Max 50 pool connections for this user

ACL System

Pigsty provides a built-in access control / ACL model. Assign these default business roles to users as required:

Role Privileges Typical Use Case
dbrole_readwrite Global read-write Primary application accounts
dbrole_readonly Global read-only Read-only application access
dbrole_admin DDL privileges Application administrators and table creation
dbrole_offline Independent read-only; instance scope controlled by HBA Ad hoc users, ETL, and analytics 
# Typical business user configuration
pg_users:
  - name: dbuser_app
    password: DBUser.App
    pgbouncer: true
    roles: [dbrole_readwrite]    # Prod account, read-write

  - name: dbuser_readonly
    password: DBUser.Readonly
    pgbouncer: true
    roles: [dbrole_readonly]     # Read-only account

  - name: dbuser_admin
    password: DBUser.Admin
    pgbouncer: true
    roles: [dbrole_admin]        # Admin, can execute DDL

  - name: dbuser_etl
    password: DBUser.ETL
    roles: [dbrole_offline]      # Analytics account; restrict instance scope through HBA

dbrole_offline does not itself restrict a user to offline instances. To establish that boundary, set role: offline on the corresponding HBA rule; see Offline Role and Instance Isolation.

To redesign your own ACL system, customize:

Pgbouncer Users

Pgbouncer is enabled by default as connection pool middleware. Pigsty adds all users in pg_users with explicit pgbouncer: true flag to the pgbouncer user list.

Users in connection pool are listed in /etc/pgbouncer/userlist.txt:

"postgres" ""
"dbuser_wiki" "SCRAM-SHA-256$4096:+77dyhrPeFDT/TptHs7/7Q==$KeatuohpKIYzHPCt/tqBu85vI11o9mar/by0hHYM2W8=:X9gig4JtjoS8Y/o1vQsIX/gY1Fns8ynTXkbWOjUfbRQ="
"dbuser_view" "SCRAM-SHA-256$4096:DFoZHU/DXsHL8MJ8regdEw==$gx9sUGgpVpdSM4o6A2R9PKAUkAsRPLhLoBDLBUYtKS0=:MujSgKe6rxcIUMv4GnyXJmV0YNbf39uFRZv724+X1FE="
"dbuser_monitor" "SCRAM-SHA-256$4096:fwU97ZMO/KR0ScHO5+UuBg==$CrNsmGrx1DkIGrtrD1Wjexb/aygzqQdirTO1oBZROPY=:L8+dJ+fqlMQh7y4PmVR/gbAOvYWOr+KINjeMZ8LlFww="
"dbuser_meta" "SCRAM-SHA-256$4096:leB2RQPcw1OIiRnPnOMUEg==$eyC+NIMKeoTxshJu314+BmbMFpCcspzI3UFZ1RYfNyU=:fJgXcykVPvOfro2MWNkl5q38oz21nSl1dTtM65uYR1Q="

User-level pool params are maintained in /etc/pgbouncer/useropts.txt:

dbuser_dba      = pool_mode=session max_user_connections=16
dbuser_monitor  = pool_mode=session max_user_connections=8

When creating users, Pgbouncer user list is refreshed via online reload - doesn’t affect existing connections.

Pgbouncer runs as same dbsu as PostgreSQL (default postgres OS user). Use pgb alias to access pgbouncer admin functions.

pgbouncer_auth_query param allows dynamic query for pool user auth - convenient when you prefer not to manually manage pool users.

Related Resources

For user management operations, see User Management.

For user access privileges, see Access Control: Role System.

8.1.5 - Database

How to define and customize PostgreSQL databases through configuration?

In this document, “database” refers to a logical object within a database cluster created with CREATE DATABASE.

A PostgreSQL cluster can serve multiple databases simultaneously. In Pigsty, you can define required databases in cluster configuration.

Pigsty customizes the template1 template database - creating default schemas, installing default extensions, configuring default privileges. Newly created databases inherit these settings from template1. You can also specify other template databases via template for instant database cloning.

By default, all business databases are 1:1 added to Pgbouncer connection pool; pg_exporter auto-discovers all business databases for in-database object monitoring. All databases are also registered as PostgreSQL datasources in Grafana on all INFRA nodes for PGCAT dashboards.

Define Database

Business databases are defined in cluster param pg_databases, an array of database definition objects. During cluster initialization, databases are created in definition order, so later databases can use earlier ones as templates.

Example from Pigsty demo pg-meta cluster:

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_databases:
      - { name: meta ,baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty] ,extensions: [{name: postgis, schema: public}, {name: timescaledb}]}
      - { name: grafana  ,owner: dbuser_grafana  ,revokeconn: true ,comment: grafana primary database }
      - { name: bytebase ,owner: dbuser_bytebase ,revokeconn: true ,comment: bytebase primary database }
      - { name: kong     ,owner: dbuser_kong     ,revokeconn: true ,comment: kong the api gateway database }
      - { name: gitea    ,owner: dbuser_gitea    ,revokeconn: true ,comment: gitea meta database }
      - { name: wiki     ,owner: dbuser_wiki     ,revokeconn: true ,comment: wiki meta database }
      - { name: noco     ,owner: dbuser_noco     ,revokeconn: true ,comment: nocodb database }

Each database definition is a complex object with fields below. Only name is required:

- name: meta                      # REQUIRED, `name` is the only mandatory field
  state: create                   # Optional, database state: create (default), absent, recreate
  baseline: cmdb.sql              # Optional, SQL baseline file path (relative to Ansible search path, e.g., files/)
  pgbouncer: true                 # Optional, add to pgbouncer database list? default true
  schemas: [pigsty]               # Optional, additional schemas to create, array of schema names
  extensions:                     # Optional, extensions to install: array of extension objects
    - { name: postgis , schema: public }  # Can specify schema, or omit (installs to first schema in search_path)
    - { name: timescaledb }               # Some extensions create and use fixed schemas
  comment: pigsty meta database   # Optional, database comment/description
  owner: postgres                 # Optional, database owner, defaults to current user
  template: template1             # Optional, template to use, default template1
  strategy: FILE_COPY             # Optional, clone strategy: FILE_COPY or WAL_LOG (PG15+)
  encoding: UTF8                  # Optional, inherits from template/cluster config (UTF8)
  locale: C                       # Optional, inherits from template/cluster config (C)
  lc_collate: C                   # Optional, inherits from template/cluster config (C)
  lc_ctype: C                     # Optional, inherits from template/cluster config (C)
  locale_provider: libc           # Optional, locale provider: libc, icu, builtin (PG15+)
  icu_locale: en-US               # Optional, ICU locale rules (PG15+)
  icu_rules: ''                   # Optional, ICU collation rules (PG16+)
  builtin_locale: C.UTF-8         # Optional, builtin locale provider rules (PG17+)
  tablespace: pg_default          # Optional, default tablespace
  is_template: false              # Optional, mark as template database
  allowconn: true                 # Optional, allow connections, default true
  revokeconn: false               # Optional; when true, retain CONNECT only for owner, admin, monitor, and replication users
  register_datasource: true       # Optional, register to grafana datasource? default true
  connlimit: -1                   # Optional, connection limit, -1 means unlimited
  parameters:                     # Optional, database-level params via ALTER DATABASE SET
    work_mem: '64MB'
    statement_timeout: '30s'
  pool_auth_user: dbuser_meta     # Optional, auth user for pgbouncer auth_query
  pool_mode: transaction          # Optional, database-level pgbouncer pool mode
  pool_size: 50                   # Optional, database-level pgbouncer default pool size
  pool_reserve: 30                # Optional, database-level pgbouncer reserve pool
  pool_size_min: 0                # Optional, database-level pgbouncer min pool size
  pool_connlimit: 100             # Optional, database-level max database connections

Since Pigsty v4.1.0, database pool fields are unified as pool_reserve and pool_connlimit; legacy aliases pool_size_reserve / pool_max_db_conn are converged.

Parameter Overview

The only required field is name - a valid, unique database name within the cluster. All other params have sensible defaults. Parameters marked “Immutable” only take effect at creation; changing them requires database recreation.

Field Category Type Attr Description
name Basic string Required Database name, must be valid and unique
state Basic enum Optional State: create (default), absent, recreate
owner Basic string Mutable Database owner, defaults to postgres
comment Basic string Mutable Database comment
template Template string Immutable Template database, default template1
strategy Template enum Immutable Clone strategy: FILE_COPY or WAL_LOG (PG15+)
encoding Encoding string Immutable Character encoding, default inherited (UTF8)
locale Encoding string Immutable Locale setting, default inherited (C)
lc_collate Encoding string Immutable Collation rule, default inherited (C)
lc_ctype Encoding string Immutable Character classification, default inherited (C)
locale_provider Encoding enum Immutable Locale provider: libc, icu, builtin (PG15+)
icu_locale Encoding string Immutable ICU locale rules (PG15+)
icu_rules Encoding string Immutable ICU collation customization (PG16+)
builtin_locale Encoding string Immutable Builtin locale rules (PG17+)
tablespace Storage string Mutable Default tablespace, change triggers data migration
is_template Privilege bool Mutable Mark as template database
allowconn Privilege bool Mutable Allow connections, default true
revokeconn Privilege bool Mutable Revoke PUBLIC CONNECT privilege
connlimit Privilege int Mutable Connection limit, -1 for unlimited
baseline Init string Mutable SQL baseline file path, runs on every provisioning
schemas Init (string|object)[] Mutable Schema definitions to create
extensions Init (string|object)[] Mutable Extension definitions to install
parameters Init object Mutable Database-level parameters
pgbouncer Pool bool Mutable Add to connection pool, default true
pool_mode Pool enum Mutable Pool mode: transaction (default)
pool_size Pool int Mutable Default pool size, default 50
pool_size_min Pool int Mutable Min pool size, default 0
pool_reserve Pool int Mutable Reserve pool size, default 30
pool_connlimit Pool int Mutable Max database connections, default 100
pool_auth_user Pool string Mutable Auth query user
register_datasource Monitor bool Mutable Register to Grafana datasource, default true

Parameter Details

name

String, required. Database name - must be unique within the cluster.

The current role does not enforce this regular expression, and SQL identifiers are double-quoted. However, the name is also used in temporary file paths and shell/SQL command assembly. For safe operation across the entire automation chain, keep it within 63 bytes, follow ^[A-Za-z_][A-Za-z0-9_$]{0,62}$, and avoid spaces, quotes, slashes, or other special characters.

- name: myapp              # Simple naming
- name: my_application     # Underscore separated
- name: app_v2             # Version included

state

Enum for database operation: create, absent, or recreate. Default create.

State Description
create Default, create or modify database, adjust mutable params if exists
absent Delete database with DROP DATABASE WITH (FORCE)
recreate Drop then create, for database reset 
- name: myapp                # state defaults to create
- name: olddb
  state: absent              # Delete database
- name: testdb
  state: recreate            # Rebuild database

owner

String. Database owner, defaults to pg_dbsu (postgres) if not specified.

Target user must exist. Changing owner executes (old owner retains existing privileges):

Database owner has full control including creating schemas, tables, extensions - useful for multi-tenant scenarios.

ALTER DATABASE "myapp" OWNER TO "new_owner";
GRANT ALL PRIVILEGES ON DATABASE "myapp" TO "new_owner";

comment

String. Database comment, defaults to business database {name}.

Set via COMMENT ON DATABASE, supports Chinese and special characters (Pigsty auto-escapes quotes). Stored in the shared-object comment catalog pg_shdescription, viewable via \l+.

COMMENT ON DATABASE "myapp" IS 'my main application database';
- name: myapp
  comment: my main application database

template

String, immutable. Template database for creation, default template1.

PostgreSQL’s CREATE DATABASE clones the template - new database inherits all objects, extensions, schemas, permissions. Pigsty customizes template1 during cluster init, so new databases inherit these settings.

Template Description
template1 Default, includes Pigsty pre-configured extensions/schemas/perms
template0 Clean template, required for non-default locale providers
Custom database Use existing database as template for cloning

When using icu or builtin locale provider, must specify template: template0 since template1 locale settings can’t be overridden.

- name: myapp_icu
  template: template0        # Required for ICU
  locale_provider: icu
  icu_locale: zh-Hans

Using template0 skips monitoring extensions/schemas and default privileges - allowing fully custom database.

strategy

Enum, immutable. Clone strategy: FILE_COPY or WAL_LOG. Available PG15+.

Strategy Description Use Case
FILE_COPY Direct file copy with checkpoints before and after Large templates, lower WAL volume
WAL_LOG Block-by-block copy written to WAL; PG15+ default Small templates, non-blocking

WAL_LOG doesn’t block template connections during clone but less efficient for large templates. Ignored on PG14 and earlier.

- name: cloned_db
  template: source_db
  strategy: WAL_LOG          # WAL-based cloning

encoding

String, immutable. Character encoding, inherits from template if unspecified (usually UTF8).

Strongly recommend UTF8 unless special requirements. Cannot be changed after creation.

- name: legacy_db
  template: template0        # Use template0 for non-default encoding
  encoding: LATIN1

locale

String, immutable. Locale setting - sets both lc_collate and lc_ctype. Inherits from template (usually C).

Determines string sort order and character classification. Use C or POSIX for best performance and cross-platform consistency; use language-specific locales (e.g., zh_CN.UTF-8) for proper language sorting.

- name: chinese_db
  template: template0
  locale: zh_CN.UTF-8        # Chinese locale
  encoding: UTF8

lc_collate

String, immutable. String collation rule. Inherits from template (usually C).

Determines ORDER BY and comparison results. Common values: C (byte order, fastest), C.UTF-8, en_US.UTF-8, zh_CN.UTF-8. Cannot be changed after creation.

- name: myapp
  template: template0
  lc_collate: en_US.UTF-8    # English collation
  lc_ctype: en_US.UTF-8

lc_ctype

String, immutable. Character classification rule for upper/lower case, digits, letters. Inherits from template (usually C).

Affects upper(), lower(), regex \w, etc. Cannot be changed after creation.

locale_provider

Enum, immutable. Locale implementation provider: libc, icu, or builtin. Available PG15+, default libc.

Provider Version Description
libc - OS C library, traditional default, varies by system
icu PG15+ ICU library, cross-platform consistent, more langs
builtin PG17+ PostgreSQL builtin, most efficient, C/C.UTF-8 only

Using icu or builtin requires template: template0 with corresponding icu_locale or builtin_locale.

- name: fast_db
  template: template0
  locale_provider: builtin   # Builtin provider, most efficient
  builtin_locale: C.UTF-8

icu_locale

String, immutable. ICU locale identifier. Available PG15+ when locale_provider: icu.

ICU identifiers follow BCP 47. Common values:

Value Description
en-US US English
en-GB British English
zh-Hans Simplified Chinese
zh-Hant Traditional Chinese
ja-JP Japanese
ko-KR Korean 
- name: chinese_app
  template: template0
  locale_provider: icu
  icu_locale: zh-Hans        # Simplified Chinese ICU collation
  encoding: UTF8

icu_rules

String, immutable. Custom ICU collation rules. Available PG16+.

Allows fine-tuning default sort behavior using ICU Collation Customization.

- name: custom_sort_db
  template: template0
  locale_provider: icu
  icu_locale: en-US
  icu_rules: '&V << w <<< W'  # Custom V/W sort order

builtin_locale

String, immutable. Builtin locale provider rules. Available PG17+ when locale_provider: builtin. Values: C or C.UTF-8.

builtin provider is PG17’s new builtin implementation - faster than libc with consistent cross-platform behavior. Suitable for C/C.UTF-8 collation only.

- name: fast_db
  template: template0
  locale_provider: builtin
  builtin_locale: C.UTF-8    # Builtin UTF-8 support
  encoding: UTF8

tablespace

String, mutable. Default tablespace, default pg_default.

Changing tablespace triggers physical data migration - PostgreSQL moves all objects to new tablespace. Can take long time for large databases, use cautiously.

- name: archive_db
  tablespace: slow_hdd       # Archive data on slow storage
ALTER DATABASE "archive_db" SET TABLESPACE "slow_hdd";

is_template

Boolean, mutable. Mark database as template, default false.

When true, any user with CREATEDB privilege can use this database as template for cloning. Template databases typically pre-install standard schemas, extensions, and data.

- name: app_template
  is_template: true          # Mark as template, allow user cloning
  schemas: [core, api]
  extensions: [postgis, pg_trgm]

Deleting is_template: true databases: Pigsty first executes ALTER DATABASE ... IS_TEMPLATE false then drops.

allowconn

Boolean, mutable. Allow connections, default true.

Setting false completely disables connections at database level - no user (including superuser) can connect. Used for maintenance or archival purposes.

- name: archive_db
  allowconn: false           # Disallow all connections
ALTER DATABASE "archive_db" ALLOW_CONNECTIONS false;

revokeconn

Boolean, mutable. Revoke PUBLIC CONNECT privilege, default false.

When true, Pigsty executes:

  • Revoke PUBLIC CONNECT, regular users can’t connect
  • Grant connect to replication user (replicator) and monitor user (dbuser_monitor)
  • Grant connect to admin user (dbuser_dba) and owner with WITH GRANT OPTION

Setting false restores PUBLIC CONNECT privilege.

- name: secure_db
  owner: dbuser_secure
  revokeconn: true           # Revoke public connect, only specified users

connlimit

Integer, mutable. Max concurrent connections, default -1 (unlimited).

Positive integer limits max simultaneous sessions. Doesn’t affect superusers.

- name: limited_db
  connlimit: 50              # Max 50 concurrent connections
ALTER DATABASE "limited_db" CONNECTION LIMIT 50;

baseline

String. SQL baseline file path executed while provisioning the database.

Baseline files typically contain schema definitions, initial data, stored procedures. Path is relative to Ansible search path, usually in files/.

Whenever baseline is defined, the current role runs the file on every provisioning pass for that database, even if the database already exists. It also runs after state: recreate. Make the baseline SQL idempotent, or avoid rerunning it against an existing database.

- name: myapp
  baseline: myapp_schema.sql  # Looks for files/myapp_schema.sql

schemas

Array, mutable (add/remove). Schema definitions to create or drop. Elements can be strings or objects.

Simple format - strings for schema names (create only):

schemas:
  - app
  - api
  - core

Full format - objects for owner and drop operations:

schemas:
  - name: app                # Schema name (required)
    owner: dbuser_app        # Schema owner (optional), generates AUTHORIZATION clause
  - name: deprecated
    state: absent            # Drop schema (CASCADE)

Create uses IF NOT EXISTS; drop uses CASCADE (deletes all objects in schema).

CREATE SCHEMA IF NOT EXISTS "app" AUTHORIZATION "dbuser_app";
DROP SCHEMA IF EXISTS "deprecated" CASCADE;

extensions

Array, mutable (add/remove). Extension definitions to install or uninstall. Elements can be strings or objects.

Simple format - strings for extension names (install only):

extensions:
  - postgis
  - pg_trgm
  - vector

Full format - objects for schema, version, and uninstall:

extensions:
  - name: vector             # Extension name (required)
    schema: public           # Install to schema (optional)
    version: '0.5.1'         # Specific version (optional)
  - name: old_extension
    state: absent            # Uninstall extension (CASCADE)

Installation uses IF NOT EXISTS ... CASCADE; PostgreSQL emits a NOTICE and skips an extension that already exists, while automatically installing dependencies when possible. Uninstallation uses CASCADE and deletes dependent objects.

CREATE EXTENSION IF NOT EXISTS "vector" WITH SCHEMA "public" VERSION '0.5.1' CASCADE;
DROP EXTENSION IF EXISTS "old_extension" CASCADE;

parameters

Object, mutable. Database-level config params via ALTER DATABASE ... SET. Applies to all sessions connecting to this database.

- name: analytics
  parameters:
    work_mem: '256MB'
    maintenance_work_mem: '512MB'
    statement_timeout: '5min'
    search_path: 'analytics,public'

Use special value DEFAULT (case-insensitive) to reset to PostgreSQL default:

parameters:
  work_mem: DEFAULT          # Reset to default
  statement_timeout: '30s'   # Set new value
ALTER DATABASE "myapp" SET "work_mem" = DEFAULT;
ALTER DATABASE "myapp" SET "statement_timeout" = '30s';

pgbouncer

Boolean, mutable. Add database to Pgbouncer pool list, default true.

Setting false excludes database from Pgbouncer - clients can’t access via connection pool. For internal management databases or direct-connect scenarios.

- name: internal_db
  pgbouncer: false           # No connection pool access

pool_mode

Enum, mutable. Pgbouncer pool mode: transaction, session, or statement. Default transaction.

Mode Description Use Case
transaction Return connection after txn Most OLTP apps, default
session Return connection after session Apps needing session state
statement Return after each statement Simple stateless queries 
- name: session_app
  pool_mode: session         # Session-level pooling

pool_size

Integer, mutable. Pgbouncer default pool size, default 50.

Pool size is the regular backend-connection limit for this database’s pool; pool_size_min controls prewarmed connections. Adjust it for the workload.

- name: high_load_db
  pool_size: 128             # Larger pool for high load

pool_size_min

Integer, mutable. Pgbouncer minimum pool size, default 0.

Values > 0 pre-create specified backend connections for connection warming, reducing first-request latency.

- name: latency_sensitive
  pool_size_min: 10          # Pre-warm 10 connections

pool_reserve

Integer, mutable. Pgbouncer reserve pool size, default 30.

When default pool exhausted, Pgbouncer can allocate up to pool_reserve additional connections for burst traffic.

- name: bursty_db
  pool_size: 50
  pool_reserve: 30           # Allow up to 30 more connections after the regular pool is exhausted

pool_connlimit

Integer, mutable. Max connections via Pgbouncer pool, default 100.

This is Pgbouncer-level limit, independent of database’s connlimit param.

- name: limited_pool_db
  pool_connlimit: 50         # Pool max 50 connections

pool_auth_user

String, mutable. User for Pgbouncer auth query.

Requires pgbouncer_auth_query enabled. When set, all Pgbouncer connections to this database use specified user for auth query password verification.

- name: myapp
  pool_auth_user: dbuser_monitor  # Use monitor user for auth query

register_datasource

Boolean, mutable. Register database to Grafana as PostgreSQL datasource, default true.

Set false to skip Grafana registration. For temp databases, test databases, or internal databases not needed in monitoring.

- name: temp_db
  register_datasource: false  # Don't register to Grafana

Template Inheritance

Many parameters inherit from template database if not explicitly specified. Default template is template1, whose encoding settings are determined by cluster init params:

Cluster Param Default Description
pg_encoding UTF8 Cluster encoding
pg_locale C / C-UTF-8 (if supported) Cluster locale
pg_lc_collate C / C-UTF-8 (if supported) Cluster collation
pg_lc_ctype C / C-UTF-8 (if supported) Cluster ctype

New databases fork from template1, which is customized during PG_PROVISION with extensions, schemas, and default privileges. Unless you explicitly use another template.

Deep Customization

Pigsty provides rich customization params. To customize template database, refer to:

If above configurations don’t meet your needs, use pg_init to specify custom cluster init scripts:

Locale Providers

PostgreSQL 15+ introduced locale_provider for different locale implementations. These are immutable after creation.

Pigsty’s configure wizard selects builtin C.UTF-8/C locale provider based on PG and OS versions. Databases inherit cluster locale by default. To specify different locale provider, you must use template0.

Using ICU provider (PG15+):

- name: myapp_icu
  template: template0        # ICU requires template0
  locale_provider: icu
  icu_locale: en-US          # ICU locale rules
  encoding: UTF8

Using builtin provider (PG17+):

- name: myapp_builtin
  template: template0
  locale_provider: builtin
  builtin_locale: C.UTF-8    # Builtin locale rules
  encoding: UTF8

Provider comparison: libc (traditional, OS-dependent), icu (PG15+, cross-platform, feature-rich), builtin (PG17+, most efficient C/C.UTF-8).

Connection Pool

Pgbouncer connection pool optimizes short-connection performance, reduces contention, prevents excessive connections from overwhelming database, and provides flexibility during migrations.

Pigsty configures 1:1 connection pool for each PostgreSQL instance, running as same pg_dbsu (default postgres OS user). Pool communicates with database via /var/run/postgresql Unix socket.

Pigsty adds all databases in pg_databases to pgbouncer by default. Set pgbouncer: false to exclude specific databases. Pgbouncer database list and config params are defined in /etc/pgbouncer/database.txt:

meta                        = host=/var/run/postgresql mode=session
grafana                     = host=/var/run/postgresql mode=transaction
bytebase                    = host=/var/run/postgresql auth_user=dbuser_meta
kong                        = host=/var/run/postgresql pool_size=32 reserve_pool=64
gitea                       = host=/var/run/postgresql min_pool_size=10
wiki                        = host=/var/run/postgresql
noco                        = host=/var/run/postgresql
mongo                       = host=/var/run/postgresql

When creating databases, Pgbouncer database list is refreshed via online reload - doesn’t affect existing connections.

8.1.6 - HBA Rules

Configuration reference for PostgreSQL and PgBouncer Host-Based Authentication (HBA) rules in Pigsty.

Overview

HBA (Host-Based Authentication) controls “who can connect to the database, from where, and how”. See Authentication for the authentication model and default rules. Pigsty manages HBA rules declaratively through pg_default_hba_rules and pg_hba_rules.

Pigsty renders the following config files during cluster init or HBA refresh:

Config File Path Description
PostgreSQL HBA /pg/data/pg_hba.conf PostgreSQL server HBA rules
PgBouncer HBA /etc/pgbouncer/pgb_hba.conf Connection pool HBA rules

HBA rules are controlled by these parameters:

Parameter Level Description
pg_default_hba_rules G PostgreSQL global default HBA
pg_hba_rules G/C/I PostgreSQL cluster/instance add
pgb_default_hba_rules G PgBouncer global default HBA
pgb_hba_rules G/C/I PgBouncer cluster/instance add

Rule features:

  • Role filtering: Rules support role field, auto-filter based on instance’s pg_role
  • Order sorting: Rules support order field, controls position in final config file
  • Two syntaxes: Supports alias form (simplified) and raw form (direct HBA text)

Refresh HBA

After modifying config, re-render config files and reload services:

bin/pgsql-hba <cls>                   # Refresh entire cluster HBA (recommended)
bin/pgsql-hba <cls> <ip>...           # Refresh specific instances in cluster

Script executes the following playbook:

./pgsql.yml -l <cls> -t pg_hba,pg_reload,pgbouncer_hba,pgbouncer_reload -e pg_reload=true

PostgreSQL only: ./pgsql.yml -l <cls> -t pg_hba,pg_reload -e pg_reload=true

PgBouncer only: ./pgsql.yml -l <cls> -t pgbouncer_hba,pgbouncer_reload

Don’t edit config files directly

Don’t directly edit /pg/data/pg_hba.conf or /etc/pgbouncer/pgb_hba.conf - they’ll be overwritten on next playbook run. All changes should be made in pigsty.yml, then execute bin/pgsql-hba to refresh.

Parameter Details

pg_default_hba_rules

PostgreSQL global default HBA rule list, usually defined in all.vars, provides base access control for all clusters.

  • Type: rule[], Level: Global (G)
pg_default_hba_rules:
  - {user: '${dbsu}'    ,db: all         ,addr: local     ,auth: ident ,title: 'dbsu access via local os user ident'  ,order: 100}
  - {user: '${dbsu}'    ,db: replication ,addr: local     ,auth: ident ,title: 'dbsu replication from local os ident' ,order: 150}
  - {user: '${repl}'    ,db: replication ,addr: localhost ,auth: pwd   ,title: 'replicator replication from localhost',order: 200}
  - {user: '${repl}'    ,db: replication ,addr: intra     ,auth: pwd   ,title: 'replicator replication from intranet' ,order: 250}
  - {user: '${repl}'    ,db: postgres    ,addr: intra     ,auth: pwd   ,title: 'replicator postgres db from intranet' ,order: 300}
  - {user: '${monitor}' ,db: all         ,addr: localhost ,auth: pwd   ,title: 'monitor from localhost with password' ,order: 350}
  - {user: '${monitor}' ,db: all         ,addr: infra     ,auth: pwd   ,title: 'monitor from infra host with password',order: 400}
  - {user: '${admin}'   ,db: all         ,addr: infra     ,auth: ssl   ,title: 'admin @ infra nodes with pwd & ssl'   ,order: 450}
  - {user: '${admin}'   ,db: all         ,addr: world     ,auth: ssl   ,title: 'admin @ everywhere with ssl & pwd'    ,order: 500}
  - {user: '+dbrole_readonly',db: all    ,addr: localhost ,auth: pwd   ,title: 'pgbouncer read/write via local socket',order: 550}
  - {user: '+dbrole_readonly',db: all    ,addr: intra     ,auth: pwd   ,title: 'read/write biz user via password'     ,order: 600}
  - {user: '+dbrole_offline' ,db: all    ,addr: intra     ,auth: pwd   ,title: 'allow etl offline tasks from intranet',order: 650}

pg_hba_rules

PostgreSQL cluster/instance-level additional HBA rules, can override at cluster or instance level, merged with default rules and sorted by order.

  • Type: rule[], Level: Global/Cluster/Instance (G/C/I), Default: []
pg_hba_rules:
  - {user: app_user, db: app_db, addr: intra, auth: pwd, title: 'app user access'}

pgb_default_hba_rules

PgBouncer global default HBA rule list, usually defined in all.vars.

  • Type: rule[], Level: Global (G)
pgb_default_hba_rules:
  - {user: '${dbsu}'    ,db: pgbouncer   ,addr: local     ,auth: peer  ,title: 'dbsu local admin access with os ident',order: 100}
  - {user: 'all'        ,db: all         ,addr: localhost ,auth: pwd   ,title: 'allow all user local access with pwd' ,order: 150}
  - {user: '${monitor}' ,db: pgbouncer   ,addr: intra     ,auth: pwd   ,title: 'monitor access via intranet with pwd' ,order: 200}
  - {user: '${monitor}' ,db: all         ,addr: world     ,auth: deny  ,title: 'reject all other monitor access addr' ,order: 250}
  - {user: '${admin}'   ,db: all         ,addr: intra     ,auth: pwd   ,title: 'admin access via intranet with pwd'   ,order: 300}
  - {user: '${admin}'   ,db: all         ,addr: world     ,auth: deny  ,title: 'reject all other admin access addr'   ,order: 350}
  - {user: 'all'        ,db: all         ,addr: intra     ,auth: pwd   ,title: 'allow all user intra access with pwd' ,order: 400}

pgb_hba_rules

PgBouncer cluster/instance-level additional HBA rules.

  • Type: rule[], Level: Global/Cluster/Instance (G/C/I), Default: []

Note: PgBouncer HBA does not support db: replication.

Rule Fields

Each HBA rule is a YAML dict supporting these fields:

Field Type Required Default Description
user string No all Username, supports all, placeholders, +rolename
db string No all Database name, supports all, replication, db name
addr string Yes* - Address alias or CIDR, see Address Aliases
auth string No pwd Auth method alias, see Auth Methods
title string No - Rule description, rendered as comment in config
role string No common Instance role filter, see Role Filtering
order int No 1000 Sort weight, lower first, see Order Sorting
rules list Yes* - Raw HBA text lines, mutually exclusive with addr

Either addr or rules must be specified. Use rules to write raw HBA format directly.

Address Aliases

Pigsty provides address aliases to simplify HBA rule writing:

Alias Expands To Description
local Unix socket Local Unix socket
localhost Unix socket + 127.0.0.1/32 + ::1/128 Loopback addresses
admin ${admin_ip}/32 Admin IP address
infra All infra group node IPs Infrastructure nodes
cluster All current cluster member IPs Same cluster instances
intra / intranet 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16 Intranet CIDRs
world / all 0.0.0.0/0 + ::/0 Any address (IPv4 + IPv6)
<CIDR> Direct use e.g., 192.168.1.0/24

Intranet CIDRs can be customized via node_firewall_intranet:

node_firewall_intranet:
  - 10.0.0.0/8
  - 172.16.0.0/12
  - 192.168.0.0/16

Auth Methods

Pigsty provides auth method aliases for simplified config:

Alias Actual Method Connection Type Description
pwd scram-sha-256 or md5 host Auto-select based on pg_pwd_enc
ssl scram-sha-256 or md5 hostssl Force SSL + password
ssl-sha scram-sha-256 hostssl Force SSL + SCRAM-SHA-256
ssl-md5 md5 hostssl Force SSL + MD5
cert cert hostssl Client certificate auth
trust trust host Unconditional trust (dangerous)
deny / reject reject host Reject connection
ident ident host OS user mapping (PostgreSQL)
peer peer local OS user mapping (PgBouncer/local)

pg_pwd_enc defaults to scram-sha-256, can be set to md5 for legacy client compatibility.

User Variables

HBA rules support these user placeholders, auto-replaced with actual usernames during rendering:

Placeholder Default Corresponding Param
${dbsu} postgres pg_dbsu
${repl} replicator pg_replication_username
${monitor} dbuser_monitor pg_monitor_username
${admin} dbuser_dba pg_admin_username

Role Filtering

The role field in HBA rules controls which instances the rule applies to:

Role Description
common Default, applies to all instances
primary Primary instance only
replica Replica instance only
offline Offline instance only (pg_role: offline or pg_offline_query: true)
standby Standby instance
delayed Delayed replica instance

Role filtering matches based on instance’s pg_role variable. Non-matching rules are commented out (prefixed with #).

pg_hba_rules:
  # Only applies on primary: writer can only connect to primary
  - {user: writer, db: all, addr: intra, auth: pwd, role: primary, title: 'writer only on primary'}

  # Only applies on offline instances: ETL dedicated network
  - {user: '+dbrole_offline', db: all, addr: '172.20.0.0/16', auth: ssl, role: offline, title: 'offline dedicated'}

Order Sorting

PostgreSQL HBA is first-match-wins, rule order is critical. Pigsty controls rule rendering order via the order field.

Order Interval Convention

Interval Usage
0 - 99 User high-priority rules (before all defaults)
100 - 650 Default rule zone (spaced by 50 for insertion)
1000+ User rule default (rules without order)

PostgreSQL Default Rules Order

Order Rule Description
100 dbsu local ident
150 dbsu replication local
200 replicator localhost
250 replicator intra replication
300 replicator intra postgres
350 monitor localhost
400 monitor infra
450 admin infra ssl
500 admin world ssl
550 dbrole_readonly localhost
600 dbrole_readonly intra
650 dbrole_offline intra

PgBouncer Default Rules Order

Order Rule Description
100 dbsu local peer
150 all localhost pwd
200 monitor pgbouncer intra
250 monitor world deny
300 admin intra pwd
350 admin world deny
400 all intra pwd

Syntax Examples

Alias Form: Using Pigsty simplified syntax

pg_hba_rules:
  - title: allow grafana view access
    role: primary
    user: dbuser_view
    db: meta
    addr: infra
    auth: ssl

Rendered result:

# allow grafana view access [primary]
hostssl  meta               dbuser_view        10.10.10.10/32     scram-sha-256

Raw Form: Using PostgreSQL HBA syntax directly

pg_hba_rules:
  - title: allow intranet password access
    role: common
    rules:
      - host all all 10.0.0.0/8 scram-sha-256
      - host all all 172.16.0.0/12 scram-sha-256
      - host all all 192.168.0.0/16 scram-sha-256

Rendered result:

# allow intranet password access [common]
host all all 10.0.0.0/8 scram-sha-256
host all all 172.16.0.0/12 scram-sha-256
host all all 192.168.0.0/16 scram-sha-256

Common Scenarios

Blacklist IP: Use order: 0 to ensure first match

pg_hba_rules:
  - {user: all, db: all, addr: '10.1.1.100/32', auth: deny, order: 0, title: 'block bad ip'}

Whitelist App Server: High priority for specific IP

pg_hba_rules:
  - {user: app_user, db: app_db, addr: '192.168.1.10/32', auth: ssl, order: 50, title: 'app server'}

Admin Force Certificate: Override default SSL password auth

pg_hba_rules:
  - {user: '${admin}', db: all, addr: world, auth: cert, order: 10, title: 'admin cert only'}

Offline Instance Dedicated Network: Only on offline instances

pg_hba_rules:
  - {user: '+dbrole_offline', db: all, addr: '172.20.0.0/16', auth: ssl-sha, role: offline, title: 'etl network'}

Restrict Access by Database: Sensitive databases limited to specific networks

pg_hba_rules:
  - {user: fin_user, db: finance_db, addr: '10.20.0.0/16', auth: ssl, title: 'finance only'}
  - {user: hr_user, db: hr_db, addr: '10.30.0.0/16', auth: ssl, title: 'hr only'}

PgBouncer Dedicated Rules: Note no db: replication support

pgb_hba_rules:
  - {user: '+dbrole_readwrite', db: all, addr: world, auth: ssl, title: 'app via pgbouncer'}

Complete Cluster Example 

pg-prod:
  hosts:
    10.10.10.11: {pg_seq: 1, pg_role: primary}
    10.10.10.12: {pg_seq: 2, pg_role: replica}
    10.10.10.13: {pg_seq: 3, pg_role: offline}
  vars:
    pg_cluster: pg-prod

    pg_hba_rules:
      # Blacklist: known malicious IP (highest priority)
      - {user: all, db: all, addr: '10.1.1.100/32', auth: deny, order: 0, title: 'blacklist'}

      # App server whitelist (high priority)
      - {user: app_user, db: app_db, addr: '192.168.1.0/24', auth: ssl, order: 50, title: 'app servers'}

      # ETL tasks: offline instances only
      - {user: etl_user, db: all, addr: '172.20.0.0/16', auth: pwd, role: offline, title: 'etl tasks'}

      # Cluster internal monitoring
      - {user: '${monitor}', db: all, addr: cluster, auth: pwd, order: 380, title: 'cluster monitor'}

    pgb_hba_rules:
      # App via connection pool
      - {user: '+dbrole_readwrite', db: all, addr: '192.168.1.0/24', auth: ssl, title: 'app via pgbouncer'}

Verification & Troubleshooting

View Current HBA Rules

psql -c "TABLE pg_hba_file_rules"         # View via SQL (recommended)
cat /pg/data/pg_hba.conf                  # View PostgreSQL HBA file
cat /etc/pgbouncer/pgb_hba.conf           # View PgBouncer HBA file
grep '^#' /pg/data/pg_hba.conf | head -20 # View rule titles (verify order)

Test Connection Auth

psql -h <host> -p 5432 -U <user> -d <db> -c "SELECT 1"

Common Issues

Error Message Possible Cause Solution
no pg_hba.conf entry for host... No matching HBA rule Add corresponding rule and refresh
password authentication failed Wrong password or enc Check password and pg_pwd_enc
Rule not taking effect Not refreshed or order Run bin/pgsql-hba, check order

Important Notes

  1. Order sensitive: PostgreSQL HBA is first-match-wins, use order wisely
  2. Role matching: Ensure role field matches target instance’s pg_role
  3. Address format: CIDR must be correct, e.g., 10.0.0.0/8 not 10.0.0.0/255.0.0.0
  4. PgBouncer limitation: Does not support db: replication
  5. TLS prerequisite: ssl and cert require server-side TLS; clients must still use verify-full to authenticate the server
  6. Test first: Validate in test environment before modifying HBA
  7. Refresh on scale: Rules using addr: cluster need refresh after cluster membership changes

Related Documentation

8.1.7 - Access Control

Configuration reference for Pigsty built-in roles, users, default privileges, and database ACLs.

Access control combines roles, object privileges, database ACLs, and HBA. This page covers configuration parameters; see Access Control Concepts for design and boundaries.

Pigsty provides a compact ACL model described by these parameters:

  • pg_default_roles: system roles and system users.
  • pg_users: application users and roles.
  • pg_default_privileges: default privileges on objects created by managed administrators and owners.
  • pg_revoke_public, pg_default_schemas, and pg_default_extensions: default behavior for template1.

Manage these parameters together with HBA and database definitions to produce reproducible access-control configuration.

Default Role System (pg_default_roles)

The defaults contain four business roles and four system users:

Name Type Description
dbrole_readonly NOLOGIN Shared read-only role with SELECT and USAGE
dbrole_readwrite NOLOGIN Inherits read-only and adds INSERT, UPDATE, and DELETE
dbrole_admin NOLOGIN Inherits pg_monitor and read-write; can create objects and triggers
dbrole_offline NOLOGIN Independent read-only role; instance scope must be restricted explicitly through HBA
postgres User System superuser; same name as pg_dbsu
replicator User Streaming replication and backup; inherits monitoring and read-only privileges
dbuser_dba User Primary administration account, also synchronized to PgBouncer
dbuser_monitor User Monitoring account with pg_monitor; records slow SQL by default

These definitions live in pg_default_roles. The parameter is a complete list. When customizing it, copy and retain the required default roles and system users, then add new roles in dependency order. If a role name changes, update references in HBA, default privileges, and scripts.

Default Users and Credential Parameters

These parameters control system-user names and passwords:

Parameter Default Purpose
pg_dbsu postgres Database and OS superuser
pg_dbsu_password Empty string dbsu password, disabled by default
pg_replication_username replicator Replication user name
pg_replication_password DBUser.Replicator Replication password
pg_admin_username dbuser_dba Administrator user name
pg_admin_password DBUser.DBA Administrator password
pg_monitor_username dbuser_monitor Monitoring user
pg_monitor_password DBUser.Monitor Monitoring password

After changing these parameters, update the corresponding user definitions in pg_default_roles so user names and role attributes remain consistent.

Application Roles and Grants (pg_users)

Declare application users with pg_users; see User Configuration for field details. The roles field grants business roles.

Example read-only and read-write users:

pg_users:
  - { name: app_reader,  password: DBUser.Reader,  roles: [dbrole_readonly],  pgbouncer: true }
  - { name: app_writer,  password: DBUser.Writer,  roles: [dbrole_readwrite], pgbouncer: true }

Application users inherit default object privileges through dbrole_*. Database CONNECT privileges and pg_hba_rules continue to control which databases and sources can connect.

For finer ACLs, use standard GRANT and REVOKE in baseline SQL or a later playbook, and include those additional grants in reviews.

Default Privilege Template (pg_default_privileges)

pg_default_privileges applies to pg_dbsu, pg_admin_username, dbrole_admin, and every declared database owner. The default template is:

pg_default_privileges:
  - GRANT USAGE      ON SCHEMAS   TO dbrole_readonly
  - GRANT SELECT     ON TABLES    TO dbrole_readonly
  - GRANT SELECT     ON SEQUENCES TO dbrole_readonly
  - GRANT EXECUTE    ON FUNCTIONS TO dbrole_readonly
  - GRANT USAGE      ON SCHEMAS   TO dbrole_offline
  - GRANT SELECT     ON TABLES    TO dbrole_offline
  - GRANT SELECT     ON SEQUENCES TO dbrole_offline
  - GRANT EXECUTE    ON FUNCTIONS TO dbrole_offline
  - GRANT INSERT     ON TABLES    TO dbrole_readwrite
  - GRANT UPDATE     ON TABLES    TO dbrole_readwrite
  - GRANT DELETE     ON TABLES    TO dbrole_readwrite
  - GRANT USAGE      ON SEQUENCES TO dbrole_readwrite
  - GRANT UPDATE     ON SEQUENCES TO dbrole_readwrite
  - GRANT TRUNCATE   ON TABLES    TO dbrole_admin
  - GRANT REFERENCES ON TABLES    TO dbrole_admin
  - GRANT TRIGGER    ON TABLES    TO dbrole_admin
  - GRANT CREATE     ON SCHEMAS   TO dbrole_admin

Objects created by these identities receive the corresponding privileges automatically. Other object creators need their own ALTER DEFAULT PRIVILEGES configuration.

Additional notes:

  • pg_revoke_public defaults to true, revoking CREATE from PUBLIC on databases and the public schema.
  • pg_default_schemas and pg_default_extensions control schemas and extensions created in template1/postgres, usually for monitoring objects such as the monitor schema and pg_stat_statements.

Common Scenarios

Read-only Account for a Partner 

pg_users:
  - name: partner_ro
    password: Partner.Read
    roles: [dbrole_readonly]
pg_hba_rules:
  - { user: partner_ro, db: analytics, addr: 203.0.113.0/24, auth: ssl }

This adds an HBA rule allowing the partner to reach analytics over TLS from the specified CIDR. pg_hba_rules does not remove broader default rules. If the account must reach only this database, also narrow the default HBA policy and configure database CONNECT privileges.

DDL for an Application Administrator 

pg_users:
  - name: app_admin
    password: DBUser.AppAdmin
    roles: [dbrole_admin]

app_admin inherits DDL privileges from dbrole_admin. To apply the default privileges configured for dbrole_admin to new objects, run SET ROLE dbrole_admin first. If app_admin is a declared database owner, it can also create objects directly as that owner.

Custom Default Privileges 

pg_default_privileges:
  - GRANT INSERT,UPDATE,DELETE ON TABLES TO dbrole_admin
  - GRANT SELECT,UPDATE ON SEQUENCES TO dbrole_admin
  - GRANT SELECT ON TABLES TO reporting_group

This parameter replaces the complete default privilege list. Referenced roles must already exist. Changes affect only objects created afterward; grant privileges separately on existing objects.

Integration with Other Components

  • HBA rules: use pg_hba_rules to bind roles, databases, and sources. To restrict dbrole_offline, set role: offline on its rule.
  • PgBouncer: users with pgbouncer: true are written to userlist.txt; pool_mode and pool_connlimit control pool-level quotas.
  • Database monitoring: dbuser_monitor receives privileges from pg_default_roles. When adding another monitoring user, grant pg_monitor and check access to the monitor schema.

These parameters can be versioned with the inventory. Continue to review effective privileges through PostgreSQL catalogs.

Related Documentation

8.1.8 - Parameters

Configure PostgreSQL parameters at cluster, instance, database, and user levels

PostgreSQL parameters can be configured at multiple levels with different scopes and precedence. Pigsty supports four configuration levels, from global to local:

Level Scope Configuration Method Storage Location
Cluster All instances in cluster Patroni DCS / Tuning Templates etcd + postgresql.conf
Instance Single PG instance pg_parameters / ALTER SYSTEM postgresql.auto.conf
Database All sessions in a DB pg_databases[].parameters pg_db_role_setting
User All sessions of a user pg_users[].parameters pg_db_role_setting

Priority from low to high: Cluster < Instance < Database < User < Session (SET command). Higher priority settings override lower ones.

For complete PostgreSQL parameter documentation, see PostgreSQL Docs: Server Configuration.

Cluster Level

Cluster-level parameters are shared across all instances (primary and replicas) in a PostgreSQL cluster. In Pigsty, cluster parameters are managed via Patroni and stored in DCS (etcd by default).

Pigsty provides four pre-configured Patroni tuning templates optimized for different workloads, specified via pg_conf:

Template Use Case Characteristics
oltp.yml OLTP transactions Low latency, high concurrency (default)
olap.yml OLAP analytics Large queries, high throughput
crit.yml Critical/Financial Max durability, safety over perf
tiny.yml Tiny instances Resource-constrained, dev/test

Template files are located in roles/pgsql/templates/ and contain auto-calculated values based on hardware specs. Templates are rendered to /etc/patroni/patroni.yml during cluster initialization. See Tuning Templates for details.

Before cluster creation, you can adjust these templates to modify initial parameters. Once initialized, parameter changes should be made via Patroni’s configuration management.

Patroni DCS Config

Patroni stores cluster config in DCS (etcd by default), ensuring consistent configuration across all members.

Storage Structure:

/pigsty/                          # namespace (patroni_namespace)
  └── pg-meta/                    # cluster name (pg_cluster)
      ├── config                  # cluster config (shared)
      ├── leader                  # current primary info
      ├── members/                # member registration
      │   ├── pg-meta-1
      │   └── pg-meta-2
      └── ...

Rendering Flow:

  1. Init: Template (e.g., oltp.yml) rendered via Jinja2 to /etc/patroni/patroni.yml
  2. Start: Patroni reads local config, writes PostgreSQL parameters to DCS
  3. Runtime: Patroni periodically syncs DCS config to local PostgreSQL

Local Cache:

Each Patroni instance caches DCS config locally at /pg/conf/<instance>.yml:

  • On start: Load from DCS, cache locally
  • Runtime: Periodically sync DCS to local cache
  • DCS unavailable: Continue with local cache (no failover possible)

Config File Hierarchy

Patroni renders DCS config to local PostgreSQL config files:

/pg/data/
├── postgresql.conf          # Main config (managed by Patroni)
├── postgresql.base.conf     # Base config (via include directive)
├── postgresql.auto.conf     # Instance overrides (ALTER SYSTEM)
├── pg_hba.conf              # Client auth config
└── pg_ident.conf            # User mapping config

Load Order (priority low to high):

  1. postgresql.conf: Dynamically generated by Patroni with DCS cluster params
  2. postgresql.base.conf: Loaded via include, static base config
  3. postgresql.auto.conf: Auto-loaded by PostgreSQL, instance overrides

Since postgresql.auto.conf loads last, its parameters override earlier files.

Instance Level

Instance-level parameters apply only to a single PostgreSQL instance, overriding cluster-level config. These are written to postgresql.auto.conf, which loads last and can override any cluster parameter.

This is a powerful technique for setting instance-specific values:

  • Set hot_standby_feedback = on on replicas
  • Adjust work_mem or maintenance_work_mem for specific instances
  • Set recovery_min_apply_delay for delayed replicas

Using pg_parameters

In Pigsty config, use pg_parameters to define instance-level parameters:

pg-meta:
  hosts:
    10.10.10.10:
      pg_seq: 1
      pg_role: primary
      pg_parameters:                              # instance-level params
        log_statement: all                        # log all SQL for this instance only
  vars:
    pg_cluster: pg-meta
    pg_parameters:                                # cluster default instance params
      log_timezone: Asia/Shanghai
      log_min_duration_statement: 1000

Use ./pgsql.yml -l <cls> -t pg_param to apply parameters, which renders to postgresql.auto.conf.

Override Hierarchy

pg_parameters can be defined at different Ansible config levels, priority low to high:

all:
  vars:
    pg_parameters:                    # global default
      log_statement: none

  children:
    pg-meta:
      vars:
        pg_parameters:                # cluster override
          log_statement: ddl
      hosts:
        10.10.10.10:
          pg_parameters:              # instance override (highest)
            log_statement: all

Using ALTER SYSTEM

You can also modify instance parameters at runtime via ALTER SYSTEM:

-- Set parameters
ALTER SYSTEM SET work_mem = '256MB';
ALTER SYSTEM SET log_min_duration_statement = 1000;

-- Reset to default
ALTER SYSTEM RESET work_mem;
ALTER SYSTEM RESET ALL;  -- Reset all ALTER SYSTEM settings

-- Reload config to take effect
SELECT pg_reload_conf();

ALTER SYSTEM writes to postgresql.auto.conf.

Note: In Pigsty-managed clusters, postgresql.auto.conf is managed by Ansible via pg_parameters. Manual ALTER SYSTEM changes may be overwritten on next playbook run. Use pg_parameters in pigsty.yml for persistent instance-level params.

List-Type Parameters

PostgreSQL has special parameters accepting comma-separated lists. In YAML config, the entire value must be quoted, otherwise YAML parses it as an array:

# Correct: quote the entire value
pg_parameters:
  shared_preload_libraries: 'timescaledb, pg_stat_statements'
  search_path: '"$user", public, app'

# Wrong: unquoted causes YAML parse error
pg_parameters:
  shared_preload_libraries: timescaledb, pg_stat_statements   # YAML parses as array!

Pigsty auto-detects these list parameters and renders them without outer quotes:

Parameter Description Example Value
shared_preload_libraries Preload shared libs 'timescaledb, pg_stat_statements'
search_path Schema search path '"$user", public, app'
local_preload_libraries Local preload libs 'auto_explain'
session_preload_libraries Session preload libs 'pg_hint_plan'
log_destination Log output targets 'csvlog, stderr'
unix_socket_directories Unix socket dirs '/var/run/postgresql, /tmp'
temp_tablespaces Temp tablespaces 'ssd_space, hdd_space'
debug_io_direct Direct I/O mode (PG16+) 'data, wal'

Rendering Example:

# pigsty.yml config (quotes required in YAML)
pg_parameters:
  shared_preload_libraries: 'timescaledb, pg_stat_statements'
  search_path: '"$user", public, app'
  work_mem: 64MB
# Rendered postgresql.auto.conf (list params unquoted)
shared_preload_libraries = timescaledb, pg_stat_statements
search_path = "$user", public, app
work_mem = '64MB'

Database Level

Database-level parameters apply to all sessions connected to a specific database. Implemented via ALTER DATABASE ... SET, stored in pg_db_role_setting.

Configuration

Use the parameters field in pg_databases:

pg_databases:
  - name: analytics
    owner: dbuser_analyst
    parameters:
      work_mem: 256MB                              # analytics needs more memory
      maintenance_work_mem: 1GB                    # large table maintenance
      statement_timeout: 10min                     # allow long queries
      search_path: '"$user", public, mart'         # list param needs quotes

Like instance-level params, list-type values must be quoted in YAML.

Rendering Rules

Database params are set via ALTER DATABASE ... SET. Pigsty auto-selects correct syntax:

List-type params (search_path, temp_tablespaces, local_preload_libraries, session_preload_libraries, log_destination) without outer quotes:

ALTER DATABASE "analytics" SET "search_path" = "$user", public, mart;

Scalar params with quoted values:

ALTER DATABASE "analytics" SET "work_mem" = '256MB';
ALTER DATABASE "analytics" SET "statement_timeout" = '10min';

Note: While log_destination is in the database whitelist, its context is sighup, so it cannot take effect at database level. Configure it at instance level (pg_parameters).

View Database Params 

-- View params for a specific database
SELECT datname, unnest(setconfig) AS setting
FROM pg_db_role_setting drs
JOIN pg_database d ON d.oid = drs.setdatabase
WHERE drs.setrole = 0 AND datname = 'analytics';

Manual Management 

-- Set params
ALTER DATABASE analytics SET work_mem = '256MB';
ALTER DATABASE analytics SET search_path = "$user", public, myschema;

-- Reset params
ALTER DATABASE analytics RESET work_mem;
ALTER DATABASE analytics RESET ALL;

User Level

User-level parameters apply to all sessions of a specific database user. Implemented via ALTER USER ... SET, also stored in pg_db_role_setting.

Configuration

Use the parameters field in pg_users or pg_default_roles:

pg_users:
  - name: dbuser_analyst
    password: DBUser.Analyst
    parameters:
      work_mem: 256MB                              # more memory for analytics
      statement_timeout: 5min                      # allow longer queries
      search_path: '"$user", public, analytics'    # list param needs quotes
      log_statement: all                           # log all SQL

Rendering Rules

Same as database-level:

List-type params (search_path, temp_tablespaces, local_preload_libraries, session_preload_libraries) without outer quotes:

ALTER USER "dbuser_analyst" SET "search_path" = "$user", public, analytics;

Scalar params with quoted values:

ALTER USER "dbuser_analyst" SET "work_mem" = '256MB';
ALTER USER "dbuser_analyst" SET "statement_timeout" = '5min';

DEFAULT Value

Use DEFAULT (case-insensitive) to reset a parameter to PostgreSQL default:

parameters:
  work_mem: DEFAULT          # reset to default
  statement_timeout: 30s     # set specific value
ALTER USER "dbuser_app" SET "work_mem" = DEFAULT;
ALTER USER "dbuser_app" SET "statement_timeout" = '30s';

View User Params 

-- View params for a specific user
SELECT rolname, unnest(setconfig) AS setting
FROM pg_db_role_setting drs
JOIN pg_roles r ON r.oid = drs.setrole
WHERE rolname = 'dbuser_analyst';

Manual Management 

-- Set params
ALTER USER dbuser_app SET work_mem = '128MB';
ALTER USER dbuser_app SET search_path = "$user", public, myschema;

-- Reset params
ALTER USER dbuser_app RESET work_mem;
ALTER USER dbuser_app RESET ALL;

Priority

When the same parameter is set at multiple levels, PostgreSQL applies this priority (low to high):

postgresql.conf           ← Cluster params (Patroni DCS)
       ↓
postgresql.auto.conf      ← Instance params (pg_parameters / ALTER SYSTEM)
       ↓
Database level            ← ALTER DATABASE SET
       ↓
User level                ← ALTER USER SET
       ↓
Session level             ← SET command

Database vs User Priority:

When a user connects to a specific database and the same parameter is set at both levels, PostgreSQL uses the user-level parameter since it has higher priority.

Example:

# Database: analytics has work_mem = 256MB
pg_databases:
  - name: analytics
    parameters:
      work_mem: 256MB

# User: analyst has work_mem = 512MB
pg_users:
  - name: analyst
    parameters:
      work_mem: 512MB
  • analyst connecting to analytics: work_mem = 512MB (user takes precedence)
  • Other users connecting to analytics: work_mem = 256MB (database applies)
  • analyst connecting to other DBs: work_mem = 512MB (user applies)

8.2 - Service/Access

Split read and write operations, route traffic correctly, and reliably deliver PostgreSQL cluster capabilities.

Split read and write operations, route traffic correctly, and reliably deliver PostgreSQL cluster capabilities.

Service is an abstraction: it is the form in which database clusters provide capabilities externally, encapsulating the details of the underlying cluster.

Service is critical for stable access in production environments, showing its value during high availability cluster automatic failovers. Personal users typically don’t need to worry about this concept.

Personal User

The concept of “service” is for production environments. Personal users/single-machine clusters can skip the complexity and directly access the database using instance names/IP addresses.

For example, Pigsty’s default single-node pg-meta.meta database can be directly connected using three different users:

psql postgres://dbuser_dba:[email protected]/meta     # Direct connection with DBA superuser
psql postgres://dbuser_meta:[email protected]/meta   # Connect with default business admin user
psql postgres://dbuser_view:DBUser.Viewer@pg-meta/meta     # Connect with default read-only user via instance domain name

Service Overview

In real-world production environments, we use primary-replica database clusters based on replication. Within the cluster, there is one and only one instance as the leader (primary) that can accept writes. Other instances (replicas) continuously fetch change logs from the cluster leader to stay synchronized. Additionally, replicas can handle read-only requests, significantly offloading the primary in read-heavy, write-light scenarios. Therefore, distinguishing between write requests and read-only requests to the cluster is a very common practice.

Moreover, for production environments with high-frequency short connections, we pool requests through connection pooling middleware (Pgbouncer) to reduce the overhead of connection and backend process creation. But for scenarios like ETL and change execution, we need to bypass the connection pool and directly access the database. At the same time, high-availability clusters may experience failover during failures, which causes a change in the cluster leader. Therefore, high-availability database solutions require write traffic to automatically adapt to cluster leader changes. These different access requirements (read-write separation, pooling vs. direct connection, automatic adaptation to failovers) ultimately abstract the concept of Service.

Typically, database clusters must provide this most basic service:

  • Read-write service (primary): Can read and write to the database

For production database clusters, at least these two services should be provided:

  • Read-write service (primary): Write data: Only carried by the primary.
  • Read-only service (replica): Read data: Can be carried by replicas, but can also be carried by the primary if no replicas are available

Additionally, depending on specific business scenarios, there might be other services, such as:

  • Default direct access service (default): Service that allows (admin) users to bypass the connection pool and directly access the database
  • Offline replica service (offline): Dedicated replica that doesn’t handle online read-only traffic, used for ETL and analytical queries
  • Synchronous replica service (standby): Read-only service with no replication delay, handled by synchronous standby/primary for read-only queries
  • Delayed replica service (delayed): Access older data from the same cluster from a certain time ago, handled by delayed replicas

Default Service

Pigsty provides four different services by default for each PostgreSQL database cluster. Here are the default services and their definitions:

Service Port Description
primary 5433 Production read-write, connect to primary pool (6432)
replica 5434 Production read-only, connect to replica pool (6432)
default 5436 Admin, ETL writes, direct access to primary (5432)
offline 5438 OLAP, ETL, personal users, interactive queries

Taking the default pg-meta cluster as an example, it provides four default services:

psql postgres://dbuser_meta:DBUser.Meta@pg-meta:5433/meta   # pg-meta-primary : production read-write via primary pgbouncer(6432)
psql postgres://dbuser_meta:DBUser.Meta@pg-meta:5434/meta   # pg-meta-replica : production read-only via replica pgbouncer(6432)
psql postgres://dbuser_dba:DBUser.DBA@pg-meta:5436/meta     # pg-meta-default : direct connection via primary postgres(5432)
psql postgres://dbuser_stats:DBUser.Stats@pg-meta:5438/meta # pg-meta-offline : direct connection via offline postgres(5432)

From the sample cluster architecture diagram, you can see how these four services work:

pigsty-ha.png

The actual DNS target of pg-meta is controlled by pg_dns_target. The default auto points to the L2 VIP when VIP is enabled; otherwise it points to the inventory primary’s IP. VIP is not enabled by default. See Access Service.

Service Implementation

In Pigsty, services are implemented using haproxy on nodes, differentiated by different ports on the host node.

Haproxy is enabled by default on every node managed by Pigsty to expose services, and database nodes are no exception. Although nodes in the cluster have primary-replica distinctions from the database perspective, from the service perspective, all nodes are the same: This means even if you access a replica node, as long as you use the correct service port, you can still use the primary’s read-write service. This design seals the complexity: as long as you can access any instance on the PostgreSQL cluster, you can fully access all services.

This design is similar to the NodePort service in Kubernetes. Similarly, in Pigsty, every service includes these two core elements:

  1. Access endpoints exposed via NodePort (port number, from where to access?)
  2. Target instances chosen through Selectors (list of instances, who will handle it?)

The boundary of Pigsty’s service delivery stops at the cluster’s HAProxy. Users can access these load balancers in various ways. Please refer to Access Service.

All services are declared through configuration files. For instance, the default PostgreSQL service is defined by the pg_default_services parameter:

pg_default_services:
- { name: primary ,port: 5433 ,dest: default  ,check: /primary   ,selector: "[]" }
- { name: replica ,port: 5434 ,dest: default  ,check: /read-only ,selector: "[]" , backup: "[? pg_role == `primary` || pg_role == `offline` ]" }
- { name: default ,port: 5436 ,dest: postgres ,check: /primary   ,selector: "[]" }
- { name: offline ,port: 5438 ,dest: postgres ,check: /replica   ,selector: "[? pg_role == `offline` || pg_offline_query ]" , backup: "[? pg_role == `replica` && !pg_offline_query]"}

You can also define additional services in pg_services. Both pg_default_services and pg_services are arrays of Service Definition objects.

Define Service

Pigsty allows you to define your own services:

  • pg_default_services: Services uniformly exposed by all PostgreSQL clusters, with four by default.
  • pg_services: Additional PostgreSQL services, can be defined at global or cluster level as needed.
  • haproxy_services: Directly customize HAProxy service content, can be used for other component access

For PostgreSQL clusters, you typically only need to focus on the first two. Each service definition generates a new configuration file in the configuration directory of all related HAProxy instances: /etc/haproxy/conf.d/<pg_cluster>-<service>.cfg Here’s a custom service example standby: When you want to provide a read-only service with no replication delay, you can add this record in pg_services:

- name: standby                   # required, service name, the actual svc name will be prefixed with `pg_cluster`, e.g: pg-meta-standby
  port: 5435                      # required, service exposed port (work as kubernetes service node port mode)
  ip: "*"                         # optional, service bind ip address, `*` for all ip by default
  selector: "[]"                  # required, service member selector, use JMESPath to filter inventory
  backup: "[? pg_role == `primary`]"  # optional, backup server selector, these instances will only be used when default selector instances are all down
  dest: default                   # optional, destination port, default|postgres|pgbouncer|<port_number>, 'default' by default, meaning pg_default_service_dest decides
  check: /sync                    # optional, health check url path, / by default, here using Patroni API: /sync, only sync standby and primary will return 200 healthy status
  maxconn: 5000                   # optional, max allowed front-end connection, default 5000
  balance: roundrobin             # optional, haproxy load balance algorithm (roundrobin by default, other options: leastconn)
  options: 'inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100'

The service definition above is rendered as /etc/haproxy/conf.d/pg-test-standby.cfg on the sample three-node pg-test cluster:

#---------------------------------------------------------------------
# service: pg-test-standby @ 10.10.10.11:5435
#---------------------------------------------------------------------
# service instances 10.10.10.11, 10.10.10.13, 10.10.10.12
# service backups   10.10.10.11
listen pg-test-standby
    bind *:5435            # <--- Binds to port 5435 on all IP addresses
    mode tcp               # <--- Load balancer works on TCP protocol
    maxconn 5000           # <--- Max connections 5000, can be increased as needed
    balance roundrobin     # <--- Load balance algorithm is rr round-robin, can also use leastconn
    option httpchk         # <--- Enable HTTP health check
    option http-keep-alive # <--- Keep HTTP connections
    http-check send meth OPTIONS uri /sync   # <---- Using /sync here, Patroni health check API, only sync standby and primary will return 200 healthy status
    http-check expect status 200             # <---- Health check return code 200 means healthy
    default-server inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100
    # servers: selector "[]" includes all three pg-test instances as pg-test-standby backends; /sync admits only the primary and synchronous standby.
    server pg-test-1 10.10.10.11:6432 check port 8008 weight 100 backup  # <----- Only primary satisfies condition pg_role == `primary`, selected by backup selector.
    server pg-test-3 10.10.10.13:6432 check port 8008 weight 100         #        Therefore acts as fallback instance: normally doesn't serve requests, only serves read-only requests after all other replicas are down, maximizing avoidance of read-write service being affected by read-only service
    server pg-test-2 10.10.10.12:6432 check port 8008 weight 100         #

Here, all three instances of the pg-test cluster are selected by selector: "[]" and rendered into the backend list of the pg-test-standby service. Because of the /sync health check, the Patroni REST API returns HTTP 200 only on the primary and synchronous standby, so only those members can actually serve requests. Additionally, the primary satisfies the condition pg_role == primary and is selected by the backup selector, marked as a backup server, and will only be used when no other instances (i.e., sync standby) can satisfy the requirement.

Primary Service

The Primary service is probably the most critical service in production environments. It provides read-write capability to the database cluster on port 5433, with the service definition as follows:

- { name: default ,port: 5436 ,dest: postgres ,check: /primary   ,selector: "[]" }
  • The selector parameter selector: "[]" means all cluster members will be included in the Primary service
  • But only the primary can pass the health check (check: /primary), actually serving Primary service traffic.
  • The destination parameter dest: default means the Primary service destination is affected by the pg_default_service_dest parameter
  • The default value of dest is default which will be replaced with the value of pg_default_service_dest, defaulting to pgbouncer.
  • By default, the Primary service destination is the connection pool on the primary, i.e., the port specified by pgbouncer_port, defaulting to 6432

If the value of pg_default_service_dest is postgres, then the primary service destination will bypass the connection pool and directly use the PostgreSQL database port (pg_port, default value 5432), which is very useful for scenarios where you don’t want to use a connection pool.

Example: pg-test-primary haproxy configuration
listen pg-test-primary
    bind *:5433         # <--- primary service defaults to port 5433
    mode tcp
    maxconn 5000
    balance roundrobin
    option httpchk
    option http-keep-alive
    http-check send meth OPTIONS uri /primary # <--- primary service defaults to using Patroni RestAPI /primary health check
    http-check expect status 200
    default-server inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100
    # servers
    server pg-test-1 10.10.10.11:6432 check port 8008 weight 100
    server pg-test-3 10.10.10.13:6432 check port 8008 weight 100
    server pg-test-2 10.10.10.12:6432 check port 8008 weight 100

Patroni’s high availability mechanism ensures that at most one instance’s /primary health check is true at any time, so the Primary service will always route traffic to the primary instance.

One benefit of using the Primary service instead of directly connecting to the database is that if the cluster experiences a split-brain situation (for example, killing the primary Patroni with kill -9 without watchdog), Haproxy can still avoid split-brain in this situation, because it only distributes traffic when Patroni is alive and returns primary status.

Replica Service

The Replica service is second only to the Primary service in importance in production environments. It provides read-only capability to the database cluster on port 5434, with the service definition as follows:

- { name: replica ,port: 5434 ,dest: default  ,check: /read-only ,selector: "[]" , backup: "[? pg_role == `primary` || pg_role == `offline` ]" }
  • The selector parameter selector: "[]" means all cluster members will be included in the Replica service
  • All instances can pass the health check (check: /read-only), serving Replica service traffic.
  • Backup selector: [? pg_role == 'primary' || pg_role == 'offline' ] marks the primary and offline replicas as backup servers.
  • Only when all regular replicas are down will the Replica service be served by the primary or offline replicas.
  • The destination parameter dest: default means the Replica service destination is also affected by the pg_default_service_dest parameter
  • The default value of dest is default which will be replaced with the value of pg_default_service_dest, defaulting to pgbouncer, same as the Primary service
  • By default, the Replica service destination is the connection pool on replicas, i.e., the port specified by pgbouncer_port, defaulting to 6432
Example: pg-test-replica haproxy configuration
listen pg-test-replica
    bind *:5434
    mode tcp
    maxconn 5000
    balance roundrobin
    option httpchk
    option http-keep-alive
    http-check send meth OPTIONS uri /read-only
    http-check expect status 200
    default-server inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100
    # servers
    server pg-test-1 10.10.10.11:6432 check port 8008 weight 100 backup
    server pg-test-3 10.10.10.13:6432 check port 8008 weight 100
    server pg-test-2 10.10.10.12:6432 check port 8008 weight 100

The Replica service is very flexible: If there are living dedicated Replica instances, it will prioritize using these instances to serve read-only requests. Only when all replica instances are down will the primary serve as a fallback for read-only requests. For the common one-primary-one-replica two-node cluster: use the replica as long as it’s alive, use the primary only when the replica is down.

Additionally, unless all dedicated read-only instances are down, the Replica service will not use dedicated Offline instances, thus avoiding mixing online fast queries with offline slow queries and their mutual interference.

Default Service

The Default service provides service on port 5436, and it’s a variant of the Primary service.

The Default service always bypasses the connection pool and directly connects to PostgreSQL on the primary, which is useful for admin connections, ETL writes, CDC change data capture, etc.

- { name: primary ,port: 5433 ,dest: default  ,check: /primary   ,selector: "[]" }

If pg_default_service_dest is changed to postgres, then the Default service is completely equivalent to the Primary service except for port and name. In this case, you can consider removing Default from default services.

Example: pg-test-default haproxy configuration
listen pg-test-default
    bind *:5436         # <--- Except for listening port/target port and service name, other configurations are the same as primary service
    mode tcp
    maxconn 5000
    balance roundrobin
    option httpchk
    option http-keep-alive
    http-check send meth OPTIONS uri /primary
    http-check expect status 200
    default-server inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100
    # servers
    server pg-test-1 10.10.10.11:5432 check port 8008 weight 100
    server pg-test-3 10.10.10.13:5432 check port 8008 weight 100
    server pg-test-2 10.10.10.12:5432 check port 8008 weight 100

Offline Service

The Offline service runs on port 5438 and bypasses the connection pool to access PostgreSQL directly. It is normally used for slow or analytical queries, ETL reads, and interactive personal queries:

- { name: offline ,port: 5438 ,dest: postgres ,check: /replica   ,selector: "[? pg_role == `offline` || pg_offline_query ]" , backup: "[? pg_role == `replica` && !pg_offline_query]"}

The Offline service routes traffic directly to dedicated offline replicas, or regular read-only instances marked with pg_offline_query.

  • The selector parameter filters two types of instances from the cluster: offline replicas with pg_role = offline, or regular read-only instances marked with pg_offline_query = true
  • The main difference between dedicated offline replicas and marked regular replicas is: the former doesn’t serve Replica service requests by default, avoiding mixing fast and slow queries, while the latter does serve by default.
  • The backup selector parameter filters one type of instance from the cluster: regular replicas without the offline mark, which means if offline instances or marked regular replicas are down, other regular replicas can be used to serve Offline service.
  • Health check /replica only returns 200 for replicas, primary returns error, so Offline service will never distribute traffic to the primary instance, even if only the primary remains in the cluster.
  • At the same time, the primary instance is neither selected by the selector nor by the backup selector, so it will never serve Offline service. Therefore, Offline service can always avoid users accessing the primary, thus avoiding impact on the primary.
Example: pg-test-offline haproxy configuration
listen pg-test-offline
    bind *:5438
    mode tcp
    maxconn 5000
    balance roundrobin
    option httpchk
    option http-keep-alive
    http-check send meth OPTIONS uri /replica
    http-check expect status 200
    default-server inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100
    # servers
    server pg-test-3 10.10.10.13:5432 check port 8008 weight 100
    server pg-test-2 10.10.10.12:5432 check port 8008 weight 100 backup

The Offline service provides restricted read-only service, typically used for two types of queries: interactive queries (personal users), slow queries and long transactions (analytics/ETL).

The Offline service requires extra care. HAProxy’s /replica health check automatically rejects the new primary after a switchover, but selector uses static pg_role / pg_offline_query labels from the inventory. In a one-primary-one-replica cluster where only the replica serves Offline queries, a switchover may temporarily leave no eligible backend. Reloading an unchanged inventory does not add the old primary to the Offline backend list. First update the inventory labels (or pg_offline_query) to match the new plan and then reload service, or switch the primary back.

If your business model is relatively simple, you can consider removing Default service and Offline service, using Primary service and Replica service to directly connect to the database.

Reload Service

Reload services when cluster membership changes, service definitions or static selector labels change, or relative weights are adjusted. Normal Primary/Replica switchover is handled by Patroni health checks and does not require a separate reload.

bin/pgsql-svc <cls> [ip...]         # reload service for lb cluster or lb instance
# ./pgsql.yml -t pg_service         # the actual ansible task to reload service

Access Service

The boundary of Pigsty’s service delivery stops at the cluster’s HAProxy. Users can access these load balancers in various ways.

The typical approach is to use DNS or VIP access, binding to all or any number of load balancers in the cluster.

pigsty-access.jpg

You can use different host & port combinations, which provide PostgreSQL services in different ways.

Host

Type Example Description
Cluster Domain Name pg-test Access via cluster domain name (resolved by dnsmasq @ infra nodes)
Cluster VIP Address 10.10.10.3 Access via L2 VIP address managed by vip-manager, bound to primary
Instance Hostname pg-test-1 Access via any instance hostname (resolved by dnsmasq @ infra nodes)
Instance IP Address 10.10.10.11 Access any instance IP address

Port

Pigsty uses different ports to distinguish pg services

Port Service Type Description
5432 postgres database Direct access to postgres server
6432 pgbouncer middleware Go through connection pool middleware before postgres
5433 primary service Access primary pgbouncer (or postgres)
5434 replica service Access replica pgbouncer (or postgres)
5436 default service Access primary postgres
5438 offline service Access offline postgres

Combinations

# Access via cluster domain (these examples assume L2 VIP is enabled; otherwise the domain points to the inventory primary by default)
postgres://test@pg-test:5432/test # DNS -> L2 VIP -> primary direct connection
postgres://test@pg-test:6432/test # DNS -> L2 VIP -> primary connection pool -> primary
postgres://test@pg-test:5433/test # DNS -> L2 VIP -> HAProxy -> Primary Connection Pool -> Primary
postgres://test@pg-test:5434/test # DNS -> L2 VIP -> HAProxy -> Replica Connection Pool -> Replica
postgres://dbuser_dba@pg-test:5436/test # DNS -> L2 VIP -> HAProxy -> Primary direct connection (for Admin)
postgres://dbuser_stats@pg-test:5438/test # DNS -> L2 VIP -> HAProxy -> offline direct connection (for ETL/personal queries)

# Direct access via cluster VIP
postgres://[email protected]:5432/test # L2 VIP -> Primary direct access
postgres://[email protected]:6432/test # L2 VIP -> Primary Connection Pool -> Primary
postgres://[email protected]:5433/test # L2 VIP -> HAProxy -> Primary Connection Pool -> Primary
postgres://[email protected]:5434/test # L2 VIP -> HAProxy -> Replica Connection Pool -> Replica
postgres://[email protected]:5436/test # L2 VIP -> HAProxy -> Primary direct connection (for Admin)
postgres://[email protected]:5438/test # L2 VIP -> HAProxy -> offline direct connect (for ETL/personal queries)

# Specify any cluster instance name directly
postgres://test@pg-test-1:5432/test # DNS -> Database Instance Direct Connect (singleton access)
postgres://test@pg-test-1:6432/test # DNS -> connection pool -> database
postgres://test@pg-test-1:5433/test # DNS -> HAProxy -> connection pool -> database read/write
postgres://test@pg-test-1:5434/test # DNS -> HAProxy -> connection pool -> database read-only
postgres://dbuser_dba@pg-test-1:5436/test # DNS -> HAProxy -> database direct connect
postgres://dbuser_stats@pg-test-1:5438/test # DNS -> HAProxy -> database offline read/write

# Directly specify any cluster instance IP access
postgres://[email protected]:5432/test # Database instance direct connection (directly specify instance, no automatic traffic distribution)
postgres://[email protected]:6432/test # Connection Pool -> Database
postgres://[email protected]:5433/test # HAProxy -> connection pool -> database read/write
postgres://[email protected]:5434/test # HAProxy -> connection pool -> database read-only
postgres://[email protected]:5436/test # HAProxy -> Database Direct Connections
postgres://[email protected]:5438/test # HAProxy -> database offline read/write

# Smart client automatic read/write separation
postgres://[email protected]:6432,10.10.10.12:6432,10.10.10.13:6432/test?target_session_attrs=primary
postgres://[email protected]:6432,10.10.10.12:6432,10.10.10.13:6432/test?target_session_attrs=prefer-standby

Override Service

You can override the default service configuration in several ways. A common requirement is to have Primary service and Replica service bypass Pgbouncer connection pool and directly access PostgreSQL database.

To achieve this, you can change pg_default_service_dest to postgres, so all services with svc.dest='default' in the service definition will use postgres instead of the default pgbouncer as the target.

If you’ve already pointed Primary service to PostgreSQL, then the default service becomes redundant and can be removed.

If you don’t need to distinguish between personal interactive queries and analytics/ETL slow queries, you can consider removing the Offline service from the default service list pg_default_services.

If you don’t need read-only replicas to share online read-only traffic, you can also remove Replica service from the default service list.

Delegate Service

Pigsty exposes PostgreSQL services with haproxy on nodes. All haproxy instances in the cluster are configured with the same service definition.

However, you can delegate pg service to a specific node group (e.g., dedicated haproxy lb cluster) rather than haproxy on PostgreSQL cluster members.

To do so, you need to override the default service definition with pg_default_services and set pg_service_provider to the proxy group name.

For example, this configuration will expose pg cluster primary service on haproxy node group proxy with port 10013.

pg_service_provider: proxy       # use load balancer on group `proxy` with port 10013
pg_default_services:  [{ name: primary ,port: 10013 ,dest: postgres  ,check: /primary   ,selector: "[]" }]

It’s user’s responsibility to make sure each delegate service port is unique among the proxy cluster.

A dedicated load balancer cluster example is provided in the 20-node production environment simulation sandbox: conf/ha/simu.yml

8.3 - PostgreSQL Security

Entry points for PostgreSQL authentication, access control, encrypted communication, data protection, and secure operations.

PostgreSQL security combines authentication, authorization, network boundaries, encrypted communication, data protection, and operational process. Pigsty provides configuration entry points for these mechanisms; operators must still harden, verify, and audit the deployment for its environment.

Concepts and Boundaries

Topic Content
Security and Compliance Default state, capability boundaries, and hardening path
Authentication HBA, SCRAM, certificate authentication, and credential management
Access Control Built-in roles, default privileges, database ACLs, and instance-access boundaries
Encrypted Communication CA, TLS, server authentication, and certificate rotation
Data Security Page checksums, replication, backup, PITR, audit, and logging
Compliance Launch checks, control mappings, and evidence requirements

Configuration Reference

Administration and Verification

The inventory describes desired state. Acceptance checks should also inspect HBA, certificates, listen ports, and sensitive files on running nodes, and verify effective roles and privileges through PostgreSQL catalogs.

8.4 - Administration

Standard Operating Procedures (SOP) for database administration tasks

8.4.1 - Managing PostgreSQL Clusters

Create/destroy PostgreSQL clusters, scale existing clusters, and clone clusters.

Quick Reference

Action Command Description
Create Cluster bin/pgsql-add <cls> Create a new PostgreSQL cluster
Expand Cluster bin/pgsql-add <cls> <ip...> Add replica to existing cluster
Shrink Cluster bin/pgsql-rm <cls> <ip...> Remove instance from cluster
Remove Cluster bin/pgsql-rm <cls> Destroy entire PostgreSQL cluster
Reload Service bin/pgsql-svc <cls> [ip...] Reload cluster load balancer config
Reload HBA bin/pgsql-hba <cls> [ip...] Reload cluster HBA access rules
Clone Cluster - Clone via standby cluster or PITR

For other management tasks, see: HA Management, Manage Users, Manage Databases.

Create Cluster

To create a new PostgreSQL cluster, first define the cluster in the inventory, then add nodes and initialize:

Script
bin/node-add  <cls>     # Add nodes in group <cls>
Playbook
./node.yml  -l <cls>    # Use Ansible playbook to add nodes in group <cls>
Example
bin/node-add pg-test    # Add nodes in pg-test group, runs ./node.yml -l pg-test

On managed nodes, create the cluster with: (Execute pgsql.yml playbook on <cls> group)

Script
bin/pgsql-add <cls>     # Create PostgreSQL cluster <cls>
Playbook
./pgsql.yml -l <cls>    # Use Ansible playbook to create PostgreSQL cluster <cls>
Example
bin/pgsql-add pg-test   # Create pg-test cluster

Example: Create 3-node PG cluster pg-test

demo/pgsql.cast
Risk: Re-running create on existing cluster

If you re-run create on an existing cluster, Pigsty won’t remove existing data files, but service configs will be overwritten and the cluster will restart! Additionally, if you specified a baseline SQL in database definition, it will re-execute - if it contains delete/overwrite logic, data loss may occur.

Expand Cluster

To add a new replica to an existing PostgreSQL cluster, add the instance definition to inventory: all.children.<cls>.hosts.

pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary } # existing member
    10.10.10.12: { pg_seq: 2, pg_role: replica } # existing member
    10.10.10.13: { pg_seq: 3, pg_role: replica } # <--- new member
  vars: { pg_cluster: pg-test }

Scaling out is similar to creating a cluster. First add the new node to Pigsty: Add Node:

Script
bin/node-add <ip>       # Add node with IP <ip>
Playbook
./node.yml -l <ip>      # Use Ansible playbook to add node <ip>
Example
bin/node-add 10.10.10.13    # Add node 10.10.10.13, runs ./node.yml -l 10.10.10.13

Then run the following on the new node to scale out (Install PGSQL module on new node with same pg_cluster):

Script
bin/pgsql-add <cls> <ip>  # Add node <ip> to cluster
Playbook
./pgsql.yml -l <ip>       # Core: Use Ansible playbook to install PGSQL module on <ip>
Example
bin/pgsql-add pg-test 10.10.10.13   # Scale out pg-test with node 10.10.10.13

After scaling, you should Reload Service to add the new member to load balancer.

Example: Add replica 10.10.10.13 to 2-node cluster pg-test

demo/pgsql-append.cast

Shrink Cluster

To remove a replica from an existing PostgreSQL cluster, remove the instance definition from inventory all.children.<cls>.hosts.

Scale-in stops the instance and deletes its data directory by default. First run pig pg list <cls> and pig pb info, verify that the target is not the primary and that a recent restorable backup exists, then have the operator enter the exact <ip> and execute only after confirmation.

First uninstall PGSQL module from target node (Execute pgsql-rm.yml on <ip>):

Script
bin/pgsql-rm <cls> <ip>   # Remove the PostgreSQL instance on <ip> from cluster <cls>
Playbook
./pgsql-rm.yml -l <ip>    # Directly remove the PostgreSQL instance on <ip> with the Ansible playbook
Example
bin/pgsql-rm pg-test 10.10.10.13  # Remove node 10.10.10.13 from pg-test

After removing PGSQL module, optionally remove the node from Pigsty: Remove Node:

Script
bin/node-rm <ip>          # Remove <ip> from Pigsty management
Playbook
./node-rm.yml -l <ip>     # Directly remove <ip> from Pigsty management with the Ansible playbook
Example
bin/node-rm 10.10.10.13   # Remove 10.10.10.13 from Pigsty management

After scaling in, remove the instance from inventory, then Reload Service to remove it from load balancer.

pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica }
    10.10.10.13: { pg_seq: 3, pg_role: replica } # <--- remove after execution
  vars: { pg_cluster: pg-test }

Example: Remove replica 10.10.10.13 from 3-node cluster pg-test

demo/pgsql-shrink.cast

Remove Cluster

To destroy a cluster, uninstall PGSQL module from all nodes (Execute pgsql-rm.yml on <cls>):

This is irreversible data deletion. Inspect pig pg list <cls> and pig pb info, verify a recent backup and any independent copy to retain, and have the operator enter the exact cluster name. The commands below perform the corresponding destruction directly.

Script
bin/pgsql-rm <cls>        # Destroy the entire PostgreSQL cluster <cls>
Playbook
./pgsql-rm.yml -l <cls>   # Directly destroy the entire PostgreSQL cluster <cls> with the Ansible playbook
Example
bin/pgsql-rm pg-test      # Destroy cluster pg-test

After destroying PGSQL, optionally remove all nodes from Pigsty: Remove Node (optional if other services exist):

Script
bin/node-rm <cls>         # Remove every node in group <cls> from Pigsty management
Playbook
./node-rm.yml -l <cls>    # Directly remove the nodes in group <cls> from Pigsty management with the Ansible playbook
Example
bin/node-rm pg-test       # Remove every node in group pg-test from Pigsty management

After removal, delete the entire cluster definition from inventory.

pg-test: # remove this cluster definition group
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica }
    10.10.10.13: { pg_seq: 3, pg_role: replica }
  vars: { pg_cluster: pg-test }

Example: Destroy 3-node PG cluster pg-test

demo/pgsql-rm.cast

Note: If pg_safeguard is configured (or globally true), pgsql-rm.yml will abort to prevent accidental removal. Override with playbook command line to force removal. By default, cluster backup repo is deleted with the cluster. To preserve backups (e.g., with centralized repo), set pg_rm_backup=false:

./pgsql-rm.yml -l pg-meta -e pg_safeguard=false    # Force removal of protected pg-meta
./pgsql-rm.yml -l pg-meta -e pg_rm_backup=false    # Preserve its backup repository while removing the cluster

Reload Service

PostgreSQL clusters expose services via HAProxy on host nodes. When service definitions, instance weights, or cluster membership change (for example, scale out or scale in), reload services to update HAProxy’s static member configuration. The default Primary and Replica services detect the current role through Patroni REST API health checks, so ordinary switchover or failover reroutes automatically and does not require regenerating HAProxy configuration.

To reload service config on entire cluster or specific instances (Execute pg_service subtask of pgsql.yml on <cls> or <ip>):

Script
bin/pgsql-svc <cls>           # Reload service config for entire cluster <cls>
bin/pgsql-svc <cls> <ip...>   # Reload service config for specific instances
Playbook
./pgsql.yml -l <cls> -t pg_service -e pg_reload=true        # Reload entire cluster
./pgsql.yml -l <ip>  -t pg_service -e pg_reload=true        # Reload specific instance
Example
bin/pgsql-svc pg-test                 # Reload pg-test cluster service config
bin/pgsql-svc pg-test 10.10.10.13     # Reload pg-test 10.10.10.13 instance service config
Note

If you use a dedicated load-balancer cluster (pg_service_provider), only reloading the cluster primary updates the load-balancer configuration.

Example: Reload pg-test cluster service config

demo/pgsql-svc.cast
Example: Reload PG Service to Remove Instance

asciicast

Reload HBA

When HBA configs change, reload HBA rules to apply. (pg_hba_rules / pgb_hba_rules) If you have inventory-role-specific HBA rules or address ranges that reference cluster member aliases, reload HBA after changing pg_role labels or scaling the cluster. Role selectors use static inventory variables and do not change automatically after a Patroni switchover.

To reload PG and Pgbouncer HBA rules on entire cluster or specific instances (Execute HBA subtasks of pgsql.yml on <cls> or <ip>):

Script
bin/pgsql-hba <cls>           # Reload HBA rules for entire cluster <cls>
bin/pgsql-hba <cls> <ip...>   # Reload HBA rules for specific instances
Playbook
./pgsql.yml -l <cls> -t pg_hba,pg_reload,pgbouncer_hba,pgbouncer_reload -e pg_reload=true   # Reload entire cluster
./pgsql.yml -l <ip>  -t pg_hba,pg_reload,pgbouncer_hba,pgbouncer_reload -e pg_reload=true   # Reload specific instance
Example
bin/pgsql-hba pg-test                 # Reload pg-test cluster HBA rules
bin/pgsql-hba pg-test 10.10.10.13     # Reload pg-test 10.10.10.13 instance HBA rules

Example: Reload pg-test cluster HBA rules

demo/pgsql-hba.cast

Config Cluster

PostgreSQL config params are managed by Patroni. Initial params are specified by Patroni config template. After cluster init, config is stored in Etcd, dynamically managed and synced by Patroni. Most Patroni config params can be modified via patronictl. Other params (e.g., etcd DCS config, log/RestAPI config) can be updated via subtasks. For example, when etcd cluster membership changes, refresh Patroni config:

./pgsql.yml -l pg-test -t pg_conf                   # Update Patroni config file
ansible pg-test -b -a 'systemctl reload patroni'    # Reload Patroni service

You can override Patroni-managed defaults at different levels: specify params per instance, specify params per user, or specify params per database.

Clone Cluster

Two ways to clone a cluster: use Standby Cluster, or use Point-in-Time Recovery. The former is simple and requires no backup repository, but it does require a reachable replication upstream and can clone only the latest state. The latter requires a centralized backup repository such as Silo and can clone to any point within the retention period.

Method Pros Cons Use Cases
Standby Cluster No backup repository needed Requires reachable upstream; latest state only DR, read-write separation, migration
PITR Recover to any point Requires centralized backup Undo mistakes, data audit

Clone via Standby Cluster

Standby Cluster continuously syncs from upstream cluster via streaming replication - the simplest cloning method. Specify pg_upstream on the new cluster primary to auto-pull data from upstream.

# pg-test is the original cluster
pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
  vars: { pg_cluster: pg-test }

# pg-test2 is standby cluster (clone) of pg-test
pg-test2:
  hosts:
    10.10.10.12: { pg_seq: 1, pg_role: primary, pg_upstream: 10.10.10.11 }  # specify upstream
    10.10.10.13: { pg_seq: 2, pg_role: replica }
  vars: { pg_cluster: pg-test2 }

Create standby cluster with:

Script
bin/pgsql-add pg-test2    # Create standby cluster, auto-clone from upstream pg-test
Playbook
./pgsql.yml -l pg-test2   # Use Ansible playbook to create standby cluster

Standby cluster follows upstream, keeping data in sync. Promote to independent cluster anytime:

Example: Promote Standby to Independent Cluster

Via Config Cluster, remove standby_cluster config to promote:

$ pg edit-config pg-test2
-standby_cluster:
-  create_replica_methods:
-  - basebackup
-  host: 10.10.10.11
-  port: 5432

Apply these changes? [y/N]: y

After promotion, pg-test2 becomes independent cluster accepting writes, forked from pg-test.

Example: Change Replication Upstream

If upstream cluster switchover occurs, change standby cluster upstream via Config Cluster:

$ pg edit-config pg-test2

 standby_cluster:
   create_replica_methods:
   - basebackup
-  host: 10.10.10.11     # <--- old upstream
+  host: 10.10.10.14     # <--- new upstream
   port: 5432

Apply these changes? [y/N]: y

Clone via PITR

Point-in-Time Recovery (PITR) allows recovery to any point within backup retention. Requires a centralized backup repository (Silo/S3), but is more powerful.

To clone via PITR, add pg_pitr param specifying recovery target:

# Clone new cluster pg-meta2 from pg-meta backup
pg-meta2:
  hosts: { 10.10.10.12: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta2
    pg_pitr:
      cluster: pg-meta                    # Recover from pg-meta backup
      time: '2025-01-10 10:00:00+00'      # Recover to specific time
      archive: false                       # Disable archiving during the independent restore
      action: promote                      # Promote after replay in this one-shot example

Execute clone with pgsql-pitr.yml playbook:

Playbook
./pgsql-pitr.yml -l pg-meta2    # Use the explicitly declared action: promote above
CLI
# Specify PITR options via command line
./pgsql-pitr.yml -l pg-meta2 -e '{"pg_pitr": {"cluster": "pg-meta", "time": "2025-01-10 10:00:00+00", "archive": false, "action": "promote"}}'

PITR supports multiple recovery target types:

Target Type Example Description
Time time: "2025-01-10 10:00:00+00" Recover to specific timestamp
XID xid: "250000" Recover to before/after txn
Name name: "before_migration" Recover to named restore point
LSN lsn: "0/4001C80" Recover to specific WAL pos
Latest pg_pitr: {} Recover to end of WAL archive
Post-PITR Processing

Pigsty v5.0 PITR keeps archiving enabled by default (archive: true). If you explicitly set archive: false for exploratory recovery, reset archive_mode, restart the cluster, and perform a new full backup after confirming the recovered data is correct:

psql -c 'ALTER SYSTEM RESET archive_mode;'
pg restart <cls>
pg-backup full    # Execute new full backup

For detailed PITR usage, see Restore Operations documentation.

8.4.2 - Managing PostgreSQL Users

User management - create, modify, delete users, manage role membership, connection pool config

Quick Start

Pigsty uses declarative management: first define users in the inventory, then use bin/pgsql-user <cls> <username> to create or modify.

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_users: [{ name: dbuser_app, password: 'DBUser.App', pgbouncer: true }]  # <--- Define user list here!
Script
bin/pgsql-user <cls> <username>    # Create/modify <username> user on <cls> cluster
Playbook
./pgsql-user.yml -l pg-meta -e username=dbuser_app    # Use playbook to create/modify user
Example
bin/pgsql-user pg-meta dbuser_app    # Create/modify dbuser_app user on pg-meta cluster

For the complete user definition reference, see User Configuration. See Access Control for roles and privileges, and Authentication for credential management.

name is the key used by pgsql-user.yml to look up a user definition; the playbook does not rename roles. For a rename, create the replacement role, migrate ownership, memberships, and client credentials, validate the cutover, and only then remove the old role. Do not treat delete-and-create as a lossless rename.

Action Command Description
Create User bin/pgsql-user <cls> <user> Create new business user or role
Modify User bin/pgsql-user <cls> <user> Modify existing user properties
Delete User bin/pgsql-user <cls> <user> Dependency-aware destructive deletion (state: absent)
demo/pgsql-user.cast

Create User

Users defined in pg_users are auto-created during PostgreSQL cluster creation in the pg_user task.

To create a new user on an existing cluster, add user definition to all.children.<cls>.pg_users, then execute:

Script
bin/pgsql-user <cls> <username>   # Create user <username>
Playbook
./pgsql-user.yml -l <cls> -e username=<username>   # Use Ansible playbook
Example
bin/pgsql-user pg-meta dbuser_app    # Create dbuser_app user in pg-meta cluster

Example: Create business user dbuser_app

#all.children.pg-meta.vars.pg_users:
  - name: dbuser_app
    password: DBUser.App
    pgbouncer: true
    roles: [dbrole_readwrite]
    comment: application user for myapp

Result: Creates dbuser_app user on primary, sets password, grants dbrole_readwrite role, adds to Pgbouncer pool, reloads Pgbouncer config on all instances.

Recommendation: Use playbook

For manual user creation, you must ensure Pgbouncer user list sync yourself.

Modify User

Same command as create - playbook is idempotent. When target user exists, Pigsty modifies properties to match config.

Script
bin/pgsql-user <cls> <user>   # Modify user <user> properties
Playbook
./pgsql-user.yml -l <cls> -e username=<user>   # Idempotent, can repeat
Example
bin/pgsql-user pg-meta dbuser_app    # Modify dbuser_app to match config

Not directly mutable: name is the identity key in the declarative definition. The playbook does not rename an existing role. Use a controlled create, ownership/privilege and client migration, validation, and old-role removal sequence.

All other properties can be modified. Common examples:

Modify password: Update password field. Logging is temporarily disabled during password change to prevent leakage.

- name: dbuser_app
  password: NewSecretPassword     # New password

Modify privilege attributes: Configure boolean flags for user privileges.

- name: dbuser_app
  superuser: false           # Superuser (use carefully!)
  createdb: true             # Allow CREATE DATABASE
  createrole: false          # Allow CREATE ROLE
  inherit: true              # Auto-inherit role privileges
  replication: false         # Allow streaming replication
  bypassrls: false           # Bypass row-level security
  connlimit: 50              # Connection limit, -1 unlimited

Modify expiration: Use expire_in for relative expiry (N days), or expire_at for absolute date. expire_in takes priority and recalculates on each playbook run - good for temp users needing periodic renewal.

- name: temp_user
  expire_in: 30                   # Expires in 30 days (relative)

- name: contractor_user
  expire_at: '2024-12-31'         # Expires on date (absolute)

- name: permanent_user
  expire_at: 'infinity'           # Never expires

Modify role membership: Use roles array with simple or extended format. Role membership is additive - won’t remove undeclared existing roles. Use state: absent to explicitly revoke.

- name: dbuser_app
  roles:
    - dbrole_readwrite                      # Simple form: grant role
    - { name: dbrole_admin, admin: true }   # With ADMIN OPTION
    - { name: pg_monitor, set: false }      # PG16+: disallow SET ROLE
    - { name: old_role, state: absent }     # Revoke role membership

Manage user parameters: Use parameters dict for user-level params, generates ALTER USER ... SET. Use DEFAULT to reset.

- name: dbuser_analyst
  parameters:
    work_mem: '256MB'
    statement_timeout: '5min'
    search_path: 'analytics,public'
    log_statement: DEFAULT        # Reset to default

Connection pool config: Set pgbouncer: true to add user to pool. Optional pool_mode and pool_connlimit.

- name: dbuser_app
  pgbouncer: true                 # Add to pool
  pool_mode: transaction          # Pool mode
  pool_connlimit: 50              # Max user connections

Delete User

Deleting a user terminates sessions, transfers object ownership, revokes grants, and runs DROP ROLE; it is irreversible. Confirm the exact cluster, role, successor owner, and a recent backup before setting the user to state: absent and applying the change.

Script
bin/pgsql-user <cls> <user>   # Actual deletion after confirmation; config must say state: absent
Playbook
./pgsql-user.yml -l <cls> -e username=<user>   # Apply the user definition with state: absent
Example
bin/pgsql-user pg-meta dbuser_old    # Delete dbuser_old (config has state: absent)

Config example:

pg_users:
  - name: dbuser_old
    state: absent

Deletion process: On the primary, the task runs pg-drop-role <user> postgres --force. It disables login, terminates active sessions, transfers database and tablespace ownership plus objects in each connectable database to postgres, runs DROP OWNED to remove grants, revokes role memberships, and finally runs DROP ROLE. A pre-change audit snapshot is written to /tmp/pg_drop_role_<user>_<timestamp>.log.

Protection: The Ansible task skips postgres and the replication, admin, and monitor usernames configured in inventory. When invoked directly, pg-drop-role protects only the hard-coded default names postgres, replicator, dbuser_dba, and dbuser_monitor; renamed system accounts are not recognized automatically.

Dependency-aware, not transactional

pg-drop-role skips DROP OWNED in a database if its preceding REASSIGN OWNED fails, but the cross-database procedure is not one transaction. A mid-run failure can leave the role NOLOGIN, some ownership already transferred, or dependencies still present. The v4.5 Ansible task also uses ignore_errors, so a playbook result is not sufficient evidence. Verify role absence, successor ownership, application cutover, and the audit log afterward.

In v4.5, pgsql-user.yml reloads Pgbouncer but does not reliably prune a deleted role from /etc/pgbouncer/userlist.txt. Check every cluster instance after deletion:

sudo -iu postgres psql -AXtwc "SELECT 1 FROM pg_roles WHERE rolname = 'dbuser_old';"
grep -n '^"dbuser_old"[[:space:]]' /etc/pgbouncer/userlist.txt

If an exact Pgbouncer entry remains, remove that single line under change control, reload Pgbouncer, and validate application connections. Do not use a broad pattern to delete entries.

Manual Deletion

For manual user deletion, use pg-drop-role script directly:

# Check dependencies (read-only)
pg-drop-role dbuser_old --check

# Preview deletion (don't execute)
pg-drop-role dbuser_old --dry-run -v

# Only after confirming a recent backup, exact role, and successor owner
pg-drop-role dbuser_old dbuser_new

# Use --force only after explicitly approving session termination
pg-drop-role dbuser_old dbuser_new --force

Common Use Cases

Common user configuration examples:

Basic business user

- name: dbuser_app
  password: DBUser.App
  pgbouncer: true
  roles: [dbrole_readwrite]
  comment: application user

Read-only user

- name: dbuser_readonly
  password: DBUser.Readonly
  pgbouncer: true
  roles: [dbrole_readonly]

Admin user (can execute DDL)

- name: dbuser_admin
  password: DBUser.Admin
  pgbouncer: true
  pool_mode: session
  roles: [dbrole_admin]
  parameters:
    log_statement: 'all'

Temp user (expires in 30 days)

- name: temp_contractor
  password: TempPassword
  expire_in: 30
  roles: [dbrole_readonly]

Role (no login, for permission grouping)

- name: custom_role
  login: false
  comment: custom role for special permissions

User with advanced role options (PG16+)

- name: dbuser_special
  password: DBUser.Special
  pgbouncer: true
  roles:
    - dbrole_readwrite
    - { name: dbrole_admin, admin: true }
    - { name: pg_monitor, set: false }
    - { name: pg_execute_server_program, inherit: false }

Query Users

Common SQL queries for user info:

List all users

SELECT rolname, rolsuper, rolinherit, rolcreaterole, rolcreatedb,
       rolcanlogin, rolreplication, rolbypassrls, rolconnlimit, rolvaliduntil
FROM pg_roles WHERE rolname NOT LIKE 'pg_%' ORDER BY rolname;

View user role membership

SELECT r.rolname AS member, g.rolname AS role, m.admin_option, m.set_option, m.inherit_option
FROM pg_auth_members m
JOIN pg_roles r ON r.oid = m.member
JOIN pg_roles g ON g.oid = m.roleid
WHERE r.rolname = 'dbuser_app';

View user-level parameters

SELECT rolname, setconfig FROM pg_db_role_setting s
JOIN pg_roles r ON r.oid = s.setrole WHERE s.setdatabase = 0;

View expiring users

SELECT rolname, rolvaliduntil, rolvaliduntil - CURRENT_TIMESTAMP AS time_remaining
FROM pg_roles WHERE rolvaliduntil IS NOT NULL
  AND rolvaliduntil < CURRENT_TIMESTAMP + INTERVAL '30 days'
ORDER BY rolvaliduntil;

Connection Pool Management

Connection pool params in user definitions are applied to Pgbouncer when creating/modifying users.

Users with pgbouncer: true are added to /etc/pgbouncer/userlist.txt. User-level pool params (pool_mode, pool_connlimit) are configured via /etc/pgbouncer/useropts.txt.

Use postgres OS user with pgb alias to access Pgbouncer admin database. For more pool management, see Pgbouncer Management.

Manage Default-User Passwords

For a business user, follow Modify User: persist the new password in its pg_users definition, preview the scoped playbook, and then apply it. The three default users require extra coordination because other services consume their credentials.

Parameter Default Role Consumers
pg_admin_password DBUser.DBA dbuser_dba Admin clients, Pgbouncer, Infra service files, pgAdmin
pg_monitor_password DBUser.Monitor dbuser_monitor Exporters, Pgbouncer, Grafana data sources
pg_replication_password DBUser.Replicator replicator Patroni replication and .pgpass files

These accounts belong to pg_default_roles, not pg_users. pgsql-user.yml looks up only pg_users, so do not rotate a default password by overriding pg_users on the command line: that changes the business-user list visible to that run and exposes plaintext in shell history.

Rotate one account at a time:

  1. Persist the new parameter in pigsty.yml or the inventory actually in use; never put the plaintext password on the command line.
  2. On the current primary, open interactive psql as a superuser and run \password <username>; the meta-command reads the secret interactively.
  3. Run the corresponding refresh playbooks below after verifying the -l cluster/node scope.
  4. Keep the current administration session open and verify direct PostgreSQL, Pgbouncer, replication, exporters, and Grafana data sources before rotating another account.
# On the current primary of the target cluster
sudo -iu postgres psql -d postgres
\password dbuser_dba       # or dbuser_monitor / replicator

Refresh every consumer for the account. Replace <cls> and constrain infra to the actual targets:

# dbuser_dba: PG-node .pgpass, Pgbouncer, Infra admin files, and pgAdmin files
./pgsql.yml -l <cls> -t pg_pass,pgbouncer_user,pgbouncer_reload -e pg_reload=true
./infra.yml -l infra -t env_pgpass,env_pgscv,env_pgadmin

# dbuser_monitor: PG-node .pgpass, Pgbouncer, exporters, and Grafana data sources
./pgsql.yml -l <cls> -t pg_pass,pgbouncer_user,pgbouncer_reload,pg_exporter,pgbouncer_exporter,add_ds -e pg_reload=true
./infra.yml -l infra -t env_pgpass

# replicator: Patroni configuration plus PostgreSQL-node and Infra .pgpass files
./pgsql.yml -l <cls> -t pg_conf,pg_pass,patroni_reload -e pg_reload=true
./infra.yml -l infra -t env_pgpass

A mismatch between the replication role and Patroni nodes prevents new replication connections, so rotate that credential in a maintenance window and validate promptly. If VIBE or another module has rendered an admin connection string into its workspace context, rerender that module’s files as well.

Check for duplicate Infra .pgpass entries

In v4.5, env_pgpass adds the new line with lineinfile; it does not remove older lines by username. Because libpq uses the first matching line, inspect every target Infra node after the refresh and remove obsolete entries through controlled editing without printing secrets:

awk -F: '$4=="dbuser_dba" || $4=="dbuser_monitor" || $4=="replicator" {print NR, $4}' ~/.pgpass

patroni_password protects the Patroni REST API; it is not a PostgreSQL role password. After changing it in inventory, refresh the target PostgreSQL cluster and Infra management side separately:

./pgsql.yml -l <cls> -t pg_conf,patroni_reload -e pg_reload=true
./infra.yml -l infra -t env_patroni

Then validate authentication and cluster state with patronictl or pig pg list <cls>.

8.4.3 - Managing PostgreSQL Databases

Database management - create, modify, delete, rebuild, and clone databases using templates

Quick Start

Pigsty uses declarative management: first define databases in the inventory, then use bin/pgsql-db <cls> <dbname> to create or modify.

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_databases: [{ name: some_db }]  # <--- Define database list here!
Script
bin/pgsql-db <cls> <dbname>    # Create/modify <dbname> database on <cls> cluster
Playbook
./pgsql-db.yml -l pg-meta -e dbname=some_db    # Use playbook to create/modify database
Example
bin/pgsql-db pg-meta some_db    # Create/modify some_db database on pg-meta cluster

For the complete database definition reference, see Database Configuration. For database access permissions, see Access Control: Database Isolation.

Note: Some parameters can only be specified at creation time. Modifying these requires recreating the database (use state: recreate).

Action Command Description
Create Database bin/pgsql-db <cls> <db> Create new business database
Modify Database bin/pgsql-db <cls> <db> Modify existing database properties
Delete Database bin/pgsql-db <cls> <db> Delete database (requires state: absent)
Rebuild Database bin/pgsql-db <cls> <db> Drop and recreate (requires state: recreate)
Clone Database bin/pgsql-db <cls> <db> Clone database using template
demo/pgsql-db.cast

Create Database

Databases defined in pg_databases are auto-created during PostgreSQL cluster creation in the pg_db task.

To create a new database on an existing cluster, add database definition to all.children.<cls>.pg_databases, then execute:

Script
bin/pgsql-db <cls> <dbname>   # Create database <dbname>
Playbook
./pgsql-db.yml -l <cls> -e dbname=<dbname>   # Use Ansible playbook
Example
bin/pgsql-db pg-meta myapp    # Create myapp database in pg-meta cluster

Example: Create business database myapp

#all.children.pg-meta.vars.pg_databases:
  - name: myapp
    owner: dbuser_myapp
    schemas: [app]
    extensions:
      - { name: pg_trgm }
      - { name: btree_gin }
    comment: my application database

Result: Creates myapp database on primary, sets owner to dbuser_myapp, creates app schema, enables pg_trgm and btree_gin extensions. Database is auto-added to Pgbouncer pool and registered as Grafana datasource.

Recommendation: Use playbook

For manual database creation, you must ensure Pgbouncer pool and Grafana datasource sync yourself.

Modify Database

Same command as create - playbook is idempotent when no baseline SQL is defined.

When target database exists, Pigsty modifies properties to match config. However, some properties can only be set at creation.

Script
bin/pgsql-db <cls> <db>   # Modify database <db> properties
Playbook
./pgsql-db.yml -l <cls> -e dbname=<db>   # Idempotent, can repeat
Example
bin/pgsql-db pg-meta myapp    # Modify myapp database to match config

Immutable properties: These can’t be modified after creation, require state: recreate:

  • name (database name), template, strategy (clone strategy)
  • encoding, locale/lc_collate/lc_ctype, locale_provider/icu_locale/icu_rules/builtin_locale

All other properties can be modified. Common examples:

Modify owner: Update owner field, executes ALTER DATABASE ... OWNER TO and grants permissions.

- name: myapp
  owner: dbuser_new_owner     # New owner

Modify connection limit: Use connlimit to limit max connections.

- name: myapp
  connlimit: 100              # Max 100 connections

Revoke public connect: Setting revokeconn: true revokes PUBLIC CONNECT privilege, allowing only owner, DBA, monitor, and replication users.

- name: myapp
  owner: dbuser_myapp
  revokeconn: true            # Revoke PUBLIC CONNECT

Manage parameters: Use parameters dict for database-level params, generates ALTER DATABASE ... SET. Use special value DEFAULT to reset.

- name: myapp
  parameters:
    work_mem: '256MB'
    maintenance_work_mem: '512MB'
    statement_timeout: '30s'
    search_path: DEFAULT      # Reset to default

Manage schemas: Use schemas array with simple or extended format. Use state: absent to drop (CASCADE).

- name: myapp
  schemas:
    - app                                   # Simple form
    - { name: core, owner: dbuser_myapp }   # Specify owner
    - { name: deprecated, state: absent }   # Drop schema

Manage extensions: Use extensions array with simple or extended format. Use state: absent to uninstall (CASCADE).

- name: myapp
  extensions:
    - postgis                                 # Simple form
    - { name: vector, schema: public }        # Specify schema
    - { name: pg_trgm, state: absent }        # Uninstall extension
CASCADE Warning

Dropping schemas or uninstalling extensions uses CASCADE, deleting all dependent objects. Understand impact before executing.

Connection pool config: By default all databases are added to Pgbouncer. Configure pgbouncer, pool_mode, pool_size, pool_reserve, pool_size_min, pool_connlimit, and pool_auth_user.

- name: myapp
  pgbouncer: true              # Add to pool (default true)
  pool_mode: transaction       # Pool mode: transaction/session/statement
  pool_size: 50                # Default pool size
  pool_reserve: 30             # Reserve pool size
  pool_size_min: 0             # Minimum pool size
  pool_connlimit: 100          # Max database connections
  pool_auth_user: dbuser_meta  # Auth query user (with pgbouncer_auth_query)

Since Pigsty v4.1.0, database pool fields are unified as pool_reserve and pool_connlimit; legacy aliases pool_size_reserve / pool_max_db_conn are converged.

Delete Database

To delete a database, set state to absent and execute:

Script
bin/pgsql-db <cls> <db>   # Delete <db> (config must have state: absent)
Playbook
./pgsql-db.yml -l <cls> -e dbname=<db>   # Use Ansible playbook
Example
bin/pgsql-db pg-meta olddb    # Delete olddb (config has state: absent)

Config example:

pg_databases:
  - name: olddb
    state: absent

Deletion process: If is_template: true, first executes ALTER DATABASE ... IS_TEMPLATE false; uses DROP DATABASE ... WITH (FORCE) (PG13+) to force drop and terminate all connections; removes from Pgbouncer pool; unregisters from Grafana datasource.

Protection: System databases postgres, template0, template1 cannot be deleted. Deletion only runs on primary - streaming replication syncs to replicas.

Danger Warning

Database deletion is irreversible - permanently deletes all data. Before executing: ensure recent backup exists, confirm no business uses the database, notify stakeholders. Pigsty is not responsible for any data loss from database deletion. Use at your own risk.

Rebuild Database

recreate state rebuilds database (drop then create):

Script
bin/pgsql-db <cls> <db>   # Rebuild <db> (config must have state: recreate)
Playbook
./pgsql-db.yml -l <cls> -e dbname=<db>   # Use Ansible playbook
Example
bin/pgsql-db pg-meta testdb    # Rebuild testdb (config has state: recreate)

Config example:

pg_databases:
  - name: testdb
    state: recreate
    owner: dbuser_test
    baseline: test_init.sql    # Execute after rebuild

Use cases: Test environment reset, clear dev database, modify immutable properties (encoding, locale), restore to initial state.

Difference from manual DROP + CREATE: Single command; auto-preserves Pgbouncer and Grafana config; auto-loads baseline init script.

Clone Database

Clone PostgreSQL databases using PG template mechanism. During cloning, no active connections to template database are allowed.

Script
bin/pgsql-db <cls> <db>   # Clone <db> (config must specify template)
Playbook
./pgsql-db.yml -l <cls> -e dbname=<db>   # Use Ansible playbook
Example
bin/pgsql-db pg-meta meta_dev    # Clone meta_dev (config has template: meta)

Config example:

pg_databases:
  - name: meta                   # Source database

  - name: meta_dev
    template: meta               # Use meta as template
    strategy: FILE_COPY          # PG15+ clone strategy, instant on PG18

Instant Clone (PG18+): If using PostgreSQL 18+, Pigsty defaults file_copy_method. With strategy: FILE_COPY, database clone completes in ~200ms without copying data files. E.g., cloning 30GB database: normal takes 18s, instant takes 200ms.

Manual clone: Ensure all connections to template are terminated:

SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname = 'meta';
CREATE DATABASE meta_dev TEMPLATE meta STRATEGY FILE_COPY;

Limitations: Instant clone only available on supported filesystems (xfs, btrfs, zfs, apfs); don’t use postgres database as template; in high-concurrency environments, all template connections must be cleared within clone window (~200ms).

Connection Pool Management

Connection pool params in database definitions are applied to Pgbouncer when creating/modifying databases.

By default all databases are added to Pgbouncer pool (pgbouncer: true). Databases are added to /etc/pgbouncer/database.txt. Database-level pool params (pool_auth_user, pool_mode, pool_size, pool_reserve, pool_size_min, pool_connlimit) are configured via this file.

Use postgres OS user with pgb alias to access Pgbouncer admin database. For more pool management, see Pgbouncer Management.

8.4.4 - Patroni HA Management

Manage PostgreSQL cluster HA with Patroni, including config changes, status check, switchover, restart, and reinit replica.

Overview

Pigsty uses Patroni to manage PostgreSQL clusters. It handles config changes, status checks, switchover, restart, reinit replicas, and more.

To use Patroni for management, you need one of the following identities:

Patroni provides patronictl CLI for management. Pigsty provides a wrapper alias pg to simplify operations.

Using patronictl via pg alias
pg ()
{
    local patroni_conf="/infra/conf/patronictl.yml";
    if [ ! -r ${patroni_conf} ]; then
        patroni_conf="/etc/patroni/patroni.yml";
        if [ ! -r ${patroni_conf} ]; then
            echo "error: patronictl config not found";
            return 1;
        fi;
    fi;
    patronictl -c ${patroni_conf} "$@"
}

Available Commands

Command Function Description
edit-config Edit Config Interactively edit cluster Patroni/PostgreSQL config
list List Status List cluster members and their status
switchover Switchover Switch primary role to specified replica (planned)
failover Failover Force failover to specified replica (emergency)
restart Restart Restart PostgreSQL instance to apply restart-required params
reload Reload Reload Patroni config (no restart needed)
reinit Reinit Replica Reinitialize replica (wipe data and re-clone)
pause Pause Auto-Failover Pause Patroni automatic failover
resume Resume Auto-Failover Resume Patroni automatic failover
history View History Show cluster failover history
show-config Show Config Display current cluster config (read-only)
query Execute Query Execute SQL query on cluster members
topology View Topology Display cluster replication topology
version View Version Display Patroni version info
remove Remove Member Remove cluster member from DCS (dangerous)

Edit Config

Use edit-config to interactively edit cluster Patroni and PostgreSQL config. This opens an editor to modify config stored in DCS, automatically applying changes to all members. You can change Patroni params (ttl, loop_wait, synchronous_mode, etc.) and PostgreSQL params in postgresql.parameters.

pg edit-config <cls>                  # Interactive edit cluster config
pg edit-config <cls> --force          # Skip confirmation and apply directly
pg edit-config <cls> -p <k>=<v>       # Modify PostgreSQL param (--pg shorthand)
pg edit-config <cls> -s <k>=<v>       # Modify Patroni param (--set shorthand)
demo/pgsql-config.cast

Common config modification examples:

# Modify PostgreSQL param: slow query threshold (prompts for confirmation)
pg edit-config pg-test -p log_min_duration_statement=1000

# Modify PostgreSQL param, skip confirmation
pg edit-config pg-test -p log_min_duration_statement=1000 --force

# Modify multiple PostgreSQL params
pg edit-config pg-test -p work_mem=256MB -p maintenance_work_mem=1GB --force

# Modify Patroni params: increase failure detection window (increase RTO)
pg edit-config pg-test -s loop_wait=15 -s ttl=60 --force

# Modify Patroni param: enable synchronous replication mode
pg edit-config pg-test -s synchronous_mode=true --force

# Modify Patroni param: enable strict synchronous mode (require at least one sync replica for writes)
pg edit-config pg-test -s synchronous_mode_strict=true --force

# Modify restart-required params (need pg restart after)
pg edit-config pg-test -p shared_buffers=4GB --force
pg edit-config pg-test -p shared_preload_libraries='timescaledb, pg_stat_statements' --force
pg edit-config pg-test -p max_connections=200 --force

Some params require PostgreSQL restart to take effect. Use pg list to check - instances marked with * need restart. Then use pg restart to apply. You can also use curl or programs to call Patroni REST API:

# View current config
curl -s 10.10.10.11:8008/config | jq .

# Modify params via API (requires auth)
curl -u 'postgres:Patroni.API' \
     -d '{"postgresql":{"parameters": {"log_min_duration_statement":200}}}' \
     -s -X PATCH http://10.10.10.11:8008/config | jq .

List Status

Use list to view cluster members and status. Output shows each instance’s name, host, role, state, timeline, and replication lag. This is the most commonly used command for checking cluster health.

pg list <cls>                         # List specified cluster status
pg list                               # List all clusters (on admin node)
pg list <cls> -e                      # Show extended info (--extended)
pg list <cls> -t                      # Show timestamp (--timestamp)
pg list <cls> -f json                 # Output as JSON (--format)
pg list <cls> -W 5                    # Refresh every 5 seconds (--watch)

Example output:

+ Cluster: pg-test (7322261897169354773) -----+----+--------------+
| Member    | Host        | Role    | State   | TL | Lag in MB    |
+-----------+-------------+---------+---------+----+--------------+
| pg-test-1 | 10.10.10.11 | Leader  | running |  1 |              |
| pg-test-2 | 10.10.10.12 | Replica | running |  1 |            0 |
| pg-test-3 | 10.10.10.13 | Replica | running |  1 |            0 |
+-----------+-------------+---------+---------+----+--------------+

Column descriptions: Member is instance name, composed of pg_cluster-pg_seq; Host is instance IP; Role is role type - Leader (primary), Replica, Sync Standby, Standby Leader (cascade primary); State is running state - running, streaming, in archive recovery, starting, stopped, etc.; TL is timeline number, incremented after each switchover; Lag in MB is replication lag in MB (not shown for primary).

Instances requiring restart show * after the name:

+ Cluster: pg-test (7322261897169354773) -------+----+--------------+
| Member      | Host        | Role    | State   | TL | Lag in MB    |
+-------------+-------------+---------+---------+----+--------------+
| pg-test-1 * | 10.10.10.11 | Leader  | running |  1 |              |
| pg-test-2 * | 10.10.10.12 | Replica | running |  1 |            0 |
+-------------+-------------+---------+---------+----+--------------+

Switchover

Use switchover for planned primary-replica switchover. Switchover is graceful: Patroni ensures replica is fully synced, demotes primary, then promotes target replica. Takes seconds with brief write unavailability. Use for primary host maintenance, upgrades, or migrating primary to better nodes.

pg switchover <cls>                   # Interactive switchover, prompts for target replica
pg switchover <cls> --leader <old>    # Specify current primary name
pg switchover <cls> --candidate <new> # Specify target replica name
pg switchover <cls> --scheduled <time> # Scheduled switchover, format: 2024-12-01T03:00
pg switchover <cls> --force           # Skip confirmation

Before switchover, ensure all replicas are healthy (running or streaming), replication lag is acceptable, and stakeholders are notified.

# Interactive switchover (recommended, shows topology and prompts for selection)
$ pg switchover pg-test
Current cluster topology
+ Cluster: pg-test (7322261897169354773) -----+----+--------------+
| Member    | Host        | Role    | State   | TL | Lag in MB    |
+-----------+-------------+---------+---------+----+--------------+
| pg-test-1 | 10.10.10.11 | Leader  | running |  1 |              |
| pg-test-2 | 10.10.10.12 | Replica | running |  1 |            0 |
| pg-test-3 | 10.10.10.13 | Replica | running |  1 |            0 |
+-----------+-------------+---------+---------+----+--------------+
Primary [pg-test-1]:
Candidate ['pg-test-2', 'pg-test-3'] []: pg-test-2
When should the switchover take place (e.g. 2024-01-01T12:00) [now]:
Are you sure you want to switchover cluster pg-test, demoting current leader pg-test-1? [y/N]: y

# Non-interactive switchover (specify primary and candidate)
pg switchover pg-test --leader pg-test-1 --candidate pg-test-2 --force

# Scheduled switchover (at 3 AM, for maintenance window)
pg switchover pg-test --leader pg-test-1 --candidate pg-test-2 --scheduled "2024-12-01T03:00"

After switchover, use pg list to confirm new cluster topology.

Failover

Use failover for emergency failover. Unlike switchover, failover is for when primary is unavailable. It directly promotes a replica without waiting for original primary confirmation. Since replicas may not be fully synced, failover may cause minor data loss. Use switchover for non-emergency situations.

pg failover <cls>                     # Interactive failover
pg failover <cls> --candidate <new>   # Specify replica to promote
pg failover <cls> --force             # Skip confirmation

Failover examples:

# Interactive failover
$ pg failover pg-test
Candidate ['pg-test-2', 'pg-test-3'] []: pg-test-2
Are you sure you want to failover cluster pg-test? [y/N]: y
Successfully failed over to "pg-test-2"

# Non-interactive failover (for emergencies)
pg failover pg-test --candidate pg-test-2 --force

Switchover vs Failover: Switchover is for planned maintenance, requires original primary online, ensures full sync before switching, no data loss; Failover is for emergency recovery, original primary can be offline, directly promotes replica, may lose unsynced data. Use Switchover for daily maintenance/upgrades; use Failover only when primary is completely down and unrecoverable.

The built-in Patroni failover subcommand currently has no --leader option. Use planned switchover --leader ... when you need to validate or name the old primary; failover accepts only the candidate replica.

Restart

Use restart to restart PostgreSQL instances, typically to apply restart-required parameter changes. When run against the whole cluster, patronictl submits each selected member in turn but does not guarantee a replica-first, leader-last order. If that order matters, restart replicas by role and then restart the leader separately.

pg restart <cls>                      # Restart all instances in cluster
pg restart <cls> <member>             # Restart specific instance
pg restart <cls> --role leader        # Restart primary only
pg restart <cls> --role replica       # Restart all replicas
pg restart <cls> --pending            # Restart only instances marked for restart
pg restart <cls> --scheduled <time>   # Scheduled restart
pg restart <cls> --timeout <sec>      # Set restart timeout (seconds)
pg restart <cls> --force              # Skip confirmation

After modifying restart-required params (shared_buffers, shared_preload_libraries, max_connections, max_worker_processes, etc.), use this command.

# Check which instances need restart (marked with *)
$ pg list pg-test
+ Cluster: pg-test (7322261897169354773) -------+----+--------------+
| Member      | Host        | Role    | State   | TL | Lag in MB    |
+-------------+-------------+---------+---------+----+--------------+
| pg-test-1 * | 10.10.10.11 | Leader  | running |  1 |              |
| pg-test-2 * | 10.10.10.12 | Replica | running |  1 |            0 |
+-------------+-------------+---------+---------+----+--------------+

# Restart single replica
pg restart pg-test pg-test-2

# Restart all cluster members (leader-last order is not guaranteed)
pg restart pg-test --force

# Restart only pending instances
pg restart pg-test --pending --force

# Explicitly restart replicas first, then the leader
pg restart pg-test --role replica --force
pg restart pg-test --role leader --force

# Scheduled restart (for maintenance window)
pg restart pg-test --scheduled "2024-12-01T03:00"

# Set restart timeout to 300 seconds
pg restart pg-test --timeout 300 --force

Reload

Use reload to reload Patroni config without restarting PostgreSQL. This re-reads config files and applies non-restart params via pg_reload_conf(). Lighter than restart - doesn’t interrupt connections or running queries.

pg reload <cls>                       # Reload entire cluster config
pg reload <cls> <member>              # Reload specific instance config
pg reload <cls> --role leader         # Reload primary only
pg reload <cls> --role replica        # Reload all replicas
pg reload <cls> --force               # Skip confirmation

Most PostgreSQL params work via reload. Only postmaster-context params (shared_buffers, max_connections, shared_preload_libraries, archive_mode, etc.) require restart.

# Reload entire cluster
pg reload pg-test

# Reload single instance
pg reload pg-test pg-test-1

# Force reload, skip confirmation
pg reload pg-test --force

Reinit Replica

Use reinit to reinitialize a replica. This deletes all data on the replica and rebuilds it according to Patroni’s create_replica_methods order. Pigsty tries basebackup (pg_basebackup) first by default; when a remote pgBackRest repository is enabled, pgbackrest is also configured as a fallback. Use this when replica data is corrupted, the replica is too far behind and required WAL has been removed, or replica configuration must be reset.

pg reinit <cls> <member>              # Reinitialize specified replica
pg reinit <cls> <member> --force      # Skip confirmation
pg reinit <cls> <member> --wait       # Wait for rebuild to complete
Destructive operation

This operation deletes all data on the target instance. Run it only on a replica, never on the primary.

# Reinitialize replica (prompts for confirmation)
$ pg reinit pg-test pg-test-2
Are you sure you want to reinitialize members pg-test-2? [y/N]: y
Success: reinitialize for member pg-test-2

# Force reinitialize, skip confirmation
pg reinit pg-test pg-test-2 --force

# Reinitialize and wait for completion
pg reinit pg-test pg-test-2 --force --wait

During rebuild, use pg list to check progress. Replica state shows creating replica:

+ Cluster: pg-test (7322261897169354773) --------------+----+------+
| Member    | Host        | Role    | State            | TL | Lag  |
+-----------+-------------+---------+------------------+----+------+
| pg-test-1 | 10.10.10.11 | Leader  | running          |  2 |      |
| pg-test-2 | 10.10.10.12 | Replica | creating replica |    |    ? |
+-----------+-------------+---------+------------------+----+------+

Pause

Use pause to pause Patroni automatic failover. When paused, Patroni won’t auto-promote replicas even if primary fails. Use for planned maintenance windows (prevent accidental triggers), debugging (prevent cluster state changes), or manual switchover timing control.

pg pause <cls>                        # Pause automatic failover
pg pause <cls> --wait                 # Pause and wait for all members to confirm
Warning

While paused, the cluster will not recover automatically if the primary fails. Run resume after maintenance.

# Pause automatic failover
$ pg pause pg-test
Success: cluster management is paused

# Check cluster status (shows Maintenance mode: on)
$ pg list pg-test
+ Cluster: pg-test (7322261897169354773) -----+----+--------------+
| Member    | Host        | Role    | State   | TL | Lag in MB    |
+-----------+-------------+---------+---------+----+--------------+
| pg-test-1 | 10.10.10.11 | Leader  | running |  1 |              |
| pg-test-2 | 10.10.10.12 | Replica | running |  1 |            0 |
+-----------+-------------+---------+---------+----+--------------+
 Maintenance mode: on

Resume

Use resume to resume Patroni automatic failover. Execute immediately after maintenance to ensure cluster auto-recovers on primary failure.

pg resume <cls>                       # Resume automatic failover
pg resume <cls> --wait                # Resume and wait for all members to confirm
# Resume automatic failover
$ pg resume pg-test
Success: cluster management is resumed

# Confirm resumed (Maintenance mode prompt disappears)
$ pg list pg-test

History

Use history to view cluster failover history. Each switchover (auto or manual) creates a new timeline record.

pg history <cls>                      # Show failover history
pg history <cls> -f json              # Output as JSON
pg history <cls> -f yaml              # Output as YAML
$ pg history pg-test
+----+-----------+------------------------------+---------------------------+
| TL |       LSN | Reason                       | Timestamp                 |
+----+-----------+------------------------------+---------------------------+
|  1 | 0/5000060 | no recovery target specified | 2024-01-15T10:30:00+08:00 |
|  2 | 0/6000000 | switchover to pg-test-2      | 2024-01-20T14:00:00+08:00 |
|  3 | 0/7000028 | failover to pg-test-1        | 2024-01-25T09:15:00+08:00 |
+----+-----------+------------------------------+---------------------------+

Column descriptions: TL is timeline number, incremented after each switchover, distinguishes primary histories; LSN is Log Sequence Number at switchover, marks WAL position; Reason is switchover reason - switchover to xxx (manual), failover to xxx (failure), or no recovery target specified (init); Timestamp is when switchover occurred.

Show Config

Use show-config to view current cluster config stored in DCS. This is read-only; use edit-config to modify.

pg show-config <cls>                  # Show cluster config
$ pg show-config pg-test
loop_wait: 10
maximum_lag_on_failover: 1048576
postgresql:
  parameters:
    archive_command: pgbackrest --stanza=pg-test archive-push %p
    max_connections: 100
    shared_buffers: 256MB
    log_min_duration_statement: 1000
  use_pg_rewind: true
  use_slots: true
retry_timeout: 10
ttl: 30
synchronous_mode: false

Query

Use query to quickly execute SQL on cluster members. Convenient for debugging - for complex production queries, use psql or applications.

pg query <cls> -c "<sql>"             # Execute on primary
pg query <cls> -c "<sql>" -m <member> # Execute on specific instance (--member)
pg query <cls> -c "<sql>" -r leader   # Execute on primary (--role)
pg query <cls> -c "<sql>" -r replica  # Execute on all replicas
pg query <cls> -f <file>              # Execute SQL from file
pg query <cls> -c "<sql>" -U <user>   # Specify username (--username)
pg query <cls> -c "<sql>" -d <db>     # Specify database (--dbname)
pg query <cls> -c "<sql>" --format json  # Output as JSON
# Check primary connection count
pg query pg-test -c "SELECT count(*) FROM pg_stat_activity"

# Check PostgreSQL version
pg query pg-test -c "SELECT version()"

# Check replication status on all replicas
pg query pg-test -c "SELECT pg_is_in_recovery(), pg_last_wal_replay_lsn()" -r replica

# Execute on specific instance
pg query pg-test -c "SELECT pg_is_in_recovery()" -m pg-test-2

# Use specific user and database
pg query pg-test -c "SELECT current_user, current_database()" -U postgres -d postgres

# Output as JSON
pg query pg-test -c "SELECT * FROM pg_stat_replication" --format json

Topology

Use topology to view cluster replication topology as a tree. More intuitive than list for showing primary-replica relationships, especially for cascading replication.

pg topology <cls>                     # Show replication topology
$ pg topology pg-test
+ Cluster: pg-test (7322261897169354773) -------+----+--------------+
| Member      | Host        | Role    | State   | TL | Lag in MB    |
+-------------+-------------+---------+---------+----+--------------+
| pg-test-1   | 10.10.10.11 | Leader  | running |  1 |              |
| + pg-test-2 | 10.10.10.12 | Replica | running |  1 |            0 |
| + pg-test-3 | 10.10.10.13 | Replica | running |  1 |            0 |
+-------------+-------------+---------+---------+----+--------------+

In cascading replication, topology clearly shows replication hierarchy - e.g., pg-test-3 replicates from pg-test-2, which replicates from primary pg-test-1.

Version

Use version to view patronictl version.

pg version                            # Show patronictl version
$ pg version
patronictl version 4.1.0

Remove

Use remove to remove cluster or member metadata from DCS. This is dangerous - only removes DCS metadata, doesn’t stop PostgreSQL or delete data files. Misuse may cause cluster state inconsistency.

pg remove <cls>                       # Remove entire cluster metadata from DCS

Normally you don’t need this command. To properly remove clusters/instances, use Pigsty’s bin/pgsql-rm script or pgsql-rm.yml playbook. Only consider remove for: orphaned DCS metadata (node physically removed but metadata remains), or cluster destroyed via other means requiring metadata cleanup.

# Remove entire cluster metadata (requires multiple confirmations)
$ pg remove pg-test
Please confirm the cluster name to remove: pg-test
You are about to remove all information in DCS for pg-test, please type: "Yes I am aware": Yes I am aware

8.4.5 - Managing PostgreSQL HBA Rules

HBA management - refresh rules, verify config, troubleshoot, Pgbouncer HBA

Quick Start

Pigsty uses declarative management: first define HBA rules in the inventory, then use bin/pgsql-hba <cls> to refresh.

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_hba_rules:                            # <--- Define HBA rules here!
      - {user: dbuser_app, db: app, addr: intra, auth: pwd, title: 'app access'}
      - {user: dbuser_api, db: all, addr: world, auth: ssl, title: 'api ssl access'}
Script
bin/pgsql-hba <cls>              # Refresh PostgreSQL and Pgbouncer HBA rules for cluster
bin/pgsql-hba <cls> <ip>...      # Refresh HBA rules for specific instances
Playbook
./pgsql.yml -l <cls> -t pg_hba,pg_reload                 # Refresh PostgreSQL HBA only
./pgsql.yml -l <cls> -t pgbouncer_hba,pgbouncer_reload   # Refresh Pgbouncer HBA only
./pgsql.yml -l <cls> -t pg_hba,pg_reload,pgbouncer_hba,pgbouncer_reload  # Refresh both
Example
bin/pgsql-hba pg-meta                      # Refresh pg-meta cluster HBA rules
bin/pgsql-hba pg-meta 10.10.10.10          # Refresh specific instance only
bin/pgsql-hba pg-meta 10.10.10.11 10.10.10.12  # Refresh multiple instances

For rule syntax, see HBA Configuration. For authentication methods, default boundaries, and credential management, see Authentication.

Action Description Risk
Refresh HBA Rules Re-render config files and reload service Low
Verify HBA Rules View current rules, test connection auth Read
Common Scenarios Add rules, block IP, role-based, post-expansion Low
Troubleshooting Connection rejected, auth failed, rules not applied -
Pgbouncer HBA Pgbouncer connection pool HBA management Low
demo/pgsql-hba.cast

Refresh HBA Rules

After modifying HBA rules in pigsty.yml, re-render config files and reload services.

Script
bin/pgsql-hba <cls>              # Refresh entire cluster HBA (PostgreSQL + Pgbouncer)
bin/pgsql-hba <cls> <ip>...      # Refresh specific instances (multiple IPs space-separated)
Playbook
./pgsql.yml -l <cls> -t pg_hba,pg_reload                 # Refresh PostgreSQL HBA only
./pgsql.yml -l <cls> -t pgbouncer_hba,pgbouncer_reload   # Refresh Pgbouncer HBA only
./pgsql.yml -l <cls> -t pg_hba,pg_reload,pgbouncer_hba,pgbouncer_reload  # Refresh both
Example
bin/pgsql-hba pg-meta                      # Refresh pg-meta cluster
bin/pgsql-hba pg-meta 10.10.10.10          # Refresh 10.10.10.10 instance only

Result: Renders PostgreSQL and Pgbouncer HBA config files based on inventory definitions, then reloads services to apply.

Config file locations

Service Config File Path Template File
PostgreSQL /pg/data/pg_hba.conf roles/pgsql/templates/pg_hba.conf
Pgbouncer /etc/pgbouncer/pgb_hba.conf roles/pgsql/templates/pgbouncer.hba
Don’t edit config files directly

Directly editing /pg/data/pg_hba.conf or /etc/pgbouncer/pgb_hba.conf works temporarily, but will be overwritten next time Ansible playbook runs. All HBA rule changes should be in pigsty.yml, then execute bin/pgsql-hba to refresh.

Related Tags

Tag Description
pg_hba Render PostgreSQL HBA config file
pg_reload Reload PostgreSQL config (needs pg_reload=true)
pgbouncer_hba Render Pgbouncer HBA config file
pgbouncer_reload Reload Pgbouncer config

Verify HBA Rules

After refreshing HBA rules, verify config is correctly applied.

View current HBA rules

SQL
-- View PostgreSQL HBA rules (recommended)
TABLE pg_hba_file_rules;

-- View matching rules for specific database
SELECT * FROM pg_hba_file_rules WHERE database @> ARRAY['mydb']::text[];
Bash
# View PostgreSQL HBA config file
cat /pg/data/pg_hba.conf

# View Pgbouncer HBA config file
cat /etc/pgbouncer/pgb_hba.conf

# View config file header (confirm if updated)
head -20 /pg/data/pg_hba.conf
Test Connection
# Test connection for specific user from specific address
psql -h <host> -p 5432 -U <user> -d <database> -c "SELECT 1"

# Test connection through Pgbouncer
psql -h <host> -p 6432 -U <user> -d <database> -c "SELECT 1"

Check HBA config syntax

# Reload config (validates syntax)
psql -c "SELECT pg_reload_conf()"

# If syntax errors, check logs
tail -f /pg/log/postgresql-*.log

Common Scenarios

Add New HBA Rule

Add rule to cluster config’s pg_hba_rules, then refresh:

pg-meta:
  vars:
    pg_hba_rules:
      - {user: new_user, db: new_db, addr: '192.168.1.0/24', auth: pwd, title: 'new app access'}
bin/pgsql-hba pg-meta

Emergency IP Block

When detecting malicious IP, add high-priority (order: 0) deny rule:

pg_hba_rules:
  - {user: all, db: all, addr: '10.1.1.100/32', auth: deny, order: 0, title: 'emergency block'}
bin/pgsql-hba pg-meta    # Refresh immediately

Role-Based Rules

Configure different HBA rules for primary and replica using role parameter:

pg_hba_rules:
  # Only primary allows write users
  - {user: writer, db: all, addr: intra, auth: pwd, role: primary, title: 'writer on primary'}
  # Replicas allow read-only users
  - {user: reader, db: all, addr: world, auth: ssl, role: replica, title: 'reader on replica'}

After refresh, rules auto-enable/disable based on instance’s pg_role.

Refresh HBA After Expansion

When cluster adds new instances, rules using addr: cluster need refresh to include new members:

./pgsql.yml -l 10.10.10.14       # Add new instance
bin/pgsql-hba pg-meta            # Refresh all instances' HBA (includes new member IPs)

Refresh HBA After Failover

After Patroni failover, instance pg_role may not match config. If HBA rules use role filtering, update config and refresh:

# Update role definitions in pigsty.yml then refresh
bin/pgsql-hba pg-meta

Troubleshooting

Connection Rejected

Symptom: FATAL: no pg_hba.conf entry for host "x.x.x.x", user "xxx", database "xxx"

Steps:

  1. Check current HBA rules, confirm if matching rule exists:
psql -c "TABLE pg_hba_file_rules"

  1. Confirm client IP, username, database matches any rule

  2. Check rule order (HBA uses first-match-wins)

  3. Add corresponding rule and refresh:

bin/pgsql-hba <cls>

Authentication Failed

Symptom: FATAL: password authentication failed for user "xxx"

Steps:

  1. Confirm password is correct
  2. Check password encryption method (pg_pwd_enc) compatibility with client
  3. Check if user exists:
SELECT * FROM pg_roles WHERE rolname = 'xxx';

HBA Rules Not Applied

Steps:

  1. Confirm refresh command was executed
  2. Check if Ansible execution succeeded
  3. Confirm PostgreSQL reloaded:
psql -c "SELECT pg_reload_conf()"
  1. Check if config file was updated:
head -20 /pg/data/pg_hba.conf

Rule Order Issues

HBA uses first-match-wins. If rules not working as expected:

  1. Check order values in rule definitions
  2. Use psql -c "TABLE pg_hba_file_rules" to view actual order
  3. Adjust order values (lower numbers = higher priority)

Pgbouncer HBA

Pgbouncer HBA management is similar to PostgreSQL, with some differences.

Config differences

Difference PostgreSQL Pgbouncer
Config file /pg/data/pg_hba.conf /etc/pgbouncer/pgb_hba.conf
Replication Supports db: replication Not supported
Local auth Uses ident Uses peer

Refresh Pgbouncer HBA

Script
bin/pgsql-hba <cls>    # Refresh both PostgreSQL and Pgbouncer
Playbook
./pgsql.yml -l <cls> -t pgbouncer_hba,pgbouncer_reload   # Refresh Pgbouncer HBA only
View
cat /etc/pgbouncer/pgb_hba.conf    # View Pgbouncer HBA rules

Best Practices

  1. Always manage in config files: Don’t edit pg_hba.conf directly - all changes through pigsty.yml
  2. Test environment first: HBA changes can cause connection issues - verify in test env first
  3. Use order for priority: Blocklist rules use order: 0 to ensure priority matching
  4. Refresh promptly: Refresh HBA after adding/removing instances or failover
  5. Principle of least privilege: Only open necessary access - avoid addr: world + auth: trust
  6. Monitor auth failures: Watch for auth failures in pg_stat_activity
  7. Backup config: Backup pigsty.yml before important changes

Related Documentation

8.4.6 - Pgbouncer Connection Pooling

Manage Pgbouncer connection pool, including pause, resume, disable, enable, reconnect, kill, and reload operations.

Overview

Pigsty uses Pgbouncer as PostgreSQL connection pooling middleware, listening on port 6432 by default, proxying access to local PostgreSQL on port 5432.

This is an optional component. If you don’t have massive connections or need transaction pooling and query metrics, you can disable it, connect directly to the database, or keep it unused.

User & Database Management

Pgbouncer users and databases are auto-managed by Pigsty, applying database config and user config when creating databases and creating users.

Database Management: Databases defined in pg_databases are auto-added to Pgbouncer by default. Set pgbouncer: false to exclude specific databases.

pg_databases:
  - name: mydb                # Added to connection pool by default
    pool_auth_user: dbuser_meta # Optional, auth query user (with pgbouncer_auth_query)
    pool_mode: transaction    # Database-level pool mode
    pool_size: 50             # Default pool size
    pool_reserve: 30          # Reserve pool size
    pool_size_min: 0          # Minimum pool size
    pool_connlimit: 100       # Max database connections
  - name: internal
    pgbouncer: false          # Excluded from connection pool

User Management: Users defined in pg_users need explicit pgbouncer: true to be added to connection pool user list.

pg_users:
  - name: dbuser_app
    password: DBUser.App
    pgbouncer: true           # Add to connection pool user list
    pool_mode: transaction    # User-level pool mode
    pool_connlimit: 50        # User-level max connections

Since Pigsty v4.1.0, database pool fields are unified as pool_reserve and pool_connlimit; legacy aliases pool_size_reserve / pool_max_db_conn are converged.

Service Management

In Pigsty, PostgreSQL cluster Primary Service and Replica Service default to Pgbouncer port 6432. To bypass connection pool and access PostgreSQL directly, customize pg_services, or set pg_default_service_dest to postgres.

Config Management

Pgbouncer config files are in /etc/pgbouncer/, generated and managed by Pigsty:

File Description
pgbouncer.ini Main config, pool-level params
database.txt Database list, database-level params
userlist.txt User password list
useropts.txt User-level pool params
pgb_hba.conf HBA access control rules

Pigsty auto-manages database.txt and userlist.txt, updating them when creating databases or creating users.

You can manually edit config then RELOAD to apply:

# Edit config
$ vim /etc/pgbouncer/pgbouncer.ini

# Reload via systemctl
$ sudo systemctl reload pgbouncer

# Reload as pg_dbsu / postgres user
$ pgb -c "RELOAD;"

Pool Management

Pgbouncer runs as the same dbsu as PostgreSQL, default postgres OS user. Pigsty provides pgb alias for easy management:

alias pgb='psql -p6432 -dpgbouncer'

Use pgb on database nodes to connect to Pgbouncer admin console for management commands and monitoring queries.

$ pgb
pgbouncer=# SHOW POOLS;
pgbouncer=# SHOW CLIENTS;
pgbouncer=# SHOW SERVERS;
Command Function Description
PAUSE Pause Pause database, wait for txn completion then disconnect
RESUME Resume Resume database paused by PAUSE/KILL/SUSPEND
DISABLE Disable Reject new client connections for database
ENABLE Enable Allow new client connections for database
RECONNECT Reconnect Gracefully close and rebuild server connections
KILL Kill Immediately disconnect all client and server connections
KILL_CLIENT Kill Client Terminate specific client connection
SUSPEND Suspend Flush buffers and stop listening, for online restart
SHUTDOWN Shutdown Shutdown Pgbouncer process
RELOAD Reload Reload config files
WAIT_CLOSE Wait Close Wait for server connections to close after RECONNECT/RELOAD
Monitor Commands Monitor View pool status, clients, servers, etc.

PAUSE

Use PAUSE to pause database connections. Pgbouncer waits for active txn/session to complete based on pool mode, then disconnects server connections. New client requests are blocked until RESUME.

PAUSE [db];           -- Pause specified database, or all if not specified

Typical use cases:

  • Online backend database switch (e.g., update connection target after switchover)
  • Maintenance operations requiring all connections disconnected
  • Combined with SUSPEND for Pgbouncer online restart
$ pgb -c "PAUSE mydb;"        # Pause mydb database
$ pgb -c "PAUSE;"             # Pause all databases

After pause, SHOW DATABASES shows paused status:

pgbouncer=# SHOW DATABASES;
   name   |   host    | port | database | ... | paused | disabled
----------+-----------+------+----------+-----+--------+----------
 mydb     | /var/run  | 5432 | mydb     | ... |      1 |        0

RESUME

Use RESUME to restore databases paused by PAUSE, KILL, or SUSPEND, allowing new connections and resuming normal service.

RESUME [db];          -- Resume specified database, or all if not specified
$ pgb -c "RESUME mydb;"       # Resume mydb database
$ pgb -c "RESUME;"            # Resume all databases

DISABLE

Use DISABLE to disable a database, rejecting all new client connection requests. Existing connections are unaffected.

DISABLE db;           -- Disable specified database (database name required)

Typical use cases:

  • Temporarily offline a database for maintenance
  • Block new connections for safe database migration
  • Gradually decommission a database being removed
$ pgb -c "DISABLE mydb;"      # Disable mydb, new connections rejected

ENABLE

Use ENABLE to enable a database previously disabled by DISABLE, accepting new client connections again.

ENABLE db;            -- Enable specified database (database name required)
$ pgb -c "ENABLE mydb;"       # Enable mydb, allow new connections

RECONNECT

Use RECONNECT to gracefully rebuild server connections. Pgbouncer closes connections when released back to pool, creating new ones when needed.

RECONNECT [db];       -- Rebuild server connections for database, or all if not specified

Typical use cases:

  • Refresh connections after backend database IP change
  • Reroute traffic after switchover
  • Rebuild connections after DNS update
$ pgb -c "RECONNECT mydb;"    # Rebuild mydb server connections
$ pgb -c "RECONNECT;"         # Rebuild all server connections

After RECONNECT, use WAIT_CLOSE to wait for old connections to fully release.

KILL

Use KILL to immediately disconnect all client and server connections for a database. Unlike PAUSE, KILL doesn’t wait for transaction completion - forces immediate disconnect.

KILL [db];            -- Kill all connections for database, or all (except admin) if not specified
$ pgb -c "KILL mydb;"         # Force disconnect all mydb connections
$ pgb -c "KILL;"              # Force disconnect all database connections (except admin)

After KILL, new connections are blocked until RESUME.

KILL_CLIENT

Use KILL_CLIENT to terminate a specific client connection. Client ID can be obtained from SHOW CLIENTS output.

KILL_CLIENT id;       -- Terminate client connection with specified ID
# View client connections
$ pgb -c "SHOW CLIENTS;"

# Terminate specific client (assuming ptr column shows ID 0x1234567890)
$ pgb -c "KILL_CLIENT 0x1234567890;"

SUSPEND

Use SUSPEND to suspend Pgbouncer. Flushes all socket buffers and stops listening until RESUME.

SUSPEND;              -- Suspend Pgbouncer

SUSPEND is mainly for Pgbouncer online restart (zero-downtime upgrade):

# 1. Suspend current Pgbouncer
$ pgb -c "SUSPEND;"

# 2. Start new Pgbouncer process (with -R option to take over sockets)
$ pgbouncer -R /etc/pgbouncer/pgbouncer.ini

# 3. New process takes over, old process exits automatically

SHUTDOWN

Use SHUTDOWN to shut down Pgbouncer process. Multiple shutdown modes supported:

SHUTDOWN;                      -- Immediate shutdown
SHUTDOWN WAIT_FOR_SERVERS;     -- Wait for server connections to release
SHUTDOWN WAIT_FOR_CLIENTS;     -- Wait for clients to disconnect (zero-downtime rolling restart)
Mode Description
SHUTDOWN Immediately shutdown Pgbouncer
WAIT_FOR_SERVERS Stop accepting new connections, wait for server release
WAIT_FOR_CLIENTS Stop accepting new connections, wait for all clients disconnect, for rolling restart 
$ pgb -c "SHUTDOWN WAIT_FOR_CLIENTS;"   # Graceful shutdown, wait for clients

RELOAD

Use RELOAD to reload Pgbouncer config files. Dynamically updates most config params without process restart.

RELOAD;               -- Reload config files
$ pgb -c "RELOAD;"              # Reload via admin console
$ systemctl reload pgbouncer    # Reload via systemd
$ kill -SIGHUP $(cat /run/postgresql/pgbouncer.pid)  # Reload via signal

Pigsty provides playbook task to reload Pgbouncer config:

./pgsql.yml -l <cls> -t pgbouncer_reload    # Reload cluster Pgbouncer config

WAIT_CLOSE

Use WAIT_CLOSE to wait for server connections to finish closing. Typically used after RECONNECT or RELOAD to ensure old connections are fully released.

WAIT_CLOSE [db];      -- Wait for server connections to close, or all if not specified
# Complete connection rebuild flow
$ pgb -c "RECONNECT mydb;"
$ pgb -c "WAIT_CLOSE mydb;"    # Wait for old connections to release

Monitoring

Pgbouncer provides rich SHOW commands for monitoring pool status:

Command Description
SHOW HELP Show available commands
SHOW DATABASES Show database config and status
SHOW POOLS Show pool statistics
SHOW CLIENTS Show client connection list
SHOW SERVERS Show server connection list
SHOW USERS Show user config
SHOW STATS Show statistics (requests, bytes)
SHOW STATS_TOTALS Show cumulative statistics
SHOW STATS_AVERAGES Show average statistics
SHOW CONFIG Show current config params
SHOW MEM Show memory usage
SHOW DNS_HOSTS Show DNS cached hostnames
SHOW DNS_ZONES Show DNS cached zones
SHOW SOCKETS Show open socket info
SHOW ACTIVE_SOCKETS Show active sockets
SHOW LISTS Show internal list counts
SHOW FDS Show file descriptor usage
SHOW STATE Show Pgbouncer running state
SHOW VERSION Show Pgbouncer version

Common monitoring examples:

# View pool status
$ pgb -c "SHOW POOLS;"

# View client connections
$ pgb -c "SHOW CLIENTS;"

# View server connections
$ pgb -c "SHOW SERVERS;"

# View statistics
$ pgb -c "SHOW STATS;"

# View database status
$ pgb -c "SHOW DATABASES;"

For more monitoring command details, see Pgbouncer official docs.

Unix Signals

Pgbouncer supports Unix signal control, useful when admin console is unavailable:

Signal Equivalent Command Description
SIGHUP RELOAD Reload config files
SIGTERM SHUTDOWN WAIT_FOR_CLIENTS Graceful shutdown, wait clients
SIGINT SHUTDOWN WAIT_FOR_SERVERS Graceful shutdown, wait servers
SIGQUIT SHUTDOWN Immediate shutdown
SIGUSR1 PAUSE Pause all databases
SIGUSR2 RESUME Resume all databases 
# Reload config via signal
$ kill -SIGHUP $(cat /run/postgresql/pgbouncer.pid)

# Graceful shutdown via signal
$ kill -SIGTERM $(cat /run/postgresql/pgbouncer.pid)

# Pause via signal
$ kill -SIGUSR1 $(cat /run/postgresql/pgbouncer.pid)

# Resume via signal
$ kill -SIGUSR2 $(cat /run/postgresql/pgbouncer.pid)

Traffic Switching

Pigsty-managed database routes live in /etc/pgbouncer/database.txt. To move one database’s Pgbouncer traffic to another node, edit that file, reload the configuration, then drain and rebuild existing server connections:

# 1. Change only mydb's backend target to 10.10.10.12
$ sed -i -E '/^mydb[[:space:]]*=/ s#host=[^[:space:]]+#host=10.10.10.12#' /etc/pgbouncer/database.txt

# 2. Reload config
$ pgb -c "RELOAD;"

# 3. Rebuild this database's connections and wait for old connections to close
$ pgb -c "RECONNECT mydb;"
$ pgb -c "WAIT_CLOSE mydb;"

The pgb-route function currently shipped in the source only edits /etc/pgbouncer/pgbouncer.ini. That file merely includes database.txt and does not contain the generated per-database host= routes, so the function does not change managed database backends. Do not use it in place of the procedure above.

8.4.7 - Managing PostgreSQL Component Services

Use systemctl to manage PostgreSQL cluster component services - start, stop, restart, reload, and status check.

Overview

Pigsty’s PGSQL module consists of multiple components, each running as a systemd service on nodes. (pgbackrest is an exception)

Understanding these components and their management is essential for maintaining production PostgreSQL clusters.

Component Port Service Name Description
Patroni 8008 patroni HA manager, manages PostgreSQL lifecycle
PostgreSQL 5432 postgres Placeholder service, not used, for emergency
Pgbouncer 6432 pgbouncer Connection pooling middleware, traffic entry
PgBackRest - - pgBackRest has no daemon service
HAProxy 543x haproxy Load balancer, exposes database services
pg_exporter 9630 pg_exporter PostgreSQL metrics exporter
pgbouncer_exporter 9631 pgbouncer_exporter Pgbouncer metrics exporter
vip-manager - vip-manager Optional, manages L2 VIP address floating
Important

Do NOT use systemctl directly to manage PostgreSQL service. PostgreSQL is managed by Patroni - use patronictl commands instead. Direct PostgreSQL operations may cause Patroni state inconsistency and trigger unexpected failover. The postgres service is an emergency escape hatch when Patroni fails.

Quick Reference

Operation Command
Start systemctl start <service>
Stop systemctl stop <service>
Restart systemctl restart <service>
Reload systemctl reload <service>
Status systemctl status <service>
Logs journalctl -u <service> -f
Enable systemctl enable <service>
Disable systemctl disable <service>

Common service names: patroni, pgbouncer, haproxy, pg_exporter, pgbouncer_exporter, vip-manager

Patroni

Patroni is PostgreSQL’s HA manager, handling startup, shutdown, failure detection, and automatic failover. It’s the core PGSQL module component. PostgreSQL process is managed by Patroni - don’t use systemctl to manage postgres service directly.

Start Patroni

systemctl start patroni     # Start Patroni (also starts PostgreSQL)

After starting, Patroni auto-launches PostgreSQL. On first start, behavior depends on role:

  • Primary: Initialize or recover data directory
  • Replica: Clone data from primary and establish replication

Stop Patroni

systemctl stop patroni      # Stop Patroni (also stops PostgreSQL)

Stopping Patroni gracefully shuts down PostgreSQL. Note: If this is primary and auto-failover isn’t paused, may trigger failover.

Restart Patroni

systemctl restart patroni   # Restart Patroni (also restarts PostgreSQL)

Restart causes brief service interruption. For production, use pg restart for rolling restart.

Reload Patroni

systemctl reload patroni    # Reload Patroni config

Reload re-reads config file and applies hot-reloadable params to PostgreSQL.

View Status & Logs

systemctl status patroni    # View Patroni service status
journalctl -u patroni -f    # Real-time Patroni logs
journalctl -u patroni -n 100 --no-pager  # Last 100 lines

Config file: /etc/patroni/patroni.yml

Best Practice: Use patronictl instead of systemctl to manage PostgreSQL clusters.

Pgbouncer

Pgbouncer is a lightweight PostgreSQL connection pooling middleware. Business traffic typically goes through Pgbouncer (6432) rather than directly to PostgreSQL (5432) for connection reuse and database protection.

Start Pgbouncer

systemctl start pgbouncer

Stop Pgbouncer

systemctl stop pgbouncer

Note: Stopping Pgbouncer disconnects all pooled business connections.

Restart Pgbouncer

systemctl restart pgbouncer

Restart disconnects all existing connections. For config changes only, use reload.

Reload Pgbouncer

systemctl reload pgbouncer

Reload re-reads config files (user list, pool params, etc.) without disconnecting existing connections.

View Status & Logs

systemctl status pgbouncer
journalctl -u pgbouncer -f

Config files:

  • Main config: /etc/pgbouncer/pgbouncer.ini
  • HBA rules: /etc/pgbouncer/pgb_hba.conf
  • User list: /etc/pgbouncer/userlist.txt
  • Database list: /etc/pgbouncer/database.txt

Admin Console

psql -p 6432 -U postgres -d pgbouncer  # Connect to Pgbouncer admin console

Common admin commands:

SHOW POOLS;      -- View pool status
SHOW CLIENTS;    -- View client connections
SHOW SERVERS;    -- View backend server connections
SHOW STATS;      -- View statistics
RELOAD;          -- Reload config
PAUSE;           -- Pause all pools
RESUME;          -- Resume all pools

HAProxy

HAProxy is a high-performance load balancer that routes traffic to correct PostgreSQL instances. Pigsty uses HAProxy to expose services, routing traffic based on role (primary/replica) and health status.

Start HAProxy

systemctl start haproxy

Stop HAProxy

systemctl stop haproxy

Note: Stopping HAProxy disconnects all load-balanced connections.

Restart HAProxy

systemctl restart haproxy

Reload HAProxy

systemctl reload haproxy

HAProxy supports graceful reload without disconnecting existing connections. Use reload for config changes.

View Status & Logs

systemctl status haproxy
journalctl -u haproxy -f

Config files: the main configuration is /etc/haproxy/haproxy.cfg; Pigsty-generated service fragments are stored under /etc/haproxy/conf.d/.

Admin Interface

HAProxy provides a web admin interface, default port 9101:

http://<node_ip>:9101/haproxy

Default auth: username admin, password configured by haproxy_admin_password.

pg_exporter

pg_exporter is PostgreSQL’s Prometheus metrics exporter for collecting database performance metrics.

Start pg_exporter

systemctl start pg_exporter

Stop pg_exporter

systemctl stop pg_exporter

After stopping, Prometheus can’t collect PostgreSQL metrics from this instance.

Restart pg_exporter

systemctl restart pg_exporter

View Status & Logs

systemctl status pg_exporter
journalctl -u pg_exporter -f

Config file: /etc/pg_exporter.yml

Verify Metrics

curl -s localhost:9630/metrics | head -20

pgbouncer_exporter

pgbouncer_exporter is Pgbouncer’s Prometheus metrics exporter.

Start/Stop/Restart

systemctl start pgbouncer_exporter
systemctl stop pgbouncer_exporter
systemctl restart pgbouncer_exporter

View Status & Logs

systemctl status pgbouncer_exporter
journalctl -u pgbouncer_exporter -f

Verify Metrics

curl -s localhost:9631/metrics | head -20

vip-manager

vip-manager is an optional component for managing L2 VIP address floating. When pg_vip_enabled is enabled, vip-manager binds VIP to current primary node.

Start vip-manager

systemctl start vip-manager

Stop vip-manager

systemctl stop vip-manager

After stopping, VIP address is released from current node.

Restart vip-manager

systemctl restart vip-manager

View Status & Logs

systemctl status vip-manager
journalctl -u vip-manager -f

Config file: /etc/default/vip-manager

Verify VIP Binding

ip addr show           # Check network interfaces, verify VIP binding
pg list <cls>          # Confirm primary location

Startup Order & Dependencies

Recommended PGSQL module component startup order:

1. patroni          # Start Patroni first (auto-starts PostgreSQL)
2. pgbouncer        # Then start connection pool
3. haproxy          # Start load balancer
4. pg_exporter      # Start metrics exporters
5. pgbouncer_exporter
6. vip-manager      # Finally start VIP manager (if enabled)

Stop order should be reversed. Pigsty playbooks handle these dependencies automatically.

Batch Start All Services

systemctl start patroni pgbouncer haproxy pg_exporter pgbouncer_exporter

Batch Stop All Services

systemctl stop pgbouncer_exporter pg_exporter haproxy pgbouncer patroni

Common Troubleshooting

Service Startup Failure

systemctl status <service>        # View service status
journalctl -u <service> -n 50     # View recent logs
journalctl -u <service> --since "5 min ago"  # Last 5 minutes logs

Patroni Won’t Start

Symptom Possible Cause Solution
Can’t connect to etcd etcd cluster unavailable Check etcd service status
Data dir permission error File ownership not postgres chown -R postgres:postgres /pg/data
Port in use Leftover PostgreSQL process pg_ctl stop -D /pg/data or kill

Pgbouncer Won’t Start

Symptom Possible Cause Solution
Config syntax error INI format error Check /etc/pgbouncer/pgbouncer.ini
Port in use Port 6432 already used lsof -i :6432
userlist.txt permissions Incorrect file permissions chmod 600 /etc/pgbouncer/userlist.txt

HAProxy Won’t Start

Symptom Possible Cause Solution
Config syntax error Invalid main configuration or service fragment haproxy -Ws -f /etc/haproxy/haproxy.cfg -f /etc/haproxy/conf.d -c -q
Port in use Service port conflict lsof -i :5433

Related Documentation

8.4.8 - Manage PostgreSQL Cron Jobs

Configure crontab to schedule PostgreSQL backups, vacuum freeze, and bloat maintenance tasks

Pigsty uses crontab to manage scheduled tasks for routine backups, freezing aging transactions, and reorganizing bloated tables and indexes.

Quick Reference

Operation Quick Command Description
Configure Cron Jobs ./pgsql.yml -t pg_crontab -l <cls> Apply pg_crontab config
View Cron Jobs crontab -l View as postgres user
Physical Backup pg-backup [full|diff|incr] Execute backup with pgBackRest
Transaction Freeze pg-vacuum [database...] Freeze aging transactions, prevent XID wraparound
Bloat Maintenance pg-repack [database...] Online reorganize bloated tables and indexes

For other management tasks, see: Backup Management, Monitoring System, HA Management.

Configure Cron Jobs

Use the pg_crontab parameter to configure cron jobs for the PostgreSQL database superuser (pg_dbsu, default postgres).

Example Configuration

The following pg-meta cluster configures a daily full backup at 1:00 AM, while pg-test configures weekly full backup on Monday with incremental backups on other days.

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_crontab:
      - '00 01 * * * /pg/bin/pg-backup'
pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica }
  vars:
    pg_cluster: pg-test
    pg_crontab:
      - '00 01 * * 1            /pg/bin/pg-backup full'
      - '00 01 * * 2,3,4,5,6,7  /pg/bin/pg-backup'

Recommended Maintenance Schedule

pg_crontab:
  - '00 01 * * * /pg/bin/pg-backup full'    # Daily full backup at 1:00 AM
  - '00 03 * * 0 /pg/bin/pg-vacuum'         # Weekly vacuum freeze on Sunday at 3:00 AM
  - '00 04 * * 1 /pg/bin/pg-repack'         # Weekly repack on Monday at 4:00 AM
Task Frequency Timing Description
pg-backup Daily Early morning Full or incremental backup, depending on business needs
pg-vacuum Weekly Sunday early morning Freeze aging transactions, prevent XID wraparound
pg-repack Weekly/Monthly Off-peak hours Reorganize bloated tables/indexes, reclaim space
Primary Only Execution

The pg-backup, pg-vacuum, and pg-repack scripts automatically detect the current node role. Only the primary will actually execute; replicas will exit directly. Therefore, you can safely configure the same cron jobs on all nodes, and after failover, the new primary will automatically continue executing maintenance tasks.

Apply Cron Jobs

Cron jobs are automatically written to the default location for the corresponding OS distribution when the pgsql.yml playbook executes (the pg_crontab task):

  • EL (RHEL/Rocky/Alma): /var/spool/cron/postgres
  • Debian/Ubuntu: /var/spool/cron/crontabs/postgres
Playbook
./pgsql.yml -l pg-meta -t pg_crontab     # Apply pg_crontab config to specified cluster
./pgsql.yml -l 10.10.10.10 -t pg_crontab # Target specific host only
Manual
# Edit cron jobs as postgres user
sudo -u postgres crontab -e

# Or edit crontab file directly
sudo vi /var/spool/cron/postgres           # EL series
sudo vi /var/spool/cron/crontabs/postgres  # Debian/Ubuntu

Each playbook execution will fully overwrite the cron job configuration.

View Cron Jobs

Execute the following command as the pg_dbsu OS user to view cron jobs:

crontab -l

# Pigsty Managed Crontab for postgres
SHELL=/bin/bash
PATH=/usr/pgsql/bin:/pg/bin:/usr/local/bin:/usr/bin:/usr/sbin:/bin:/sbin
MAILTO=""
00 01 * * * /pg/bin/pg-backup

If you’re not familiar with crontab syntax, refer to Crontab Guru for explanations.

pg-backup

pg-backup is Pigsty’s physical backup script based on pgBackRest, supporting full, differential, and incremental backup modes.

Basic Usage

pg-backup                # Execute incremental backup (default), auto full if no existing full backup
pg-backup full           # Execute full backup
pg-backup diff           # Execute differential backup (based on most recent full backup)
pg-backup incr           # Execute incremental backup (based on most recent any backup)

Backup Types

Type Parameter Description
Full Backup full Complete backup of all data, only this backup needed for recovery
Differential diff Backup changes since last full backup, recovery needs full + diff
Incremental incr Backup changes since last any backup, recovery needs complete chain

Execution Requirements

  • Script must run on primary as postgres user
  • Script auto-detects current node role, exits (exit 1) when run on replica
  • Auto-retrieves stanza name from /etc/pgbackrest/pgbackrest.conf

Common Cron Configurations

Daily Full
pg_crontab:
  - '00 01 * * * /pg/bin/pg-backup full'    # Daily full backup at 1:00 AM
Weekly Full + Daily Incr
pg_crontab:
  - '00 01 * * 1            /pg/bin/pg-backup full'  # Monday full backup
  - '00 01 * * 2,3,4,5,6,7  /pg/bin/pg-backup'       # Other days incremental
Weekly Full + Daily Diff
pg_crontab:
  - '00 01 * * 1            /pg/bin/pg-backup full'  # Monday full backup
  - '00 01 * * 2,3,4,5,6,7  /pg/bin/pg-backup diff'  # Other days differential

For more backup and recovery operations, see the Backup Management section.

pg-vacuum

pg-vacuum is Pigsty’s transaction freeze script for executing VACUUM FREEZE operations to prevent database shutdown from transaction ID (XID) wraparound.

Basic Usage

Basic
pg-vacuum                    # Freeze aging tables in all databases
pg-vacuum mydb               # Process specified database only
pg-vacuum mydb1 mydb2        # Process multiple databases
Options
pg-vacuum -n mydb            # Dry run mode, display only without executing
pg-vacuum -a 80000000 mydb   # Use custom age threshold (default 100M)
pg-vacuum -r 50 mydb         # Use custom aging ratio threshold (default 40%)
Manual SQL
-- Execute VACUUM FREEZE on entire database
VACUUM FREEZE;

-- Execute VACUUM FREEZE on specific table
VACUUM FREEZE schema.table_name;

Command Options

Option Description Default
-h, --help Show help message -
-n, --dry-run Dry run mode, display only false
-a, --age Age threshold, tables exceeding need freeze 100000000
-r, --ratio Aging ratio threshold, full freeze if exceeded (%) 40

Logic

  1. Check database datfrozenxid age, skip database if below threshold
  2. Calculate aging page ratio (percentage of table pages exceeding age threshold of total pages)
  3. If aging ratio > 40%, execute full database VACUUM FREEZE ANALYZE
  4. Otherwise, only execute VACUUM FREEZE ANALYZE on tables exceeding age threshold

Script sets vacuum_cost_limit = 10000 and vacuum_cost_delay = 1ms to control I/O impact.

Execution Requirements

  • Script must run on primary as postgres user
  • Uses file lock /tmp/pg-vacuum.lock to prevent concurrent execution
  • Auto-skips template0, template1, postgres system databases

Common Cron Configuration

pg_crontab:
  - '00 03 * * 0 /pg/bin/pg-vacuum'     # Weekly Sunday at 3:00 AM

pg-repack

pg-repack is Pigsty’s bloat maintenance script based on the pg_repack extension for online reorganization of bloated tables and indexes.

Basic Usage

Basic
pg-repack                    # Reorganize bloated tables and indexes in all databases
pg-repack mydb               # Reorganize specified database only
pg-repack mydb1 mydb2        # Reorganize multiple databases
Options
pg-repack -n mydb            # Dry run mode, display only without executing
pg-repack -t mydb            # Reorganize tables only
pg-repack -i mydb            # Reorganize indexes only
pg-repack -T 30 -j 4 mydb    # Custom lock timeout (seconds) and parallelism
Manual
# Use pg_repack command directly to reorganize specific table
pg_repack dbname -t schema.table

# Use pg_repack command directly to reorganize specific index
pg_repack dbname -i schema.index

Command Options

Option Description Default
-h, --help Show help message -
-n, --dry-run Dry run mode, display only false
-t, --table Reorganize tables only false
-i, --index Reorganize indexes only false
-T, --timeout Lock wait timeout (seconds) 10
-j, --jobs Parallel jobs 2

Auto-Selection Thresholds

Script auto-selects objects to reorganize based on table/index size and bloat ratio:

Table Bloat Thresholds

Size Range Bloat Threshold Max Count
< 256MB > 40% 64
256MB - 2GB > 30% 16
2GB - 8GB > 20% 4
8GB - 64GB > 15% 1

Index Bloat Thresholds

Size Range Bloat Threshold Max Count
< 128MB > 40% 64
128MB - 1GB > 35% 16
1GB - 8GB > 30% 4
8GB - 64GB > 20% 1

Tables/indexes over 64GB are skipped with a warning and require manual handling.

Execution Requirements

  • Script must run on primary as postgres user
  • Requires pg_repack extension installed (installed by default in Pigsty)
  • Requires pg_table_bloat and pg_index_bloat views in monitor schema
  • Uses file lock /tmp/pg-repack.lock to prevent concurrent execution
  • Auto-skips template0, template1, postgres system databases
Lock Waiting

Normal reads/writes are not affected during reorganization, but the final switch moment requires acquiring AccessExclusive lock on the table, blocking all access. For high-throughput workloads, recommend running during off-peak hours or maintenance windows.

Common Cron Configuration

pg_crontab:
  - '00 04 * * 1 /pg/bin/pg-repack'     # Weekly Monday at 4:00 AM

You can confirm database bloat through Pigsty’s PGCAT Database - Table Bloat panel and select high-bloat tables and indexes for reorganization.

For more details see: Managing Relation Bloat

Remove Cron Jobs

When using the pgsql-rm.yml playbook to remove a PostgreSQL cluster, it automatically deletes the postgres user’s crontab file.

./pgsql-rm.yml -l <cls> -t pg_crontab    # Remove cron jobs only
./pgsql-rm.yml -l <cls> --check          # Preflight full removal; execute only after backup and target confirmation

Related Documentation

8.4.9 - Managing PostgreSQL Extensions

Extension management - download, install, configure, enable, update, and remove extensions

Quick Start

Pigsty provides 575 extensions. Using extensions involves four steps: Download, Install, Configure, Enable.

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_extensions: [ postgis, timescaledb, pgvector ]           # <--- Install extension packages
    pg_libs: 'timescaledb, pg_stat_statements, auto_explain'    # <--- Configure preload extensions
    pg_databases:
      - name: meta
        extensions: [ postgis, timescaledb, vector ]            # <--- Enable in database
Script
bin/pgsql-ext <cls>           # Install extensions defined in config on <cls> cluster
bin/pgsql-ext <cls> [ext...]  # Install extensions specified on command line
Playbook
./pgsql.yml -l pg-meta -t pg_ext    # Use playbook to install extensions
Example
bin/pgsql-ext pg-meta                         # Install defined extensions on pg-meta cluster
bin/pgsql-ext pg-meta pg_duckdb pg_mooncake   # Install specified extensions

For complete extension reference, see Extensions. For available extensions, see Extension Catalog.

Action Command Description
Download Extensions ./infra.yml -t repo_build Download extensions to local repo
Install Extensions bin/pgsql-ext <cls> Install extension packages on cluster
Configure Extensions pg edit-config <cls> -p Add to preload libs (requires restart)
Enable Extensions psql -c 'CREATE EXT ...' Create extension objects in database
Update Extensions ALTER EXTENSION UPDATE Update packages and extension objects
Remove Extensions DROP EXTENSION Drop extension objects, uninstall pkgs
demo/pgsql-ext.cast

Install Extensions

Extensions defined in pg_extensions are auto-installed during PostgreSQL cluster creation in the pg_extension task.

To install extensions on an existing cluster, add extensions to all.children.<cls>.pg_extensions, then execute:

Script
bin/pgsql-ext <cls>   # Install extensions on <cls> cluster
Playbook
./pgsql.yml -l <cls> -t pg_extension   # Use Ansible playbook
Example
bin/pgsql-ext pg-meta    # Install extensions defined in config on pg-meta

Example: Install PostGIS, TimescaleDB and PGVector on cluster

#all.children.pg-meta.vars:
pg_extensions: [ postgis, timescaledb, pgvector ]

Result: Installs extension packages on all cluster nodes. Pigsty auto-translates package aliases to actual package names for OS and PG version.

Ensure repos available before install

Before installing, ensure nodes have correct repos configured - extensions downloaded to local repo, or upstream repos configured.

Manual Install

If you don’t want to use Pigsty config to manage extensions, pass extension list directly on command line:

Script
bin/pgsql-ext pg-meta pg_duckdb pg_mooncake   # Install specified extensions on pg-meta
Playbook
./pgsql.yml -l pg-meta -t pg_ext -e '{"pg_extensions": ["pg_duckdb", "pg_mooncake"]}'

You can also use pig package manager CLI to install extensions on single node, with auto package alias resolution.

pig install postgis timescaledb       # Install multiple extensions
pig install pgvector -v 18            # Install for specific PG major version

ansible pg-test -b -a 'pig install pg_duckdb'   # Batch install on cluster with Ansible

You can also use OS package manager directly (apt/dnf), but you must know the exact RPM/DEB package name for your OS/PG:

# EL systems (RHEL, Rocky, Alma, Oracle Linux)
sudo yum install -y pgvector_18*

# Debian / Ubuntu
sudo apt install -y postgresql-18-pgvector

Download Extensions

To install extensions, ensure node’s extension repos contain the extension:

Pigsty’s default config auto-downloads mainstream extensions during installation. For additional extensions, add to repo_extra_packages and rebuild repo:

repo_extra_packages: [ pgvector, postgis, timescaledb ]
Script
make repo         # Shortcut = repo-build + node-repo
make repo-build   # Rebuild Infra repo (download packages and deps)
make node-repo    # Refresh node repo cache, update Infra repo reference
Playbook
./deploy.yml -t repo_build,node_repo  # Execute both tasks at once
./infra.yml -t repo_build     # Re-download packages to local repo
./node.yml  -t node_repo      # Refresh node repo cache

Configure Repos

You can also let all nodes use upstream repos directly (not recommended for production), skipping download and installing from upstream extension repos:

./node.yml -t node_repo -e node_repo_modules=node,pgsql   # Add PGDG and Pigsty upstream repos

Configure Extensions

Some extensions require preloading to shared_preload_libraries, requiring database restart after modification.

Use pg_libs as its default value to configure preload extensions, but this only takes effect during cluster init - later modifications are ineffective.

pg-meta:
  vars:
    pg_cluster: pg-meta
    pg_libs: 'timescaledb, pg_stat_statements, auto_explain'   # Preload extensions
    pg_extensions: [ timescaledb, postgis, pgvector ]          # Install packages

For existing clusters, refer to Modify Config to modify shared_preload_libraries:

pg edit-config pg-meta --force -p shared_preload_libraries='timescaledb, pg_stat_statements, auto_explain'
pg restart pg-meta   # Modify pg-meta params and restart to apply

Ensure extension packages are correctly installed before adding preload config. If extension in shared_preload_libraries doesn’t exist or fails to load, PostgreSQL won’t start. Also, manage cluster config changes through Patroni - avoid using ALTER SYSTEM or pg_parameters to modify instance config separately. If primary and replica configs differ, it may cause startup failure or replication interruption.

Enable Extensions

After installing packages, execute CREATE EXTENSION in database to use extension features.

Enable during cluster init

Declare extensions to enable in database definition via extensions array:

pg_databases:
  - name: meta
    extensions:
      - vector                             # Simple form
      - { name: postgis, schema: public }  # Specify schema

Manual enable

SQL
CREATE EXTENSION vector;                      -- Create extension
CREATE EXTENSION postgis SCHEMA public;       -- Specify schema
CREATE EXTENSION IF NOT EXISTS vector;        -- Idempotent creation
CREATE EXTENSION postgis_topology CASCADE;    -- Auto-install dependencies
psql
psql -d meta -c 'CREATE EXTENSION vector;'                  # Create extension in meta database
psql -d meta -c 'CREATE EXTENSION postgis SCHEMA public;'   # Specify schema
Playbook
# After modifying database definition, use playbook to enable extensions
bin/pgsql-db pg-meta meta    # Creating/modifying database auto-enables defined extensions

Result: Creates extension objects (functions, types, operators, index methods, etc.) in database, enabling use of extension features.

Update Extensions

Extension updates involve two layers: package update and extension object update.

Update packages

pig
pig update pgvector                           # Update extension with pig
yum
sudo yum update pgvector_18 # EL
apt
sudo apt upgrade postgresql-18-pgvector  # Debian/Ubuntu

Update extension objects

-- View upgradeable extensions
SELECT name, installed_version, default_version FROM pg_available_extensions
WHERE installed_version IS NOT NULL AND installed_version <> default_version;

-- Update extension to latest version
ALTER EXTENSION vector UPDATE;

-- Update to specific version
ALTER EXTENSION vector UPDATE TO '0.8.1';
Update Notes

Backup database before updating extensions. Preloaded extensions may require PostgreSQL restart after update. Some extension version upgrades may be incompatible - check extension docs.

Remove Extensions

Removing extensions involves two layers: drop extension objects and uninstall packages.

Drop extension objects

DROP EXTENSION vector;              -- Drop extension
DROP EXTENSION vector CASCADE;      -- Cascade drop (drops dependent objects)

Remove from preload

For preloaded extensions, remove from shared_preload_libraries and restart:

pg edit-config pg-meta --force -p shared_preload_libraries='pg_stat_statements, auto_explain'
pg restart pg-meta   # Restart to apply config

Uninstall packages (optional)

pig
pig remove pgvector                           # Uninstall with pig
yum
sudo yum remove pgvector_18*                  # EL systems
apt
sudo apt remove postgresql-18-pgvector        # Debian/Ubuntu
CASCADE Warning

Using CASCADE to drop extensions also drops all objects depending on that extension (tables, indexes, views, etc.). Check dependencies before executing.

Query Extensions

Common SQL queries for extension info:

View enabled extensions

SELECT extname, extversion, nspname AS schema
FROM pg_extension e JOIN pg_namespace n ON e.extnamespace = n.oid
ORDER BY extname;

View available extensions

SELECT name, default_version, installed_version, comment
FROM pg_available_extensions
WHERE installed_version IS NOT NULL   -- Only show installed
ORDER BY name;

Check if extension is available

SELECT * FROM pg_available_extensions WHERE name = 'vector';

View extension dependencies

SELECT e.extname, d.refobjid::regclass AS depends_on
FROM pg_extension e
JOIN pg_depend d ON d.objid = e.oid
WHERE d.deptype = 'e' AND e.extname = 'postgis_topology';

View extension objects

SELECT classid::regclass, objid, deptype
FROM pg_depend
WHERE refobjid = (SELECT oid FROM pg_extension WHERE extname = 'vector');

psql shortcuts

\dx                    # List enabled extensions
\dx+ vector            # Show extension details

Add Repos

To install directly from upstream, manually add repos.

Using Pigsty playbook

./node.yml -t node_repo -e node_repo_modules=node,pgsql        # Add PGDG and Pigsty repos
./node.yml -t node_repo -e node_repo_modules=node,pgsql,local  # Including local repo

YUM repos (EL systems)

# Pigsty repo
curl -fsSL https://repo.pigsty.io/key | sudo tee /etc/pki/rpm-gpg/RPM-GPG-KEY-pigsty >/dev/null
curl -fsSL https://repo.pigsty.io/yum/repo | sudo tee /etc/yum.repos.d/pigsty.repo >/dev/null

# China mainland mirror
curl -fsSL https://repo.pigsty.cc/key | sudo tee /etc/pki/rpm-gpg/RPM-GPG-KEY-pigsty >/dev/null
curl -fsSL https://repo.pigsty.cc/yum/repo | sudo tee /etc/yum.repos.d/pigsty.repo >/dev/null

APT repos (Debian/Ubuntu)

curl -fsSL https://repo.pigsty.io/key | sudo gpg --dearmor -o /etc/apt/keyrings/pigsty.gpg
sudo tee /etc/apt/sources.list.d/pigsty.list > /dev/null <<EOF
deb [signed-by=/etc/apt/keyrings/pigsty.gpg] https://repo.pigsty.io/apt/infra generic main
deb [signed-by=/etc/apt/keyrings/pigsty.gpg] https://repo.pigsty.io/apt/pgsql $(lsb_release -cs) main
EOF
sudo apt update

# China mainland mirror: replace repo.pigsty.io with repo.pigsty.cc

FAQ

Difference between extension name and package name

Name Description Example
Extension name Name used with CREATE EXTENSION vector
Package alias Standardized name in Pigsty config pgvector
Package name Actual OS package name pgvector_18* or postgresql-18-pgvector

Preloaded extension prevents startup

If extension in shared_preload_libraries doesn’t exist or fails to load, PostgreSQL won’t start. Solutions:

  1. Ensure extension package is correctly installed
  2. Or remove extension from shared_preload_libraries (edit /pg/data/postgresql.conf)

Extension dependencies

Some extensions depend on others, requiring sequential creation or using CASCADE:

CREATE EXTENSION postgis;                    -- Create base extension first
CREATE EXTENSION postgis_topology;           -- Then create dependent extension
-- Or
CREATE EXTENSION postgis_topology CASCADE;   -- Auto-create dependencies

Extension version incompatibility

View extension versions supported by current PostgreSQL:

SELECT * FROM pg_available_extension_versions WHERE name = 'vector';

Related Resources

8.4.10 - Upgrading PostgreSQL Major/Minor Versions

Version upgrade - minor version rolling upgrade, major version migration, extension upgrade

Quick Start

PostgreSQL version upgrades fall into two types: minor version upgrade and major version upgrade, with very different risk and complexity.

Type Example Downtime Data Compatibility Risk
Minor upgrade 17.2 → 17.3 Seconds (rolling) Fully compatible Low
Major upgrade 17 → 18 Minutes Requires data dir upgrade Medium
Minor
# Rolling upgrade: replicas first, then primary
ansible <cls> -b -a 'yum upgrade -y postgresql17*'
pg restart --role replica --force <cls>
pg switchover <cls>
pg restart <cls> <old-primary> --force
Major
# Recommended: Logical replication migration
bin/pgsql-add pg-new              # Create new version cluster
# Configure logical replication to sync data...
# Switch traffic to new cluster
Extension
ansible <cls> -b -a 'yum upgrade -y postgis36_17*'
psql -c 'ALTER EXTENSION postgis UPDATE;'

For detailed online migration process, see Online Migration documentation.

Action Description Risk
Minor Version Upgrade Update packages, rolling restart Low
Minor Version Downgrade Rollback to previous minor version Low
Major Version Upgrade Logical replication or pg_upgrade Medium
Extension Upgrade Upgrade extension packages and objects Low

Minor Version Upgrade

Minor version upgrades (e.g., 17.2 → 17.3) are the most common upgrade scenario, typically for security patches and bug fixes. Data directory is fully compatible, completed via rolling restart.

Strategy: Recommended rolling upgrade: upgrade replicas first, then switchover to upgrade original primary - minimizes service interruption.

1. Update repo → 2. Upgrade replica packages → 3. Restart replicas
4. Switchover → 5. Upgrade original primary packages → 6. Restart original primary

Step 1: Prepare packages

Ensure local repo has latest PostgreSQL packages and refresh node cache:

Repo
cd ~/pigsty
./infra.yml -t repo_upstream      # Add upstream repos (needs internet)
./infra.yml -t repo_build         # Rebuild local repo
EL
ansible <cls> -b -a 'yum clean all'
ansible <cls> -b -a 'yum makecache'
Debian
ansible <cls> -b -a 'apt clean'
ansible <cls> -b -a 'apt update'

Step 2: Upgrade replicas

Upgrade packages on all replicas and verify version:

EL
ansible <cls> -b -a 'yum upgrade -y postgresql17*'
ansible <cls> -b -a '/usr/pgsql/bin/pg_ctl --version'
Debian
ansible <cls> -b -a 'apt install -y postgresql-17'
ansible <cls> -b -a '/usr/lib/postgresql/17/bin/pg_ctl --version'

Restart all replicas to apply new version:

pg restart --role replica --force <cls>

Step 3: Switchover

Execute switchover to transfer primary role to upgraded replica:

pg switchover <cls>
# Or non-interactive:
pg switchover --leader <old-primary> --candidate <new-primary> --scheduled=now --force <cls>

Step 4: Upgrade original primary

Original primary is now replica - upgrade packages and restart:

EL
ansible <old-primary-ip> -b -a 'yum upgrade -y postgresql17*'
Debian
ansible <old-primary-ip> -b -a 'apt install -y postgresql-17'
pg restart <cls> <old-primary-name> --force

Step 5: Verify

Confirm all instances have consistent version:

pg list <cls>
pg query <cls> -c "SELECT version()"

Minor Version Downgrade

In rare cases (e.g., new version introduces bugs), may need to downgrade PostgreSQL to previous version.

Step 1: Get old version packages

EL
cd ~/pigsty; ./infra.yml -t repo_upstream     # Add upstream repos
cd /www/pigsty; repotrack postgresql17-*-17.1 # Download specific version packages
cd ~/pigsty; ./infra.yml -t repo_create       # Rebuild repo metadata
Refresh Cache
ansible <cls> -b -a 'yum clean all'
ansible <cls> -b -a 'yum makecache'

Step 2: Execute downgrade

EL
ansible <cls> -b -a 'yum downgrade -y postgresql17*'
Debian
ansible <cls> -b -a 'apt install -y postgresql-17=17.1*'

Step 3: Restart cluster

pg restart --force <cls>

Major Version Upgrade

Major version upgrades (e.g., 17 → 18) involve data format changes, requiring specialized tools for data migration.

Method Downtime Complexity Use Case
Logical Replication Migration Seconds (switch) High Production, minimal downtime required
pg_upgrade In-Place Upgrade Minutes~Hours Medium Test env, smaller data
Recommended Approach

For production, we recommend logical replication migration: create new version cluster, sync data via logical replication, then blue-green switch. Shortest downtime and rollback-ready. See Online Migration.

Logical Replication Migration

Logical replication is the recommended approach for production major version upgrades. Core steps:

1. Create new version target cluster → 2. Configure logical replication → 3. Verify data consistency
4. Switch app traffic to new cluster → 5. Decommission old cluster

Step 1: Create new version cluster

pg-meta-new:
  hosts:
    10.10.10.12: { pg_seq: 1, pg_role: primary }
  vars:
    pg_cluster: pg-meta-new
    pg_version: 18                    # New version
bin/pgsql-add pg-meta-new

Step 2: Configure logical replication

-- Source cluster (old version) primary: create publication
CREATE PUBLICATION upgrade_pub FOR ALL TABLES;

-- Target cluster (new version) primary: create subscription
CREATE SUBSCRIPTION upgrade_sub
  CONNECTION 'host=10.10.10.11 port=5432 dbname=mydb user=replicator password=xxx'
  PUBLICATION upgrade_pub;

Step 3: Wait for sync completion

-- Target cluster: check subscription status
SELECT * FROM pg_stat_subscription;

-- Source cluster: check replication slot LSN
SELECT slot_name, confirmed_flush_lsn FROM pg_replication_slots;

Step 4: Switch traffic

After confirming data sync complete: stop app writes to source → wait for final sync → switch app connections to new cluster → drop subscription, decommission source.

-- Target cluster: drop subscription
DROP SUBSCRIPTION upgrade_sub;

For detailed migration process, see Online Migration documentation.

pg_upgrade In-Place Upgrade

pg_upgrade is PostgreSQL’s official major version upgrade tool, suitable for test environments or scenarios accepting longer downtime.

Important Warning

In-place upgrade causes longer downtime and is difficult to rollback. For production, prefer logical replication migration.

Step 1: Install new version packages

./pgsql.yml -l <cls> -t pg_pkg -e pg_version=18

Step 2: Stop Patroni

pg pause <cls>                        # Pause auto-failover
systemctl stop patroni                # Stop Patroni (stops PostgreSQL)

Step 3: Run pg_upgrade

sudo su - postgres
mkdir -p /data/postgres/pg-meta-18/data

# Pre-check (-c parameter: check only, don't execute)
/usr/pgsql-18/bin/pg_upgrade \
  -b /usr/pgsql-17/bin -B /usr/pgsql-18/bin \
  -d /data/postgres/pg-meta-17/data \
  -D /data/postgres/pg-meta-18/data \
  -v -c

# Execute upgrade
/usr/pgsql-18/bin/pg_upgrade \
  -b /usr/pgsql-17/bin -B /usr/pgsql-18/bin \
  -d /data/postgres/pg-meta-17/data \
  -D /data/postgres/pg-meta-18/data \
  --link -j 8 -v

Step 4: Update links and start

rm -rf /usr/pgsql && ln -s /usr/pgsql-18 /usr/pgsql
rm -rf /pg && ln -s /data/postgres/pg-meta-18 /pg
# Edit /etc/patroni/patroni.yml to update paths
systemctl start patroni
pg resume <cls>

Step 5: Post-processing

/usr/pgsql-18/bin/vacuumdb --all --analyze-in-stages
./delete_old_cluster.sh   # Cleanup script generated by pg_upgrade

Extension Upgrade

When upgrading PostgreSQL version, typically also need to upgrade related extensions.

Upgrade extension packages

EL
ansible <cls> -b -a 'yum upgrade -y postgis36_17 timescaledb-2-postgresql-17* pgvector_17*'
Debian
ansible <cls> -b -a 'apt install -y postgresql-17-postgis-3 postgresql-17-pgvector'

Upgrade extension objects

After package upgrade, execute extension upgrade in database:

-- View upgradeable extensions
SELECT name, installed_version, default_version FROM pg_available_extensions
WHERE installed_version IS NOT NULL AND installed_version <> default_version;

-- Upgrade extensions
ALTER EXTENSION postgis UPDATE;
ALTER EXTENSION timescaledb UPDATE;
ALTER EXTENSION vector UPDATE;

-- Check extension versions
SELECT extname, extversion FROM pg_extension;
Extension Compatibility

Before major version upgrade, confirm all extensions support target PostgreSQL version. Some extensions may require uninstall/reinstall - check extension documentation.

Important Notes

  1. Backup first: Always perform complete backup before any upgrade
  2. Test verify: Verify upgrade process in test environment first
  3. Extension compatibility: Confirm all extensions support target version
  4. Rollback plan: Prepare rollback plan, especially for major upgrades
  5. Monitor closely: Monitor database performance and error logs after upgrade
  6. Document: Record all operations and issues during upgrade

Related Documentation

8.5 - Backup & Restore

Configure repositories and policies, manage pgBackRest backups, and perform point-in-time recovery safely.

Pigsty uses pgBackRest for PostgreSQL backups. It supports full, differential, and incremental backups, parallel processing, encryption, and Silo/S3 object storage. Every PGSQL cluster is configured for backup and WAL archiving by default.

This chapter is the operational manual for backup configuration, management, recovery, and drills. For design concepts and tradeoffs, see Point-in-Time Recovery.

All backup and recovery operations ultimately invoke pgBackRest. Pigsty provides several wrapper layers:

Layer Interface Form Scope
Cluster orchestration pg_pitr + pgsql-pitr.yml Ansible playbook HA, etcd, and multi-node recovery
Instance orchestration pig pitr CLI Local-node recovery without the admin node
Command primitives pig pb, pb, and pg-backup pgBackRest wrappers Backup, inspection, expiry, and unmanaged restore
Engine pgbackrest Native CLI Underlying backup, archive, and restore engine
Section Content
Mechanism Stanzas, repositories, retention, timelines, and Pigsty wrapper mapping
Policy Scheduling, recovery windows, and storage planning
Repository Local, Silo, and external S3 repositories; encryption, versioning, and locking
Administration Backup commands, inspection, expiration, and stanza management
Restore Recovery targets, staged PITR, and complete parameter reference
Clone Restore production history into another cluster and perform drills
Tutorial A sandbox restore using pgBackRest primitives
Disclaimer

Pigsty makes every effort to provide a reliable PITR solution, but accepts no liability for data loss caused by restore operations. If you need assistance, consider professional services.

Recovery overwrites target data

Before PITR, inspect pig pg list <target-cluster> and pig pb info, verify a recent usable backup and recovery window, have the operator state the exact target cluster and recovery point, then run the target-scoped ./pgsql-pitr.yml -l <target-cluster> ... command. pgsql-pitr.yml prints a plan but does not pause for approval. Production recovery also requires a maintenance window and an independently verified backup.

Quick Start

  1. Design a backup policy: declare scheduled backups in pg_crontab and select a repository with pgbackrest_repo.
  2. Manage backups: run pg-backup and inspect recovery coverage with pb info.
  3. Perform recovery: declare pg_pitr, then run pgsql-pitr.yml.
pg_crontab: [ '00 01 * * * /pg/bin/pg-backup full' ]
./pgsql-pitr.yml -l pg-meta -e '{"pg_pitr": { "time": "2025-07-13 10:00:00+00", "action": "promote" }}'

8.5.1 - Backup Policy

Design backup policies according to your needs

The chart below combines the “Recovery Window” and “Backup Storage Usage” on a single timeline (0~108h) so they can be inspected together.

Under the same assumptions (database size 100GB, daily writes 10GB), it shows how both metrics evolve over 30 days with “weekly full + daily incremental” backups and 14-day full-backup retention.

  • When: Backup schedule
  • Where: Backup repository
  • How: Backup method

When to Backup

The first question is when to backup your database - this is a tradeoff between backup frequency and recovery time. Since you need to replay WAL logs from the last backup to the recovery target point, the more frequent the backups, the less WAL logs need to be replayed, and the faster the recovery.

Daily Full Backup

For production databases, it’s recommended to start with the simplest daily full backup strategy. This is also Pigsty’s default backup strategy, implemented via crontab.

pg_crontab: [ '00 01 * * * /pg/bin/pg-backup full' ]
pgbackrest_method: local          # Choose backup repository method: `local`, `minio`, or other custom repository
pgbackrest_repo:                  # pgbackrest repository configuration: https://pgbackrest.org/configuration.html#section-repository
  local:                          # Default pgbackrest repository using local POSIX filesystem
    path: /pg/backup              # Local backup directory, defaults to `/pg/backup`
    retention_full_type: count    # Retain full backups by count
    retention_full: 2             # Keep 2, up to 3 full backups when using local filesystem repository

Assume your database size is 100GB, daily writes are 10GB, and each full backup takes 1 hour. Under this daily-full local-repo strategy, recovery window and backup storage evolve as shown below:

The recovery window cycles between 25-49 hours, and storage usage is roughly 2 full backups plus around 2 days of WAL archives. In practice, prepare at least 3~5 times the base database size as backup disk capacity for the default policy.

tooltip: { trigger: axis, formatter: $fn:tipMerged, axisPointer: { type: line, snap: true, label: { show: false } } }
axisPointer: { link: [ { xAxisIndex: [0, 1] } ] }
legend: { show: false, bottom: 10, itemGap: 18, data: ["Primary Backup", "Secondary Backup", "WAL Archive", "Transient Backup"] }
grid:
  - { left: 82, right: "10%", top: 42, height: 218, containLabel: false }
  - { left: 82, right: "10%", top: 286, height: 218, containLabel: false }
xAxis:
  - type: category
    gridIndex: 0
    position: bottom
    boundaryGap: false
    data: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108]
    name: Time h
    nameLocation: end
    nameGap: 10
    nameTextStyle: { align: left, verticalAlign: top, padding: [8, 0, 0, 0] }
    axisLabel: { interval: 11, formatter: $fn:fmtHour }
    axisLine: { show: true, symbol: [none, arrow], symbolSize: [10, 14], lineStyle: { width: 1.6, color: "#4b5563" } }
    axisTick: { show: true, length: 6 }
    splitLine: { show: true, lineStyle: { type: dashed, width: 1, opacity: 0.28, color: "#9ca3af" } }
    minorTick: { show: true, splitNumber: 12, length: 3 }
    minorSplitLine: { show: true, lineStyle: { type: dotted, width: 1, opacity: 0.14, color: "#9ca3af" } }
  - type: category
    gridIndex: 1
    position: top
    boundaryGap: true
    data: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108]
    axisLabel: { show: false }
    axisLine: { show: true, lineStyle: { width: 1.6, color: "#4b5563" } }
    axisTick: { show: true, length: 6 }
    splitLine: { show: true, lineStyle: { type: dashed, width: 1, opacity: 0.22, color: "#9ca3af" } }
yAxis:
  - type: value
    gridIndex: 0
    min: 0
    max: 52
    interval: 5
    name: Recovery Window h
    nameLocation: end
    nameRotate: 0
    nameGap: 8
    nameTextStyle: { align: left, verticalAlign: bottom, padding: [0, 0, 8, 4] }
    axisLabel: { formatter: $fn:fmtWin }
    axisLine: { show: true, symbol: [none, arrow], symbolSize: [10, 14], lineStyle: { width: 1.6, color: "#4b5563" } }
    axisTick: { show: true, length: 6 }
    splitLine: { show: true, lineStyle: { type: dashed, width: 1, opacity: 0.35, color: "#9ca3af" } }
    minorTick: { show: true, splitNumber: 5, length: 3 }
    minorSplitLine: { show: true, lineStyle: { type: dotted, width: 1, opacity: 0.18, color: "#9ca3af" } }
  - type: value
    gridIndex: 1
    min: 0
    max: 350
    interval: 50
    inverse: true
    name: Backup Storage GB
    nameLocation: end
    nameRotate: 0
    nameGap: 8
    nameTextStyle: { align: left, verticalAlign: top, padding: [10, 0, 0, 4] }
    axisLabel: { formatter: $fn:fmtGbTick }
    axisLine: { show: true, symbol: [arrow, none], symbolSize: [10, 14], lineStyle: { width: 1.6, color: "#4b5563" } }
    axisTick: { show: true, length: 6 }
    splitLine: { show: true, lineStyle: { type: dashed, width: 1, opacity: 0.32, color: "#9ca3af" } }
series: [ { name: Recovery Window, type: line, smooth: false, symbol: none, showSymbol: false, xAxisIndex: 0, yAxisIndex: 0, lineStyle: { width: 3, color: "#f2a000" }, itemStyle: { color: "#f2a000" }, data: [[0,0],[1,0],[1,1],[2,2],[3,3],[4,4],[5,5],[6,6],[7,7],[8,8],[9,9],[10,10],[11,11],[12,12],[13,13],[14,14],[15,15],[16,16],[17,17],[18,18],[19,19],[20,20],[21,21],[22,22],[23,23],[24,24],[25,25],[26,26],[27,27],[28,28],[29,29],[30,30],[31,31],[32,32],[33,33],[34,34],[35,35],[36,36],[37,37],[38,38],[39,39],[40,40],[41,41],[42,42],[43,43],[44,44],[45,45],[46,46],[47,47],[48,48],[49,49],[49,25],[50,26],[51,27],[52,28],[53,29],[54,30],[55,31],[56,32],[57,33],[58,34],[59,35],[60,36],[61,37],[62,38],[63,39],[64,40],[65,41],[66,42],[67,43],[68,44],[69,45],[70,46],[71,47],[72,48],[73,49],[73,25],[74,26],[75,27],[76,28],[77,29],[78,30],[79,31],[80,32],[81,33],[82,34],[83,35],[84,36],[85,37],[86,38],[87,39],[88,40],[89,41],[90,42],[91,43],[92,44],[93,45],[94,46],[95,47],[96,48],[97,49],[97,25],[98,26],[99,27],[100,28],[101,29],[102,30],[103,31],[104,32],[105,33],[106,34],[107,35],[108,36]], markLine: { symbol: none, label: { show: false }, data: [ { xAxis: 0, lineStyle: { color: "#59a14f", type: "solid", width: 1.4, opacity: 0.75 } }, { xAxis: 24, lineStyle: { color: "#59a14f", type: "solid", width: 1.4, opacity: 0.75 } }, { xAxis: 48, lineStyle: { color: "#59a14f", type: "solid", width: 1.4, opacity: 0.75 } }, { xAxis: 72, lineStyle: { color: "#59a14f", type: "solid", width: 1.4, opacity: 0.75 } }, { xAxis: 96, lineStyle: { color: "#59a14f", type: "solid", width: 1.4, opacity: 0.75 } }, { xAxis: 1, lineStyle: { color: "#336791", type: "solid", width: 1.4, opacity: 0.8 } }, { xAxis: 25, lineStyle: { color: "#336791", type: "solid", width: 1.4, opacity: 0.8 } }, { xAxis: 49, lineStyle: { color: "#336791", type: "solid", width: 1.4, opacity: 0.8 } }, { xAxis: 73, lineStyle: { color: "#336791", type: "solid", width: 1.4, opacity: 0.8 } }, { xAxis: 97, lineStyle: { color: "#336791", type: "solid", width: 1.4, opacity: 0.8 } }, { yAxis: 25, label: { show: true, formatter: "lower 25h", position: "end", distance: 12, color: "#2563eb" }, lineStyle: { color: "#2563eb", type: "dashdot", width: 1.4, opacity: 0.75 } }, { yAxis: 49, label: { show: true, formatter: "upper 49h", position: "end", distance: 12, color: "#7c3aed" }, lineStyle: { color: "#7c3aed", type: "dashdot", width: 1.4, opacity: 0.75 } } ] } }, { name: Primary Backup, type: bar, stack: used, xAxisIndex: 1, yAxisIndex: 1, barWidth: 5, itemStyle: { color: "#59a14f" }, data: [0,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100] }, { name: Secondary Backup, type: bar, stack: used, xAxisIndex: 1, yAxisIndex: 1, barWidth: 5, itemStyle: { color: "#4e79a7" }, data: [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100] }, { name: WAL Archive, type: bar, stack: used, xAxisIndex: 1, yAxisIndex: 1, barWidth: 5, itemStyle: { color: "#edc949" }, data: [0,0,0.42,0.83,1.25,1.67,2.08,2.5,2.92,3.33,3.75,4.17,4.58,5,5.42,5.83,6.25,6.67,7.08,7.5,7.92,8.33,8.75,9.17,9.58,10,10.42,10.83,11.25,11.67,12.08,12.5,12.92,13.33,13.75,14.17,14.58,15,15.42,15.83,16.25,16.67,17.08,17.5,17.92,18.33,18.75,19.17,19.58,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20] }, { name: Transient Backup, type: bar, stack: used, xAxisIndex: 1, yAxisIndex: 1, barWidth: 5, itemStyle: { color: "#9ca3af", opacity: 0.75 }, data: [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,100,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,100,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,100,0,0,0,0,0,0,0,0,0,0,0,0] } ]

Full + Incremental Backup

You can optimize backup space usage by adjusting these parameters.

If using Silo / S3 as a centralized backup repository, storage is no longer limited by the local disk. In this case, consider using full + incremental backup with a 2-week retention policy:

pg_crontab:  # Full backup at 1 AM on Monday, incremental backups on weekdays
  - '00 01 * * 1           /pg/bin/pg-backup full'
  - '00 01 * * 2,3,4,5,6,7 /pg/bin/pg-backup'
pgbackrest_method: minio
pgbackrest_repo:                  # pgbackrest repository configuration: https://pgbackrest.org/configuration.html#section-repository
  minio:                          # Optional S3-compatible repository preset
    type: s3                      # Silo uses the S3-compatible repository type
    s3_endpoint: sss.pigsty       # object-storage endpoint, `sss.pigsty` by default
    s3_region: us-east-1          # compatibility region, `us-east-1` by default
    s3_bucket: pgsql              # backup bucket, `pgsql` by default
    s3_key: pgbackrest            # pgBackRest access key
    s3_key_secret: S3User.Backup  # object-storage user secret
    s3_uri_style: path            # use path-style rather than host-style URIs
    path: /pgbackrest             # backup path, `/pgbackrest` by default
    storage_port: 9000            # Silo service port, 9000 by default
    storage_ca_file: /etc/pki/ca.crt  # Pigsty CA path, `/etc/pki/ca.crt` by default
    block: y                      # Enable block-level incremental backup
    bundle: y                     # Bundle small files into a single file
    bundle_limit: 20MiB           # Bundle size limit, recommended 20MiB for object storage
    bundle_size: 128MiB           # Bundle target size, recommended 128MiB for object storage
    cipher_type: aes-256-cbc      # Enable AES encryption for remote backup repository
    cipher_pass: pgBackRest       # AES encryption password, defaults to 'pgBackRest'
    retention_full_type: time     # Retain full backups by time
    retention_full: 14            # Keep full backups from the last 14 days

With weekly full backups and time-based retention of 14 days, the steady-state recovery window is roughly 14–21 days. The exact window still depends on successful backup and WAL archival runs.

Assuming your database size is 100GB and writes 10GB of data per day, the backup size is as follows:

tooltip: { trigger: axis, formatter: $fn:tipMerged30, axisPointer: { type: line, snap: true, label: { show: false } } }
axisPointer: { link: [ { xAxisIndex: [0, 1] } ] }
legend: { show: false, bottom: 10, itemGap: 18, data: ["Primary Backup", "Secondary Backup", "Incremental Backup", "WAL Archive", "Transient Backup"] }
grid:
  - { left: 82, right: "10%", top: 42, height: 218, containLabel: false }
  - { left: 82, right: "10%", top: 302, height: 218, containLabel: false }
xAxis:
  - type: value
    gridIndex: 0
    position: bottom
    boundaryGap: false
    min: 0
    max: 31
    interval: 1
    name: Time
    nameLocation: end
    nameGap: 10
    nameTextStyle: { align: left, verticalAlign: top, padding: [8, 0, 0, 0] }
    axisLabel: { formatter: $fn:fmtDay30 }
    axisLine: { show: true, symbol: [none, arrow], symbolSize: [10, 14], lineStyle: { width: 1.6, color: "#4b5563" } }
    axisTick: { show: true, length: 6 }
    splitLine: { show: true, lineStyle: { type: dashed, width: 1, opacity: 0.28, color: "#9ca3af" } }
    minorTick: { show: true, splitNumber: 4, length: 3 }
    minorSplitLine: { show: true, lineStyle: { type: dotted, width: 1, opacity: 0.14, color: "#9ca3af" } }
  - type: category
    gridIndex: 1
    position: top
    boundaryGap: true
    z: 10
    data: [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30]
    axisLabel: { show: false }
    axisLine: { show: true, lineStyle: { width: 1.6, color: "#4b5563" } }
    axisTick: { show: true, alignWithLabel: true, length: 6 }
    splitLine: { show: true, lineStyle: { type: dashed, width: 1, opacity: 0.22, color: "#9ca3af" } }
yAxis:
  - type: value
    gridIndex: 0
    min: 0
    max: 360
    interval: 48
    name: Recovery Window h
    nameLocation: end
    nameRotate: 0
    nameGap: 8
    nameTextStyle: { align: left, verticalAlign: bottom, padding: [0, 0, 8, 4] }
    axisLabel: { formatter: $fn:fmtWin30 }
    axisLine: { show: true, symbol: [none, arrow], symbolSize: [10, 14], lineStyle: { width: 1.6, color: "#4b5563" } }
    axisTick: { show: true, length: 6 }
    splitLine: { show: true, lineStyle: { type: dashed, width: 1, opacity: 0.35, color: "#9ca3af" } }
    minorTick: { show: true, splitNumber: 4, length: 3 }
    minorSplitLine: { show: true, lineStyle: { type: dotted, width: 1, opacity: 0.18, color: "#9ca3af" } }
  - type: value
    gridIndex: 1
    min: 0
    max: 600
    interval: 50
    inverse: true
    z: 10
    name: Storage GB
    nameLocation: end
    nameRotate: 0
    nameGap: 8
    nameTextStyle: { align: left, verticalAlign: top, padding: [10, 0, 0, 4] }
    axisLabel: { formatter: $fn:fmtGbTick30 }
    axisLine: { show: true, symbol: [arrow, none], symbolSize: [10, 14], lineStyle: { width: 1.6, color: "#4b5563" } }
    axisTick: { show: true, length: 6 }
    splitLine: { show: true, lineStyle: { type: dashed, width: 1, opacity: 0.32, color: "#9ca3af" } }
series:
  - { name: Recovery Window, type: line, smooth: false, symbol: none, showSymbol: false, xAxisIndex: 0, yAxisIndex: 0, lineStyle: { width: 3, color: "#f28e2c" }, itemStyle: { color: "#f28e2c" }, data: [[1,24],[2,48],[3,72],[4,96],[5,120],[6,144],[7,168],[8,192],[9,216],[10,240],[11,264],[12,288],[13,312],[14,336],[14,168],[15,192],[16,216],[17,240],[18,264],[19,288],[20,312],[21,336],[21,168],[22,192],[23,216],[24,240],[25,264],[26,288],[27,312],[28,336],[28,168],[29,192],[30,216]], markLine: { symbol: none, label: { show: false }, data: [ { xAxis: 7, lineStyle: { color: "#336791", type: "solid", width: 1.4, opacity: 0.65 } }, { xAxis: 14, lineStyle: { color: "#336791", type: "solid", width: 1.4, opacity: 0.65 } }, { xAxis: 21, lineStyle: { color: "#336791", type: "solid", width: 1.4, opacity: 0.65 } }, { xAxis: 28, lineStyle: { color: "#336791", type: "solid", width: 1.4, opacity: 0.65 } }, { yAxis: 168, label: { show: true, formatter: "lower 7d", position: "end", distance: 12, color: "#2563eb" }, lineStyle: { color: "#2563eb", type: "dashdot", width: 1.4, opacity: 0.72 } }, { yAxis: 336, label: { show: true, formatter: "upper 14d", position: "end", distance: 12, color: "#7c3aed" }, lineStyle: { color: "#7c3aed", type: "dashdot", width: 1.4, opacity: 0.72 } } ] } }
  - { name: Primary Backup, type: bar, stack: used, xAxisIndex: 1, yAxisIndex: 1, barWidth: 16, itemStyle: { color: "#59a14f" }, data: [100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100] }
  - { name: Secondary Backup, type: bar, stack: used, xAxisIndex: 1, yAxisIndex: 1, barWidth: 16, itemStyle: { color: "#4e79a7" }, data: [0,0,0,0,0,0,0,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100,100] }
  - { name: Incremental Backup, type: bar, stack: used, xAxisIndex: 1, yAxisIndex: 1, barWidth: 16, itemStyle: { color: "#76b7b2" }, data: [0,10,20,30,40,50,60,70,80,90,100,110,120,130,70,80,90,100,110,120,130,70,80,90,100,110,120,130,70,80] }
  - { name: WAL Archive, type: bar, stack: used, xAxisIndex: 1, yAxisIndex: 1, barWidth: 16, itemStyle: { color: "#edc949" }, data: [10,20,30,40,50,60,70,80,90,100,110,120,130,140,80,90,100,110,120,130,140,80,90,100,110,120,130,140,80,90] }
  - { name: Transient Backup, type: bar, stack: used, xAxisIndex: 1, yAxisIndex: 1, barWidth: 16, itemStyle: { color: "#9ca3af", opacity: 0.75 }, data: [0,0,0,0,0,0,0,0,0,0,0,0,0,110,0,0,0,0,0,0,110,0,0,0,0,0,0,110,0,0] }

Backup Location

By default, Pigsty provides two default backup repository definitions: local and minio backup repositories.

  • local: Default option, uses local /pg/backup directory (symlink to pg_fs_backup: /data/backups)
  • minio: Uses SNSD single-node MinIO cluster (supported by Pigsty, but not enabled by default)
pgbackrest_method: local          # Choose backup repository method: `local`, `minio`, or other custom repository
pgbackrest_repo:                  # pgbackrest repository configuration: https://pgbackrest.org/configuration.html#section-repository
  local:                          # Default pgbackrest repository using local POSIX filesystem
    path: /pg/backup              # Local backup directory, defaults to `/pg/backup`
    retention_full_type: count    # Retain full backups by count
    retention_full: 2             # Keep 2, up to 3 full backups when using local filesystem repository
  minio:                          # Optional minio repository
    type: s3                      # minio is S3 compatible
    s3_endpoint: sss.pigsty       # minio endpoint domain, defaults to `sss.pigsty`
    s3_region: us-east-1          # minio region, defaults to us-east-1, meaningless for minio
    s3_bucket: pgsql              # minio bucket name, defaults to `pgsql`
    s3_key: pgbackrest            # minio user access key for pgbackrest
    s3_key_secret: S3User.Backup  # minio user secret for pgbackrest
    s3_uri_style: path            # minio uses path-style URIs instead of host-style
    path: /pgbackrest             # minio backup path, defaults to `/pgbackrest`
    storage_port: 9000            # minio port, defaults to 9000
    storage_ca_file: /etc/pki/ca.crt  # minio CA certificate path, defaults to `/etc/pki/ca.crt`
    block: y                      # Enable block-level incremental backup
    bundle: y                     # Bundle small files into a single file
    bundle_limit: 20MiB           # Bundle size limit, recommended 20MiB for object storage
    bundle_size: 128MiB           # Bundle target size, recommended 128MiB for object storage
    cipher_type: aes-256-cbc      # Enable AES encryption for remote backup repository
    cipher_pass: pgBackRest       # AES encryption password, defaults to 'pgBackRest'
    retention_full_type: time     # Retain full backups by time
    retention_full: 14            # Keep full backups from the last 14 days

8.5.2 - Restore Operations

Perform PITR with pgsql-pitr.yml, pig pitr, or pig pb restore; select targets, run stages, and verify the result.

Pigsty provides three restore entry points. They share the same parameter semantics, but serve different scopes:

Entry point Use case What it controls
pgsql-pitr.yml Production cluster recovery HA pause, multiple nodes, etcd cleanup, restore, and restart
pig pitr A local database node Single-instance orchestration without the admin node
pig pb restore An instance not managed by Patroni A direct pgBackRest restore wrapper

For a hands-on sandbox drill, see Manual Recovery. To recover into another cluster without changing production, see Clone a PG Cluster.

PITR overwrites the target cluster

pgsql-pitr.yml pauses HA, stops Patroni/PostgreSQL, overwrites the target data directory with pgbackrest --force restore, then deletes the target cluster’s etcd prefix and rebuilds HA. It prints a plan but does not wait for confirmation. Before any real restore, inspect the topology with pig pg list <target-cluster>, verify a recent usable backup and recovery window with pig pb info, and have the operator state and confirm the exact target cluster and recovery point. Schedule a maintenance window and retain an independently verified backup for production recovery.

Quick Start

To roll pg-meta back to an earlier time, declare pg_pitr:

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_pitr: { time: '2025-07-13 10:00:00+00', action: promote }

Run the same target through the safety gate before executing it:

pig pg list pg-meta
pig pb info
./pgsql-pitr.yml -l pg-meta

You can pass the same object temporarily on the command line:

./pgsql-pitr.yml -l pg-meta -e '{"pg_pitr": { "time": "2025-07-13 10:00:00+00", "action": "promote" }}'
Use valid JSON for command-line variables

The -e value must be valid JSON: quote keys and string values, for example {"pg_pitr": {"time": "...", "archive": true}}. Booleans are not quoted. Invalid quoting can fail parsing or silently produce the wrong value.

The playbook pauses Patroni HA, stops the cluster, performs a delta pgBackRest restore, starts PostgreSQL and waits for a consistent recovery state, prints control data, removes old etcd metadata, and starts the cluster under Patroni again. It prints the source, target, and restore command first, but has no interactive approval gate. A one-shot targeted recovery should therefore declare action: promote explicitly. To inspect data at the target, use step-by-step execution with action: pause.

Recovery Targets

pg_pitr supports six recovery target forms. The four target values are mutually exclusive.

Recovery target types

default/latest
pg_pitr: { }  # Replay to the end of the WAL archive stream
time
pg_pitr: { time: "2025-07-13 10:00:00+00" }
lsn
pg_pitr: { lsn: "0/4001C80" }
xid
pg_pitr: { xid: "250000" }
name
pg_pitr: { name: "some_restore_point" }
immediate
pg_pitr: { type: "immediate" }

With no target, recovery replays all archived WAL to the latest available state (Pigsty’s internal type is default). immediate stops at the first consistent point, which is useful for obtaining a usable instance as quickly as possible or testing a backup.

Recover by Time

Use a valid PostgreSQL TIMESTAMP; an explicit time zone is strongly recommended:

./pgsql-pitr.yml -l pg-meta -e '{"pg_pitr": { "time": "2025-07-13 10:00:00+00", "action": "promote" }}'

Recover by Name

Create an unambiguous marker before a risky change with pg_create_restore_point:

SELECT pg_create_restore_point('before_migration');
./pgsql-pitr.yml -l pg-meta -e '{"pg_pitr": { "name": "before_migration", "action": "promote" }}'

Recover by Transaction ID

If the offending transaction ID is known from monitoring or CSVLOG’s TXID field, use exclusive to stop before that transaction:

./pgsql-pitr.yml -l pg-meta -e '{"pg_pitr": { "xid": "250000", "exclusive": true, "action": "promote" }}'

Recover by LSN

An LSN identifies a position in the WAL stream. It is also visible in Pigsty’s PG LSN dashboard panel. Set timeline when the desired position is on a particular timeline; the default is latest.

./pgsql-pitr.yml -l pg-meta -e '{"pg_pitr": { "lsn": "0/4001C80", "timeline": "1", "action": "promote" }}'
Inclusive and exclusive targets

Targets are inclusive by default, so the target transaction is replayed. exclusive: true excludes the exact target. It applies only to time, xid, and lsn, and maps to PostgreSQL’s recovery_target_inclusive.

Recovery Source

Recovery uses the target cluster’s own backup by default. Three fields can select another source:

  • cluster: the source stanza, including another cluster in a shared repository
  • repo: a temporary repository definition in the same format as a pgbackrest_repo entry
  • set: a specific backup label; otherwise pgBackRest selects a suitable set

For example, recover pg-meta2 from pg-meta:

./pgsql-pitr.yml -l pg-meta2 -e '{"pg_pitr": { "cluster": "pg-meta", "archive": false, "action": "promote" }}'
./pgsql-pitr.yml -l pg-meta2 -e '{"pg_pitr": { "cluster": "pg-meta", "time": "2025-07-14 08:00:00+00", "archive": false, "action": "promote" }}'

Step-by-Step Execution

In an incident, use tags to retain an explicit human gate between stages. After confirming the backup, recovery point, and exact target, run the stages in order:

./pgsql-pitr.yml -l pg-meta -t down  # Pause HA; stop Patroni and PostgreSQL
./pgsql-pitr.yml -l pg-meta -t pitr  # Restore, replay WAL, and print control information
./pgsql-pitr.yml -l pg-meta -t up    # Remove etcd metadata; start the cluster and resume HA
# down                 : # stop HA and PostgreSQL
#   - pause            : # pause Patroni automatic failover
#   - stop             : # stop Patroni and PostgreSQL
# pitr                 : # perform PITR
#   - config           : # render pgBackRest config and restore script
#   - backup           : # optionally move PGDATA to /pg/data-backup
#   - restore          : # run pgBackRest restore
#   - recovery         : # start PostgreSQL and replay WAL
#   - verify           : # print recovered control data
# up                   : # rebuild HA
#   - etcd             : # remove old cluster metadata
#   - start            : # start Patroni/PostgreSQL
#   - resume           : # resume Patroni automatic failover

After down, confirm the processes are stopped. After pitr, inspect /pg/tmp/recovery.log and query the recovery state before checking narrowly authorized business data. pg_controldata /pg/data reports checkpoint and timeline metadata; it does not by itself prove that a time, XID, or LSN target was reached.

SELECT pg_is_in_recovery(), pg_is_wal_replay_paused(),
       pg_last_wal_replay_lsn(), pg_last_xact_replay_timestamp();

With action: pause, promote only after validation, then run up. If the target is wrong, adjust pg_pitr and repeat pitr before up. pause or shutdown creates a meaningful human gate only in this staged workflow; use action: promote explicitly for one-shot targeted recovery.

pg_ctl -D /pg/data promote            # only after validating an action: pause recovery
./pgsql-pitr.yml -l pg-meta -t up     # rebuild Patroni HA
Repeating the pitr stage

With backup: true, the playbook moves the current data directory to /pg/data-backup, but deletes any existing /pg/data-backup before doing so. The staged workflow is supported; a restore using backup: true is not generally idempotent.

PITR Parameter Definition

Declare the target, action, and treatment of existing data explicitly:

pg_pitr:
  cluster: pg-meta                 # source cluster/stanza; defaults to pg_cluster
  type: default                    # default | time | xid | name | lsn | immediate
  time: "2025-07-13 10:00:00+00"  # mutually exclusive with xid, name, and lsn
  name: "some_restore_point"       # mutually exclusive with time, xid, and lsn
  xid: "250000"                    # mutually exclusive with time, name, and lsn
  lsn: "0/4001C80"                 # mutually exclusive with time, xid, and name
  exclusive: false                 # exclude the exact target; time/xid/lsn only
  timeline: latest                 # target timeline; integer or latest
  set: latest                      # backup label; auto-select by default
  action: pause                    # pause | promote | shutdown
                                   # a specified target defaults to pause
  archive: true                    # preserve archiving; false sets archive-mode=off
  backup: false                    # move old PGDATA to /pg/data-backup before restore
  db_exclude: []                   # databases to exclude
  db_include: []                   # databases to include
  link_map:                        # tablespace/WAL link remapping
    pg_wal: '/data/wal'
    pg_xact: '/data/pg_xact'
  process: 4                       # restore workers; defaults to node_cpu
  repo: {}                        # temporary source repository definition
  data: /pg/data                  # target data directory
  port: 5432                      # recovery instance port

See Parameter Mapping for the corresponding pgBackRest options.

Single Instance: pig pitr

pig pitr performs a local-node workflow without Ansible: validate the target, stanza, and backup; stop Patroni/PostgreSQL; restore; optionally start PostgreSQL; and print follow-up guidance.

pig pitr -t "2025-07-13 10:00:00+00"    # Recover to a point in time
pig pitr --xid 250000 -X                # Recover to before transaction 250000
pig pitr --name before_migration        # Recover to a named restore point
pig pitr -d                             # Recover to the end of the WAL archive stream
pig pitr -I --no-restart                # Restore for immediate recovery and leave PostgreSQL stopped
pig pitr -t "..." --plan                # Print the execution plan without making changes

Use -b/--set for a backup set, -T/--target-timeline for a timeline, --target-action for the post-target action, and -D/--data with --no-restart for a side restore. The command normally attempts a fast stop and aborts if that fails; only explicit --force-stop permits immediate shutdown and a kill fallback. For managed PGDATA it leaves Patroni stopped. Validate the instance before pig pt start. It does not remove etcd metadata, rebuild replicas, or rejoin the instance to an HA cluster.

Primitive: pig pb restore

For an instance not managed by Patroni (or one deliberately taken out of management), pig pb restore directly wraps pgbackrest restore. It validates the environment, requires PostgreSQL to be stopped, displays the plan, and asks for confirmation.

pig pb restore --time "2025-07-13 10:00"
pig pb restore --set 20250715-013657F
pig pb restore -d

It rejects a live Patroni-managed target because Patroni could restart a half-restored instance, and it rejects any running PostgreSQL target. Arguments after --, such as --tablespace-map or --link-all, pass through to pgBackRest, but wrapped options such as target, stanza, and repository cannot be overridden there.

Post-Recovery

After restore:

  1. Verify the recovery state and the smallest authorized set of application checks.

  2. After a cross-cluster clone, complete stanza cleanup. Create a full backup on the new timeline as soon as practical:

    pg-backup full

  3. If exploratory recovery used archive: false, restore archiving. Because archive_mode is a postmaster setting, first confirm the maintenance window, current primary, and replication state, then obtain explicit approval for the restart:

    psql -c 'ALTER SYSTEM RESET archive_mode;'
pg restart pg-meta
pg-backup full

8.5.3 - Clone a PG Cluster

Restore one cluster’s historical state into another for data recovery, restore drills, and forensic inspection.

Cloning is one of the safest and most useful applications of recovery: leave production untouched and restore its historical state into another cluster. You can recover accidentally deleted data from the clone, validate backups in a drill, inspect a historical state, or reset a test environment to a production snapshot.

The target must be able to access the source backup repository, may be overwritten, and must use a compatible PostgreSQL major version. With a shared Silo/S3 repository, each cluster’s backups are isolated by a stanza and visible to targets holding the required credentials.

A clone overwrites the target cluster

Inspect the target topology with pig pg list <target-cluster>, verify the source stanza’s recent backups and recovery window with pig pb info, and have the operator confirm the exact source cluster, target cluster, and recovery point before performing the restore. Existing data on the target is overwritten; production work still requires a maintenance window and an independently verified backup.

Clone an Existing Cluster

Assume the four-node sandbox contains pg-meta and pg-test, sharing a Silo repository. To reset pg-test to the latest state of pg-meta, point pg_pitr to the pg-meta stanza:

pig pg list pg-test
pig pb info
./pgsql-pitr.yml -l pg-test -e '{"pg_pitr": { "cluster": "pg-meta", "archive": false, "action": "promote" }}'

Add a recovery target to clone any state inside the recovery window. For example, reset to 15:30 on December 26, 2025:

./pgsql-pitr.yml -l pg-test -e '{"pg_pitr": { "cluster": "pg-meta", "time": "2025-12-26 15:30:00+08", "archive": false, "action": "promote" }}'

These cross-cluster examples set archive: false to keep the exploratory recovery from archiving under the target stanza. After Patroni takes control, complete the stanza and archive cleanup below.

The target may also be a newly initialized empty cluster, such as pg-meta2. Create it through the normal cluster creation workflow, then perform cross-cluster PITR.

pgBackRest restore uses delta mode and rewrites only files that differ from the backup. Repeated drills, or a target already synchronized through a standby cluster, can therefore restore much faster than a first full restore.

For accidental deletion, validate the clone and use pg_dump to export only the affected tables or database back into production. An in-place rollback of the entire production cluster should be the last resort, not the first response.

Post-Clone Cleanup

The clone contains the source cluster’s data, while the target stanza may still record the target’s old PostgreSQL system identifier. pgBackRest refuses a backup when the identifiers do not match, preventing the new cluster from contaminating the source history.

After validating the clone, complete these steps. Restarting the cluster is a service change: first inspect the primary and replication state, schedule the maintenance window, and obtain explicit approval.

pb stanza-upgrade                           # accept the new system-id; cross-cluster clone only
psql -c 'ALTER SYSTEM RESET archive_mode;' # undo archive: false
pg restart pg-test                         # archive_mode requires a restart
pg-backup full                             # establish a recovery point on the new timeline

Until this is complete, scheduled backups can fail the identity check, and a clone restored with archive: false produces no new WAL archive:

postgres@pg-test-1:~$ pb backup
INFO: backup command begin 2.57.0: --annotation=pg_cluster=pg-test ... --stanza=pg-test --start-fast
ERROR: [051]: PostgreSQL version 18, system-id 7588470953413201282 do not match stanza version 18, system-id 7588470974940466058
       HINT: is this the correct stanza?
INFO: backup command end: aborted with exception [051]

Rebuild Backup Identity

stanza-upgrade lets the new cluster continue writing under its existing stanza. If the clone should start a completely independent backup history, rebuild that stanza instead.

Declarative workflow:

pig pb info -s pg-test
pig pb delete -s pg-test                 # type the exact stanza when prompted
./pgsql.yml -t pg_backup -l pg-test      # create a fresh stanza
pg-backup full

Equivalent low-level workflow:

pig pb stop
pig pb delete -s pg-test                 # confirm the exact stanza
pig pb start
pig pb create -s pg-test
pig pb backup full -s pg-test
Rebuilding permanently discards old recovery history

Delete only after checking recent backups, retaining any required independent recovery copy, and having the operator confirm the exact pg-test stanza. Object-locked versions can remain and continue consuming storage; a successful deletion command does not prove that every underlying version has been physically erased.

Online Copies: Standby Clusters

A PITR clone is a static snapshot. Use a streaming-replication standby cluster for a continuously following online copy, or a delayed cluster for a fixed rollback window such as one hour.

The three methods complement each other: standby clusters provide a live copy, delayed clusters preserve a fixed delay, and PITR clones expose any historical state inside the recovery window without requiring a pre-existing online replica.

Restore Drills

A clone is an end-to-end restore drill that does not touch production, although it does overwrite the designated drill target. Run one quarterly and after major backup changes:

  1. Select a point inside the production recovery window.
  2. Restore it into the drill cluster and record elapsed time as the measured PITR RTO.
  3. Validate integrity with authorized row-count checks, critical-table checks, and application connectivity.
  4. Complete post-clone cleanup and verify that the drill cluster can create a new backup.
  5. Record timing, failures, and any difference between the runbook and reality.

See Manual Recovery for a sandbox exercise using pgBackRest primitives, or Fork an Instance for an XFS snapshot-based local test copy.

8.5.4 - Backup Mechanism

pgBackRest concepts—stanzas, repositories, backup chains, retention, and timelines—and how Pigsty maps parameters to commands.

Pigsty’s backup and restore operations ultimately execute pgBackRest commands. Using them safely requires both pgBackRest’s model and the mapping from Pigsty’s orchestration layers to native options.

Core pgBackRest Concepts

Stanza: the Cluster’s Backup Identity

A stanza names one PostgreSQL backup configuration and isolates that cluster inside a repository. Pigsty maps it directly from pg_cluster: the pg-meta stanza stores data under backup/pg-meta/ and archive/pg-meta/, so several clusters can share one repository.

The stanza records the source system identifier and major version and checks them before a backup. That identity check is why a cross-cluster clone needs stanza-upgrade afterward. Pigsty creates the stanza during cluster initialization; stanza-upgrade updates it after a major-version change or clone.

Repository: Where Backups Live

A repository stores backup files and WAL archives. repo1-type selects POSIX, S3, Azure, GCS, or SFTP; repo1-path, repo1-cipher-*, and repo1-retention-* define location, encryption, and retention. Pigsty renders these from pgbackrest_repo; see Backup Repository.

Backup Chains and Labels

Type Contents Label suffix
Full Complete database-cluster copy F
Differential Changes since the latest full D
Incremental Changes since the latest backup of any type I

Labels encode the chain. 20250715-013657F is a full backup; 20250715-013657F_20250715-013724D and ..._20250715-013730I depend on the full identified before the underscore. --set chooses the starting backup explicitly; otherwise pgBackRest selects the newest usable set before the target.

Retention: Keeping the Repository Bounded

repo1-retention-full and repo1-retention-full-type (count or time) decide when full chains expire. Dependent differential/incremental backups and WAL needed only by that chain expire with the full. Pigsty enables expire-auto, and pig pb expire --plan previews a manual run.

Time retention is a minimum window, not “keep only fulls newer than N days.” An old full expires only when another retained full has reached that age. A 14-day setting with weekly fulls therefore commonly retains three full chains and roughly 14–21 days of history.

WAL Archiving

PostgreSQL invokes archive-push when a WAL segment fills or archive_timeout elapses. During recovery, restore_command calls archive-get. Pigsty enables asynchronous archiving through /pg/spool so a temporary repository delay does not block the primary directly.

Timelines

Each promotion after recovery or failover creates a new timeline. Older timeline history remains in the repository, and --target-timeline chooses the recovery branch (latest by default). See the conceptual PITR mechanism.

What restore Actually Does

restore first reconstructs the data directory. Pigsty enables --delta, so pgBackRest validates existing files and rewrites only mismatches. It then writes recovery state (recovery.signal, restore_command, and recovery_target_*). Actual WAL replay happens after PostgreSQL starts.

Consequently, a successful restore command is only half of PITR. --target-action controls what happens when replay reaches the target: pause, promote, or shutdown.

pgbackrest --stanza=pg-meta restore
pgbackrest --stanza=pg-meta --type=immediate restore
pgbackrest --stanza=pg-meta --type=time --target='2025-07-13 10:00:00+00' restore
pgbackrest --stanza=pg-meta --type=xid --target='250000' --target-exclusive restore
pgbackrest --stanza=pg-meta --type=name --target='my-restore-point' restore
pgbackrest --stanza=pg-meta --type=lsn --target='0/4001C80' --target-action=promote restore

Observe a Backup Chain

Run read-only info after full, differential, and incremental backups to inspect labels, size, WAL bounds, and references:

pig pb info

A representative sequence looks like:

full backup: 20250715-013657F
diff backup: 20250715-013657F_20250715-013724D
    backup reference total: 1 full
incr backup: 20250715-013657F_20250715-013730I
    backup reference total: 1 full, 1 diff

Pigsty’s Wrapper Layers

Layer Interface What it does
Cluster orchestration pg_pitr + pgsql-pitr.yml Pause HA, stop nodes, render configuration, restore/replay, inspect control data, clean etcd, and rebuild HA
Instance orchestration pig pitr Preflight, keep one target offline, restore, optionally start PostgreSQL, and leave Patroni stopped for inspection
Command primitive pig pb, pb, pg-backup Supply stanza/DBSU context and call the corresponding pgBackRest command
Engine pgbackrest Read /etc/pgbackrest/pgbackrest.conf and perform backup, archive, and restore operations

Command Primitives

pb is a login-shell function that reads the first stanza from the local configuration and forwards arguments:

pb() (
    stanza=$(grep -o '\[[^][]*]' /etc/pgbackrest/pgbackrest.conf | head -n1 | sed 's/.*\[\([^]]*\)].*/\1/')
    pgbackrest --stanza="${stanza}" "$@"
)
pb info     # pgbackrest --stanza=pg-meta info
pb backup   # pgbackrest --stanza=pg-meta backup

pg-backup adds a primary-role check for scheduled use:

pg-backup full
pg-backup diff
pg-backup incr   # the default; pgBackRest promotes it when no full exists

pig pb adds stanza detection, DBSU privilege handling, primary checks for backup, and plan/confirmation guards for destructive primitives. See Admin Commands.

Parameter Mapping

pg_pitr field pig pitr option pgBackRest option Meaning
cluster --stanza --stanza Source cluster/stanza
type plus time/xid/lsn/name corresponding target option --type + --target Recovery target
default --default no --type/--target Replay to archive end
immediate --immediate --type=immediate Stop at the first consistent point
exclusive --exclusive / -X --target-exclusive Stop before the target
action --target-action --target-action pause, promote, or shutdown
timeline --target-timeline / -T --target-timeline Target timeline
set --set / -b --set Starting backup set
db_include / db_exclude — --db-include / --db-exclude Select databases in a physical restore
link_map — --link-map Remap directory or tablespace links
process — process-max Parallel restore processes
data --data / -D --pg1-path Target data directory
repo repository number only in pig pitr rendered repo1-* Override repository definition in the playbook

A selective restore is still physical. Excluded databases receive sparse zeroed files so PostgreSQL can complete recovery, but those databases are inaccessible and must be removed explicitly afterward; this is not a logical subset like pg_dump.

How Configuration Is Rendered

The entry selected by pgbackrest_method is rendered to /etc/pgbackrest/pgbackrest.conf: underscores become hyphens and keys receive the repo1- prefix.

pgbackrest_repo:
  minio:
    type: s3                  # repo1-type=s3
    s3_endpoint: sss.pigsty   # repo1-s3-endpoint=sss.pigsty
    cipher_type: aes-256-cbc  # repo1-cipher-type=aes-256-cbc
    retention_full: 14        # repo1-retention-full=14

pgsql-pitr.yml renders a separate temporary /pg/conf/pitr.conf; PostgreSQL recovery output goes to /pg/tmp/recovery.log.

Scheduled Backups

pg_crontab entries are installed for the postgres OS user on every cluster node. Because pg-backup checks the current role, only the primary backs up, and a promoted primary takes over future schedules.

pg_crontab:
  - '00 01 * * 1 /pg/bin/pg-backup full'
  - '00 01 * * 2,3,4,5,6,7 /pg/bin/pg-backup'
./pgsql.yml -t pg_crontab -l pg-meta

See Backup Policy for frequency and retention design.

Deployment Details

The pg_backup subtask installs/configures pgBackRest, creates the stanza, and—when pgbackrest_init_backup is enabled—attempts an initial full backup. /etc/pgbackrest/initial.done is written only after that backup succeeds.

Path Purpose
/usr/bin/pgbackrest pgBackRest binary
/etc/pgbackrest/pgbackrest.conf Main stanza and repository configuration
/pg/backup Local repository path
/pg/spool Asynchronous archive spool
/pg/log/pgbackrest/ Backup, archive, and restore logs
/pg/conf/pitr.conf Temporary PITR configuration
/pg/tmp/recovery.log PostgreSQL recovery log

pgbackrest_exporter listens on pgbackrest_exporter_port, 9854 by default, and exports backup metrics. Disable it with pgbackrest_exporter_enabled or customize it with pgbackrest_exporter_options.

8.5.5 - Backup Repository

Configure local, Silo, and external S3 backup repositories, including retention, encryption, versioning, and Object Lock.

Two parameters decide where backups are stored: pgbackrest_repo defines candidate repositories, while pgbackrest_method selects one. Repository keys are rendered deterministically as pgBackRest repo1-* options, so any supported pgBackRest repository option can be used directly. Pigsty v4.5.0 renders only the entry selected by pgbackrest_method as repo1; keeping several candidate keys does not enable multi-repository backup.

Default Repositories

Pigsty supplies two definitions: local and minio.

  • local is the default. /pg/backup points at pg_fs_backup, /data/backups by default.
  • minio uses Silo from the MINIO module or another compatible S3 service. It is supported but not selected by default.
pgbackrest_method: local
pgbackrest_repo:
  local:
    path: /pg/backup
    retention_full_type: count
    retention_full: 2
  minio:
    type: s3
    s3_endpoint: sss.pigsty
    s3_region: us-east-1
    s3_bucket: pgsql
    s3_key: pgbackrest
    s3_key_secret: S3User.Backup
    s3_uri_style: path
    path: /pgbackrest
    storage_port: 9000
    storage_ca_file: /etc/pki/ca.crt
    block: y
    bundle: y
    bundle_limit: 20MiB
    bundle_size: 128MiB
    cipher_type: aes-256-cbc
    cipher_pass: pgBackRest
    retention_full_type: time
    retention_full: 14

The presets deliberately differ. local favors simple, fast restores with count-based retention and no encryption or bundling. minio enables AES-256-CBC encryption, bundles small files, uses block incremental backup, and retains full backups by time.

Replace public defaults in production

For a remote repository, replace both cipher_pass and s3_key_secret. pgBackRest and S3User.Backup are public example defaults. Losing the encryption passphrase makes the repository unrecoverable, so store it separately from the backups under controlled recovery procedures; see Deployment Security.

Retention Policy

pgBackRest applies retention after each backup (expire-auto). When a full backup expires, dependent differential/incremental backups and the WAL needed only by that chain expire with it.

  • retention_full_type: count plus retention_full: 2 keeps the two newest full chains; a third can exist briefly while a new full completes.
  • retention_full_type: time plus retention_full: 14 establishes a minimum time window. An old full does not expire until another full is at least 14 days old; with weekly full backups this usually leaves three chains and roughly 14–21 days of recovery history.

See Backup Policy for recovery-window and space calculations, and Admin Commands for a safe expiration preview.

Use a Silo Repository

The MINIO module currently deploys Silo, an S3-compatible object store. It provides an independent disaster-recovery copy only when deployed outside the database host or site failure domain. Deploy it, then select the minio preset:

all:
  vars:
    pgbackrest_method: minio
  children:
    minio: { hosts: { 10.10.10.10: { minio_seq: 1 }} ,vars: { minio_cluster: minio, minio_type: silo }}

The preset uses the HTTPS endpoint sss.pigsty by default and validates it with /etc/pki/ca.crt. MINIO initialization creates the default pgsql bucket and pgbackrest user.

For serious production use, deploy and test a fault-tolerant multi-node/multi-drive object store; see MINIO Configuration. The preset name minio does not require the server to be managed by Pigsty: independently operated MinIO, RustFS, or another compatible implementation can use it, but that service’s installation, upgrades, certificates, and lifecycle remain outside the MINIO role’s support boundary.

Use S3 or Cloud Object Storage

For a single database node, an off-host cloud object store is often the most valuable repository. Define a new entry and select it:

pgbackrest_method: s3
pgbackrest_repo:
  s3:
    type: s3
    s3_endpoint: oss-cn-beijing-internal.aliyuncs.com
    s3_region: oss-cn-beijing
    s3_bucket: <your_bucket_name>
    s3_key: <your_access_key>
    s3_key_secret: <your_secret_key>
    s3_uri_style: host
    path: /pgbackrest
    bundle: y
    bundle_limit: 20MiB
    bundle_size: 128MiB
    cipher_type: aes-256-cbc
    cipher_pass: <separate-strong-passphrase>
    retention_full_type: time
    retention_full: 14
  local:
    path: /pg/backup
    retention_full_type: count
    retention_full: 2

pgBackRest also supports Azure, GCS, and SFTP repositories.

Share a Repository Across Clusters

A centralized repository can serve several PostgreSQL clusters. Each stanza, mapped from pg_cluster, isolates one cluster’s backup and archive history. This also enables cross-cluster restore.

Cluster names must therefore be globally unique within a shared repository, even across otherwise separate deployment environments.

Repository Versioning

Object-store versioning can preserve earlier versions after an overwrite or deletion. It still shares the same storage system and control plane, so it does not replace an independent off-site or offline copy. Enable it for a bucket when it is created:

minio_buckets:
  - { name: pgsql, versioning: true }
  - { name: meta, versioning: true }
  - { name: data }

pgBackRest’s repo-target-time option can read the repository as it existed at an earlier time when the backend retains those versions.

Repository Locking

Some S3-compatible services support Object Lock/WORM. A retained object version cannot be changed or permanently deleted until its retention period ends. A normal delete can still create a delete marker that hides the current object while historical versions remain and continue consuming capacity.

The lock flag enables versioning and lock capability only when Pigsty creates the bucket:

minio_buckets:
  - { name: pgsql, lock: true }
  - { name: meta, versioning: true }
  - { name: data }

It does not set a default retention period. Configure GOVERNANCE or COMPLIANCE retention with mcli retention set or the storage console, then verify with mcli retention info. A sufficiently privileged principal can bypass GOVERNANCE; even root cannot shorten COMPLIANCE retention.

Locking changes expiration and stanza deletion: pgBackRest may expire objects logically while retained historical versions remain until their deadline. Test backup, expiration, delete-marker cleanup, and version recovery on a non-production bucket before enabling it.

Switch Repositories

After changing a repository definition or pgbackrest_method, rerender configuration, initialize the stanza, and create a recovery point in the new repository promptly:

./pgsql.yml -t pg_backup -l pg-meta           # Apply the repository definition and create the stanza
sudo -iu postgres pg-backup full              # Run on the current primary to establish the first recovery point

Existing backups are not migrated automatically. While retained, the old repository can still be selected as a restore source through pg_pitr.repo.

8.5.6 - Admin Commands

Backup administration reference covering setup and removal, manual backups, inspection, expiration, stanza management, logs, and alternative tools.

Run backup commands as the database superuser (pg_dbsu, postgres by default) on a database node. You can use any of these entry points:

  • pig pb: the PIG CLI wrapper, with automatic stanza detection, DBSU switching, and safety checks; this is the recommended interface
  • pb: a login-shell function that supplies --stanza and forwards arguments to pgBackRest
  • pgbackrest: the native command; see the pgBackRest command reference

Command Overview

PIG command Alias Native pgBackRest command Purpose
pig pb info i info Show backup and archive status
pig pb list ls — List repositories, stanzas, or backup sets
pig pb backup [full/diff/incr] b backup Create a backup after checking the primary role
pig pb restore r restore Low-level restore primitive; see Restore Operations
pig pb expire e expire Expire backups according to retention (--plan previews)
pig pb create c stanza-create Create a stanza
pig pb upgrade u stanza-upgrade Upgrade a stanza after a major-version change or clone
pig pb delete d stanza-delete Delete a stanza and all of its backups
pig pb check ck check Verify configuration, repository access, and archiving
pig pb start up start Re-enable pgBackRest operations
pig pb stop dw stop Stop new pgBackRest operations
pig pb log [list/show/tail] l — Inspect pgBackRest logs

Enable Backup

If pgbackrest_enabled is true when the cluster is created, backup is enabled automatically. If it was disabled at creation time, or repository settings have changed, run the pg_backup subtask:

./pgsql.yml -t pg_backup -l pg-meta   # Configure pgBackRest and create the stanza

After cluster initialization, Pigsty attempts an initial full backup. It writes /etc/pgbackrest/initial.done only after the backup command succeeds; the playbook ignores a failed attempt and leaves no marker. This file only prevents the initialization task from repeating, so always verify actual repository state with pig pb info or pgbackrest info. Define scheduled backups with pg_crontab; see Backup Policy.

Remove Backup

pig pb delete is the preferred interface when only a backup stanza must be removed. It asks for interactive confirmation; with a multi-stanza configuration, the target must also be explicit. Verify the exact target first:

pig pb info -s pg-meta           # Verify the backup chain and recovery window
pig pb delete -s pg-meta         # Executes only after typing the exact stanza name

When a primary instance (pg_role = primary) is removed, pgsql-rm.yml also tries to delete the cluster’s backup stanza by default. Every command below changes or deletes state; never execute one merely by copying the example:

./pgsql-rm.yml -l pg-meta                          # Remove the cluster and its backups
./pgsql-rm.yml -l pg-meta -e pg_rm_backup=false    # Remove the cluster while preserving backups
./pgsql-rm.yml -l pg-meta -t pg_backup             # Remove backup-related state only

Before execution, verify a recent usable backup, record the recovery requirement, and have the operator re-enter the exact cluster/stanza name. Set pg_rm_backup to false to preserve backups while removing the cluster.

pgsql-rm.yml -t pg_backup forcibly runs pgbackrest stanza-delete on the primary, removes the local repository directory in local mode, then removes the pgBackRest configuration and initial-backup marker. The task ignores some deletion errors, so a successful playbook result does not prove that repository objects were physically removed. Prefer pig pb delete when only the stanza needs deletion because it supplies a plan and confirmation guard.

With object versioning and object-lock retention, deletion may create a delete marker while locked historical versions continue consuming storage until their retention period expires.

Backup Deletion

Deleting backups can permanently destroy recovery options. Confirm the cluster/stanza, verify a recent backup and an alternative recovery copy, and retain the pig pb info output and deletion plan as an audit record.

Manual Backup

You can trigger a backup outside the crontab schedule. Both pg-backup and pig pb backup check that the current instance is primary and exit on a replica:

pg-backup            # Incremental; pgBackRest promotes it to full if no full backup exists
pg-backup full       # Full backup
pg-backup diff       # Differential backup relative to the latest full
pg-backup incr       # Incremental backup relative to the latest backup

pig pb backup full   # Equivalent PIG wrapper with stanza and DBSU detection

Backup consumes disk I/O and network bandwidth. Pigsty limits parallelism to a small number of processes, but production runs should still be scheduled for low-traffic periods.

Inspect Backups

pb info shows backups and WAL archive status for the current stanza:

pb info          # pgbackrest --stanza=pg-meta info
pig pb info      # Equivalent PIG wrapper
pig pb list      # List backup sets; use "list stanza" for all stanzas

Backup labels ending in F, D, and I identify full, differential, and incremental backups. The portion before an underscore identifies the full backup anchoring that chain. The WAL archive range and the oldest usable full backup together bound the recovery window.

The pgbackrest_exporter service on port 9854 continuously exports metrics such as the latest backup time, type, size, and error status.

Expire Old Backups

The configured retention policy is applied automatically after backups (expire-auto). Preview or run expiration manually with:

pig pb expire --plan    # Show what would be removed; no deletion
pig pb expire           # Apply the configured retention policy

Stanza Management

A stanza records a cluster’s backup identity, including its system identifier and major version. Manual management is occasionally required:

pig pb create                    # Create a stanza; cluster initialization normally does this
pig pb upgrade                   # Update after a major-version upgrade or clone recovery
pig pb delete -s pg-meta         # Delete all backups and archives after confirmation; destructive

The usual manual upgrade case is post-clone cleanup: after restoring another cluster’s backup into a new cluster, update the stanza identity before new backups can be written.

Check and Control 

pig pb check     # End-to-end check of configuration, archive push, and repository access
pig pb stop      # Stop new pgBackRest operations for a maintenance window
pig pb start     # Re-enable pgBackRest operations

check performs an archive-path check rather than being purely local or read-only; it verifies that WAL can reach the repository.

Logs 

pig pb log              # Show the latest log snapshot
pig pb log list         # List log files
pig pb log tail         # Follow the latest log
ls /pg/log/pgbackrest/  # Directory containing backup, archive, and restore logs

For pgsql-pitr.yml, PostgreSQL recovery output is written to /pg/tmp/recovery.log.

Alternative Backup Tools

pg-basebackup

The legacy /pg/bin/pg-basebackup script creates a single-file physical backup using native pg_basebackup, an lz4-compressed tar stream, and /pg/backup by default. Use it only for a simple local copy when a pgBackRest repository is unavailable:

pg-basebackup                                      # /pg/backup/backup_<tag>_<date>.tar.lz4
pg-basebackup --dst /tmp --file backup.tar.lz4     # Explicit destination and filename

mkdir -p /tmp/data
cat /pg/backup/backup_pg-meta_20250713.tar.lz4 | unlz4 -d -c | tar -xC /tmp/data
Legacy Encryption

pg-basebackup -e uses the obsolete OpenSSL RC4 cipher and must not be treated as confidentiality protection. For encrypted backups, use a pgBackRest repository configured with AES-256 (cipher_type: aes-256-cbc).

Logical Backup

Logical backups made with pg_dump cannot provide PITR, but they are appropriate for cross-major-version migration, partial exports, and long-term logical snapshots. Production recovery plans commonly use logical and physical backups together. See the PostgreSQL documentation.

8.6 - Data Migration

How to migrate an existing PostgreSQL cluster to a new Pigsty-managed PostgreSQL cluster with minimal downtime?

Pigsty includes a built-in playbook pgsql-migration.yml that implements online database migration based on logical replication.

With pre-generated automation scripts, application downtime can be reduced to just a few seconds. However, note that logical replication requires PostgreSQL 10 or later to work.

Of course, if you have sufficient downtime budget, you can always use the pg_dump | psql approach for offline migration.

Defining Migration Tasks

To use Pigsty’s online migration playbook, you need to create a definition file that describes the migration task details.

Refer to the task definition file example: files/migration/pg-meta.yml.

This migration task will online migrate pg-meta.meta to pg-test.test, where the former is called the Source Cluster (SRC) and the latter is called the Destination Cluster (DST).

pg-meta-1	10.10.10.10  --> pg-test-1	10.10.10.11 (10.10.10.12,10.10.10.13)

Logical replication-based migration works on a per-database basis. You need to specify the database name to migrate, as well as the IP addresses of the source and destination cluster primary nodes and superuser connection information.

---
#-----------------------------------------------------------------
# PG_MIGRATION
#-----------------------------------------------------------------
context_dir: ~/migration  # Directory for migration manual & scripts
#-----------------------------------------------------------------
# SRC Cluster (Old Cluster)
#-----------------------------------------------------------------
src_cls: pg-meta      # Source cluster name                  <Required>
src_db: meta          # Source database name                 <Required>
src_ip: 10.10.10.10   # Source cluster primary IP            <Required>
#src_pg: ''            # If defined, use this as source dbsu pgurl instead of:
#                      # postgres://{{ pg_admin_username }}@{{ src_ip }}/{{ src_db }}
#                      # e.g.: 'postgres://dbuser_dba:[email protected]:5432/meta'
#sub_conn: ''          # If defined, use this as subscription connection string instead of:
#                      # host={{ src_ip }} dbname={{ src_db }} user={{ pg_replication_username }}'
#                      # e.g.: 'host=10.10.10.10 dbname=meta user=replicator password=DBUser.Replicator'
#-----------------------------------------------------------------
# DST Cluster (New Cluster)
#-----------------------------------------------------------------
dst_cls: pg-test      # Destination cluster name             <Required>
dst_db: test          # Destination database name            <Required>
dst_ip: 10.10.10.11   # Destination cluster primary IP       <Required>
#dst_pg: ''            # If defined, use this as destination dbsu pgurl instead of:
#                      # postgres://{{ pg_admin_username }}@{{ dst_ip }}/{{ dst_db }}
#                      # e.g.: 'postgres://dbuser_dba:[email protected]:5432/test'
#-----------------------------------------------------------------
# PGSQL
#-----------------------------------------------------------------
pg_dbsu: postgres
pg_replication_username: replicator
pg_replication_password: DBUser.Replicator
pg_admin_username: dbuser_dba
pg_admin_password: DBUser.DBA
pg_monitor_username: dbuser_monitor
pg_monitor_password: DBUser.Monitor
#-----------------------------------------------------------------
...

By default, the superuser connection strings on both source and destination sides are constructed using the global admin user and the respective primary IP addresses, but you can always override these defaults through the src_pg and dst_pg parameters. Similarly, you can override the subscription connection string default through the sub_conn parameter.

Generating Migration Plan

This playbook does not actively perform cluster migration, but it generates the operation manual and automation scripts needed for migration.

By default, you will find the migration context directory at ~/migration/pg-meta.meta. Follow the instructions in README.md and execute these scripts in sequence to complete the database migration!

# Activate migration context: enable related environment variables
. ~/migration/pg-meta.meta/activate

# These scripts check src cluster status and help generate new cluster definitions in pigsty
./check-user     # Check src users
./check-db       # Check src databases
./check-hba      # Check src hba rules
./check-repl     # Check src replication identity
./check-misc     # Check src special objects

# These scripts establish logical replication between existing src cluster and pigsty-managed dst cluster, data except sequences will sync in real-time
./copy-schema    # Copy schema to destination
./create-pub     # Create publication on src
./create-sub     # Create subscription on dst
./copy-progress  # Print logical replication progress
./copy-diff      # Quick compare src and dst differences by counting tables

# These scripts run during online migration, which stops src cluster and copies sequence numbers (logical replication doesn't replicate sequences!)
./copy-seq [n]   # Sync sequence numbers, if n is given, apply additional offset

# You must switch application traffic to the new cluster based on your access method (dns,vip,haproxy,pgbouncer,etc.)!
#./disable-src   # Restrict src cluster access to admin nodes and new cluster (your implementation)
#./re-routing    # Re-route application traffic from SRC to DST! (your implementation)

# Then cleanup to remove subscription and publication
./drop-sub       # Drop subscription on dst after migration
./drop-pub       # Drop publication on src after migration

Notes

If you’re worried about primary key conflicts when copying sequence numbers, you can advance all sequences forward by some distance when copying, for example +1000. You can use ./copy-seq with a parameter 1000 to achieve this.

You must implement your own ./re-routing script to route your application traffic from src to dst. Because we don’t know how your traffic is routed (e.g., dns, VIP, haproxy, or pgbouncer). Of course, you can also do this manually…

You can implement a ./disable-src script to restrict application access to the src cluster—this is optional: if you can ensure all application traffic is cleanly switched in ./re-routing, you don’t really need this step.

But if you have various access from unknown sources that can’t be cleanly sorted out, it’s better to use more thorough methods: change HBA rules and reload to implement (recommended), or simply stop the postgres, pgbouncer, or haproxy processes on the source primary.

8.7 - Tutorials

Step-by-step guides for common PostgreSQL tasks and scenarios.

This section provides step-by-step tutorials for common PostgreSQL tasks and scenarios.

  • Citus Cluster: Deploy and manage Citus distributed clusters
  • Disaster Drill: Emergency recovery when 2 of 3 nodes fail
  • PG VIP: Configure L2 VIP for PostgreSQL clusters

8.7.1 - Troubleshooting

Common failures and analysis troubleshooting approaches

This document lists potential failures in PostgreSQL and Pigsty, as well as SOPs for locating, handling, and analyzing issues.

Disk Space Exhausted

Disk space exhaustion is the most common type of failure.

Symptoms

When the disk space where the database resides is exhausted, PostgreSQL will not work normally and may exhibit the following symptoms: database logs repeatedly report “no space left on device” errors, new data cannot be written, and PostgreSQL may even trigger a PANIC and force shutdown.

Pigsty includes a NodeFsSpaceFull alert rule that triggers when filesystem available space is less than 10%. Use the monitoring system’s NODE Instance panel to review the FS metrics panel to locate the issue.

Diagnosis

You can also log into the database node and use df -h to view the usage of each mounted partition to determine which partition is full. For database nodes, focus on checking the following directories and their sizes to determine which category of files has filled up the space:

  • Data directory (/pg/data/base): Stores data files for tables and indexes; pay attention to heavy writes and temporary files
  • WAL directory (e.g., pg/data/pg_wal): Stores PG WAL; WAL accumulation/replication slot retention is a common cause of disk exhaustion.
  • Database log directory (e.g., pg/log): If PG logs are not rotated in time and large amounts of errors are written, they may also consume significant space.
  • Local backup directory (e.g., data/backups): When using pgBackRest or similar tools to save backups locally, this may also fill up the disk.

If the issue occurs on the Pigsty admin node or monitoring node, also consider:

  • Monitoring data: VictoriaMetrics time-series metrics and VictoriaLogs log storage both consume disk space; check retention policies.
  • Object storage data: Pigsty’s integrated Silo object storage may be used for PG backup storage.

After identifying the directory consuming the most space, you can further use du -sh <directory> to drill down and find specific large files or subdirectories.

Resolution

Disk exhaustion is an emergency issue requiring immediate action to free up space and ensure the database continues to operate. When the data disk is not separated from the system disk, a full disk may prevent shell commands from executing. In this case, you can delete the /pg/dummy placeholder file to free up a small amount of emergency space so shell commands can work again. If the database has crashed due to pg_wal filling up, you need to restart the database service after clearing space and carefully check data integrity.

Transaction ID Wraparound

PostgreSQL cyclically uses 32-bit transaction IDs (XIDs), and when exhausted, a “transaction ID wraparound” failure occurs (XID Wraparound).

Symptoms

The typical sign in the first phase is when the age saturation in the PGSQL Persist - Age Usage panel enters the warning zone. Database logs begin to show messages like: WARNING: database "postgres" must be vacuumed within xxxxxxxx transactions.

If the problem continues to worsen, PostgreSQL enters protection mode: when remaining transaction IDs drop to about 1 million, the database switches to read-only mode; when reaching the limit of about 2.1 billion (2^31), it refuses any new transactions and forces the server to shut down to avoid data corruption.

Diagnosis

PostgreSQL and Pigsty enable automatic garbage collection (AutoVacuum) by default, so the occurrence of this type of failure usually has deeper root causes. Common causes include: very long transactions (SAGE), misconfigured Autovacuum, replication slot blockage, insufficient resources, storage engine/extension bugs, disk bad blocks.

First identify the database with the highest age, then use the Pigsty PGCAT Database - Tables panel to confirm the age distribution of tables. Also review the database error logs, which usually contain clues to locate the root cause.

Resolution

  1. Immediately freeze old transactions: If the database has not yet entered read-only protection mode, immediately execute a manual VACUUM FREEZE on the affected database. You can start by freezing the most severely aged tables one by one rather than doing the entire database at once to accelerate the effect. Connect to the database as a superuser and run VACUUM FREEZE table_name; on tables identified with the largest relfrozenxid, prioritizing tables with the highest XID age. This can quickly reclaim large amounts of transaction ID space.
  2. Single-user mode rescue: If the database is already refusing writes or has crashed for protection, you need to start the database in single-user mode to perform freeze operations. In single-user mode, run VACUUM FREEZE database_name; to freeze and clean the entire database. After completion, restart the database in multi-user mode. This can lift the wraparound lock and make the database writable again. Be very careful when operating in single-user mode and ensure sufficient transaction ID margin to complete the freeze.
  3. Standby node takeover: In some complex scenarios (e.g., when hardware issues prevent vacuum from completing), consider promoting a read-only standby node in the cluster to primary to obtain a relatively clean environment for handling the freeze. For example, if the primary cannot vacuum due to bad blocks, you can manually failover to promote the standby to the new primary, then perform emergency vacuum freeze on it. After ensuring the new primary has frozen old transactions, switch the load back.

Connection Exhaustion

PostgreSQL has a maximum connections configuration (max_connections). When client connections exceed this limit, new connection requests will be rejected. The typical symptom is that applications cannot connect to the database and report errors like FATAL: remaining connection slots are reserved for non-replication superuser connections or too many clients already. This indicates that regular connections are exhausted, leaving only slots reserved for superusers or replication.

Diagnosis

Connection exhaustion is usually caused by a large number of concurrent client requests. You can directly review the database’s current active sessions through PGCAT Instance / PGCAT Database / PGCAT Locks. Determine what types of queries are filling the system and proceed with further handling. Pay special attention to whether there are many connections in the “Idle in Transaction” state and long-running transactions (as well as slow queries).

Resolution

Kill queries: For situations where exhaustion has already blocked business operations, typically use pg_terminate_backend(pid) immediately for emergency pressure relief. For cases using connection pooling, you can adjust the connection pool size parameters and execute a reload to reduce the number of connections at the database level.

You can also modify the max_connections parameter to a larger value, but this parameter requires a database restart to take effect.

etcd Quota Exhausted

An exhausted etcd quota will cause the PG high availability control plane to fail and prevent configuration changes.

Diagnosis

Pigsty uses etcd as the distributed configuration store (DCS) when implementing high availability. etcd itself has a storage quota (default is about 2GB). When etcd storage usage reaches the quota limit, etcd will refuse write operations and report “etcdserver: mvcc: database space exceeded”. In this case, Patroni cannot write heartbeats or update configuration to etcd, causing cluster management functions to fail.

Resolution

Versions between Pigsty v2.0.0 and v2.5.1 are affected by this issue by default. Pigsty v2.6.0 added auto-compaction configuration for deployed etcd. If you only use it for PG high availability leases, this issue will no longer occur in regular use cases.

Defective Storage Engine

Currently, TimescaleDB’s experimental storage engine Hypercore has been proven to have defects, with cases of VACUUM being unable to reclaim leading to XID wraparound failures. Users using this feature should migrate to PostgreSQL native tables or TimescaleDB’s default engine promptly.

Detailed introduction: PG New Storage Engine Failure Case (Chinese)

8.7.2 - Manual PITR Drill

Run, validate, and complete PostgreSQL point-in-time recovery in explicit stages inside an isolated sandbox.

This tutorial drills PostgreSQL point-in-time recovery in Pigsty v4.5.0’s four-node sandbox. The main path runs pgsql-pitr.yml as down → pitr → up, giving the operator a separate validation gate before data overwrite, timeline promotion, and HA reconstruction.

For one current node, use pig pitr. For direct pgBackRest control, see the low-level pg-pitr utility.

Follow this literally only in a disposable sandbox

Recovery stops Patroni/PostgreSQL and overwrites the target PGDATA with pgbackrest --force restore; the up stage also deletes the target cluster’s etcd prefix and rebuilds Patroni state. The playbook prints a plan but has no interactive confirmation. Before production use, the operator must state and confirm the exact cluster and recovery point, verify a recent usable backup that has been independently tested, run --check with exactly the same -l, variables, and tags, and schedule a maintenance window. This tutorial does not authorize running these commands in any production environment.

Prepare an Isolated Sandbox

Use Vagrant or another disposable four-node lab and select the ha/full template, which includes a Silo backup repository:

curl -fsSL https://repo.pigsty.io/get | bash
cd ~/pigsty
./configure -c ha/full
./deploy.yml

ha/full defines the single-node pg-meta, three-node pg-test, and a Silo/pgBackRest repository. The rest of this tutorial uses the exact target pg-meta; do not copy that selector into another environment without resolving its inventory first.

Initial deployment and backup both change sandbox state. Production environments require their own deployment and backup approval process.

Establish Recovery Evidence

Start with read-only topology, backup-chain, and WAL-range checks:

pig pg list pg-meta
sudo -iu postgres pig pb info
sudo -iu postgres pig pb check

info must show at least one usable backup with status: ok, and archived WAL must cover the intended target. check validates the current stanza and archive path, but it does not replace a real restore drill or independent-copy validation.

In the sandbox, run Pigsty’s heartbeat helper to create an easy-to-verify time series:

sudo -iu postgres /pg/bin/pg-heartbeat

Record, then stop the workload:

  • the timezone-qualified timestamp you intend to recover to;
  • heartbeat, LSN, and transaction boundaries around that point;
  • current primary, timeline, and backup label;
  • target cluster pg-meta and target node.

Inspecting real application tables requires separate authorization. This tutorial uses only sandbox heartbeat data.

Declare the Recovery Task

Declare the target under pg-meta.vars in the sandbox inventory:

pg_pitr:
  cluster: pg-meta
  time: "2026-08-13 10:00:00+08"
  action: pause
  archive: true
  backup: false
  • cluster is the source backup stanza and defaults to the target pg_cluster.
  • action: pause pauses PostgreSQL at the target for a human validation gate.
  • archive: true preserves archive settings.
  • backup: true is not a safe-backup substitute: it deletes an existing <pg_data>-backup before moving current PGDATA, so this drill keeps it false.

The same object can be supplied temporarily with -e, but preflight and all three stages must repeat the exact same valid JSON to prevent variable drift.

Full Preflight

Before any stop or write action, check the complete workflow against the same target:

./pgsql-pitr.yml -l pg-meta --check

Confirm that Ansible resolves exactly pg-meta, then review the output for:

  • source stanza, recovery type, time, timeline, and action;
  • destination pg_data, port, and repository;
  • tablespace/link mappings;
  • archive and backup behavior.

--check validates inventory, variables, and task selection. It cannot prove that a pgBackRest backup is restorable. Any change to target, backup, or variables requires a new preflight.

Stage One: Stop

Only after the operator reconfirms exact target pg-meta, recovery point, and maintenance window, run:

./pgsql-pitr.yml -l pg-meta -t down

down attempts to pause Patroni automatic failover, stops Patroni on every target member, and uses immediate shutdown if PostgreSQL remains running. Then verify every target node rather than trusting only the playbook result:

sudo systemctl is-active patroni
sudo -iu postgres pg_ctl -D /pg/data status

Expected results are inactive and “server is not running.” If any member remains active, stop and diagnose; do not enter restore.

Stage Two: Restore and Validate

Recheck pg_pitr and the target nodes before running the destructive stage:

./pgsql-pitr.yml -l pg-meta -t pitr

This stage:

  1. renders /pg/conf/pitr.conf and /pg/bin/pg-restore;
  2. optionally moves old PGDATA according to backup;
  3. creates the destination and runs pgBackRest restore with --force and delta=y;
  4. starts PostgreSQL directly and waits for a consistent recovery state in the log;
  5. prints a pg_controldata summary.

Control data proves only that the directory has readable control state; it does not prove that a time, XID, or application boundary is correct. With action: pause, confirm WAL reached and paused near the target:

sudo -iu postgres psql -p 5432 -Atqc \
  'SELECT pg_is_in_recovery(), pg_is_wal_replay_paused(), pg_last_wal_replay_lsn(), pg_last_xact_replay_timestamp()'

Then inspect only the smallest authorized data scope; in the sandbox, check heartbeat rows. If the target is wrong:

  1. keep every Patroni member stopped;
  2. stop the manually started PostgreSQL;
  3. adjust the target and rerun the complete --check;
  4. rerun the pitr stage.

Do not run up or allow replicas from the old timeline to rejoin.

Promote and Stage Three: Rebuild HA

Promote only after the operator accepts the recovered result and the creation of a new timeline:

sudo -iu postgres pg_ctl -D /pg/data promote
sudo -iu postgres psql -p 5432 -Atqc 'SELECT pg_is_in_recovery()'

The expected result is f. Promotion is not read-only validation and cannot be losslessly undone.

With every Patroni member still stopped and exact target still pg-meta, run:

./pgsql-pitr.yml -l pg-meta -t up

up deletes the /pg/pg-meta/ prefix from etcd for the primary (the effective prefix also depends on pg_namespace and Citus settings), stops the manually started PostgreSQL, starts Patroni on the primary, then starts replicas one by one and resumes HA. The etcd deletion task tolerates errors, so a successful playbook result does not prove that stale DCS state was removed correctly.

Post-Recovery Acceptance

Verify each layer; “service started” is not recovery completion:

pig pg list pg-meta
sudo -iu postgres psql -Atqc \
  "SELECT pg_is_in_recovery(), pg_current_wal_lsn(), current_setting('archive_mode')"
sudo -iu postgres pig pb check

Also confirm:

  • exactly the intended member is primary and replicas stream from the new timeline;
  • HAProxy/VIP/DNS and application traffic point only to accepted instances;
  • data and event boundaries around the recovery target are correct;
  • archive_mode, archive_command, and new-WAL archiving work;
  • monitoring, alerts, and the backup repository contain no stale cluster state.

After the new timeline is stable, create and verify a new full backup under the applicable approval process:

sudo -iu postgres pg-backup full
sudo -iu postgres pig pb info

If the recovery explicitly used archive: false, it wrote archive-mode=off. Reset that override and perform a controlled restart only after validating recovery and confirming a maintenance window; default archive: true does not require this step.

Multi-Node and Cross-Cluster Recovery

  • Old-timeline replicas must not rejoin without validation. up starts replicas one at a time and waits for clone/recovery; monitor them to completion.
  • In cross-stanza recovery, pg_pitr.cluster is the source while -l is the destination being overwritten. Record and state both separately.
  • Cross-cluster recovery should normally use archive: false so a test destination cannot write WAL into the source stanza. Enable its own archiving only after acceptance and post-clone stanza cleanup.
  • link_map, data, port, and a temporary repo change the actual storage and data targets; include all of them in both --check and human review.

Related

8.7.3 - Enabling HugePage for PostgreSQL

Enabling HugePage for PostgreSQL to reduce memory fragmentation and improve performance.

Use node_hugepage_count and node_hugepage_ratio or /pg/bin/pg-tune-hugepage

If you plan to enable HugePages, consider using node_hugepage_count and node_hugepage_ratio, and apply with ./node.yml -t node_tune.

HugePages have pros and cons for databases. The advantage is that memory is managed exclusively, eliminating concerns about being reallocated and reducing database OOM risk. The disadvantage is that it may negatively impact performance in certain scenarios.

Before PostgreSQL starts, you need to allocate enough huge pages. The wasted portion can be reclaimed using the pg-tune-hugepage script, but this script is only available for PostgreSQL 15+.

If your PostgreSQL is already running, you can enable huge pages using the following method (PG15+ only):

sync; echo 3 > /proc/sys/vm/drop_caches   # Flush disk, release system cache (be prepared for database perf impact)
sudo /pg/bin/pg-tune-hugepage             # Write nr_hugepages to /etc/sysctl.d/hugepage.conf
pg restart <cls>                          # Restart postgres to use hugepage

8.7.4 - Clone and Side-Restore a PostgreSQL Instance

Create a local physical copy with pg-fork and run low-level pgBackRest recovery against a stopped data directory with pg-pitr.

Pigsty v4.5.0 provides two local shell utilities:

  • pg-fork copies a PostgreSQL data directory and gives the copy a separate port.
  • pg-pitr invokes pgBackRest to restore a stopped data directory to a selected target.

They are useful for sandbox drills, side-channel investigation, and temporary testing. They are not complete Patroni-cluster recovery orchestrators. Prefer pig pitr for a managed instance and staged pgsql-pitr.yml for a multi-node cluster.

Verify paths, backups, and stopped state first

pg-fork recursively removes an existing destination directory; pg-pitr overwrites the destination with backup data. Both can execute without a prompt in a non-interactive environment. Before a real run, verify source and destination absolute paths, ports, tablespaces, exact cluster/instance identity, and an independent recent backup that has been tested. A newly created CoW clone is not an independent backup.

pg-fork

pg-fork copies a PostgreSQL data directory on the current node. Run it as the database OS user—normally postgres, or at least a member of the postgres group:

pg-fork 1                         # /pg/data -> /pg/data1, destination port 15432
pg-fork 2 -d /pg/data1            # /pg/data1 -> /pg/data2, destination port 25432
pg-fork 3 -D /srv/pg-clone -P 55432

Parameters 

pg-fork <FORK_ID> [options]
Parameter Meaning Default
<FORK_ID> One digit from 1 to 9, used to derive the default directory and port Required
-d, --data <path> Source data directory $PG_DATA or /pg/data
-D, --dst <path> Destination data directory /pg/data<FORK_ID>
-p, --port <port> Source instance port $PG_PORT or 5432
-P, --dst-port <port> Destination instance port <FORK_ID>5432
-s, --skip Skip the online-backup API and force cold-copy mode No
-y, --yes Skip interactive confirmation No

The script rejects equal normalized source and destination paths, but it cannot know whether a custom destination contains other important data. If the destination directory exists, it is recursively removed before copying.

Hot Backup and Cold Copy

By default, the script connects to the source port and uses one psql session to run:

  1. CHECKPOINT;
  2. pg_backup_start();
  3. rm -rf <destination> followed by cp -a --reflink=auto;
  4. pg_backup_stop(wait_for_archive => false).

If the source cannot be reached on the selected port, the script automatically falls back to a cold copy instead of aborting. -s also forces a cold copy. A cold copy is safe only after you independently confirm that the source instance is fully stopped; a postmaster.pid warning is only a clue, not proof of process state.

On the same filesystem, the implementation recognizes these as fast CoW modes: XFS with reflink enabled, Btrfs, Bcachefs, and OCFS2. Other filesystems or cross-filesystem destinations still use cp --reflink=auto but may fall back to a full copy. The script’s help text mentions ZFS more broadly than its detector; the v4.5.0 implementation does not classify ZFS as a confirmed fast-CoW mode.

Clone Configuration

After a successful copy, pg-fork:

  • removes postmaster.pid, postmaster.opts, and standby.signal from the destination;
  • clears physical replication-slot files in the destination;
  • writes a separate port, archive_mode=off, and a local log_directory to the destination’s postgresql.auto.conf;
  • removes primary_conninfo, primary_slot_name, and old recovery_target* overrides.

It does not check whether the destination port is free or resize memory settings. Before starting the copy, inspect at least:

postgres -D /pg/data1 -C port
postgres -D /pg/data1 -C archive_mode
postgres -D /pg/data1 -C shared_buffers
pg_ctl -D /pg/data1 status
External tablespaces are not isolated

cp -a preserves symlinks under pg_tblspc; pg-fork does not copy or remap tablespaces outside PGDATA. Starting such a clone can access or modify the source instance’s tablespaces. If external tablespaces exist, independently copy and remap every one of them, or do not use this script to create a writable clone.

Interaction Boundary

The script asks Proceed with fork? [y/N] only when standard input is a terminal and -y was not used. Pipes, CI, cron, and other non-interactive invocations do not receive that prompt. Automation must therefore enforce a strict absolute-path allowlist and destination-existence check before invoking the script; do not add -y by default merely for convenience.

pg-pitr

pg-pitr is a low-level pgBackRest restore wrapper. It does not pause or start Patroni, stop or start PostgreSQL, clear DCS state, or rebuild replicas.

Recovery Targets

Understand and select a recovery target before execution. Invoking the command without arguments only shows help:

Parameter pgBackRest semantics
-d, --default Set no stop target and replay to the available end of WAL
-i, --immediate Stop when the selected backup becomes consistent
-t, --time <timestamp> Recover to a timestamp
-n, --name <restore-point> Recover to a named restore point
-l, --lsn <lsn> Recover to an LSN
-x, --xid <xid> Recover to a transaction ID

-S/--set (with compatibility alias -b/--backup) only selects which backup set recovery starts from; it is not a stop target. For example, -S 20251225-120000F -d still replays to the end of WAL. Combine -S ... -i to stop as soon as that backup becomes consistent.

For time, name, lsn, xid, and immediate, pgBackRest’s effective default action is to pause at the target; -P/--promote changes it to automatic promotion. Use -X/--exclusive only with a precise boundary such as time, lsn, or xid.

Other Options

Parameter Meaning
-D, --data <path> Absolute destination data directory; default /pg/data
-s, --stanza <name> pgBackRest stanza; defaults to the first non-global stanza in the config
-T, --timeline <value> latest, current, or a positive numeric timeline
-P, --promote Automatically promote recovery methods that have a stop target
-v, --verbose Enable pgBackRest info-level console logging
-c, --check, --dry-run Print the command without executing it
-y, --yes Skip the five-second countdown
-- <args> Pass additional arguments directly to pgBackRest

-c is a command-rendering check. It does not prove that the backup/WAL is usable or that PostgreSQL and Patroni are stopped. The wrapper also does not filter conflicting native arguments; review the final command carefully when passing repository, tablespace, or link-mapping options.

Safe Execution Sequence

This example shows only the low-level flow for one isolated destination. Use the complete runbook for production cluster recovery:

# 1. Read-only verification of backup sets and the recovery window
pig pb info

# 2. Confirm the exact target is stopped; stop Patroni first for a managed instance
pg_ctl -D /pg/data1 status

# 3. Render and review the exact command without writing data
pg-pitr -D /pg/data1 -t "2026-08-13 10:00:00+08" -c

# 4. Remove -c only after the operator reconfirms destination, backup, and stopped state
pg-pitr -D /pg/data1 -t "2026-08-13 10:00:00+08"

Real execution refuses root and aborts whenever postmaster.pid exists in the destination. Even a stale PID file requires the operator to confirm that PostgreSQL is stopped before removing it. There is no y/N question: an interactive terminal gets only an interruptible five-second countdown; a non-interactive invocation skips the countdown and enters restore immediately.

After restore, the operator starts and validates the instance:

pg_ctl -D /pg/data1 start
psql -p 15432 -Atqc \
  'SELECT pg_is_in_recovery(), pg_is_wal_replay_paused(), pg_last_xact_replay_timestamp()'

Promote only after the recovery target, authorized application data, timeline, and archive settings are all verified. Promotion creates a new timeline; it is not a reversible “inspect” operation. pg-pitr does not itself disable archiving. Do not mechanically follow its generic final “enable archive_mode” hint—inspect the effective value first and correct only an override introduced by this recovery.

Additional Side-Restore Risks

When restoring into a custom directory such as /pg/data1, pgBackRest can restore postgresql.auto.conf from the backup and overwrite the separate port written by pg-fork. Recheck port, archive_mode, sockets, logging, and memory settings before startup.

If the backup contains external tablespaces or links, a side restore can also use the original paths. For isolation, pass reviewed pgBackRest --tablespace-map, --link-map, or related arguments after -- and inspect the rendered command. Otherwise, do not start the restored copy on the same host as production.

Recommended Clone-Validation Flow

  1. Verify the source instance, destination absolute path, destination port, tablespaces, and independent backup.
  2. Run pg-fork <id> in an interactive terminal and confirm that the plan shows hot backup rather than an unintended cold-copy fallback.
  3. Before starting the clone, run pg-pitr -D <clone> ... -c and inspect the recovery command.
  4. Execute restore only after explicit destination confirmation, then recheck the clone’s port and every external path.
  5. Start the clone on the isolated port and verify recovery state and only the application data you are authorized to inspect.
  6. Promote only if the clone is intentionally becoming a new primary; otherwise stop it and clean up only the exact, verified path.

Side validation reduces direct writes to the current PGDATA, but still uses the same backup repository, consumes host resources, and may touch external tablespaces. It is not a risk-free sandbox.

Related

8.7.5 - Accidental Deletion

Handling accidental data deletion, table deletion, and database deletion

Accidental Data Deletion

If it’s a small-scale DELETE misoperation, you can consider using the pg_surgery or pg_dirtyread extension for in-place surgical recovery.

-- Immediately disable Auto Vacuum on this table and abort Auto Vacuum worker processes for this table
ALTER TABLE public.some_table SET (autovacuum_enabled = off, toast.autovacuum_enabled = off);

CREATE EXTENSION pg_dirtyread;
SELECT * FROM pg_dirtyread('tablename') AS t(col1 type1, col2 type2, ...);

If the deleted data has already been reclaimed by VACUUM, then use the general accidental deletion recovery process.

Accidental Object Deletion

When DROP/DELETE type misoperations occur, typically decide on a recovery plan according to the following process:

  1. Confirm whether this data can be recovered from the business system or other data systems. If yes, recover directly from the business side.
  2. Confirm whether there is a delayed replica. If yes, advance the delayed replica to the time point before deletion and query the data for recovery.
  3. If the data has been confirmed deleted, confirm backup information and whether the backup range covers the deletion time point. If it does, start PITR.
  4. Confirm whether to perform in-place cluster PITR rollback, or start a new server for replay, or use a replica for replay, and execute the recovery strategy.

Accidental Cluster Deletion

If an entire database cluster is accidentally deleted through Pigsty management commands, for example, incorrectly executing the pgsql-rm.yml playbook or the bin/pgsql-rm command. Unless you have set the pg_rm_backup parameter to false, the backup will be deleted along with the database cluster.

Warning: In this situation, your data will be unrecoverable! Please think three times before proceeding!

Recommendation: For production environments, you can globally configure this parameter to false in the configuration manifest to preserve backups when removing clusters.

8.7.6 - HA Drill: Handling 2-of-3 Node Failure

HA scenario response plan: When two of three nodes fail and auto-failover doesn’t work, how to recover from the emergency state?

If a classic 3-node HA deployment experiences simultaneous failure of two nodes (majority), the system typically cannot complete automatic failover and requires manual intervention.

First, assess the status of the other two servers. If they can be brought up quickly, prioritize recovering those two servers. Otherwise, enter the Emergency Recovery Procedure.

The Emergency Recovery Procedure assumes your admin node has failed and only a single regular database node survives. In this case, the fastest recovery process is:

  • Adjust HAProxy configuration to direct traffic to the primary.
  • Stop Patroni and manually promote the PostgreSQL replica to primary.

Adjust HAProxy Configuration

If you access the cluster bypassing HAProxy, you can skip this step. If you access the database cluster through HAProxy, you need to adjust the load balancer configuration to manually direct read/write traffic to the primary.

  • Edit /etc/haproxy/conf.d/<pg_cluster>-primary.cfg, where <pg_cluster> is your PostgreSQL cluster name, e.g., pg-meta.
  • Comment out the health check configuration options to stop health checks.
  • Comment out the other two failed machines in the server list, keeping only the current primary server.
listen pg-meta-primary
    bind *:5433
    mode tcp
    maxconn 5000
    balance roundrobin

    # Comment out the following four health check lines
    #option httpchk                               # <---- remove this
    #option http-keep-alive                       # <---- remove this
    #http-check send meth OPTIONS uri /primary    # <---- remove this
    #http-check expect status 200                 # <---- remove this

    default-server inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100
    server pg-meta-1 10.10.10.10:6432 check port 8008 weight 100

    # Comment out the other two failed machines
    #server pg-meta-2 10.10.10.11:6432 check port 8008 weight 100 <---- comment this
    #server pg-meta-3 10.10.10.12:6432 check port 8008 weight 100 <---- comment this

After adjusting the configuration, don’t rush to execute systemctl reload haproxy to reload. Wait until after promoting the primary, then execute together. The effect of this configuration is that HAProxy will no longer perform primary health checks (which by default use Patroni), but will directly direct write traffic to the current primary.

Manually Promote Replica

Log in to the target server, switch to the dbsu user, execute CHECKPOINT to flush to disk, stop Patroni, restart PostgreSQL, and execute Promote.

sudo su - postgres                     # Switch to database dbsu user
psql -c 'checkpoint; checkpoint;'      # Two Checkpoints to flush dirty pages, avoid long PG restart
sudo systemctl stop patroni            # Stop Patroni
pg-restart                             # Restart PostgreSQL
pg-promote                             # Promote PostgreSQL replica to primary
psql -c 'SELECT pg_is_in_recovery();'  # If result is f, it has been promoted to primary

If you adjusted the HAProxy configuration above, you can now execute systemctl reload haproxy to reload the HAProxy configuration and direct traffic to the new primary.

systemctl reload haproxy                # Reload HAProxy configuration to direct write traffic to current instance

Avoid Split Brain

After emergency recovery, the second priority is: Avoid Split Brain. Users should prevent the other two servers from coming back online and forming a split brain with the current primary, causing data inconsistency.

Simple approaches:

  • Power off/disconnect network the other two servers to ensure they don’t come online uncontrollably.
  • Adjust the database connection string used by applications to point directly to the surviving server’s primary.

Then decide the next steps based on the specific situation:

  • A: The two servers have temporary failures (e.g., network/power outage) and can be repaired in place to continue service.
  • B: The two failed servers have permanent failures (e.g., hardware damage) and will be removed and decommissioned.

Recovery After Temporary Failure

If the other two servers have temporary failures and can be repaired to continue service, follow these steps for repair and rebuild:

  • Handle one failed server at a time, prioritize the admin node / INFRA node.
  • Start the failed server and stop Patroni after startup.

After the ETCD cluster quorum is restored, it will resume work. Then start Patroni on the surviving server (current primary) to take over the existing PostgreSQL and regain cluster leadership. After Patroni starts, enter maintenance mode.

sudo systemctl restart patroni
pg pause <pg_cluster>

On the other two instances, create the touch /pg/data/standby.signal marker file as the postgres user to mark them as replicas, then start Patroni:

sudo -iu postgres touch /pg/data/standby.signal
sudo systemctl restart patroni

After confirming Patroni cluster identity/roles are correct, exit maintenance mode:

pg resume <pg_cluster>

Recovery After Permanent Failure

After permanent failure, first recover the ~/pigsty directory on the admin node. The key files needed are pigsty.yml and files/pki/ca/ca.key.

If you cannot retrieve or don’t have backups of these two files, you can deploy a new Pigsty and migrate the existing cluster to the new deployment via Backup Cluster.

Please regularly backup the pigsty directory (e.g., using Git for version control). Learn from this and avoid such mistakes in the future.

Configuration Repair

You can use the surviving node as the new admin node, copy the ~/pigsty directory to the new admin node, then start adjusting the configuration. For example, replace the original default admin node 10.10.10.10 with the surviving node 10.10.10.12:

all:
  vars:
    admin_ip: 10.10.10.12               # Use new admin node address
    node_etc_hosts: [10.10.10.12 h.pigsty a.pigsty p.pigsty g.pigsty sss.pigsty]
    infra_portal: {}                    # Also modify other configs referencing old admin IP (10.10.10.10)

  children:

    infra:                              # Adjust Infra cluster
      hosts:
        # 10.10.10.10: { infra_seq: 1 } # Old Infra node
        10.10.10.12: { infra_seq: 3 }   # New Infra node

    etcd:                               # Adjust ETCD cluster
      hosts:
        #10.10.10.10: { etcd_seq: 1 }   # Comment out this failed node
        #10.10.10.11: { etcd_seq: 2 }   # Comment out this failed node
        10.10.10.12: { etcd_seq: 3 }    # Keep surviving node
      vars:
        etcd_cluster: etcd

    pg-meta:                            # Adjust PGSQL cluster configuration
      hosts:
        #10.10.10.10: { pg_seq: 1, pg_role: primary }
        #10.10.10.11: { pg_seq: 2, pg_role: replica }
        #10.10.10.12: { pg_seq: 3, pg_role: replica , pg_offline_query: true }
        10.10.10.12: { pg_seq: 3, pg_role: primary , pg_offline_query: true }
      vars:
        pg_cluster: pg-meta

ETCD Repair

Then execute the following command to reset ETCD to a single-node cluster:

./etcd.yml -e etcd_safeguard=false -e etcd_clean=true

Follow the instructions in ETCD Reload Configuration to adjust ETCD Endpoint references.

INFRA Repair

If the surviving node doesn’t have the INFRA module, configure and install a new INFRA module on the current node. Execute the following command to deploy the INFRA module to the surviving node:

./infra.yml -l 10.10.10.12

Repair monitoring on the current node:

./node.yml -t monitor

PGSQL Repair 

./pgsql.yml -t pg_conf                            # Regenerate PG configuration files
systemctl reload patroni                          # Reload Patroni configuration on surviving node

After repairing each module, you can follow the standard expansion process to add new nodes to the cluster and restore cluster high availability.

8.7.7 - Bind a L2 VIP to PostgreSQL Primary with VIP-Manager

You can define an OPTIONAL L2 VIP on a PostgreSQL cluster, provided that all nodes in the cluster are in the same L2 network.

This VIP works on Master-Backup mode and always points to the node where the primary instance of the database cluster is located.

This VIP is managed by the VIP-Manager, which reads the Leader Key written by Patroni from DCS (etcd) to determine whether it is the master.

Enable VIP

Define pg_vip_enabled parameter as true in the cluster level to enable the VIP component on the cluster. You can also enable this configuration in the global configuration.

# pgsql 3 node ha cluster: pg-test
pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }   # primary instance, leader of cluster
    10.10.10.12: { pg_seq: 2, pg_role: replica }   # replica instance, follower of leader
    10.10.10.13: { pg_seq: 3, pg_role: replica, pg_offline_query: true } # replica with offline access
  vars:
    pg_cluster: pg-test           # define pgsql cluster name
    pg_users:  [{ name: test , password: test , pgbouncer: true , roles: [ dbrole_admin ] }]
    pg_databases: [{ name: test }]

    # Enable L2 VIP
    pg_vip_enabled: true
    pg_vip_address: 10.10.10.3/24
    #pg_vip_interface: auto

Beware that pg_vip_address must be a valid IP address with subnet and available in the current L2 network.

pg_vip_interface defaults to auto, in which case Pigsty detects the interface for each instance from the IPv4 address in the inventory.

If auto-detection is unsuitable for non-standard or policy-routing environments, explicitly specify a valid interface name for each instance, for example:

pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary , pg_vip_interface: eth0  }
    10.10.10.12: { pg_seq: 2, pg_role: replica , pg_vip_interface: eth1  }
    10.10.10.13: { pg_seq: 3, pg_role: replica , pg_vip_interface: ens33 }
  vars:
    pg_cluster: pg-test           # define pgsql cluster name
    pg_users:  [{ name: test , password: test , pgbouncer: true , roles: [ dbrole_admin ] }]
    pg_databases: [{ name: test }]

    # Enable L2 VIP
    pg_vip_enabled: true
    pg_vip_address: 10.10.10.3/24

To refresh the VIP configuration and restart the VIP-Manager, use the following command:

./pgsql.yml -t pg_vip

8.7.8 - Deploy HA Citus Cluster

How to deploy a Citus high-availability distributed cluster?

Citus is a PostgreSQL extension that transforms PostgreSQL into a distributed database, enabling horizontal scaling across multiple nodes to handle large amounts of data and queries.

Patroni v3.0+ provides native high-availability support for Citus, simplifying the setup of Citus clusters. Pigsty also provides native support for this.

Note

Citus 13.x supports PostgreSQL 18, 17, 16, 15, and 14. The Pigsty extension repository provides Citus ARM64 packages.

Citus Cluster

Pigsty natively supports Citus. The current complete configuration template is conf/ha/citus.yml.

The simplified four-node topology below illustrates the key parameters: a two-node coordinator cluster pg-citus0 and two single-node Worker clusters, pg-citus1 and pg-citus2. It is not a line-for-line excerpt from the current complete template.

pg-citus:
  hosts:
    10.10.10.10: { pg_group: 0, pg_cluster: pg-citus0 ,pg_vip_address: 10.10.10.2/24 ,pg_seq: 1, pg_role: primary }
    10.10.10.11: { pg_group: 0, pg_cluster: pg-citus0 ,pg_vip_address: 10.10.10.2/24 ,pg_seq: 2, pg_role: replica }
    10.10.10.12: { pg_group: 1, pg_cluster: pg-citus1 ,pg_vip_address: 10.10.10.3/24 ,pg_seq: 1, pg_role: primary }
    10.10.10.13: { pg_group: 2, pg_cluster: pg-citus2 ,pg_vip_address: 10.10.10.4/24 ,pg_seq: 1, pg_role: primary }
  vars:
    pg_mode: citus                            # pgsql cluster mode: citus
    pg_version: 18                            # citus 13.x supports PG 14-18
    pg_shard: pg-citus                        # citus shard name: pg-citus
    pg_primary_db: citus                      # primary database used by citus
    pg_vip_enabled: true                      # enable vip for citus cluster
    pg_vip_interface: auto                    # auto detect vip interface for every member
    pg_dbsu_password: DBUser.Postgres         # all dbsu password access for citus cluster
    pg_extensions: [ citus, postgis, pgvector, topn, pg_cron, hll ]  # install these extensions
    pg_libs: 'citus, pg_cron, pg_stat_statements' # explicitly preload citus and keep it first
    pg_users: [{ name: dbuser_citus ,password: DBUser.Citus ,pgbouncer: true ,roles: [ dbrole_admin ]    }]
    pg_databases: [{ name: citus ,owner: dbuser_citus ,extensions: [ citus, vector, topn, pg_cron, hll ] }]
    pg_parameters:
      cron.database_name: citus
      citus.node_conninfo: 'sslrootcert=/pg/cert/ca.crt sslmode=verify-full'
    pg_hba_rules:
      - { user: 'all' ,db: all  ,addr: 127.0.0.1/32  ,auth: ssl   ,title: 'all user ssl access from localhost' }
      - { user: 'all' ,db: all  ,addr: intra         ,auth: ssl   ,title: 'all user ssl access from intranet'  }

Compared to standard PostgreSQL clusters, Citus cluster configuration has some special requirements. First, you need to ensure the Citus extension is downloaded, installed, loaded, and enabled, which involves the following four parameters:

  • repo_packages: Must include the citus extension, or you need to use a PostgreSQL offline package that includes Citus.
  • pg_extensions: Must include the citus extension, i.e., you must install the citus extension on each node.
  • pg_libs: Must explicitly include citus in the first position; current Patroni templates use this parameter directly for shared_preload_libraries.
  • pg_databases: Define a primary database that must have the citus extension installed.

Second, you need to ensure the Citus cluster is configured correctly:

  • pg_mode: Must be set to citus to tell Patroni to use Citus mode.
  • pg_primary_db: Must specify the name of the primary database with citus extension, named citus here.
  • pg_shard: Must specify a unified name as the cluster name prefix for all horizontal shard PG clusters, pg-citus here.
  • pg_group: Must specify a shard number, integers starting from zero. 0 represents the coordinator cluster, others are Worker clusters.
  • pg_cluster: Must be unique among physical PostgreSQL clusters. Using pg_shard plus a sequence number is the usual naming convention, but the current role does not require it to equal a string concatenation of pg_shard and pg_group.
  • pg_dbsu_password: Must be set to a non-empty plaintext password, otherwise Citus will not work properly.
  • pg_parameters: Recommended to set citus.node_conninfo to enforce SSL access and require node-to-node client certificate verification.

After configuration, you can deploy the Citus cluster using pgsql.yml just like a regular PostgreSQL cluster.

Manage Citus Cluster

After defining the Citus cluster, deploy it using the pgsql.yml playbook:

./pgsql.yml -l pg-citus    # Deploy Citus cluster pg-citus

Using any member’s DBSU (postgres) user, you can list the Citus cluster status with patronictl (alias: pg):

$ pg list
+ Citus cluster: pg-citus ----------+---------+-----------+----+-----------+--------------------+
| Group | Member      | Host        | Role    | State     | TL | Lag in MB | Tags               |
+-------+-------------+-------------+---------+-----------+----+-----------+--------------------+
|     0 | pg-citus0-1 | 10.10.10.10 | Leader  | running   |  1 |           | clonefrom: true    |
|       |             |             |         |           |    |           | conf: tiny.yml     |
|       |             |             |         |           |    |           | spec: 20C.40G.125G |
|       |             |             |         |           |    |           | version: '16'      |
+-------+-------------+-------------+---------+-----------+----+-----------+--------------------+
|     1 | pg-citus1-1 | 10.10.10.11 | Leader  | running   |  1 |           | clonefrom: true    |
|       |             |             |         |           |    |           | conf: tiny.yml     |
|       |             |             |         |           |    |           | spec: 10C.20G.125G |
|       |             |             |         |           |    |           | version: '16'      |
+-------+-------------+-------------+---------+-----------+----+-----------+--------------------+
|     2 | pg-citus2-1 | 10.10.10.12 | Leader  | running   |  1 |           | clonefrom: true    |
|       |             |             |         |           |    |           | conf: tiny.yml     |
|       |             |             |         |           |    |           | spec: 10C.20G.125G |
|       |             |             |         |           |    |           | version: '16'      |
+-------+-------------+-------------+---------+-----------+----+-----------+--------------------+
|     2 | pg-citus2-2 | 10.10.10.13 | Replica | streaming |  1 |         0 | clonefrom: true    |
|       |             |             |         |           |    |           | conf: tiny.yml     |
|       |             |             |         |           |    |           | spec: 10C.20G.125G |
|       |             |             |         |           |    |           | version: '16'      |
+-------+-------------+-------------+---------+-----------+----+-----------+--------------------+

You can treat each horizontal shard cluster as an independent PGSQL cluster and manage them with the pg (patronictl) command. Note that when using the pg command to manage Citus clusters, you need to use the --group parameter to specify the cluster shard number:

pg list pg-citus --group 0   # Use --group 0 to specify cluster shard number

Citus has a system table called pg_dist_node that records Citus cluster node information. Patroni automatically maintains this table.

PGURL=postgres://postgres:[email protected]/citus

psql $PGURL -c 'SELECT * FROM pg_dist_node;'       # View node information
 nodeid | groupid |  nodename   | nodeport | noderack | hasmetadata | isactive | noderole  | nodecluster | metadatasynced | shouldhaveshards
--------+---------+-------------+----------+----------+-------------+----------+-----------+-------------+----------------+------------------
      1 |       0 | 10.10.10.10 |     5432 | default  | t           | t        | primary   | default     | t              | f
      4 |       1 | 10.10.10.12 |     5432 | default  | t           | t        | primary   | default     | t              | t
      5 |       2 | 10.10.10.13 |     5432 | default  | t           | t        | primary   | default     | t              | t
      6 |       0 | 10.10.10.11 |     5432 | default  | t           | t        | secondary | default     | t              | f

You can also view user authentication information (superuser access only):

$ psql $PGURL -c 'SELECT * FROM pg_dist_authinfo;'   # View node auth info (superuser only)

Then you can use a regular business user (e.g., dbuser_citus with DDL privileges) to access the Citus cluster:

psql postgres://dbuser_citus:[email protected]/citus -c 'SELECT * FROM pg_dist_node;'

Using Citus Cluster

When using Citus clusters, we strongly recommend reading the Citus official documentation to understand its architecture and core concepts.

The key is understanding the five types of tables in Citus and their characteristics and use cases:

  • Distributed Table
  • Reference Table
  • Local Table
  • Local Management Table
  • Schema Table

On the coordinator node, you can create distributed tables and reference tables and query them from any data node. Since 11.2, any Citus database node can act as a coordinator.

We can use pgbench to create some tables and distribute the main table (pgbench_accounts) across nodes, then use other small tables as reference tables:

PGURL=postgres://dbuser_citus:[email protected]/citus
pgbench -i $PGURL

psql $PGURL <<-EOF
SELECT create_distributed_table('pgbench_accounts', 'aid'); SELECT truncate_local_data_after_distributing_table('public.pgbench_accounts');
SELECT create_reference_table('pgbench_branches')         ; SELECT truncate_local_data_after_distributing_table('public.pgbench_branches');
SELECT create_reference_table('pgbench_history')          ; SELECT truncate_local_data_after_distributing_table('public.pgbench_history');
SELECT create_reference_table('pgbench_tellers')          ; SELECT truncate_local_data_after_distributing_table('public.pgbench_tellers');
EOF

Run read/write tests:

pgbench -nv -P1 -c10 -T500 postgres://dbuser_citus:[email protected]/citus      # Direct connect to coordinator port 5432
pgbench -nv -P1 -c10 -T500 postgres://dbuser_citus:[email protected]:6432/citus # Through connection pool, reduce client connection pressure
pgbench -nv -P1 -c10 -T500 postgres://dbuser_citus:[email protected]/citus      # Any primary node can act as coordinator
pgbench --select-only -nv -P1 -c10 -T500 postgres://dbuser_citus:[email protected]/citus # Read-only queries

Production Deployment

For production use of Citus, you typically need to set up streaming replication physical replicas for the Coordinator and each Worker cluster.

The current conf/ha/citus.yml defines one pg-meta instance plus 12 Citus instances across 13 hosts: six two-node physical clusters with pg_group values 0-5. The 10-node fragment below is a separate production-topology example, not the current template.

pg-citus: # citus group
  hosts:
    10.10.10.50: { pg_group: 0, pg_cluster: pg-citus0 ,pg_vip_address: 10.10.10.60/24 ,pg_seq: 0, pg_role: primary }
    10.10.10.51: { pg_group: 0, pg_cluster: pg-citus0 ,pg_vip_address: 10.10.10.60/24 ,pg_seq: 1, pg_role: replica }
    10.10.10.52: { pg_group: 1, pg_cluster: pg-citus1 ,pg_vip_address: 10.10.10.61/24 ,pg_seq: 0, pg_role: primary }
    10.10.10.53: { pg_group: 1, pg_cluster: pg-citus1 ,pg_vip_address: 10.10.10.61/24 ,pg_seq: 1, pg_role: replica }
    10.10.10.54: { pg_group: 2, pg_cluster: pg-citus2 ,pg_vip_address: 10.10.10.62/24 ,pg_seq: 0, pg_role: primary }
    10.10.10.55: { pg_group: 2, pg_cluster: pg-citus2 ,pg_vip_address: 10.10.10.62/24 ,pg_seq: 1, pg_role: replica }
    10.10.10.56: { pg_group: 3, pg_cluster: pg-citus3 ,pg_vip_address: 10.10.10.63/24 ,pg_seq: 0, pg_role: primary }
    10.10.10.57: { pg_group: 3, pg_cluster: pg-citus3 ,pg_vip_address: 10.10.10.63/24 ,pg_seq: 1, pg_role: replica }
    10.10.10.58: { pg_group: 4, pg_cluster: pg-citus4 ,pg_vip_address: 10.10.10.64/24 ,pg_seq: 0, pg_role: primary }
    10.10.10.59: { pg_group: 4, pg_cluster: pg-citus4 ,pg_vip_address: 10.10.10.64/24 ,pg_seq: 1, pg_role: replica }
  vars:
    pg_mode: citus                            # pgsql cluster mode: citus
    pg_version: 18                            # citus 13.x supports PG 14-18
    pg_shard: pg-citus                        # citus shard name: pg-citus
    pg_primary_db: citus                      # primary database used by citus
    pg_vip_enabled: true                      # enable vip for citus cluster
    pg_vip_interface: auto                    # auto detect vip interface for every member
    pg_dbsu_password: DBUser.Postgres         # enable dbsu password access for citus
    pg_extensions: [ citus, postgis, pgvector, topn, pg_cron, hll ]  # install these extensions
    pg_libs: 'citus, pg_cron, pg_stat_statements' # citus will be added by patroni automatically
    pg_users: [{ name: dbuser_citus ,password: DBUser.Citus ,pgbouncer: true ,roles: [ dbrole_admin ]    }]
    pg_databases: [{ name: citus ,owner: dbuser_citus ,extensions: [ citus, vector, topn, pg_cron, hll ] }]
    pg_parameters:
      cron.database_name: citus
      citus.node_conninfo: 'sslrootcert=/pg/cert/ca.crt sslmode=verify-full'
    pg_hba_rules:
      - { user: 'all' ,db: all  ,addr: 127.0.0.1/32  ,auth: ssl   ,title: 'all user ssl access from localhost' }
      - { user: 'all' ,db: all  ,addr: intra         ,auth: ssl   ,title: 'all user ssl access from intranet'  }

We will cover a series of advanced Citus topics in subsequent tutorials:

  • Read/write separation
  • Failure handling
  • Consistent backup and recovery
  • Advanced monitoring and diagnostics
  • Connection pooling

8.8 - Monitoring

Overview of Pigsty’s monitoring system architecture and how to monitor existing PostgreSQL instances

This document introduces Pigsty’s monitoring system architecture, including metrics, logs, and target management. It also covers how to monitor existing PG clusters and remote RDS services.

Monitoring Overview

Pigsty uses a modern observability stack for PostgreSQL monitoring:

  • Grafana for metrics visualization and PostgreSQL datasource
  • VictoriaMetrics for collecting metrics from PostgreSQL / Pgbouncer / Patroni / HAProxy / Node
  • VictoriaLogs for logging PostgreSQL / Pgbouncer / Patroni / pgBackRest and host component logs
  • Battery-included Grafana dashboards showcasing all aspects of PostgreSQL; use the live demo and the dashboard links in this page because the set evolves with each release

Metrics

PostgreSQL monitoring metrics are fully defined by the pg_exporter configuration file: roles/pg_monitor/templates/pg_exporter.yml. They are further processed by VictoriaMetrics/vmalert-compatible recording and alerting rules: files/victoria/rules/pgsql.yml.

Pigsty uses three identity labels: cls, ins, ip, which are attached to all metrics and logs. Additionally, metrics from Pgbouncer, host nodes (NODE), and load balancers are also used by Pigsty, with the same labels used whenever possible for correlation analysis.

- { cls: pg-meta, ins: pg-meta-1, ip: 10.10.10.10 }
- { cls: pg-test, ins: pg-test-1, ip: 10.10.10.11 }
- { cls: pg-test, ins: pg-test-2, ip: 10.10.10.12 }
- { cls: pg-test, ins: pg-test-3, ip: 10.10.10.13 }

Logs

PostgreSQL-related logs are collected by Vector and sent to the VictoriaLogs log storage/query service on infra nodes.

Target Management

VictoriaMetrics monitoring targets are defined in static files under /infra/targets/pgsql/, with each instance having a corresponding file. Taking pg-meta-1 as an example:

# pg-meta-1 [primary] @ 10.10.10.10
- labels: { cls: pg-meta, ins: pg-meta-1, ip: 10.10.10.10 }
  targets:
    - 10.10.10.10:9630    # <--- pg_exporter for PostgreSQL metrics
    - 10.10.10.10:9631    # <--- pgbouncer_exporter for PgBouncer metrics
    - 10.10.10.10:8008    # <--- Patroni metrics (when API SSL is not enabled)
    - 10.10.10.10:9854    # <--- pgbackrest_exporter for backup metrics

When the global flag patroni_ssl_enabled is set, Patroni targets are written separately to /infra/targets/patroni/<ins>.yml, because they use an HTTPS scrape endpoint. When monitoring RDS instances, monitoring targets are placed in /infra/targets/pgrds/ and managed by cluster.

When removing a cluster using bin/pgsql-rm or pgsql-rm.yml, the corresponding monitoring targets are removed. You can also use:

bin/pgmon-rm <cls|ins>    # Remove monitoring targets from all infra nodes

Remote RDS monitoring targets are placed in /infra/targets/pgrds/<cls>.yml, created by the pgsql-monitor.yml playbook or bin/pgmon-add script.

Monitoring Modes

Pigsty provides three monitoring modes to suit different monitoring needs.

Item \ Level L1 L2 L3
Name Basic Managed Standard
Abbr RDS MANAGED FULL
Scenario Connection string only, e.g., RDS Existing DB, nodes manageable Instances created by Pigsty
PGCAT Features ✅ Fully Available ✅ Fully Available ✅ Fully Available
PGSQL Features ✅ PG metrics only ✅ PG & node metrics only ✅ Full Features
Connection Pool Metrics ❌ Not Available ⚠️ Optional ✅ Pre-installed
Load Balancer Metrics ❌ Not Available ⚠️ Optional ✅ Pre-installed
PGLOG Features ❌ Not Available ⚠️ Optional ✅ Pre-installed
PG Exporter ⚠️ On infra nodes ✅ On DB nodes ✅ On DB nodes
Node Exporter ❌ Not deployed ✅ On DB nodes ✅ On DB nodes
Intrusiveness ✅ Non-intrusive ⚠️ Install Exporter ⚠️ Fully managed by Pigsty
Monitor Existing Instances ✅ Supported ✅ Supported ❌ For Pigsty-managed only
Monitoring Users & Views Manual setup Manual setup Auto-created by Pigsty
Deployment Playbook bin/pgmon-add <cls> Partial pgsql.yml/node.yml pgsql.yml
Required Permissions Connectable PGURL from infra SSH & sudo on DB nodes SSH & sudo on DB nodes
Feature Summary PGCAT + PGRDS Most features Full features

Databases fully managed by Pigsty are automatically monitored with the best support and typically require no configuration. For existing PostgreSQL clusters or RDS services, if the target DB nodes can be managed by Pigsty (ssh accessible, sudo available), you can consider managed deployment for a monitoring experience similar to native Pigsty. If you can only access the target database via PGURL (database connection string), such as remote RDS services, you can use basic mode to monitor the target database.

Monitor Existing Cluster

If the target DB nodes can be managed by Pigsty (ssh accessible and sudo available), you can use the pg_exporter task in the pgsql.yml playbook to deploy monitoring components (PG Exporter) on target nodes in the same way as standard deployments. You can also use the pgbouncer and pgbouncer_exporter tasks from that playbook to deploy connection pools and their monitoring on existing instance nodes. Additionally, you can use node_exporter, haproxy, and vector from node.yml to deploy host monitoring, load balancing, and log collection components, achieving an experience identical to native Pigsty database instances.

The definition method for existing clusters is exactly the same as for clusters managed by Pigsty. You selectively execute partial tasks from the pgsql.yml playbook instead of running the entire playbook.

./node.yml  -l <cls> -t node_repo,node_pkg           # Add YUM repos from INFRA nodes and install packages on host nodes
./node.yml  -l <cls> -t node_exporter,node_register  # Configure host monitoring and add to VictoriaMetrics
./node.yml  -l <cls> -t vector                       # Configure host log collection and send to VictoriaLogs
./pgsql.yml -l <cls> -t pg_exporter,pg_register      # Configure PostgreSQL monitoring and register with VictoriaMetrics/Grafana

Since the target database cluster already exists, you need to manually create monitoring users, schemas, and extensions on the target database cluster.

Monitor RDS

If you can only access the target database via PGURL (database connection string), you can configure according to the instructions here. In this mode, Pigsty deploys corresponding PG Exporters on INFRA nodes to scrape remote database metrics, as shown below:

------ infra ------
|                 |
| victoria-metrics|            v---- pg-foo-1 ----v
|       ^         |  metrics   |         ^        |
|   pg_exporter <-|------------|----  postgres    |
|   (port: 20001) |            | 10.10.10.10:5432 |
|       ^         |            ^------------------^
|       ^         |                      ^
|       ^         |            v---- pg-foo-2 ----v
|       ^         |  metrics   |         ^        |
|   pg_exporter <-|------------|----  postgres    |
|   (port: 20002) |            | 10.10.10.11:5433 |
-------------------            ^------------------^

In this mode, the monitoring system will not have metrics from hosts, connection pools, load balancers, or high availability components, but the database itself and real-time status information from the data catalog are still available. Pigsty provides two dedicated monitoring dashboards focused on PostgreSQL metrics: PGRDS Cluster and PGRDS Instance, while overview and database-level monitoring reuses existing dashboards. Since Pigsty cannot manage your RDS, users need to configure monitoring objects on the target database in advance.

Limitations when monitoring external Postgres instances
  • PgBouncer connection pool metrics are not available
  • Patroni high availability component metrics are not available
  • Host node monitoring metrics are not available, including node HAProxy and Keepalived metrics
  • Log collection and log-derived metrics are not available

Here we use the sandbox environment as an example: suppose the pg-meta cluster is an RDS instance pg-foo-1 to be monitored, and the pg-test cluster is an RDS cluster pg-bar to be monitored:

  1. Create monitoring schemas, users, and permissions on the target. Refer to Monitor Setup for details

  2. Declare the cluster in the configuration inventory. For example, if we want to monitor “remote” pg-meta & pg-test clusters:

    infra:            # Infra cluster for proxies, monitoring, alerts, etc.
  hosts: { 10.10.10.10: { infra_seq: 1 } }
  vars:           # Install pg_exporter on group 'infra' for remote postgres RDS
    pg_exporters: # List all remote instances here, assign a unique unused local port for k
      20001: { pg_cluster: pg-foo, pg_seq: 1, pg_host: 10.10.10.10 , pg_databases: [{ name: meta }] } # Register meta database as Grafana datasource

      20002: { pg_cluster: pg-bar, pg_seq: 1, pg_host: 10.10.10.11 , pg_port: 5432 } # Different connection string methods
      20003: { pg_cluster: pg-bar, pg_seq: 2, pg_host: 10.10.10.12 , pg_exporter_url: 'postgres://dbuser_monitor:[email protected]:5432/postgres?sslmode=disable'}
      20004: { pg_cluster: pg-bar, pg_seq: 3, pg_host: 10.10.10.13 , pg_monitor_username: dbuser_monitor, pg_monitor_password: DBUser.Monitor }

    Databases listed in the pg_databases field will be registered in Grafana as PostgreSQL datasources, providing data support for PGCAT monitoring dashboards. If you don’t want to use PGCAT and register databases in Grafana, simply set pg_databases to an empty array or leave it blank.

    pigsty-monitor.jpg

  3. Execute the add monitoring command: bin/pgmon-add <clsname>

    bin/pgmon-add pg-foo  # Bring pg-foo cluster into monitoring
bin/pgmon-add pg-bar  # Bring pg-bar cluster into monitoring

  4. To remove remote cluster monitoring targets, use bin/pgmon-rm <clsname>

    bin/pgmon-rm pg-foo  # Remove pg-foo from Pigsty monitoring
bin/pgmon-rm pg-bar  # Remove pg-bar from Pigsty monitoring

You can use more parameters to override default pg_exporter options. Here’s an example configuration for monitoring Aliyun RDS for PostgreSQL and PolarDB with Pigsty:

Example: Monitoring Aliyun RDS for PostgreSQL and PolarDB

For details, refer to: remote.yml

infra:            # Infra cluster for proxies, monitoring, alerts, etc.
  hosts: { 10.10.10.10: { infra_seq: 1 } }
  vars:
    pg_exporters:   # List all remote RDS PG instances to be monitored here

      20001:        # Assign a unique unused local port for local monitoring agent, this is a PolarDB primary
        pg_cluster: pg-polar                  # RDS cluster name (identity parameter, manually assigned name in monitoring system)
        pg_seq: 1                             # RDS instance number (identity parameter, manually assigned name in monitoring system)
        pg_host: pc-2ze379wb1d4irc18x.polardbpg.rds.aliyuncs.com # RDS host address
        pg_port: 1921                         # RDS port (from console connection info)
        pg_exporter_auto_discovery: true      # Enable new database auto-discovery
        pg_exporter_include_database: 'test'  # Only monitor databases in this list (comma-separated)
        pg_monitor_username: dbuser_monitor   # Monitoring username, overrides global config
        pg_monitor_password: DBUser_Monitor   # Monitoring password, overrides global config
        pg_databases: [{ name: test }]        # List of databases to enable PGCAT for, only name field needed, set register_datasource to false to not register

      20002:       # This is a PolarDB standby
        pg_cluster: pg-polar                  # RDS cluster name (identity parameter, manually assigned name in monitoring system)
        pg_seq: 2                             # RDS instance number (identity parameter, manually assigned name in monitoring system)
        pg_host: pe-2ze7tg620e317ufj4.polarpgmxs.rds.aliyuncs.com # RDS host address
        pg_port: 1521                         # RDS port (from console connection info)
        pg_exporter_auto_discovery: true      # Enable new database auto-discovery
        pg_exporter_include_database: 'test,postgres'  # Only monitor databases in this list (comma-separated)
        pg_monitor_username: dbuser_monitor   # Monitoring username
        pg_monitor_password: DBUser_Monitor   # Monitoring password
        pg_databases: [ { name: test } ]        # List of databases to enable PGCAT for, only name field needed, set register_datasource to false to not register

      20004: # This is a basic single-node RDS for PostgreSQL instance
        pg_cluster: pg-rds                    # RDS cluster name (identity parameter, manually assigned name in monitoring system)
        pg_seq: 1                             # RDS instance number (identity parameter, manually assigned name in monitoring system)
        pg_host: pgm-2zern3d323fe9ewk.pg.rds.aliyuncs.com  # RDS host address
        pg_port: 5432                         # RDS port (from console connection info)
        pg_exporter_auto_discovery: true      # Enable new database auto-discovery
        pg_exporter_include_database: 'rds'   # Only monitor databases in this list (comma-separated)
        pg_monitor_username: dbuser_monitor   # Monitoring username
        pg_monitor_password: DBUser_Monitor   # Monitoring password
        pg_databases: [ { name: rds } ]       # List of databases to enable PGCAT for, only name field needed, set register_datasource to false to not register

      20005: # This is a high-availability RDS for PostgreSQL cluster primary
        pg_cluster: pg-rdsha                  # RDS cluster name (identity parameter, manually assigned name in monitoring system)
        pg_seq: 1                             # RDS instance number (identity parameter, manually assigned name in monitoring system)
        pg_host: pgm-2ze3d35d27bq08wu.pg.rds.aliyuncs.com  # RDS host address
        pg_port: 5432                         # RDS port (from console connection info)
        pg_exporter_include_database: 'rds'   # Only monitor databases in this list (comma-separated)
        pg_databases: [ { name: rds }, {name : test} ]  # Include these two databases in PGCAT management, register as Grafana datasources

      20006: # This is a high-availability RDS for PostgreSQL cluster read-only instance (standby)
        pg_cluster: pg-rdsha                  # RDS cluster name (identity parameter, manually assigned name in monitoring system)
        pg_seq: 2                             # RDS instance number (identity parameter, manually assigned name in monitoring system)
        pg_host: pgr-2zexqxalk7d37edt.pg.rds.aliyuncs.com  # RDS host address
        pg_port: 5432                         # RDS port (from console connection info)
        pg_exporter_include_database: 'rds'   # Only monitor databases in this list (comma-separated)
        pg_databases: [ { name: rds }, {name : test} ]  # Include these two databases in PGCAT management, register as Grafana datasources

Monitor Setup

When you want to monitor existing instances, whether RDS or self-built PostgreSQL instances, you need to configure the target database so that Pigsty can access them.

To monitor an external existing PostgreSQL instance, you need a connection string that can access that instance/cluster. Any accessible connection string (business user, superuser) can be used, but we recommend using a dedicated monitoring user to avoid permission leaks.

  • Monitor User: The default username is dbuser_monitor, which should belong to the pg_monitor role group or have access to relevant views
  • Monitor Authentication: Default password authentication is used; ensure HBA policies allow the monitoring user to access databases from the admin node or DB node locally
  • Monitor Schema: Fixed schema name monitor is used for installing additional monitoring views and extension plugins; optional but recommended
  • Monitor Extension: Strongly recommended to enable the built-in monitoring extension pg_stat_statements
  • Monitor Views: Monitoring views are optional but can provide additional metric support

Monitor User

Using the default monitoring user dbuser_monitor as an example, create the following user on the target database cluster.

CREATE USER dbuser_monitor;                                       -- Create monitoring user
COMMENT ON ROLE dbuser_monitor IS 'system monitor user';          -- Comment on monitoring user
GRANT pg_monitor TO dbuser_monitor;                               -- Grant pg_monitor privilege to monitoring user, otherwise some metrics cannot be collected

ALTER USER dbuser_monitor PASSWORD 'DBUser.Monitor';              -- Modify monitoring user password as needed (strongly recommended! but keep consistent with Pigsty config)
ALTER USER dbuser_monitor SET log_min_duration_statement = 1000;  -- Recommended to avoid logs filling up with monitoring slow queries
ALTER USER dbuser_monitor SET search_path = monitor,public;       -- Recommended to ensure pg_stat_statements extension works properly

Please note that the monitoring user and password created here should be consistent with pg_monitor_username and pg_monitor_password.

Monitor Authentication

Configure the database pg_hba.conf file, adding the following rules to allow the monitoring user to access all databases from localhost and the admin machine using password authentication.

# allow local role monitor with password
local   all  dbuser_monitor                    md5
host    all  dbuser_monitor  127.0.0.1/32      md5
host    all  dbuser_monitor  <admin_machine_IP>/32 md5

If your RDS doesn’t support defining HBA, simply whitelist the internal IP address of the machine running Pigsty.

Monitor Schema

The monitoring schema is optional; even without it, the main functionality of Pigsty’s monitoring system can work properly, but we strongly recommend creating this schema.

CREATE SCHEMA IF NOT EXISTS monitor;               -- Create dedicated monitoring schema
GRANT USAGE ON SCHEMA monitor TO dbuser_monitor;   -- Allow monitoring user to use it

Monitor Extension

The monitoring extension is optional, but we strongly recommend enabling the pg_stat_statements extension, which provides important data about query performance.

Note: This extension must be listed in the database parameter shared_preload_libraries to take effect, and modifying that parameter requires a database restart.

CREATE EXTENSION IF NOT EXISTS "pg_stat_statements" WITH SCHEMA "monitor";

Please note that you should install this extension in the default admin database postgres. Sometimes RDS doesn’t allow you to create a monitoring schema in the postgres database. In such cases, you can install the pg_stat_statements plugin in the default public schema, as long as you ensure the monitoring user’s search_path is configured as above so it can find the pg_stat_statements view.

CREATE EXTENSION IF NOT EXISTS "pg_stat_statements";
ALTER USER dbuser_monitor SET search_path = monitor,public; -- Recommended to ensure pg_stat_statements extension works properly

Monitor Views

Monitoring views provide several commonly used pre-processed results and encapsulate permissions for monitoring metrics that require high privileges (such as shared memory allocation), making them convenient for querying and use. Strongly recommended to create in all databases requiring monitoring.

Monitoring schema and monitoring view definitions

The SQL below is provided to explain the monitored objects. The complete definitions rendered by current Pigsty are authoritative in roles/pgsql/templates/pg-init-template.sql, which also includes additional hardening for secure search paths and privilege boundaries.

----------------------------------------------------------------------
-- Table bloat estimate : monitor.pg_table_bloat
----------------------------------------------------------------------
DROP VIEW IF EXISTS monitor.pg_table_bloat CASCADE;
CREATE OR REPLACE VIEW monitor.pg_table_bloat AS
SELECT CURRENT_CATALOG AS datname, nspname, relname , tblid , bs * tblpages AS size,
       CASE WHEN tblpages - est_tblpages_ff > 0 THEN (tblpages - est_tblpages_ff)/tblpages::FLOAT ELSE 0 END AS ratio
FROM (
         SELECT ceil( reltuples / ( (bs-page_hdr)*fillfactor/(tpl_size*100) ) ) + ceil( toasttuples / 4 ) AS est_tblpages_ff,
                tblpages, fillfactor, bs, tblid, nspname, relname, is_na
         FROM (
                  SELECT
                      ( 4 + tpl_hdr_size + tpl_data_size + (2 * ma)
                          - CASE WHEN tpl_hdr_size % ma = 0 THEN ma ELSE tpl_hdr_size % ma END
                          - CASE WHEN ceil(tpl_data_size)::INT % ma = 0 THEN ma ELSE ceil(tpl_data_size)::INT % ma END
                          ) AS tpl_size, (heappages + toastpages) AS tblpages, heappages,
                      toastpages, reltuples, toasttuples, bs, page_hdr, tblid, nspname, relname, fillfactor, is_na
                  FROM (
                           SELECT
                               tbl.oid AS tblid, ns.nspname , tbl.relname, tbl.reltuples,
                               tbl.relpages AS heappages, coalesce(toast.relpages, 0) AS toastpages,
                               coalesce(toast.reltuples, 0) AS toasttuples,
                               coalesce(substring(array_to_string(tbl.reloptions, ' ') FROM 'fillfactor=([0-9]+)')::smallint, 100) AS fillfactor,
                               current_setting('block_size')::numeric AS bs,
                               CASE WHEN version()~'mingw32' OR version()~'64-bit|x86_64|ppc64|ia64|amd64' THEN 8 ELSE 4 END AS ma,
                               24 AS page_hdr,
                               23 + CASE WHEN MAX(coalesce(s.null_frac,0)) > 0 THEN ( 7 + count(s.attname) ) / 8 ELSE 0::int END
                                   + CASE WHEN bool_or(att.attname = 'oid' and att.attnum < 0) THEN 4 ELSE 0 END AS tpl_hdr_size,
                               sum( (1-coalesce(s.null_frac, 0)) * coalesce(s.avg_width, 0) ) AS tpl_data_size,
                               bool_or(att.atttypid = 'pg_catalog.name'::regtype)
                                   OR sum(CASE WHEN att.attnum > 0 THEN 1 ELSE 0 END) <> count(s.attname) AS is_na
                           FROM pg_attribute AS att
                                    JOIN pg_class AS tbl ON att.attrelid = tbl.oid
                                    JOIN pg_namespace AS ns ON ns.oid = tbl.relnamespace
                                    LEFT JOIN pg_stats AS s ON s.schemaname=ns.nspname AND s.tablename = tbl.relname AND s.inherited=false AND s.attname=att.attname
                                    LEFT JOIN pg_class AS toast ON tbl.reltoastrelid = toast.oid
                           WHERE NOT att.attisdropped AND tbl.relkind = 'r' AND nspname NOT IN ('pg_catalog','information_schema')
                           GROUP BY 1,2,3,4,5,6,7,8,9,10
                       ) AS s
              ) AS s2
     ) AS s3
WHERE NOT is_na;
COMMENT ON VIEW monitor.pg_table_bloat IS 'postgres table bloat estimate';

GRANT SELECT ON monitor.pg_table_bloat TO pg_monitor;

----------------------------------------------------------------------
-- Index bloat estimate : monitor.pg_index_bloat
----------------------------------------------------------------------
DROP VIEW IF EXISTS monitor.pg_index_bloat CASCADE;
CREATE OR REPLACE VIEW monitor.pg_index_bloat AS
SELECT CURRENT_CATALOG AS datname, nspname, idxname AS relname, tblid, idxid, relpages::BIGINT * bs AS size,
       COALESCE((relpages - ( reltuples * (6 + ma - (CASE WHEN index_tuple_hdr % ma = 0 THEN ma ELSE index_tuple_hdr % ma END)
                                               + nulldatawidth + ma - (CASE WHEN nulldatawidth % ma = 0 THEN ma ELSE nulldatawidth % ma END))
                                  / (bs - pagehdr)::FLOAT  + 1 )), 0) / relpages::FLOAT AS ratio
FROM (
         SELECT nspname,idxname,indrelid AS tblid,indexrelid AS idxid,
                reltuples,relpages,
                current_setting('block_size')::INTEGER                                                               AS bs,
                (CASE WHEN version() ~ 'mingw32' OR version() ~ '64-bit|x86_64|ppc64|ia64|amd64' THEN 8 ELSE 4 END)  AS ma,
                24                                                                                                   AS pagehdr,
                (CASE WHEN max(COALESCE(pg_stats.null_frac, 0)) = 0 THEN 2 ELSE 6 END)                               AS index_tuple_hdr,
                sum((1.0 - COALESCE(pg_stats.null_frac, 0.0)) *
                    COALESCE(pg_stats.avg_width, 1024))::INTEGER                                                     AS nulldatawidth
         FROM pg_attribute
                  JOIN (
             SELECT pg_namespace.nspname,
                    ic.relname                                                   AS idxname,
                    ic.reltuples,
                    ic.relpages,
                    pg_index.indrelid,
                    pg_index.indexrelid,
                    tc.relname                                                   AS tablename,
                    regexp_split_to_table(pg_index.indkey::TEXT, ' ') :: INTEGER AS attnum,
                    pg_index.indexrelid                                          AS index_oid
             FROM pg_index
                      JOIN pg_class ic ON pg_index.indexrelid = ic.oid
                      JOIN pg_class tc ON pg_index.indrelid = tc.oid
                      JOIN pg_namespace ON pg_namespace.oid = ic.relnamespace
                      JOIN pg_am ON ic.relam = pg_am.oid
             WHERE pg_am.amname = 'btree' AND ic.relpages > 0 AND nspname NOT IN ('pg_catalog', 'information_schema')
         ) ind_atts ON pg_attribute.attrelid = ind_atts.indexrelid AND pg_attribute.attnum = ind_atts.attnum
                  JOIN pg_stats ON pg_stats.schemaname = ind_atts.nspname
             AND ((pg_stats.tablename = ind_atts.tablename AND pg_stats.attname = pg_get_indexdef(pg_attribute.attrelid, pg_attribute.attnum, TRUE))
                 OR (pg_stats.tablename = ind_atts.idxname AND pg_stats.attname = pg_attribute.attname))
         WHERE pg_attribute.attnum > 0
         GROUP BY 1, 2, 3, 4, 5, 6
     ) est;
COMMENT ON VIEW monitor.pg_index_bloat IS 'postgres index bloat estimate (btree-only)';

GRANT SELECT ON monitor.pg_index_bloat TO pg_monitor;

----------------------------------------------------------------------
-- Relation Bloat : monitor.pg_bloat
----------------------------------------------------------------------
DROP VIEW IF EXISTS monitor.pg_bloat CASCADE;
CREATE OR REPLACE VIEW monitor.pg_bloat AS
SELECT coalesce(ib.datname, tb.datname)                                                   AS datname,
       coalesce(ib.nspname, tb.nspname)                                                   AS nspname,
       coalesce(ib.tblid, tb.tblid)                                                       AS tblid,
       coalesce(tb.nspname || '.' || tb.relname, ib.nspname || '.' || ib.tblid::RegClass) AS tblname,
       tb.size                                                                            AS tbl_size,
       CASE WHEN tb.ratio < 0 THEN 0 ELSE round(tb.ratio::NUMERIC, 6) END                 AS tbl_ratio,
       (tb.size * (CASE WHEN tb.ratio < 0 THEN 0 ELSE tb.ratio::NUMERIC END)) ::BIGINT    AS tbl_wasted,
       ib.idxid,
       ib.nspname || '.' || ib.relname                                                    AS idxname,
       ib.size                                                                            AS idx_size,
       CASE WHEN ib.ratio < 0 THEN 0 ELSE round(ib.ratio::NUMERIC, 5) END                 AS idx_ratio,
       (ib.size * (CASE WHEN ib.ratio < 0 THEN 0 ELSE ib.ratio::NUMERIC END)) ::BIGINT    AS idx_wasted
FROM monitor.pg_index_bloat ib
         FULL OUTER JOIN monitor.pg_table_bloat tb ON ib.tblid = tb.tblid;

COMMENT ON VIEW monitor.pg_bloat IS 'postgres relation bloat detail';
GRANT SELECT ON monitor.pg_bloat TO pg_monitor;

----------------------------------------------------------------------
-- monitor.pg_index_bloat_human
----------------------------------------------------------------------
DROP VIEW IF EXISTS monitor.pg_index_bloat_human CASCADE;
CREATE OR REPLACE VIEW monitor.pg_index_bloat_human AS
SELECT idxname                            AS name,
       tblname,
       idx_wasted                         AS wasted,
       pg_size_pretty(idx_size)           AS idx_size,
       round(100 * idx_ratio::NUMERIC, 2) AS idx_ratio,
       pg_size_pretty(idx_wasted)         AS idx_wasted,
       pg_size_pretty(tbl_size)           AS tbl_size,
       round(100 * tbl_ratio::NUMERIC, 2) AS tbl_ratio,
       pg_size_pretty(tbl_wasted)         AS tbl_wasted
FROM monitor.pg_bloat
WHERE idxname IS NOT NULL;
COMMENT ON VIEW monitor.pg_index_bloat_human IS 'postgres index bloat info in human-readable format';
GRANT SELECT ON monitor.pg_index_bloat_human TO pg_monitor;


----------------------------------------------------------------------
-- monitor.pg_table_bloat_human
----------------------------------------------------------------------
DROP VIEW IF EXISTS monitor.pg_table_bloat_human CASCADE;
CREATE OR REPLACE VIEW monitor.pg_table_bloat_human AS
SELECT tblname                                          AS name,
       idx_wasted + tbl_wasted                          AS wasted,
       pg_size_pretty(idx_wasted + tbl_wasted)          AS all_wasted,
       pg_size_pretty(tbl_wasted)                       AS tbl_wasted,
       pg_size_pretty(tbl_size)                         AS tbl_size,
       tbl_ratio,
       pg_size_pretty(idx_wasted)                       AS idx_wasted,
       pg_size_pretty(idx_size)                         AS idx_size,
       round(idx_wasted::NUMERIC * 100.0 / idx_size, 2) AS idx_ratio
FROM (SELECT datname,
             nspname,
             tblname,
             coalesce(max(tbl_wasted), 0)                         AS tbl_wasted,
             coalesce(max(tbl_size), 1)                           AS tbl_size,
             round(100 * coalesce(max(tbl_ratio), 0)::NUMERIC, 2) AS tbl_ratio,
             coalesce(sum(idx_wasted), 0)                         AS idx_wasted,
             coalesce(sum(idx_size), 1)                           AS idx_size
      FROM monitor.pg_bloat
      WHERE tblname IS NOT NULL
      GROUP BY 1, 2, 3
     ) d;
COMMENT ON VIEW monitor.pg_table_bloat_human IS 'postgres table bloat info in human-readable format';
GRANT SELECT ON monitor.pg_table_bloat_human TO pg_monitor;


----------------------------------------------------------------------
-- Activity Overview: monitor.pg_session
----------------------------------------------------------------------
DROP VIEW IF EXISTS monitor.pg_session CASCADE;
CREATE OR REPLACE VIEW monitor.pg_session AS
SELECT coalesce(datname, 'all') AS datname, numbackends, active, idle, ixact, max_duration, max_tx_duration, max_conn_duration
FROM (
         SELECT datname,
                count(*)                                         AS numbackends,
                count(*) FILTER ( WHERE state = 'active' )       AS active,
                count(*) FILTER ( WHERE state = 'idle' )         AS idle,
                count(*) FILTER ( WHERE state = 'idle in transaction'
                    OR state = 'idle in transaction (aborted)' ) AS ixact,
                max(extract(epoch from now() - state_change))
                FILTER ( WHERE state = 'active' )                AS max_duration,
                max(extract(epoch from now() - xact_start))      AS max_tx_duration,
                max(extract(epoch from now() - backend_start))   AS max_conn_duration
         FROM pg_stat_activity
         WHERE backend_type = 'client backend'
           AND pid <> pg_backend_pid()
         GROUP BY ROLLUP (1)
         ORDER BY 1 NULLS FIRST
     ) t;
COMMENT ON VIEW monitor.pg_session IS 'postgres activity group by session';
GRANT SELECT ON monitor.pg_session TO pg_monitor;


----------------------------------------------------------------------
-- Sequential Scan: monitor.pg_seq_scan
----------------------------------------------------------------------
DROP VIEW IF EXISTS monitor.pg_seq_scan CASCADE;
CREATE OR REPLACE VIEW monitor.pg_seq_scan AS
SELECT schemaname                                                        AS nspname,
       relname,
       seq_scan,
       seq_tup_read,
       seq_tup_read / seq_scan                                           AS seq_tup_avg,
       idx_scan,
       n_live_tup + n_dead_tup                                           AS tuples,
       round(n_live_tup * 100.0::NUMERIC / (n_live_tup + n_dead_tup), 2) AS live_ratio
FROM pg_stat_user_tables
WHERE seq_scan > 0
  and (n_live_tup + n_dead_tup) > 0
ORDER BY seq_scan DESC;
COMMENT ON VIEW monitor.pg_seq_scan IS 'table that have seq scan';
GRANT SELECT ON monitor.pg_seq_scan TO pg_monitor;
Function for viewing shared memory allocation (PG13 and above)
DROP FUNCTION IF EXISTS monitor.pg_shmem() CASCADE;
CREATE OR REPLACE FUNCTION monitor.pg_shmem() RETURNS SETOF
    pg_shmem_allocations SET search_path = '' AS $$ SELECT * FROM pg_shmem_allocations;$$ LANGUAGE SQL SECURITY DEFINER;
COMMENT ON FUNCTION monitor.pg_shmem() IS 'security wrapper for system view pg_shmem';
REVOKE ALL ON FUNCTION monitor.pg_shmem() FROM PUBLIC;
REVOKE ALL ON FUNCTION monitor.pg_shmem() FROM dbrole_readonly;
REVOKE ALL ON FUNCTION monitor.pg_shmem() FROM dbrole_offline;
GRANT EXECUTE ON FUNCTION monitor.pg_shmem() TO pg_monitor;

8.9 - Dashboard

Pigsty provides numerous out-of-the-box Grafana monitoring dashboards for PostgreSQL

Pigsty provides numerous out-of-the-box Grafana monitoring dashboards for PostgreSQL: Demo & Gallery.

The current source provides 31 PostgreSQL-related dashboards: 29 PostgreSQL / PGCAT dashboards under files/grafana/pgsql, plus two PGLOG dashboards under files/grafana/app. They are organized by hierarchy into Overview, Cluster, Instance, and Database categories, and by data source into PGSQL, PGCAT, and PGLOG.

pigsty-dashboard.jpg

Overview

Overview Cluster Instance Database
PGSQL Overview PGSQL Cluster PGSQL Instance PGSQL Database
PGSQL Alert PGRDS Cluster PGRDS Instance PGCAT Database
PGSQL Shard PGSQL Activity PGCAT Instance PGSQL Tables
PGSQL Replication PGSQL Persist PGSQL Table
PGSQL Service PGSQL Proxy PGCAT Table
PGSQL Databases PGSQL Pgbouncer PGSQL Query
PGSQL Patroni PGSQL Session PGCAT Query
PGSQL PITR PGSQL Xacts PGCAT Locks
PGSQL Exporter PGCAT Schema

Overview

  • pgsql-overview: Main dashboard for the PGSQL module
  • pgsql-alert: Global key metrics and alert events for PGSQL
  • pgsql-shard: Overview of horizontally sharded PGSQL clusters (e.g., Citus/GPSQL)

Cluster

  • pgsql-cluster: Main dashboard for a PGSQL cluster
  • pgrds-cluster: RDS version of PGSQL Cluster, focusing on PostgreSQL-native metrics
  • pgsql-activity: Session/load/QPS/TPS/locks for PGSQL cluster
  • pgsql-replication: Replication, slots, and pub/sub for PGSQL cluster
  • pgsql-service: Service, proxy, routing, and load balancing for PGSQL cluster
  • pgsql-databases: Database CRUD, slow queries, and table statistics across all instances
  • pgsql-patroni: HA status and Patroni component status for cluster
  • pgsql-pitr: PITR context for point-in-time recovery assistance

Instance

  • pgsql-instance: Main dashboard for a single PGSQL instance
  • pgrds-instance: RDS version of PGSQL Instance, focusing on PostgreSQL-native metrics
  • pgcat-instance: Instance info retrieved directly from database catalog
  • pgsql-proxy: Detailed metrics for a single HAProxy load balancer
  • pgsql-pgbouncer: Metrics overview for a single Pgbouncer connection pooler
  • pgsql-persist: Persistence metrics: WAL, XID, checkpoint, archive, IO
  • pgsql-session: Session and active/idle time metrics for a single instance
  • pgsql-xacts: Transaction, lock, TPS/QPS related metrics
  • pgsql-exporter: Self-monitoring metrics for Postgres and Pgbouncer exporters

Database

  • pgsql-database: Main dashboard for a single PGSQL database
  • pgcat-database: Database info retrieved directly from database catalog
  • pgsql-tables: Table/index access metrics within a single database
  • pgsql-table: Detailed info for a single table (QPS/RT/index/sequence…)
  • pgcat-table: Detailed table info from database catalog (stats/bloat…)
  • pgsql-query: Detailed info for a query type (QPS/RT)
  • pgcat-query: Query details from database catalog (SQL/stats)
  • pgcat-schema: Schema info from database catalog (tables/indexes/sequences…)
  • pgcat-locks: Activity and lock wait info from database catalog

Overview

PGSQL Overview: Main dashboard for the PGSQL module

PGSQL Overview

pgsql-overview.jpg

PGSQL Alert: Global core metrics overview and alert events

PGSQL Alert

pgsql-alert.jpg

PGSQL Shard: Cross-shard metric comparison for horizontally sharded PGSQL clusters (e.g., CITUS/GPSQL)

PGSQL Shard

pgsql-shard.jpg

Cluster

PGSQL Cluster: Main dashboard for a PGSQL cluster

PGSQL Cluster

pgsql-cluster.jpg

PGRDS Cluster: RDS version of PGSQL Cluster, focusing on PostgreSQL-native metrics

PGRDS Cluster

pgrds-cluster.jpg

PGSQL Service: Service, proxy, routing, and load balancing for PGSQL cluster

PGSQL Service

pgsql-service.jpg

PGSQL Activity: Session/load/QPS/TPS/locks for PGSQL cluster

PGSQL Activity

pgsql-activity.jpg

PGSQL Replication: Replication, slots, and pub/sub for PGSQL cluster

PGSQL Replication

pgsql-replication.jpg

PGSQL Databases: Database CRUD, slow queries, and table statistics across all instances

PGSQL Databases

pgsql-databases.jpg

PGSQL Patroni: HA status and Patroni component status for cluster

PGSQL Patroni

pgsql-patroni.jpg

PGSQL PITR: PITR context for point-in-time recovery assistance

PGSQL PITR

pgsql-patroni.jpg

Instance

PGSQL Instance: Main dashboard for a single PGSQL instance

PGSQL Instance

pgsql-instance.jpg

PGRDS Instance: RDS version of PGSQL Instance, focusing on PostgreSQL-native metrics

PGRDS Instance

pgrds-instance.jpg

PGSQL Proxy: Detailed metrics for a single HAProxy load balancer

PGSQL Proxy

pgsql-proxy.jpg

PGSQL Pgbouncer: Metrics overview for a single Pgbouncer connection pooler

PGSQL Pgbouncer

pgsql-pgbouncer.jpg

PGSQL Persist: Persistence metrics: WAL, XID, checkpoint, archive, IO

PGSQL Persist

pgsql-persist.jpg

PGSQL Xacts: Transaction, lock, TPS/QPS related metrics

PGSQL Xacts

pgsql-xacts.jpg

PGSQL Session: Session and active/idle time metrics for a single instance

PGSQL Session

pgsql-session.jpg

PGSQL Exporter: Self-monitoring metrics for Postgres/Pgbouncer exporters

PGSQL Exporter

pgsql-exporter.jpg

Database

PGSQL Database: Main dashboard for a single PGSQL database

PGSQL Database

pgsql-database.jpg

PGSQL Tables: Table/index access metrics within a single database

PGSQL Tables

pgsql-tables.jpg

PGSQL Table: Detailed info for a single table (QPS/RT/index/sequence…)

PGSQL Table

pgsql-table.jpg

PGSQL Query: Detailed info for a query type (QPS/RT)

PGSQL Query

pgsql-query.jpg

PGCAT

PGCAT Instance: Instance info retrieved directly from database catalog

PGCAT Instance

pgcat-instance.jpg

PGCAT Database: Database info retrieved directly from database catalog

PGCAT Database

pgcat-database.jpg

PGCAT Schema: Schema info from database catalog (tables/indexes/sequences…)

PGCAT Schema

pgcat-schema.jpg

PGCAT Table: Detailed table info from database catalog (stats/bloat…)

PGCAT Table

pgcat-table.jpg

PGCAT Query: Query details from database catalog (SQL/stats)

PGCAT Query

pgcat-query.jpg

PGCAT Locks: Activity and lock wait info from database catalog

PGCAT Locks

pgcat-locks.jpg

PGLOG

PGLOG Overview: Overview of CSV log samples in Pigsty CMDB

PGLOG Overview

pglog-overview.jpg

PGLOG Session: Log details for a single session in CSV log samples

PGLOG Session

pglog-session.jpg

Gallery

See pigsty/wiki/gallery for details.

PGSQL Overview

pgsql-overview.jpg

PGSQL Shard

pgsql-shard.jpg

PGSQL Cluster

pgsql-cluster.jpg

PGSQL Service

pgsql-service.jpg

PGSQL Activity

pgsql-activity.jpg

PGSQL Replication

pgsql-replication.jpg

PGSQL Databases

pgsql-databases.jpg

PGSQL Instance

pgsql-instance.jpg

PGSQL Proxy

pgsql-proxy.jpg

PGSQL Pgbouncer

pgsql-pgbouncer.jpg

PGSQL Session

pgsql-session.jpg

PGSQL Xacts

pgsql-xacts.jpg

PGSQL Persist

pgsql-persist.jpg

PGSQL Database

pgsql-database.jpg

PGSQL Tables

pgsql-tables.jpg

PGSQL Table

pgsql-table.jpg

PGSQL Query

pgsql-query.jpg

PGCAT Instance

pgcat-instance.jpg

PGCAT Database

pgcat-database.jpg

PGCAT Schema

pgcat-schema.jpg

PGCAT Table

pgcat-table.jpg

PGCAT Lock

pgcat-locks.jpg

PGCAT Query

pgcat-query.jpg

PGLOG Overview

pglog-overview.jpg

PGLOG Session

pglog-session.jpg

8.9.1 - Overview

PostgreSQL module global overview monitoring dashboards

PostgreSQL module global overview monitoring dashboards, including:

8.9.1.1 - PGSQL Overview

Main dashboard for the PGSQL module

Main dashboard for the PGSQL module: Demo

PGSQL Overview is the main dashboard for the PostgreSQL module, providing a global overview of the entire PGSQL module.

pgsql-overview

8.9.1.2 - PGSQL Alert

Global key metrics and alert events for PGSQL

Global key metrics and alert events for PGSQL: Demo

PGSQL Alert provides a global overview of core metrics and alert events for PostgreSQL clusters.

pgsql-alert

8.9.1.3 - PGSQL Shard

Overview of horizontally sharded PGSQL clusters

Overview of horizontally sharded PGSQL clusters: Demo

PGSQL Shard provides cross-shard metric comparison for horizontally sharded PGSQL clusters such as CITUS or GPSQL.

pgsql-shard

8.9.2 - Cluster

PostgreSQL cluster-level monitoring dashboards

PostgreSQL cluster-level monitoring dashboards, including:

  • PGSQL Cluster: Main dashboard for a PGSQL cluster
  • PGRDS Cluster: RDS version of PGSQL Cluster, focusing on PostgreSQL-native metrics
  • PGSQL Activity: Session/load/QPS/TPS/locks for PGSQL cluster
  • PGSQL Replication: Replication, slots, and pub/sub for PGSQL cluster
  • PGSQL Service: Service, proxy, routing, and load balancing for PGSQL cluster
  • PGSQL Databases: Database CRUD, slow queries, and table statistics across all instances
  • PGSQL Patroni: HA status and Patroni component status for cluster
  • PGSQL PITR: PITR context for point-in-time recovery assistance

8.9.2.1 - PGSQL Cluster

Main dashboard for a PGSQL cluster

Main dashboard for a PGSQL cluster: Demo

PGSQL Cluster is the main dashboard for a single PostgreSQL cluster, providing cluster-level core metrics overview.

pgsql-cluster

8.9.2.2 - PGRDS Cluster

RDS version of PGSQL Cluster focusing on PostgreSQL-native metrics

RDS version of PGSQL Cluster: Demo

PGRDS Cluster is the RDS version of PGSQL Cluster, focusing on PostgreSQL-native metrics without host-level metrics.

pgrds-cluster

8.9.2.3 - PGSQL Activity

Session/load/QPS/TPS/locks for PGSQL cluster

Session/load/QPS/TPS/locks for PGSQL cluster: Demo

PGSQL Activity focuses on session activity, load, QPS, TPS, and lock status for a PostgreSQL cluster.

pgsql-activity

8.9.2.4 - PGSQL Replication

Replication, slots, and pub/sub for PGSQL cluster

Replication, slots, and pub/sub for PGSQL cluster: Demo

PGSQL Replication focuses on replication status, replication slots, and logical replication (pub/sub) for a PostgreSQL cluster.

pgsql-replication

8.9.2.5 - PGSQL Service

Service, proxy, routing, and load balancing for PGSQL cluster

Service, proxy, routing, and load balancing for PGSQL cluster: Demo

PGSQL Service focuses on service endpoints, proxy routing, and load balancing status for a PostgreSQL cluster.

pgsql-service

8.9.2.6 - PGSQL Databases

Database CRUD, slow queries, and table statistics across all instances

Database CRUD, slow queries, and table statistics: Demo

PGSQL Databases focuses on database-level CRUD operations, slow queries, and table statistics across all instances in a cluster.

pgsql-databases

8.9.2.7 - PGSQL Patroni

HA status and Patroni component status for cluster

HA status and Patroni component status: Demo

PGSQL Patroni focuses on high-availability status and Patroni component health for a PostgreSQL cluster.

pgsql-patroni

8.9.2.8 - PGSQL PITR

PITR context for point-in-time recovery assistance

PITR context for point-in-time recovery: Demo

PGSQL PITR provides context information for point-in-time recovery operations, showing backup status and WAL timeline.

pgsql-pitr

8.9.3 - Instance

PostgreSQL instance-level monitoring dashboards

PostgreSQL instance-level monitoring dashboards, including:

  • PGSQL Instance: Main dashboard for a single PGSQL instance
  • PGRDS Instance: RDS version of PGSQL Instance, focusing on PostgreSQL-native metrics
  • PGCAT Instance: Instance info retrieved directly from database catalog
  • PGSQL Persist: Persistence metrics: WAL, XID, checkpoint, archive, IO
  • PGSQL Proxy: Detailed metrics for a single HAProxy load balancer
  • PGSQL Pgbouncer: Metrics overview for a single Pgbouncer connection pooler
  • PGSQL Session: Session and active/idle time metrics for a single instance
  • PGSQL Xacts: Transaction, lock, TPS/QPS related metrics
  • PGSQL Exporter: Self-monitoring metrics for Postgres and Pgbouncer exporters

8.9.3.1 - PGSQL Instance

Main dashboard for a single PGSQL instance

Main dashboard for a single PGSQL instance: Demo

PGSQL Instance is the main dashboard for a single PostgreSQL instance, providing comprehensive instance-level metrics.

pgsql-instance

8.9.3.2 - PGRDS Instance

RDS version of PGSQL Instance focusing on PostgreSQL-native metrics

RDS version of PGSQL Instance: Demo

PGRDS Instance is the RDS version of PGSQL Instance, focusing on PostgreSQL-native metrics without host-level metrics.

pgrds-instance

8.9.3.3 - PGCAT Instance

Instance info retrieved directly from database catalog

Instance info from database catalog: Demo

PGCAT Instance shows instance-level information retrieved directly from PostgreSQL system catalog.

pgcat-instance

8.9.3.4 - PGSQL Persist

Persistence metrics - WAL, XID, checkpoint, archive, IO

Persistence metrics for PGSQL instance: Demo

PGSQL Persist focuses on persistence-related metrics: WAL generation, XID consumption, checkpoints, archiving, and I/O patterns.

pgsql-persist

8.9.3.5 - PGSQL Proxy

Detailed metrics for a single HAProxy load balancer

Detailed metrics for HAProxy: Demo

PGSQL Proxy shows detailed metrics for a single HAProxy load balancer instance serving PostgreSQL traffic.

pgsql-proxy

8.9.3.6 - PGSQL Pgbouncer

Metrics overview for a single Pgbouncer connection pooler

Metrics overview for Pgbouncer: Demo

PGSQL Pgbouncer shows connection pooling metrics for a single Pgbouncer instance.

pgsql-pgbouncer

8.9.3.7 - PGSQL Session

Session and active/idle time metrics for a single instance

Session and active/idle time metrics: Demo

PGSQL Session focuses on session statistics and active/idle time distribution for a single PostgreSQL instance.

pgsql-session

8.9.3.8 - PGSQL Xacts

Transaction, lock, TPS/QPS related metrics

Transaction, lock, TPS/QPS metrics: Demo

PGSQL Xacts focuses on transaction processing, lock activity, and TPS/QPS metrics for a single PostgreSQL instance.

pgsql-xacts

8.9.3.9 - PGSQL Exporter

Self-monitoring metrics for Postgres and Pgbouncer exporters

Self-monitoring metrics for exporters: Demo

PGSQL Exporter shows self-monitoring metrics for the Postgres exporter and Pgbouncer exporter components.

pgsql-exporter

8.9.4 - Database

PostgreSQL database-level monitoring dashboards

PostgreSQL database-level monitoring dashboards, including:

  • PGSQL Database: Main dashboard for a single PGSQL database
  • PGCAT Database: Database info retrieved directly from database catalog
  • PGSQL Tables: Table/index access metrics within a single database
  • PGSQL Table: Detailed info for a single table (QPS/RT/index/sequence…)
  • PGCAT Table: Detailed table info from database catalog
  • PGSQL Query: Detailed info for a query type (QPS/RT)
  • PGCAT Query: Query details from database catalog
  • PGCAT Locks: Activity and lock wait info from database catalog
  • PGCAT Schema: Schema info from database catalog

8.9.4.1 - PGSQL Database

Main dashboard for a single PGSQL database

Main dashboard for a single PGSQL database: Demo

PGSQL Database is the main dashboard for a single PostgreSQL database, providing comprehensive database-level metrics.

pgsql-database

8.9.4.2 - PGCAT Database

Database info retrieved directly from database catalog

Database info from database catalog: Demo

PGCAT Database shows database-level information retrieved directly from PostgreSQL system catalog.

pgcat-database

8.9.4.3 - PGSQL Tables

Table/index access metrics within a single database

Table/index access metrics: Demo

PGSQL Tables shows table and index access metrics for all objects within a single PostgreSQL database.

pgsql-tables

8.9.4.4 - PGSQL Table

Detailed info for a single table (QPS/RT/index/sequence)

Detailed info for a single table: Demo

PGSQL Table shows detailed metrics for a single table including QPS, response time, index usage, and sequence info.

pgsql-table

8.9.4.5 - PGCAT Table

Detailed table info from database catalog

Detailed table info from catalog: Demo

PGCAT Table shows detailed table information from database catalog including statistics and bloat analysis.

pgcat-table

8.9.4.6 - PGSQL Query

Detailed info for a query type (QPS/RT)

Detailed info for a query type: Demo

PGSQL Query shows detailed metrics for a specific query type including QPS and response time distribution.

pgsql-query

8.9.4.7 - PGCAT Query

Query details from database catalog

Query details from database catalog: Demo

PGCAT Query shows query details from database catalog including SQL text and execution statistics.

pgcat-query

8.9.4.8 - PGCAT Locks

Activity and lock wait info from database catalog

Activity and lock wait info: Demo

PGCAT Locks shows active sessions and lock wait information from database catalog.

pgcat-locks

8.9.4.9 - PGCAT Schema

Schema info from database catalog

Schema info from database catalog: Demo

PGCAT Schema shows schema-level information from database catalog including tables, indexes, and sequences.

pgcat-schema

8.10 - Metrics

Complete monitoring metrics reference for the Pigsty PGSQL module

The PGSQL module provides 638 available monitoring metrics.

Metric Name Type Labels Description
ALERTS Unknown category, job, level, ins, severity, ip, alertname, alertstate, instance, cls N/A
ALERTS_FOR_STATE Unknown category, job, level, ins, severity, ip, alertname, instance, cls N/A
cls:pressure1 Unknown job, cls N/A
cls:pressure15 Unknown job, cls N/A
cls:pressure5 Unknown job, cls N/A
go_gc_duration_seconds summary job, ins, ip, instance, quantile, cls A summary of the pause duration of garbage collection cycles.
go_gc_duration_seconds_count Unknown job, ins, ip, instance, cls N/A
go_gc_duration_seconds_sum Unknown job, ins, ip, instance, cls N/A
go_goroutines gauge job, ins, ip, instance, cls Number of goroutines that currently exist.
go_info gauge version, job, ins, ip, instance, cls Information about the Go environment.
go_memstats_alloc_bytes gauge job, ins, ip, instance, cls Number of bytes allocated and still in use.
go_memstats_alloc_bytes_total counter job, ins, ip, instance, cls Total number of bytes allocated, even if freed.
go_memstats_buck_hash_sys_bytes gauge job, ins, ip, instance, cls Number of bytes used by the profiling bucket hash table.
go_memstats_frees_total counter job, ins, ip, instance, cls Total number of frees.
go_memstats_gc_sys_bytes gauge job, ins, ip, instance, cls Number of bytes used for garbage collection system metadata.
go_memstats_heap_alloc_bytes gauge job, ins, ip, instance, cls Number of heap bytes allocated and still in use.
go_memstats_heap_idle_bytes gauge job, ins, ip, instance, cls Number of heap bytes waiting to be used.
go_memstats_heap_inuse_bytes gauge job, ins, ip, instance, cls Number of heap bytes that are in use.
go_memstats_heap_objects gauge job, ins, ip, instance, cls Number of allocated objects.
go_memstats_heap_released_bytes gauge job, ins, ip, instance, cls Number of heap bytes released to OS.
go_memstats_heap_sys_bytes gauge job, ins, ip, instance, cls Number of heap bytes obtained from system.
go_memstats_last_gc_time_seconds gauge job, ins, ip, instance, cls Number of seconds since 1970 of last garbage collection.
go_memstats_lookups_total counter job, ins, ip, instance, cls Total number of pointer lookups.
go_memstats_mallocs_total counter job, ins, ip, instance, cls Total number of mallocs.
go_memstats_mcache_inuse_bytes gauge job, ins, ip, instance, cls Number of bytes in use by mcache structures.
go_memstats_mcache_sys_bytes gauge job, ins, ip, instance, cls Number of bytes used for mcache structures obtained from system.
go_memstats_mspan_inuse_bytes gauge job, ins, ip, instance, cls Number of bytes in use by mspan structures.
go_memstats_mspan_sys_bytes gauge job, ins, ip, instance, cls Number of bytes used for mspan structures obtained from system.
go_memstats_next_gc_bytes gauge job, ins, ip, instance, cls Number of heap bytes when next garbage collection will take place.
go_memstats_other_sys_bytes gauge job, ins, ip, instance, cls Number of bytes used for other system allocations.
go_memstats_stack_inuse_bytes gauge job, ins, ip, instance, cls Number of bytes in use by the stack allocator.
go_memstats_stack_sys_bytes gauge job, ins, ip, instance, cls Number of bytes obtained from system for stack allocator.
go_memstats_sys_bytes gauge job, ins, ip, instance, cls Number of bytes obtained from system.
go_threads gauge job, ins, ip, instance, cls Number of OS threads created.
ins:pressure1 Unknown job, ins, ip, cls N/A
ins:pressure15 Unknown job, ins, ip, cls N/A
ins:pressure5 Unknown job, ins, ip, cls N/A
patroni_cluster_unlocked gauge job, ins, ip, instance, cls, scope Value is 1 if the cluster is unlocked, 0 if locked.
patroni_dcs_last_seen gauge job, ins, ip, instance, cls, scope Epoch timestamp when DCS was last contacted successfully by Patroni.
patroni_failsafe_mode_is_active gauge job, ins, ip, instance, cls, scope Value is 1 if failsafe mode is active, 0 if inactive.
patroni_is_paused gauge job, ins, ip, instance, cls, scope Value is 1 if auto failover is disabled, 0 otherwise.
patroni_master gauge job, ins, ip, instance, cls, scope Value is 1 if this node is the leader, 0 otherwise.
patroni_pending_restart gauge job, ins, ip, instance, cls, scope Value is 1 if the node needs a restart, 0 otherwise.
patroni_postgres_in_archive_recovery gauge job, ins, ip, instance, cls, scope Value is 1 if Postgres is replicating from archive, 0 otherwise.
patroni_postgres_running gauge job, ins, ip, instance, cls, scope Value is 1 if Postgres is running, 0 otherwise.
patroni_postgres_server_version gauge job, ins, ip, instance, cls, scope Version of Postgres (if running), 0 otherwise.
patroni_postgres_streaming gauge job, ins, ip, instance, cls, scope Value is 1 if Postgres is streaming, 0 otherwise.
patroni_postgres_timeline counter job, ins, ip, instance, cls, scope Postgres timeline of this node (if running), 0 otherwise.
patroni_postmaster_start_time gauge job, ins, ip, instance, cls, scope Epoch seconds since Postgres started.
patroni_primary gauge job, ins, ip, instance, cls, scope Value is 1 if this node is the leader, 0 otherwise.
patroni_replica gauge job, ins, ip, instance, cls, scope Value is 1 if this node is a replica, 0 otherwise.
patroni_standby_leader gauge job, ins, ip, instance, cls, scope Value is 1 if this node is the standby_leader, 0 otherwise.
patroni_sync_standby gauge job, ins, ip, instance, cls, scope Value is 1 if this node is a sync standby replica, 0 otherwise.
patroni_up Unknown job, ins, ip, instance, cls N/A
patroni_version gauge job, ins, ip, instance, cls, scope Patroni semver without periods.
patroni_xlog_location counter job, ins, ip, instance, cls, scope Current location of the Postgres transaction log, 0 if this node is not the leader.
patroni_xlog_paused gauge job, ins, ip, instance, cls, scope Value is 1 if the Postgres xlog is paused, 0 otherwise.
patroni_xlog_received_location counter job, ins, ip, instance, cls, scope Current location of the received Postgres transaction log, 0 if this node is not a replica.
patroni_xlog_replayed_location counter job, ins, ip, instance, cls, scope Current location of the replayed Postgres transaction log, 0 if this node is not a replica.
patroni_xlog_replayed_timestamp gauge job, ins, ip, instance, cls, scope Current timestamp of the replayed Postgres transaction log, 0 if null.
pg:cls:active_backends Unknown job, cls N/A
pg:cls:active_time_rate15m Unknown job, cls N/A
pg:cls:active_time_rate1m Unknown job, cls N/A
pg:cls:active_time_rate5m Unknown job, cls N/A
pg:cls:age Unknown job, cls N/A
pg:cls:buf_alloc_rate1m Unknown job, cls N/A
pg:cls:buf_clean_rate1m Unknown job, cls N/A
pg:cls:buf_flush_backend_rate1m Unknown job, cls N/A
pg:cls:buf_flush_checkpoint_rate1m Unknown job, cls N/A
pg:cls:cpu_count Unknown job, cls N/A
pg:cls:cpu_usage Unknown job, cls N/A
pg:cls:cpu_usage_15m Unknown job, cls N/A
pg:cls:cpu_usage_1m Unknown job, cls N/A
pg:cls:cpu_usage_5m Unknown job, cls N/A
pg:cls:db_size Unknown job, cls N/A
pg:cls:file_size Unknown job, cls N/A
pg:cls:ixact_backends Unknown job, cls N/A
pg:cls:ixact_time_rate1m Unknown job, cls N/A
pg:cls:lag_bytes Unknown job, cls N/A
pg:cls:lag_seconds Unknown job, cls N/A
pg:cls:leader Unknown job, ins, ip, instance, cls N/A
pg:cls:load1 Unknown job, cls N/A
pg:cls:load15 Unknown job, cls N/A
pg:cls:load5 Unknown job, cls N/A
pg:cls:lock_count Unknown job, cls N/A
pg:cls:locks Unknown job, cls, mode N/A
pg:cls:log_size Unknown job, cls N/A
pg:cls:lsn_rate1m Unknown job, cls N/A
pg:cls:members Unknown job, ins, ip, cls N/A
pg:cls:num_backends Unknown job, cls N/A
pg:cls:partition Unknown job, cls N/A
pg:cls:receiver Unknown state, slot_name, job, appname, ip, cls, sender_host, sender_port N/A
pg:cls:rlock_count Unknown job, cls N/A
pg:cls:saturation1 Unknown job, cls N/A
pg:cls:saturation15 Unknown job, cls N/A
pg:cls:saturation5 Unknown job, cls N/A
pg:cls:sender Unknown pid, usename, address, job, ins, appname, ip, cls N/A
pg:cls:session_time_rate1m Unknown job, cls N/A
pg:cls:size Unknown job, cls N/A
pg:cls:slot_count Unknown job, cls N/A
pg:cls:slot_retained_bytes Unknown job, cls N/A
pg:cls:standby_count Unknown job, cls N/A
pg:cls:sync_state Unknown job, cls N/A
pg:cls:timeline Unknown job, cls N/A
pg:cls:tup_deleted_rate1m Unknown job, cls N/A
pg:cls:tup_fetched_rate1m Unknown job, cls N/A
pg:cls:tup_inserted_rate1m Unknown job, cls N/A
pg:cls:tup_modified_rate1m Unknown job, cls N/A
pg:cls:tup_returned_rate1m Unknown job, cls N/A
pg:cls:wal_size Unknown job, cls N/A
pg:cls:xact_commit_rate15m Unknown job, cls N/A
pg:cls:xact_commit_rate1m Unknown job, cls N/A
pg:cls:xact_commit_rate5m Unknown job, cls N/A
pg:cls:xact_rollback_rate15m Unknown job, cls N/A
pg:cls:xact_rollback_rate1m Unknown job, cls N/A
pg:cls:xact_rollback_rate5m Unknown job, cls N/A
pg:cls:xact_total_rate15m Unknown job, cls N/A
pg:cls:xact_total_rate1m Unknown job, cls N/A
pg:cls:xact_total_sigma15m Unknown job, cls N/A
pg:cls:xlock_count Unknown job, cls N/A
pg:db:active_backends Unknown datname, job, ins, ip, instance, cls N/A
pg:db:active_time_rate15m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:active_time_rate1m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:active_time_rate5m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:age Unknown datname, job, ins, ip, instance, cls N/A
pg:db:age_deriv1h Unknown datname, job, ins, ip, instance, cls N/A
pg:db:age_exhaust Unknown datname, job, ins, ip, instance, cls N/A
pg:db:blk_io_time_seconds_rate1m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:blk_read_time_seconds_rate1m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:blk_write_time_seconds_rate1m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:blks_access_1m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:blks_hit_1m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:blks_hit_ratio1m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:blks_read_1m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:conn_limit Unknown datname, job, ins, ip, instance, cls N/A
pg:db:conn_usage Unknown datname, job, ins, ip, instance, cls N/A
pg:db:db_size Unknown datname, job, ins, ip, instance, cls N/A
pg:db:ixact_backends Unknown datname, job, ins, ip, instance, cls N/A
pg:db:ixact_time_rate1m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:lock_count Unknown datname, job, ins, ip, instance, cls N/A
pg:db:num_backends Unknown datname, job, ins, ip, instance, cls N/A
pg:db:rlock_count Unknown datname, job, ins, ip, instance, cls N/A
pg:db:session_time_rate1m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:temp_bytes_rate1m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:temp_files_1m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:tup_deleted_rate1m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:tup_fetched_rate1m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:tup_inserted_rate1m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:tup_modified_rate1m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:tup_returned_rate1m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:wlock_count Unknown datname, job, ins, ip, instance, cls N/A
pg:db:xact_commit_rate15m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:xact_commit_rate1m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:xact_commit_rate5m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:xact_rollback_rate15m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:xact_rollback_rate1m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:xact_rollback_rate5m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:xact_total_rate15m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:xact_total_rate1m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:xact_total_rate5m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:xact_total_sigma15m Unknown datname, job, ins, ip, instance, cls N/A
pg:db:xlock_count Unknown datname, job, ins, ip, instance, cls N/A
pg:env:active_backends Unknown job N/A
pg:env:active_time_rate15m Unknown job N/A
pg:env:active_time_rate1m Unknown job N/A
pg:env:active_time_rate5m Unknown job N/A
pg:env:age Unknown job N/A
pg:env:cpu_count Unknown job N/A
pg:env:cpu_usage Unknown job N/A
pg:env:cpu_usage_15m Unknown job N/A
pg:env:cpu_usage_1m Unknown job N/A
pg:env:cpu_usage_5m Unknown job N/A
pg:env:ixact_backends Unknown job N/A
pg:env:ixact_time_rate1m Unknown job N/A
pg:env:lag_bytes Unknown job N/A
pg:env:lag_seconds Unknown job N/A
pg:env:lsn_rate1m Unknown job N/A
pg:env:session_time_rate1m Unknown job N/A
pg:env:tup_deleted_rate1m Unknown job N/A
pg:env:tup_fetched_rate1m Unknown job N/A
pg:env:tup_inserted_rate1m Unknown job N/A
pg:env:tup_modified_rate1m Unknown job N/A
pg:env:tup_returned_rate1m Unknown job N/A
pg:env:xact_commit_rate15m Unknown job N/A
pg:env:xact_commit_rate1m Unknown job N/A
pg:env:xact_commit_rate5m Unknown job N/A
pg:env:xact_rollback_rate15m Unknown job N/A
pg:env:xact_rollback_rate1m Unknown job N/A
pg:env:xact_rollback_rate5m Unknown job N/A
pg:env:xact_total_rate15m Unknown job N/A
pg:env:xact_total_rate1m Unknown job N/A
pg:env:xact_total_sigma15m Unknown job N/A
pg:ins:active_backends Unknown job, ins, ip, instance, cls N/A
pg:ins:active_time_rate15m Unknown job, ins, ip, instance, cls N/A
pg:ins:active_time_rate1m Unknown job, ins, ip, instance, cls N/A
pg:ins:active_time_rate5m Unknown job, ins, ip, instance, cls N/A
pg:ins:age Unknown job, ins, ip, instance, cls N/A
pg:ins:blks_hit_ratio1m Unknown job, ins, ip, instance, cls N/A
pg:ins:buf_alloc_rate1m Unknown job, ins, ip, instance, cls N/A
pg:ins:buf_clean_rate1m Unknown job, ins, ip, instance, cls N/A
pg:ins:buf_flush_backend_rate1m Unknown job, ins, ip, instance, cls N/A
pg:ins:buf_flush_checkpoint_rate1m Unknown job, ins, ip, instance, cls N/A
pg:ins:ckpt_1h Unknown job, ins, ip, instance, cls N/A
pg:ins:ckpt_req_1m Unknown job, ins, ip, instance, cls N/A
pg:ins:ckpt_timed_1m Unknown job, ins, ip, instance, cls N/A
pg:ins:conn_limit Unknown job, ins, ip, instance, cls N/A
pg:ins:conn_usage Unknown job, ins, ip, instance, cls N/A
pg:ins:cpu_count Unknown job, ins, ip, instance, cls N/A
pg:ins:cpu_usage Unknown job, ins, ip, instance, cls N/A
pg:ins:cpu_usage_15m Unknown job, ins, ip, instance, cls N/A
pg:ins:cpu_usage_1m Unknown job, ins, ip, instance, cls N/A
pg:ins:cpu_usage_5m Unknown job, ins, ip, instance, cls N/A
pg:ins:db_size Unknown job, ins, ip, instance, cls N/A
pg:ins:file_size Unknown job, ins, ip, instance, cls N/A
pg:ins:fs_size Unknown job, ins, ip, instance, cls N/A
pg:ins:is_leader Unknown job, ins, ip, instance, cls N/A
pg:ins:ixact_backends Unknown job, ins, ip, instance, cls N/A
pg:ins:ixact_time_rate1m Unknown job, ins, ip, instance, cls N/A
pg:ins:lag_bytes Unknown job, ins, ip, instance, cls N/A
pg:ins:lag_seconds Unknown job, ins, ip, instance, cls N/A
pg:ins:load1 Unknown job, ins, ip, instance, cls N/A
pg:ins:load15 Unknown job, ins, ip, instance, cls N/A
pg:ins:load5 Unknown job, ins, ip, instance, cls N/A
pg:ins:lock_count Unknown job, ins, ip, instance, cls N/A
pg:ins:locks Unknown job, ins, ip, mode, instance, cls N/A
pg:ins:log_size Unknown job, ins, ip, instance, cls N/A
pg:ins:lsn_rate1m Unknown job, ins, ip, instance, cls N/A
pg:ins:mem_size Unknown job, ins, ip, instance, cls N/A
pg:ins:num_backends Unknown job, ins, ip, instance, cls N/A
pg:ins:rlock_count Unknown job, ins, ip, instance, cls N/A
pg:ins:saturation1 Unknown job, ins, ip, cls N/A
pg:ins:saturation15 Unknown job, ins, ip, cls N/A
pg:ins:saturation5 Unknown job, ins, ip, cls N/A
pg:ins:session_time_rate1m Unknown job, ins, ip, instance, cls N/A
pg:ins:slot_retained_bytes Unknown job, ins, ip, instance, cls N/A
pg:ins:space_usage Unknown job, ins, ip, instance, cls N/A
pg:ins:status Unknown job, ins, ip, instance, cls N/A
pg:ins:sync_state Unknown job, ins, instance, cls N/A
pg:ins:target_count Unknown job, cls, ins N/A
pg:ins:timeline Unknown job, ins, ip, instance, cls N/A
pg:ins:tup_deleted_rate1m Unknown job, ins, ip, instance, cls N/A
pg:ins:tup_fetched_rate1m Unknown job, ins, ip, instance, cls N/A
pg:ins:tup_inserted_rate1m Unknown job, ins, ip, instance, cls N/A
pg:ins:tup_modified_rate1m Unknown job, ins, ip, instance, cls N/A
pg:ins:tup_returned_rate1m Unknown job, ins, ip, instance, cls N/A
pg:ins:wal_size Unknown job, ins, ip, instance, cls N/A
pg:ins:wlock_count Unknown job, ins, ip, instance, cls N/A
pg:ins:xact_commit_rate15m Unknown job, ins, ip, instance, cls N/A
pg:ins:xact_commit_rate1m Unknown job, ins, ip, instance, cls N/A
pg:ins:xact_commit_rate5m Unknown job, ins, ip, instance, cls N/A
pg:ins:xact_rollback_rate15m Unknown job, ins, ip, instance, cls N/A
pg:ins:xact_rollback_rate1m Unknown job, ins, ip, instance, cls N/A
pg:ins:xact_rollback_rate5m Unknown job, ins, ip, instance, cls N/A
pg:ins:xact_total_rate15m Unknown job, ins, ip, instance, cls N/A
pg:ins:xact_total_rate1m Unknown job, ins, ip, instance, cls N/A
pg:ins:xact_total_rate5m Unknown job, ins, ip, instance, cls N/A
pg:ins:xact_total_sigma15m Unknown job, ins, ip, instance, cls N/A
pg:ins:xlock_count Unknown job, ins, ip, instance, cls N/A
pg:query:call_rate1m Unknown datname, query, job, ins, ip, instance, cls N/A
pg:query:rt_1m Unknown datname, query, job, ins, ip, instance, cls N/A
pg:table:scan_rate1m Unknown datname, relname, job, ins, ip, instance, cls N/A
pg_activity_count gauge datname, state, job, ins, ip, instance, cls Count of connection among (datname,state)
pg_activity_max_conn_duration gauge datname, state, job, ins, ip, instance, cls Max backend session duration since state change among (datname, state)
pg_activity_max_duration gauge datname, state, job, ins, ip, instance, cls Max duration since last state change among (datname, state)
pg_activity_max_tx_duration gauge datname, state, job, ins, ip, instance, cls Max transaction duration since state change among (datname, state)
pg_archiver_failed_count counter job, ins, ip, instance, cls Number of failed attempts for archiving WAL files
pg_archiver_finish_count counter job, ins, ip, instance, cls Number of WAL files that have been successfully archived
pg_archiver_last_failed_time counter job, ins, ip, instance, cls Time of the last failed archival operation
pg_archiver_last_finish_time counter job, ins, ip, instance, cls Time of the last successful archive operation
pg_archiver_reset_time gauge job, ins, ip, instance, cls Time at which archive statistics were last reset
pg_backend_count gauge type, job, ins, ip, instance, cls Database backend process count by backend_type
pg_bgwriter_buffers_alloc counter job, ins, ip, instance, cls Number of buffers allocated
pg_bgwriter_buffers_backend counter job, ins, ip, instance, cls Number of buffers written directly by a backend
pg_bgwriter_buffers_backend_fsync counter job, ins, ip, instance, cls Number of times a backend had to execute its own fsync call
pg_bgwriter_buffers_checkpoint counter job, ins, ip, instance, cls Number of buffers written during checkpoints
pg_bgwriter_buffers_clean counter job, ins, ip, instance, cls Number of buffers written by the background writer
pg_bgwriter_checkpoint_sync_time counter job, ins, ip, instance, cls Total amount of time that has been spent in the portion of checkpoint processing where files are synchronized to disk, in seconds
pg_bgwriter_checkpoint_write_time counter job, ins, ip, instance, cls Total amount of time that has been spent in the portion of checkpoint processing where files are written to disk, in seconds
pg_bgwriter_checkpoints_req counter job, ins, ip, instance, cls Number of requested checkpoints that have been performed
pg_bgwriter_checkpoints_timed counter job, ins, ip, instance, cls Number of scheduled checkpoints that have been performed
pg_bgwriter_maxwritten_clean counter job, ins, ip, instance, cls Number of times the background writer stopped a cleaning scan because it had written too many buffers
pg_bgwriter_reset_time counter job, ins, ip, instance, cls Time at which bgwriter statistics were last reset
pg_boot_time gauge job, ins, ip, instance, cls unix timestamp when postmaster boot
pg_checkpoint_checkpoint_lsn counter job, ins, ip, instance, cls Latest checkpoint location
pg_checkpoint_elapse gauge job, ins, ip, instance, cls Seconds elapsed since latest checkpoint in seconds
pg_checkpoint_full_page_writes gauge job, ins, ip, instance, cls Latest checkpoint’s full_page_writes enabled
pg_checkpoint_newest_commit_ts_xid counter job, ins, ip, instance, cls Latest checkpoint’s newestCommitTsXid
pg_checkpoint_next_multi_offset counter job, ins, ip, instance, cls Latest checkpoint’s NextMultiOffset
pg_checkpoint_next_multixact_id counter job, ins, ip, instance, cls Latest checkpoint’s NextMultiXactId
pg_checkpoint_next_oid counter job, ins, ip, instance, cls Latest checkpoint’s NextOID
pg_checkpoint_next_xid counter job, ins, ip, instance, cls Latest checkpoint’s NextXID xid
pg_checkpoint_next_xid_epoch counter job, ins, ip, instance, cls Latest checkpoint’s NextXID epoch
pg_checkpoint_oldest_active_xid counter job, ins, ip, instance, cls Latest checkpoint’s oldestActiveXID
pg_checkpoint_oldest_commit_ts_xid counter job, ins, ip, instance, cls Latest checkpoint’s oldestCommitTsXid
pg_checkpoint_oldest_multi_dbid gauge job, ins, ip, instance, cls Latest checkpoint’s oldestMulti’s DB OID
pg_checkpoint_oldest_multi_xid counter job, ins, ip, instance, cls Latest checkpoint’s oldestMultiXid
pg_checkpoint_oldest_xid counter job, ins, ip, instance, cls Latest checkpoint’s oldestXID
pg_checkpoint_oldest_xid_dbid gauge job, ins, ip, instance, cls Latest checkpoint’s oldestXID’s DB OID
pg_checkpoint_prev_tli counter job, ins, ip, instance, cls Latest checkpoint’s PrevTimeLineID
pg_checkpoint_redo_lsn counter job, ins, ip, instance, cls Latest checkpoint’s REDO location
pg_checkpoint_time counter job, ins, ip, instance, cls Time of latest checkpoint
pg_checkpoint_tli counter job, ins, ip, instance, cls Latest checkpoint’s TimeLineID
pg_conf_reload_time gauge job, ins, ip, instance, cls seconds since last configuration reload
pg_db_active_time counter datname, job, ins, ip, instance, cls Time spent executing SQL statements in this database, in seconds
pg_db_age gauge datname, job, ins, ip, instance, cls Age of database calculated from datfrozenxid
pg_db_allow_conn gauge datname, job, ins, ip, instance, cls If false(0) then no one can connect to this database.
pg_db_blk_read_time counter datname, job, ins, ip, instance, cls Time spent reading data file blocks by backends in this database, in seconds
pg_db_blk_write_time counter datname, job, ins, ip, instance, cls Time spent writing data file blocks by backends in this database, in seconds
pg_db_blks_access counter datname, job, ins, ip, instance, cls Number of times disk blocks that accessed read+hit
pg_db_blks_hit counter datname, job, ins, ip, instance, cls Number of times disk blocks were found already in the buffer cache
pg_db_blks_read counter datname, job, ins, ip, instance, cls Number of disk blocks read in this database
pg_db_cks_fail_time gauge datname, job, ins, ip, instance, cls Time at which the last data page checksum failure was detected in this database
pg_db_cks_fails counter datname, job, ins, ip, instance, cls Number of data page checksum failures detected in this database, -1 for not enabled
pg_db_confl_confl_bufferpin counter datname, job, ins, ip, instance, cls Number of queries in this database that have been canceled due to pinned buffers
pg_db_confl_confl_deadlock counter datname, job, ins, ip, instance, cls Number of queries in this database that have been canceled due to deadlocks
pg_db_confl_confl_lock counter datname, job, ins, ip, instance, cls Number of queries in this database that have been canceled due to lock timeouts
pg_db_confl_confl_snapshot counter datname, job, ins, ip, instance, cls Number of queries in this database that have been canceled due to old snapshots
pg_db_confl_confl_tablespace counter datname, job, ins, ip, instance, cls Number of queries in this database that have been canceled due to dropped tablespaces
pg_db_conflicts counter datname, job, ins, ip, instance, cls Number of queries canceled due to conflicts with recovery in this database
pg_db_conn_limit gauge datname, job, ins, ip, instance, cls Sets maximum number of concurrent connections that can be made to this database. -1 means no limit.
pg_db_datid gauge datname, job, ins, ip, instance, cls OID of the database
pg_db_deadlocks counter datname, job, ins, ip, instance, cls Number of deadlocks detected in this database
pg_db_frozen_xid gauge datname, job, ins, ip, instance, cls All transaction IDs before this one have been frozened
pg_db_is_template gauge datname, job, ins, ip, instance, cls If true(1), then this database can be cloned by any user with CREATEDB privileges
pg_db_ixact_time counter datname, job, ins, ip, instance, cls Time spent idling while in a transaction in this database, in seconds
pg_db_numbackends gauge datname, job, ins, ip, instance, cls Number of backends currently connected to this database
pg_db_reset_time counter datname, job, ins, ip, instance, cls Time at which database statistics were last reset
pg_db_session_time counter datname, job, ins, ip, instance, cls Time spent by database sessions in this database, in seconds
pg_db_sessions counter datname, job, ins, ip, instance, cls Total number of sessions established to this database
pg_db_sessions_abandoned counter datname, job, ins, ip, instance, cls Number of database sessions to this database that were terminated because connection to the client was lost
pg_db_sessions_fatal counter datname, job, ins, ip, instance, cls Number of database sessions to this database that were terminated by fatal errors
pg_db_sessions_killed counter datname, job, ins, ip, instance, cls Number of database sessions to this database that were terminated by operator intervention
pg_db_temp_bytes counter datname, job, ins, ip, instance, cls Total amount of data written to temporary files by queries in this database.
pg_db_temp_files counter datname, job, ins, ip, instance, cls Number of temporary files created by queries in this database
pg_db_tup_deleted counter datname, job, ins, ip, instance, cls Number of rows deleted by queries in this database
pg_db_tup_fetched counter datname, job, ins, ip, instance, cls Number of rows fetched by queries in this database
pg_db_tup_inserted counter datname, job, ins, ip, instance, cls Number of rows inserted by queries in this database
pg_db_tup_modified counter datname, job, ins, ip, instance, cls Number of rows modified by queries in this database
pg_db_tup_returned counter datname, job, ins, ip, instance, cls Number of rows returned by queries in this database
pg_db_tup_updated counter datname, job, ins, ip, instance, cls Number of rows updated by queries in this database
pg_db_xact_commit counter datname, job, ins, ip, instance, cls Number of transactions in this database that have been committed
pg_db_xact_rollback counter datname, job, ins, ip, instance, cls Number of transactions in this database that have been rolled back
pg_db_xact_total counter datname, job, ins, ip, instance, cls Number of transactions in this database
pg_downstream_count gauge state, job, ins, ip, instance, cls Count of corresponding state
pg_exporter_agent_up Unknown job, ins, ip, instance, cls N/A
pg_exporter_last_scrape_time gauge job, ins, ip, instance, cls seconds exporter spending on scrapping
pg_exporter_query_cache_ttl gauge datname, query, job, ins, ip, instance, cls times to live of query cache
pg_exporter_query_scrape_duration gauge datname, query, job, ins, ip, instance, cls seconds query spending on scrapping
pg_exporter_query_scrape_error_count gauge datname, query, job, ins, ip, instance, cls times the query failed
pg_exporter_query_scrape_hit_count gauge datname, query, job, ins, ip, instance, cls numbers been scrapped from this query
pg_exporter_query_scrape_metric_count gauge datname, query, job, ins, ip, instance, cls numbers of metrics been scrapped from this query
pg_exporter_query_scrape_total_count gauge datname, query, job, ins, ip, instance, cls times exporter server was scraped for metrics
pg_exporter_scrape_duration gauge job, ins, ip, instance, cls seconds exporter spending on scrapping
pg_exporter_scrape_error_count counter job, ins, ip, instance, cls times exporter was scraped for metrics and failed
pg_exporter_scrape_total_count counter job, ins, ip, instance, cls times exporter was scraped for metrics
pg_exporter_server_scrape_duration gauge datname, job, ins, ip, instance, cls seconds exporter server spending on scrapping
pg_exporter_server_scrape_error_count Unknown datname, job, ins, ip, instance, cls N/A
pg_exporter_server_scrape_total_count gauge datname, job, ins, ip, instance, cls times exporter server was scraped for metrics
pg_exporter_server_scrape_total_seconds gauge datname, job, ins, ip, instance, cls seconds exporter server spending on scrapping
pg_exporter_up gauge job, ins, ip, instance, cls always be 1 if your could retrieve metrics
pg_exporter_uptime gauge job, ins, ip, instance, cls seconds since exporter primary server inited
pg_flush_lsn counter job, ins, ip, instance, cls primary only, location of current wal syncing
pg_func_calls counter datname, funcname, job, ins, ip, instance, cls Number of times this function has been called
pg_func_self_time counter datname, funcname, job, ins, ip, instance, cls Total time spent in this function itself, not including other functions called by it, in ms
pg_func_total_time counter datname, funcname, job, ins, ip, instance, cls Total time spent in this function and all other functions called by it, in ms
pg_in_recovery gauge job, ins, ip, instance, cls server is in recovery mode? 1 for yes 0 for no
pg_index_idx_blks_hit counter datname, relname, job, ins, relid, ip, instance, cls, idxname Number of buffer hits in this index
pg_index_idx_blks_read counter datname, relname, job, ins, relid, ip, instance, cls, idxname Number of disk blocks read from this index
pg_index_idx_scan counter datname, relname, job, ins, relid, ip, instance, cls, idxname Number of index scans initiated on this index
pg_index_idx_tup_fetch counter datname, relname, job, ins, relid, ip, instance, cls, idxname Number of live table rows fetched by simple index scans using this index
pg_index_idx_tup_read counter datname, relname, job, ins, relid, ip, instance, cls, idxname Number of index entries returned by scans on this index
pg_index_relpages gauge datname, relname, job, ins, relid, ip, instance, cls, idxname Size of the on-disk representation of this index in pages
pg_index_reltuples gauge datname, relname, job, ins, relid, ip, instance, cls, idxname Estimate relation tuples
pg_insert_lsn counter job, ins, ip, instance, cls primary only, location of current wal inserting
pg_io_evictions counter type, job, ins, object, ip, context, instance, cls Number of times a block has been written out from a shared or local buffer
pg_io_extend_time counter type, job, ins, object, ip, context, instance, cls Time spent in extend operations in seconds
pg_io_extends counter type, job, ins, object, ip, context, instance, cls Number of relation extend operations, each of the size specified in op_bytes.
pg_io_fsync_time counter type, job, ins, object, ip, context, instance, cls Time spent in fsync operations in seconds
pg_io_fsyncs counter type, job, ins, object, ip, context, instance, cls Number of fsync calls. These are only tracked in context normal
pg_io_hits counter type, job, ins, object, ip, context, instance, cls The number of times a desired block was found in a shared buffer.
pg_io_op_bytes gauge type, job, ins, object, ip, context, instance, cls The number of bytes per unit of I/O read, written, or extended. 8192 by default
pg_io_read_time counter type, job, ins, object, ip, context, instance, cls Time spent in read operations in seconds
pg_io_reads counter type, job, ins, object, ip, context, instance, cls Number of read operations, each of the size specified in op_bytes.
pg_io_reset_time gauge type, job, ins, object, ip, context, instance, cls Timestamp at which these statistics were last reset
pg_io_reuses counter type, job, ins, object, ip, context, instance, cls The number of times an existing buffer in reused
pg_io_write_time counter type, job, ins, object, ip, context, instance, cls Time spent in write operations in seconds
pg_io_writeback_time counter type, job, ins, object, ip, context, instance, cls Time spent in writeback operations in seconds
pg_io_writebacks counter type, job, ins, object, ip, context, instance, cls Number of units of size op_bytes which the process requested the kernel write out to permanent storage.
pg_io_writes counter type, job, ins, object, ip, context, instance, cls Number of write operations, each of the size specified in op_bytes.
pg_is_in_recovery gauge job, ins, ip, instance, cls 1 if in recovery mode
pg_is_wal_replay_paused gauge job, ins, ip, instance, cls 1 if wal play paused
pg_lag gauge job, ins, ip, instance, cls replica only, replication lag in seconds
pg_last_replay_time gauge job, ins, ip, instance, cls time when last transaction been replayed
pg_lock_count gauge datname, job, ins, ip, mode, instance, cls Number of locks of corresponding mode and database
pg_lsn counter job, ins, ip, instance, cls log sequence number, current write location
pg_meta_info gauge cls, extensions, version, job, ins, primary_conninfo, conf_path, hba_path, ip, cluster_id, instance, listen_port, wal_level, ver_num, cluster_name, data_dir constant 1
pg_query_calls counter datname, query, job, ins, ip, instance, cls Number of times the statement was executed
pg_query_exec_time counter datname, query, job, ins, ip, instance, cls Total time spent executing the statement, in seconds
pg_query_io_time counter datname, query, job, ins, ip, instance, cls Total time the statement spent reading and writing blocks, in seconds
pg_query_rows counter datname, query, job, ins, ip, instance, cls Total number of rows retrieved or affected by the statement
pg_query_sblk_dirtied counter datname, query, job, ins, ip, instance, cls Total number of shared blocks dirtied by the statement
pg_query_sblk_hit counter datname, query, job, ins, ip, instance, cls Total number of shared block cache hits by the statement
pg_query_sblk_read counter datname, query, job, ins, ip, instance, cls Total number of shared blocks read by the statement
pg_query_sblk_written counter datname, query, job, ins, ip, instance, cls Total number of shared blocks written by the statement
pg_query_wal_bytes counter datname, query, job, ins, ip, instance, cls Total amount of WAL bytes generated by the statement
pg_receive_lsn counter job, ins, ip, instance, cls replica only, location of wal synced to disk
pg_recovery_backup_end_lsn counter job, ins, ip, instance, cls Backup end location
pg_recovery_backup_start_lsn counter job, ins, ip, instance, cls Backup start location
pg_recovery_min_lsn counter job, ins, ip, instance, cls Minimum recovery ending location
pg_recovery_min_timeline counter job, ins, ip, instance, cls Min recovery ending loc’s timeline
pg_recovery_prefetch_block_distance gauge job, ins, ip, instance, cls How many blocks ahead the prefetcher is looking
pg_recovery_prefetch_hit counter job, ins, ip, instance, cls Number of blocks not prefetched because they were already in the buffer pool
pg_recovery_prefetch_io_depth gauge job, ins, ip, instance, cls How many prefetches have been initiated but are not yet known to have completed
pg_recovery_prefetch_prefetch counter job, ins, ip, instance, cls Number of blocks prefetched because they were not in the buffer pool
pg_recovery_prefetch_reset_time counter job, ins, ip, instance, cls Time at which these recovery prefetch statistics were last reset
pg_recovery_prefetch_skip_fpw gauge job, ins, ip, instance, cls Number of blocks not prefetched because a full page image was included in the WAL
pg_recovery_prefetch_skip_init counter job, ins, ip, instance, cls Number of blocks not prefetched because they would be zero-initialized
pg_recovery_prefetch_skip_new counter job, ins, ip, instance, cls Number of blocks not prefetched because they didn’t exist yet
pg_recovery_prefetch_skip_rep counter job, ins, ip, instance, cls Number of blocks not prefetched because they were already recently prefetched
pg_recovery_prefetch_wal_distance gauge job, ins, ip, instance, cls How many bytes ahead the prefetcher is looking
pg_recovery_require_record gauge job, ins, ip, instance, cls End-of-backup record required
pg_recv_flush_lsn counter state, slot_name, job, ins, ip, instance, cls, sender_host, sender_port Last write-ahead log location already received and flushed to disk
pg_recv_flush_tli counter state, slot_name, job, ins, ip, instance, cls, sender_host, sender_port Timeline number of last write-ahead log location received and flushed to disk
pg_recv_init_lsn counter state, slot_name, job, ins, ip, instance, cls, sender_host, sender_port First write-ahead log location used when WAL receiver is started
pg_recv_init_tli counter state, slot_name, job, ins, ip, instance, cls, sender_host, sender_port First timeline number used when WAL receiver is started
pg_recv_msg_recv_time gauge state, slot_name, job, ins, ip, instance, cls, sender_host, sender_port Receipt time of last message received from origin WAL sender
pg_recv_msg_send_time gauge state, slot_name, job, ins, ip, instance, cls, sender_host, sender_port Send time of last message received from origin WAL sender
pg_recv_pid gauge state, slot_name, job, ins, ip, instance, cls, sender_host, sender_port Process ID of the WAL receiver process
pg_recv_reported_lsn counter state, slot_name, job, ins, ip, instance, cls, sender_host, sender_port Last write-ahead log location reported to origin WAL sender
pg_recv_reported_time gauge state, slot_name, job, ins, ip, instance, cls, sender_host, sender_port Time of last write-ahead log location reported to origin WAL sender
pg_recv_time gauge state, slot_name, job, ins, ip, instance, cls, sender_host, sender_port Time of current snapshot
pg_recv_write_lsn counter state, slot_name, job, ins, ip, instance, cls, sender_host, sender_port Last write-ahead log location already received and written to disk, but not flushed.
pg_relkind_count gauge datname, job, ins, ip, instance, cls, relkind Number of relations of corresponding relkind
pg_repl_backend_xmin counter pid, usename, address, job, ins, appname, ip, instance, cls This standby’s xmin horizon reported by hot_standby_feedback.
pg_repl_client_port gauge pid, usename, address, job, ins, appname, ip, instance, cls TCP port number that the client is using for communication with this WAL sender, or -1 if a Unix socket is used
pg_repl_flush_diff gauge pid, usename, address, job, ins, appname, ip, instance, cls Last log position flushed to disk by this standby server diff with current lsn
pg_repl_flush_lag gauge pid, usename, address, job, ins, appname, ip, instance, cls Time elapsed between flushing recent WAL locally and receiving notification that this standby server has written and flushed it
pg_repl_flush_lsn counter pid, usename, address, job, ins, appname, ip, instance, cls Last write-ahead log location flushed to disk by this standby server
pg_repl_launch_time counter pid, usename, address, job, ins, appname, ip, instance, cls Time when this process was started, i.e., when the client connected to this WAL sender
pg_repl_lsn counter pid, usename, address, job, ins, appname, ip, instance, cls Current log position on this server
pg_repl_replay_diff gauge pid, usename, address, job, ins, appname, ip, instance, cls Last log position replayed into the database on this standby server diff with current lsn
pg_repl_replay_lag gauge pid, usename, address, job, ins, appname, ip, instance, cls Time elapsed between flushing recent WAL locally and receiving notification that this standby server has written, flushed and applied it
pg_repl_replay_lsn counter pid, usename, address, job, ins, appname, ip, instance, cls Last write-ahead log location replayed into the database on this standby server
pg_repl_reply_time gauge pid, usename, address, job, ins, appname, ip, instance, cls Send time of last reply message received from standby server
pg_repl_sent_diff gauge pid, usename, address, job, ins, appname, ip, instance, cls Last log position sent to this standby server diff with current lsn
pg_repl_sent_lsn counter pid, usename, address, job, ins, appname, ip, instance, cls Last write-ahead log location sent on this connection
pg_repl_state gauge pid, usename, address, job, ins, appname, ip, instance, cls Current WAL sender encoded state 0-4 for streaming startup catchup backup stopping
pg_repl_sync_priority gauge pid, usename, address, job, ins, appname, ip, instance, cls Priority of this standby server for being chosen as the synchronous standby
pg_repl_sync_state gauge pid, usename, address, job, ins, appname, ip, instance, cls Encoded synchronous state of this standby server, 0-3 for async potential sync quorum
pg_repl_time counter pid, usename, address, job, ins, appname, ip, instance, cls Current timestamp in unix epoch
pg_repl_write_diff gauge pid, usename, address, job, ins, appname, ip, instance, cls Last log position written to disk by this standby server diff with current lsn
pg_repl_write_lag gauge pid, usename, address, job, ins, appname, ip, instance, cls Time elapsed between flushing recent WAL locally and receiving notification that this standby server has written it
pg_repl_write_lsn counter pid, usename, address, job, ins, appname, ip, instance, cls Last write-ahead log location written to disk by this standby server
pg_replay_lsn counter job, ins, ip, instance, cls replica only, location of wal applied
pg_seq_blks_hit counter datname, job, ins, ip, instance, cls, seqname Number of buffer hits in this sequence
pg_seq_blks_read counter datname, job, ins, ip, instance, cls, seqname Number of disk blocks read from this sequence
pg_seq_last_value counter datname, job, ins, ip, instance, cls, seqname The last sequence value written to disk
pg_setting_block_size gauge job, ins, ip, instance, cls pg page block size, 8192 by default
pg_setting_data_checksums gauge job, ins, ip, instance, cls whether data checksum is enabled, 1 enabled 0 disabled
pg_setting_max_connections gauge job, ins, ip, instance, cls number of concurrent connections to the database server
pg_setting_max_locks_per_transaction gauge job, ins, ip, instance, cls no more than this many distinct objects can be locked at any one time
pg_setting_max_prepared_transactions gauge job, ins, ip, instance, cls maximum number of transactions that can be in the prepared state simultaneously
pg_setting_max_replication_slots gauge job, ins, ip, instance, cls maximum number of replication slots
pg_setting_max_wal_senders gauge job, ins, ip, instance, cls maximum number of concurrent connections from standby servers
pg_setting_max_worker_processes gauge job, ins, ip, instance, cls maximum number of background processes that the system can support
pg_setting_wal_log_hints gauge job, ins, ip, instance, cls whether wal_log_hints is enabled, 1 enabled 0 disabled
pg_size_bytes gauge datname, job, ins, ip, instance, cls File size in bytes
pg_slot_active gauge slot_name, job, ins, ip, instance, cls True(1) if this slot is currently actively being used
pg_slot_catalog_xmin counter slot_name, job, ins, ip, instance, cls The oldest transaction affecting the system catalogs that this slot needs the database to retain.
pg_slot_confirm_lsn counter slot_name, job, ins, ip, instance, cls The address (LSN) up to which the logical slot’s consumer has confirmed receiving data.
pg_slot_reset_time counter slot_name, job, ins, ip, instance, cls When statistics were last reset
pg_slot_restart_lsn counter slot_name, job, ins, ip, instance, cls The address (LSN) of oldest WAL which still might be required by the consumer of this slot
pg_slot_retained_bytes gauge slot_name, job, ins, ip, instance, cls Size of bytes that retained for this slot
pg_slot_safe_wal_size gauge slot_name, job, ins, ip, instance, cls bytes that can be written to WAL which will not make slot into lost
pg_slot_spill_bytes counter slot_name, job, ins, ip, instance, cls Bytes that spilled to disk due to logical decode mem exceeding
pg_slot_spill_count counter slot_name, job, ins, ip, instance, cls Xacts that spilled to disk due to logical decode mem exceeding (a xact can be spilled multiple times)
pg_slot_spill_txns counter slot_name, job, ins, ip, instance, cls Xacts that spilled to disk due to logical decode mem exceeding (subtrans included)
pg_slot_stream_bytes counter slot_name, job, ins, ip, instance, cls Bytes that streamed to decoding output plugin after mem exceed
pg_slot_stream_count counter slot_name, job, ins, ip, instance, cls Xacts that streamed to decoding output plugin after mem exceed (a xact can be streamed multiple times)
pg_slot_stream_txns counter slot_name, job, ins, ip, instance, cls Xacts that streamed to decoding output plugin after mem exceed
pg_slot_temporary gauge slot_name, job, ins, ip, instance, cls True(1) if this is a temporary replication slot.
pg_slot_total_bytes counter slot_name, job, ins, ip, instance, cls Number of decoded bytes sent to the decoding output plugin for this slot
pg_slot_total_txns counter slot_name, job, ins, ip, instance, cls Number of decoded xacts sent to the decoding output plugin for this slot
pg_slot_wal_status gauge slot_name, job, ins, ip, instance, cls WAL reserve status 0-3 means reserved,extended,unreserved,lost, -1 means other
pg_slot_xmin counter slot_name, job, ins, ip, instance, cls The oldest transaction that this slot needs the database to retain.
pg_slru_blks_exists counter job, ins, ip, instance, cls Number of blocks checked for existence for this SLRU
pg_slru_blks_hit counter job, ins, ip, instance, cls Number of times disk blocks were found already in the SLRU, so that a read was not necessary
pg_slru_blks_read counter job, ins, ip, instance, cls Number of disk blocks read for this SLRU
pg_slru_blks_written counter job, ins, ip, instance, cls Number of disk blocks written for this SLRU
pg_slru_blks_zeroed counter job, ins, ip, instance, cls Number of blocks zeroed during initializations
pg_slru_flushes counter job, ins, ip, instance, cls Number of flushes of dirty data for this SLRU
pg_slru_reset_time counter job, ins, ip, instance, cls Time at which these statistics were last reset
pg_slru_truncates counter job, ins, ip, instance, cls Number of truncates for this SLRU
pg_ssl_disabled gauge job, ins, ip, instance, cls Number of client connection that does not use ssl
pg_ssl_enabled gauge job, ins, ip, instance, cls Number of client connection that use ssl
pg_sync_standby_enabled gauge job, ins, ip, names, instance, cls Synchronous commit enabled, 1 if enabled, 0 if disabled
pg_table_age gauge datname, relname, job, ins, ip, instance, cls Age of this table in vacuum cycles
pg_table_analyze_count counter datname, relname, job, ins, ip, instance, cls Number of times this table has been manually analyzed
pg_table_autoanalyze_count counter datname, relname, job, ins, ip, instance, cls Number of times this table has been analyzed by the autovacuum daemon
pg_table_autovacuum_count counter datname, relname, job, ins, ip, instance, cls Number of times this table has been vacuumed by the autovacuum daemon
pg_table_frozenxid counter datname, relname, job, ins, ip, instance, cls All txid before this have been frozen on this table
pg_table_heap_blks_hit counter datname, relname, job, ins, ip, instance, cls Number of buffer hits in this table
pg_table_heap_blks_read counter datname, relname, job, ins, ip, instance, cls Number of disk blocks read from this table
pg_table_idx_blks_hit counter datname, relname, job, ins, ip, instance, cls Number of buffer hits in all indexes on this table
pg_table_idx_blks_read counter datname, relname, job, ins, ip, instance, cls Number of disk blocks read from all indexes on this table
pg_table_idx_scan counter datname, relname, job, ins, ip, instance, cls Number of index scans initiated on this table
pg_table_idx_tup_fetch counter datname, relname, job, ins, ip, instance, cls Number of live rows fetched by index scans
pg_table_kind gauge datname, relname, job, ins, ip, instance, cls Relation kind r/table/114
pg_table_n_dead_tup gauge datname, relname, job, ins, ip, instance, cls Estimated number of dead rows
pg_table_n_ins_since_vacuum gauge datname, relname, job, ins, ip, instance, cls Estimated number of rows inserted since this table was last vacuumed
pg_table_n_live_tup gauge datname, relname, job, ins, ip, instance, cls Estimated number of live rows
pg_table_n_mod_since_analyze gauge datname, relname, job, ins, ip, instance, cls Estimated number of rows modified since this table was last analyzed
pg_table_n_tup_del counter datname, relname, job, ins, ip, instance, cls Number of rows deleted
pg_table_n_tup_hot_upd counter datname, relname, job, ins, ip, instance, cls Number of rows HOT updated (i.e with no separate index update required)
pg_table_n_tup_ins counter datname, relname, job, ins, ip, instance, cls Number of rows inserted
pg_table_n_tup_mod counter datname, relname, job, ins, ip, instance, cls Number of rows modified (insert + update + delete)
pg_table_n_tup_newpage_upd counter datname, relname, job, ins, ip, instance, cls Number of rows updated where the successor version goes onto a new heap page
pg_table_n_tup_upd counter datname, relname, job, ins, ip, instance, cls Number of rows updated (includes HOT updated rows)
pg_table_ncols gauge datname, relname, job, ins, ip, instance, cls Number of columns in the table
pg_table_pages gauge datname, relname, job, ins, ip, instance, cls Size of the on-disk representation of this table in pages
pg_table_relid gauge datname, relname, job, ins, ip, instance, cls Relation oid of this table
pg_table_seq_scan counter datname, relname, job, ins, ip, instance, cls Number of sequential scans initiated on this table
pg_table_seq_tup_read counter datname, relname, job, ins, ip, instance, cls Number of live rows fetched by sequential scans
pg_table_size_bytes gauge datname, relname, job, ins, ip, instance, cls Total bytes of this table (including toast, index, toast index)
pg_table_size_indexsize gauge datname, relname, job, ins, ip, instance, cls Bytes of all related indexes of this table
pg_table_size_relsize gauge datname, relname, job, ins, ip, instance, cls Bytes of this table itself (main, vm, fsm)
pg_table_size_toastsize gauge datname, relname, job, ins, ip, instance, cls Bytes of toast tables of this table
pg_table_tbl_scan counter datname, relname, job, ins, ip, instance, cls Number of scans initiated on this table
pg_table_tup_read counter datname, relname, job, ins, ip, instance, cls Number of live rows fetched by scans
pg_table_tuples counter datname, relname, job, ins, ip, instance, cls All txid before this have been frozen on this table
pg_table_vacuum_count counter datname, relname, job, ins, ip, instance, cls Number of times this table has been manually vacuumed (not counting VACUUM FULL)
pg_timestamp gauge job, ins, ip, instance, cls database current timestamp
pg_up gauge job, ins, ip, instance, cls last scrape was able to connect to the server: 1 for yes, 0 for no
pg_uptime gauge job, ins, ip, instance, cls seconds since postmaster start
pg_version gauge job, ins, ip, instance, cls server version number
pg_wait_count gauge datname, job, ins, event, ip, instance, cls Count of WaitEvent on target database
pg_wal_buffers_full counter job, ins, ip, instance, cls Number of times WAL data was written to disk because WAL buffers became full
pg_wal_bytes counter job, ins, ip, instance, cls Total amount of WAL generated in bytes
pg_wal_fpi counter job, ins, ip, instance, cls Total number of WAL full page images generated
pg_wal_records counter job, ins, ip, instance, cls Total number of WAL records generated
pg_wal_reset_time counter job, ins, ip, instance, cls When statistics were last reset
pg_wal_sync counter job, ins, ip, instance, cls Number of times WAL files were synced to disk via issue_xlog_fsync request
pg_wal_sync_time counter job, ins, ip, instance, cls Total amount of time spent syncing WAL files to disk via issue_xlog_fsync request, in seconds
pg_wal_write counter job, ins, ip, instance, cls Number of times WAL buffers were written out to disk via XLogWrite request.
pg_wal_write_time counter job, ins, ip, instance, cls Total amount of time spent writing WAL buffers to disk via XLogWrite request in seconds
pg_write_lsn counter job, ins, ip, instance, cls primary only, location of current wal writing
pg_xact_xmax counter job, ins, ip, instance, cls First as-yet-unassigned txid. txid >= this are invisible.
pg_xact_xmin counter job, ins, ip, instance, cls Earliest txid that is still active
pg_xact_xnum gauge job, ins, ip, instance, cls Current active transaction count
pgbouncer:cls:load1 Unknown job, cls N/A
pgbouncer:cls:load15 Unknown job, cls N/A
pgbouncer:cls:load5 Unknown job, cls N/A
pgbouncer:db:conn_usage Unknown datname, job, ins, ip, instance, host, cls, real_datname, port N/A
pgbouncer:db:conn_usage_reserve Unknown datname, job, ins, ip, instance, host, cls, real_datname, port N/A
pgbouncer:db:pool_current_conn Unknown datname, job, ins, ip, instance, host, cls, real_datname, port N/A
pgbouncer:db:pool_disabled Unknown datname, job, ins, ip, instance, host, cls, real_datname, port N/A
pgbouncer:db:pool_max_conn Unknown datname, job, ins, ip, instance, host, cls, real_datname, port N/A
pgbouncer:db:pool_paused Unknown datname, job, ins, ip, instance, host, cls, real_datname, port N/A
pgbouncer:db:pool_reserve_size Unknown datname, job, ins, ip, instance, host, cls, real_datname, port N/A
pgbouncer:db:pool_size Unknown datname, job, ins, ip, instance, host, cls, real_datname, port N/A
pgbouncer:ins:free_clients Unknown job, ins, ip, instance, cls N/A
pgbouncer:ins:free_servers Unknown job, ins, ip, instance, cls N/A
pgbouncer:ins:load1 Unknown job, ins, ip, instance, cls N/A
pgbouncer:ins:load15 Unknown job, ins, ip, instance, cls N/A
pgbouncer:ins:load5 Unknown job, ins, ip, instance, cls N/A
pgbouncer:ins:login_clients Unknown job, ins, ip, instance, cls N/A
pgbouncer:ins:pool_databases Unknown job, ins, ip, instance, cls N/A
pgbouncer:ins:pool_users Unknown job, ins, ip, instance, cls N/A
pgbouncer:ins:pools Unknown job, ins, ip, instance, cls N/A
pgbouncer:ins:used_clients Unknown job, ins, ip, instance, cls N/A
pgbouncer_database_current_connections gauge datname, job, ins, ip, instance, host, cls, real_datname, port Current number of connections for this database
pgbouncer_database_disabled gauge datname, job, ins, ip, instance, host, cls, real_datname, port True(1) if this database is currently disabled, else 0
pgbouncer_database_max_connections gauge datname, job, ins, ip, instance, host, cls, real_datname, port Maximum number of allowed connections for this database
pgbouncer_database_min_pool_size gauge datname, job, ins, ip, instance, host, cls, real_datname, port Minimum number of server connections
pgbouncer_database_paused gauge datname, job, ins, ip, instance, host, cls, real_datname, port True(1) if this database is currently paused, else 0
pgbouncer_database_pool_size gauge datname, job, ins, ip, instance, host, cls, real_datname, port Maximum number of server connections
pgbouncer_database_reserve_pool gauge datname, job, ins, ip, instance, host, cls, real_datname, port Maximum number of additional connections for this database
pgbouncer_exporter_agent_up Unknown job, ins, ip, instance, cls N/A
pgbouncer_exporter_last_scrape_time gauge job, ins, ip, instance, cls seconds exporter spending on scrapping
pgbouncer_exporter_query_cache_ttl gauge datname, query, job, ins, ip, instance, cls times to live of query cache
pgbouncer_exporter_query_scrape_duration gauge datname, query, job, ins, ip, instance, cls seconds query spending on scrapping
pgbouncer_exporter_query_scrape_error_count gauge datname, query, job, ins, ip, instance, cls times the query failed
pgbouncer_exporter_query_scrape_hit_count gauge datname, query, job, ins, ip, instance, cls numbers been scrapped from this query
pgbouncer_exporter_query_scrape_metric_count gauge datname, query, job, ins, ip, instance, cls numbers of metrics been scrapped from this query
pgbouncer_exporter_query_scrape_total_count gauge datname, query, job, ins, ip, instance, cls times exporter server was scraped for metrics
pgbouncer_exporter_scrape_duration gauge job, ins, ip, instance, cls seconds exporter spending on scrapping
pgbouncer_exporter_scrape_error_count counter job, ins, ip, instance, cls times exporter was scraped for metrics and failed
pgbouncer_exporter_scrape_total_count counter job, ins, ip, instance, cls times exporter was scraped for metrics
pgbouncer_exporter_server_scrape_duration gauge datname, job, ins, ip, instance, cls seconds exporter server spending on scrapping
pgbouncer_exporter_server_scrape_total_count gauge datname, job, ins, ip, instance, cls times exporter server was scraped for metrics
pgbouncer_exporter_server_scrape_total_seconds gauge datname, job, ins, ip, instance, cls seconds exporter server spending on scrapping
pgbouncer_exporter_up gauge job, ins, ip, instance, cls always be 1 if your could retrieve metrics
pgbouncer_exporter_uptime gauge job, ins, ip, instance, cls seconds since exporter primary server inited
pgbouncer_in_recovery gauge job, ins, ip, instance, cls server is in recovery mode? 1 for yes 0 for no
pgbouncer_list_items gauge job, ins, ip, instance, list, cls Number of corresponding pgbouncer object
pgbouncer_pool_active_cancel_clients gauge datname, job, ins, ip, instance, user, cls, pool_mode Client connections that have forwarded query cancellations to the server and are waiting for the server response.
pgbouncer_pool_active_cancel_servers gauge datname, job, ins, ip, instance, user, cls, pool_mode Server connections that are currently forwarding a cancel request
pgbouncer_pool_active_clients gauge datname, job, ins, ip, instance, user, cls, pool_mode Client connections that are linked to server connection and can process queries
pgbouncer_pool_active_servers gauge datname, job, ins, ip, instance, user, cls, pool_mode Server connections that are linked to a client
pgbouncer_pool_cancel_clients gauge datname, job, ins, ip, instance, user, cls, pool_mode Client connections that have not forwarded query cancellations to the server yet.
pgbouncer_pool_cancel_servers gauge datname, job, ins, ip, instance, user, cls, pool_mode cancel requests have completed that were sent to cancel a query on this server
pgbouncer_pool_idle_servers gauge datname, job, ins, ip, instance, user, cls, pool_mode Server connections that are unused and immediately usable for client queries
pgbouncer_pool_login_servers gauge datname, job, ins, ip, instance, user, cls, pool_mode Server connections currently in the process of logging in
pgbouncer_pool_maxwait gauge datname, job, ins, ip, instance, user, cls, pool_mode How long the first(oldest) client in the queue has waited, in seconds, key metric
pgbouncer_pool_maxwait_us gauge datname, job, ins, ip, instance, user, cls, pool_mode Microsecond part of the maximum waiting time.
pgbouncer_pool_tested_servers gauge datname, job, ins, ip, instance, user, cls, pool_mode Server connections that are currently running reset or check query
pgbouncer_pool_used_servers gauge datname, job, ins, ip, instance, user, cls, pool_mode Server connections that have been idle for more than server_check_delay (means have to run check query)
pgbouncer_pool_waiting_clients gauge datname, job, ins, ip, instance, user, cls, pool_mode Client connections that have sent queries but have not yet got a server connection
pgbouncer_stat_avg_query_count gauge datname, job, ins, ip, instance, cls Average queries per second in last stat period
pgbouncer_stat_avg_query_time gauge datname, job, ins, ip, instance, cls Average query duration, in seconds
pgbouncer_stat_avg_recv gauge datname, job, ins, ip, instance, cls Average received (from clients) bytes per second
pgbouncer_stat_avg_sent gauge datname, job, ins, ip, instance, cls Average sent (to clients) bytes per second
pgbouncer_stat_avg_wait_time gauge datname, job, ins, ip, instance, cls Time spent by clients waiting for a server, in seconds (average per second).
pgbouncer_stat_avg_xact_count gauge datname, job, ins, ip, instance, cls Average transactions per second in last stat period
pgbouncer_stat_avg_xact_time gauge datname, job, ins, ip, instance, cls Average transaction duration, in seconds
pgbouncer_stat_total_query_count gauge datname, job, ins, ip, instance, cls Total number of SQL queries pooled by pgbouncer
pgbouncer_stat_total_query_time counter datname, job, ins, ip, instance, cls Total number of seconds spent when executing queries
pgbouncer_stat_total_received counter datname, job, ins, ip, instance, cls Total volume in bytes of network traffic received by pgbouncer
pgbouncer_stat_total_sent counter datname, job, ins, ip, instance, cls Total volume in bytes of network traffic sent by pgbouncer
pgbouncer_stat_total_wait_time counter datname, job, ins, ip, instance, cls Time spent by clients waiting for a server, in seconds
pgbouncer_stat_total_xact_count gauge datname, job, ins, ip, instance, cls Total number of SQL transactions pooled by pgbouncer
pgbouncer_stat_total_xact_time counter datname, job, ins, ip, instance, cls Total number of seconds spent when in a transaction
pgbouncer_up gauge job, ins, ip, instance, cls last scrape was able to connect to the server: 1 for yes, 0 for no
pgbouncer_version gauge job, ins, ip, instance, cls server version number
process_cpu_seconds_total counter job, ins, ip, instance, cls Total user and system CPU time spent in seconds.
process_max_fds gauge job, ins, ip, instance, cls Maximum number of open file descriptors.
process_open_fds gauge job, ins, ip, instance, cls Number of open file descriptors.
process_resident_memory_bytes gauge job, ins, ip, instance, cls Resident memory size in bytes.
process_start_time_seconds gauge job, ins, ip, instance, cls Start time of the process since unix epoch in seconds.
process_virtual_memory_bytes gauge job, ins, ip, instance, cls Virtual memory size in bytes.
process_virtual_memory_max_bytes gauge job, ins, ip, instance, cls Maximum amount of virtual memory available in bytes.
promhttp_metric_handler_requests_in_flight gauge job, ins, ip, instance, cls Current number of scrapes being served.
promhttp_metric_handler_requests_total counter code, job, ins, ip, instance, cls Total number of scrapes by HTTP status code.
scrape_duration_seconds Unknown job, ins, ip, instance, cls N/A
scrape_samples_post_metric_relabeling Unknown job, ins, ip, instance, cls N/A
scrape_samples_scraped Unknown job, ins, ip, instance, cls N/A
scrape_series_added Unknown job, ins, ip, instance, cls N/A
up Unknown job, ins, ip, instance, cls N/A

8.11 - Parameters

Customize PostgreSQL clusters with 120 parameters in the PGSQL module

The PGSQL module needs to be installed on nodes managed by Pigsty (i.e., nodes that have the NODE module configured), and also requires an available ETCD cluster in your deployment to store cluster metadata.

Installing the PGSQL module on a single node will create a standalone PGSQL server/instance, i.e., a primary instance. Installing on additional nodes will create read replicas, which can serve as standby instances and handle read-only requests. You can also create offline instances for ETL/OLAP/interactive queries, use sync standby and quorum commit to improve data consistency, or even set up standby clusters and delayed clusters to quickly respond to data loss caused by human errors and software defects.

You can define multiple PGSQL clusters and further organize them into a horizontal sharding cluster: Pigsty natively supports Citus cluster groups, allowing you to upgrade your standard PGSQL cluster in-place to a distributed database cluster.

Section Description
PG_ID PostgreSQL cluster and instance identity parameters
PG_BUSINESS Business users, databases, services and access control rule definition
PG_INSTALL PostgreSQL installation: version, paths, packages
PG_BOOTSTRAP PostgreSQL cluster initialization: Patroni high availability
PG_PROVISION PostgreSQL cluster template provisioning: roles, privileges, extensions
PG_BACKUP pgBackRest backup and recovery configuration
PG_ACCESS Service exposure, connection pooling, VIP, DNS client access config
PG_MONITOR PostgreSQL monitoring exporter configuration
PG_REMOVE PostgreSQL instance cleanup and uninstall configuration

Parameter Overview

PG_ID parameters are used to define PostgreSQL cluster and instance identity, including cluster name, instance sequence number, role, shard, and other core identity parameters.

Parameter Type Level Description
pg_mode enum C pgsql cluster mode: pgsql,citus,mssql,mysql,ivory,pgtde,polar,gpsql,agens,oriole,pgedge
pg_cluster string C pgsql cluster name, REQUIRED identity parameter
pg_seq int I pgsql instance seq number, REQUIRED identity parameter
pg_role enum I pgsql instance role: primary, replica, standby, offline, or delayed
pg_instances dict I define multiple pg instances on node in {port:ins_vars} format
pg_upstream ip I replication upstream IP for a standby cluster or cascade replica
pg_shard string C pgsql shard name; specify explicitly for horizontal sharding
pg_group int C non-negative pgsql shard index; specify explicitly for horizontal sharding
gp_role enum C greenplum role of this cluster, could be master or segment
pg_exporters dict C additional pg_exporters to monitor remote postgres instances
pg_offline_query bool I set to true to mark this replica as offline instance for offline queries

PG_BUSINESS parameters are used to define business users, databases, services and access control rules, as well as default system user credentials.

Parameter Type Level Description
pg_users user[] C postgres business users
pg_databases database[] C postgres business databases
pg_services service[] C postgres business services
pg_hba_rules hba[] C business hba rules for postgres
pgb_hba_rules hba[] C business hba rules for pgbouncer
pg_crontab string[] C crontab entries for postgres dbsu
pg_replication_username username G postgres replication username, replicator by default
pg_replication_password password G postgres replication password, DBUser.Replicator by default
pg_admin_username username G postgres admin username, dbuser_dba by default
pg_admin_password password G postgres admin password in plain text, DBUser.DBA by default
pg_monitor_username username G postgres monitor username, dbuser_monitor by default
pg_monitor_password password G postgres monitor password, DBUser.Monitor by default
pg_dbsu_password password G/C dbsu password, empty string disables it by default, best not set

PG_INSTALL parameters are used to configure PostgreSQL installation options, including version, paths, packages, and extensions.

Parameter Type Level Description
pg_dbsu username C os dbsu name, postgres by default, better not change it
pg_dbsu_uid int C os dbsu uid and gid, 26 for default postgres user and group
pg_dbsu_sudo enum C dbsu sudo privilege, none,limit,all,nopass. limit by default
pg_dbsu_home path C postgresql home directory, /var/lib/pgsql by default
pg_dbsu_ssh_exchange bool C exchange postgres dbsu ssh key among same pgsql cluster
pg_version enum C postgres major version to be installed, 18 by default
pg_bin_dir path C postgres binary dir, /usr/pgsql/bin by default
pg_log_dir path C postgres log dir, /pg/log/postgres by default
pg_packages string[] C pg packages to be installed, ${pg_version} will be replaced
pg_extensions string[] C pg extensions to be installed, ${pg_version} will be replaced

PG_BOOTSTRAP parameters are used to configure PostgreSQL cluster initialization, including Patroni high availability, storage paths, networking, encoding, and other core settings.

Parameter Type Level Description
pg_data path C PostgreSQL data directory, /pg/data by default
pg_fs_main path C mountpoint/path for pg main data, /data/postgres by default
pg_fs_backup path C mountpoint/path for pg backup data, /data/backups by default
pg_storage_type enum C storage type for pg main data, SSD,HDD. SSD by default
pg_dummy_filesize size C size of /pg/dummy, hold 64MB disk space for emergency use
pg_listen ip(s) C/I postgres/pgbouncer listen addr, comma separated list, 0.0.0.0
pg_port port C postgres listen port, 5432 by default
pg_localhost path C postgres unix socket dir for localhost connection
pg_namespace path C top level key namespace in etcd, used by patroni & vip
patroni_enabled bool C if disabled, no postgres cluster will be created during init
patroni_mode enum C patroni working mode: default,pause,remove
patroni_port port C patroni listen port, 8008 by default
patroni_log_dir path C patroni log dir, /pg/log/patroni by default
patroni_ssl_enabled bool G secure patroni RestAPI communications with SSL?
patroni_watchdog_mode enum C patroni watchdog mode: automatic,required,off. off by default
patroni_username username C patroni restapi username, postgres by default
patroni_password password C patroni restapi password, Patroni.API by default
pg_primary_db string C primary database name, used by citus,etc. postgres by default
pg_parameters dict C extra parameters in postgresql.auto.conf
pg_files path[] C extra files to be copied to PGDATA (e.g. license files)
pg_conf enum C config template: oltp,olap,crit,tiny. oltp.yml by default
pg_max_conn int C postgres max connections, auto will use recommended value
pg_shared_buffer_ratio float C postgres shared buffer memory ratio, 0.25 by default, 0.1~0.4
pg_rto enum C RTO mode: fast, norm, safe, or wide; default norm
pg_rto_plan dict G RTO presets for Patroni HA and HAProxy health-check timeouts
pg_rpo int C sampled lag threshold for Patroni failover candidates; default 1MiB
pg_libs string C preloaded libraries, pg_stat_statements,auto_explain by default
pg_delay interval I WAL replay apply delay for standby cluster, for delayed replica
pg_checksum bool C enable data checksum for postgres cluster?
pg_pwd_enc enum C password encryption algorithm: fixed to scram-sha-256
pg_encoding enum C database cluster encoding, UTF8 by default
pg_locale enum C database cluster locale, C by default
pg_lc_collate enum C database cluster collate, C by default
pg_lc_ctype enum C database character type, C by default
pg_io_method enum C PostgreSQL IO method: auto, sync, worker, io_uring
pg_etcd_password password C etcd password for this PostgreSQL cluster, cluster name by default
pgsodium_key string C pgsodium encryption master key, 64 hex digits, sha256(pg_cluster)
pgsodium_getkey_script path C pgsodium getkey script path, uses template pgsodium_getkey

PG_PROVISION parameters are used to configure PostgreSQL cluster template provisioning, including default roles, privileges, schemas, extensions, and HBA rules.

Parameter Type Level Description
pg_provision bool C provision postgres cluster content after bootstrap?
pg_init string G/C init script for cluster template, pg-init by default
pg_default_roles role[] G/C default predefined roles and system users in postgres
pg_default_privileges string[] G/C default privileges when created by admin user
pg_default_schemas string[] G/C default schemas to be created
pg_default_extensions extension[] G/C default extensions to be created
pg_reload bool A reload postgres config after hba changes?
pg_default_hba_rules hba[] G/C postgres default host-based auth rules, global default HBA
pgb_default_hba_rules hba[] G/C pgbouncer default host-based auth rules, global default HBA

PG_BACKUP parameters are used to configure pgBackRest backup and recovery, including repository type, paths, and retention policies.

Parameter Type Level Description
pgbackrest_enabled bool C enable pgbackrest on pgsql host?
pgbackrest_log_dir path C pgbackrest log dir, /pg/log/pgbackrest by default
pgbackrest_method enum C pgbackrest repo method: local,minio,etc…
pgbackrest_init_backup bool C perform full backup after init? true by default
pgbackrest_repo dict G/C pgbackrest repo definition

PG_ACCESS parameters are used to configure service exposure, connection pooling, VIP, DNS, and other client access options.

Parameter Type Level Description
pgbouncer_enabled bool C if disabled, pgbouncer will not be configured
pgbouncer_port port C pgbouncer listen port, 6432 by default
pgbouncer_log_dir path C pgbouncer log dir, /pg/log/pgbouncer by default
pgbouncer_auth_query bool C use AuthQuery to get unlisted business users from postgres?
pgbouncer_poolmode enum C pool mode: transaction,session,statement. transaction by default
pgbouncer_sslmode enum C pgbouncer client ssl mode, disabled by default
pgbouncer_ignore_param string[] C pgbouncer ignore startup parameters list
pg_weight int I relative load balancing weight in service, 0-255, 100 by default
pg_service_provider string G/C dedicated haproxy node group name, or use local haproxy
pg_default_service_dest enum G/C default service dest if svc.dest=‘default’: postgres or pgbouncer
pg_default_services service[] G/C postgres default service definition list, shared globally
pg_vip_enabled bool C enable L2 VIP for pgsql primary? disabled by default
pg_vip_address cidr4 C vip address in <ipv4>/<mask> format, required if vip enabled
pg_vip_interface string C/I vip network interface to bind, auto by default
pg_dns_suffix string C pgsql dns suffix, empty by default
pg_dns_target enum C PG DNS resolves to: auto, primary, vip, none, or specific IP

PG_MONITOR parameters are used to configure PostgreSQL monitoring exporters, including pg_exporter, pgbouncer_exporter, and pgbackrest_exporter.

Parameter Type Level Description
pg_exporter_enabled bool C enable pg_exporter on pgsql host?
pg_exporter_config string C pg_exporter config file/template name
pg_exporter_cache_ttls string C pg_exporter collector ttl stages, ‘1,10,60,300’ by default
pg_exporter_port port C pg_exporter listen port, 9630 by default
pg_exporter_params string C extra URL parameters for pg_exporter dsn
pg_exporter_url pgurl C overwrite auto-generated postgres DSN connection string
pg_exporter_auto_discovery bool C enable auto database discovery for monitoring? enabled
pg_exporter_exclude_database string C excluded database list when auto-discovery, comma separated
pg_exporter_include_database string C only monitor these databases when auto-discovery enabled
pg_exporter_connect_timeout int C pg_exporter connect timeout in ms, 200 by default
pg_exporter_options arg C extra command line options for pg_exporter
pgbouncer_exporter_enabled bool C enable pgbouncer_exporter on pgsql host?
pgbouncer_exporter_port port C pgbouncer_exporter listen port, 9631 by default
pgbouncer_exporter_url pgurl C overwrite auto-generated pgbouncer dsn connection string
pgbouncer_exporter_options arg C extra command line options for pgbouncer_exporter
pgbackrest_exporter_enabled bool C enable pgbackrest_exporter on pgsql host?
pgbackrest_exporter_port port C pgbackrest_exporter listen port, 9854 by default
pgbackrest_exporter_options arg C extra command line options for pgbackrest_exporter

PG_REMOVE parameters are used to configure PostgreSQL instance cleanup and uninstall behavior, including data directory, backup, and package removal control.

Parameter Type Level Description
pg_rm_data bool G/C/A remove postgres data directory when removing instance?
pg_rm_backup bool G/C/A remove pgbackrest backup when removing primary?
pg_rm_pkg bool G/C/A uninstall related packages when removing pgsql instance?
pg_safeguard bool G/C/A prevent accidental pgsql cleanup operations? false

PG_ID

Here are commonly used parameters for identifying entities in the PGSQL module: clusters, instances, services, etc…

# pg_cluster:           #CLUSTER  # pgsql cluster name, required identity parameter
# pg_seq: 0             #INSTANCE # pgsql instance seq number, required identity parameter
# pg_role: replica      #INSTANCE # pgsql role, required, could be primary,replica,offline
# pg_instances: {}      #INSTANCE # define multiple pg instances on node in `{port:ins_vars}` format
# pg_upstream:          #INSTANCE # repl upstream ip addr for standby cluster or cascade replica
# pg_shard:             #CLUSTER  # pgsql shard name, optional identity for sharding clusters
# pg_group: 0           #CLUSTER  # pgsql shard index number, optional identity for sharding clusters
# gp_role: master       #CLUSTER  # greenplum role of this cluster, could be master or segment
pg_offline_query: false #INSTANCE # set to true to enable offline query on this instance

You must explicitly specify these identity parameters, they have no default values:

Name Type Level Description
pg_cluster string C PG cluster name
pg_seq number I PG instance ID
pg_role enum I PG instance role
pg_shard string C Shard name
pg_group number C Shard index
  • pg_cluster: Identifies the cluster name, configured at cluster level.
  • pg_role: Configured at instance level, identifies the role of the instance. Only primary role is treated specially. If not specified, defaults to replica role, with special delayed and offline roles.
  • pg_seq: Used to identify instances within a cluster, typically an integer starting from 0 or 1, once assigned it doesn’t change.
  • {{ pg_cluster }}-{{ pg_seq }} uniquely identifies an instance, i.e., pg_instance.
  • {{ pg_cluster }}-{{ pg_role }} identifies services within the cluster, i.e., pg_service.
  • pg_shard and pg_group are used for horizontal sharding clusters, only for citus, greenplum, and matrixdb.

pg_cluster, pg_role, pg_seq are core identity parameters, required for any Postgres cluster and must be explicitly specified. Here is an example:

pg-test:
  hosts:
    10.10.10.11: {pg_seq: 1, pg_role: replica}
    10.10.10.12: {pg_seq: 2, pg_role: primary}
    10.10.10.13: {pg_seq: 3, pg_role: replica}
  vars:
    pg_cluster: pg-test

All other parameters can be inherited from global or default configuration, but identity parameters must be explicitly specified and manually assigned.

pg_mode

Parameter Name: pg_mode, Type: enum, Level: C

PostgreSQL cluster mode, default value is pgsql, i.e., standard PostgreSQL cluster.

Available mode options include:

  • pgsql: Standard PostgreSQL cluster
  • citus: Citus distributed database cluster
  • mssql: Babelfish MSSQL wire protocol compatible kernel
  • mysql: OpenHalo/HaloDB MySQL wire protocol compatible kernel
  • ivory: IvorySQL Oracle compatible kernel
  • pgtde: Percona PostgreSQL with pg_tde
  • polar: PolarDB for PostgreSQL kernel
  • gpsql: Greenplum parallel database cluster (monitoring)
  • agens: AgensGraph graph database kernel
  • oriole: OrioleDB storage-engine kernel
  • pgedge: pgEdge distributed-replication kernel

pg_shard and pg_group default to pg_cluster and 0, respectively. When pg_mode is citus or gpsql and the sharded system contains multiple physical clusters, set both explicitly to define the horizontal-sharding identity.

In both cases, each PostgreSQL cluster is part of a larger business unit.

pg_cluster

Parameter Name: pg_cluster, Type: string, Level: C

PostgreSQL cluster name, required identity parameter, no default value.

The cluster name is used as the namespace for resources.

The current role validation accepts names matching ^[A-Za-z0-9-]+$ and rejects root. To keep DNS names, service names, and operation scripts consistent, lowercase names beginning with a letter and containing only lowercase letters, digits, and hyphens are still recommended.

pg_seq

Parameter Name: pg_seq, Type: int, Level: I

PostgreSQL instance sequence number, required identity parameter, no default value.

The sequence number of this instance, uniquely assigned within its cluster, typically using natural numbers starting from 0 or 1, usually not recycled or reused.

pg_role

Parameter Name: pg_role, Type: enum, Level: I

PostgreSQL instance role, required identity parameter with no default. Current validation accepts primary, replica, standby, offline, and delayed.

The commonly used service-membership labels are:

  • primary: Primary instance, there is one and only one in a cluster.
  • replica: Replica for serving online read-only traffic, may have slight replication delay under high load (10ms~100ms, 100KB).
  • offline: Offline replica for handling offline read-only traffic, such as analytics/ETL/personal queries.

standby and delayed are also valid inventory role values, but these strings alone do not create a standby cluster or delayed replication. Configure the topology with pg_upstream and pg_delay. Role-filtered HBA rules use these inventory labels directly.

pg_instances

Parameter Name: pg_instances, Type: dict, Level: I

Define multiple PostgreSQL instances on a single host using {port:ins_vars} format.

This parameter is reserved for multi-instance deployment on a single node. Pigsty has not yet implemented this feature and strongly recommends dedicated node deployment.

pg_upstream

Parameter Name: pg_upstream, Type: ip, Level: I

Upstream instance IP address for standby cluster or cascade replica.

Setting pg_upstream on the primary instance of a cluster indicates this cluster is a standby cluster, and this instance will act as a standby leader, receiving and applying changes from the upstream cluster.

Setting pg_upstream on a non-primary instance specifies a specific instance as the upstream for physical replication. If different from the primary instance IP address, this instance becomes a cascade replica. It is the user’s responsibility to ensure the upstream IP address is another instance in the same cluster.

pg_shard

Parameter Name: pg_shard, Type: string, Level: C

PostgreSQL horizontal shard name, defaulting to pg_cluster. Specify it explicitly for horizontal-sharding systems made up of multiple physical clusters, such as Citus.

When multiple standard PostgreSQL clusters serve the same business together in a horizontal sharding manner, Pigsty marks this group of clusters as a horizontal sharding cluster.

pg_shard is the shard group name. It is typically a prefix of pg_cluster.

For example, if we have a shard group pg-citus with 4 clusters, their identity parameters would be:

cls pg_shard: pg-citus
cls pg_group = 0:   pg-citus0
cls pg_group = 1:   pg-citus1
cls pg_group = 2:   pg-citus2
cls pg_group = 3:   pg-citus3

pg_group

Parameter Name: pg_group, Type: int, Level: C

PostgreSQL horizontal-sharding cluster index, defaulting to 0. Specify it explicitly for horizontal-sharding systems made up of multiple physical clusters, such as Citus.

This parameter is used in conjunction with pg_shard, typically using non-negative integers as index numbers.

gp_role

Parameter Name: gp_role, Type: enum, Level: C

Greenplum/Matrixdb role of the PostgreSQL cluster, can be master or segment.

  • master: Marks the postgres cluster as a greenplum master instance (coordinator node), this is the default value.
  • segment: Marks the postgres cluster as a greenplum segment cluster (data node).

This parameter is only used for Greenplum/MatrixDB databases (pg_mode is gpsql) and has no meaning for regular PostgreSQL clusters.

pg_exporters

Parameter Name: pg_exporters, Type: dict, Level: C

Additional exporter definitions for monitoring remote PostgreSQL instances, default value: {}

If you want to monitor remote PostgreSQL instances, define them in the pg_exporters parameter on the cluster where the monitoring system resides (Infra node), and use the pgsql-monitor.yml playbook to complete the deployment.

pg_exporters: # list all remote instances here, alloc a unique unused local port as k
    20001: { pg_cluster: pg-foo, pg_seq: 1, pg_host: 10.10.10.10 }
    20004: { pg_cluster: pg-foo, pg_seq: 2, pg_host: 10.10.10.11 }
    20002: { pg_cluster: pg-bar, pg_seq: 1, pg_host: 10.10.10.12 }
    20003: { pg_cluster: pg-bar, pg_seq: 1, pg_host: 10.10.10.13 }

pg_offline_query

Parameter Name: pg_offline_query, Type: bool, Level: I

Set to true to mark this instance as eligible for offline queries. The default is false.

The flag adds the instance to the default offline service candidate set and activates HBA rules with role: offline on that instance. It does not grant connection access by itself; effective access still depends on generated HBA rules, database CONNECT privileges, and role attributes.

Instances with this flag have an effect similar to setting pg_role = offline for the instance, with the only difference being that offline instances by default do not serve replica service requests and exist as dedicated offline/analytics replica instances.

If no dedicated offline instance is available, enable the parameter on a regular replica. To restrict dbrole_offline to these instances, also set role: offline explicitly on the corresponding HBA rule.

PG_BUSINESS

Customize cluster templates: users, databases, services, and permission rules.

Users should pay close attention to this section of parameters, as this is where business declares its required database objects.

Default database users and their credentials. Their passwords must be changed in production.

# postgres business object definition, overwrite in group vars
pg_users: []                      # postgres business users
pg_databases: []                  # postgres business databases
pg_services: []                   # postgres business services
pg_hba_rules: []                  # business hba rules for postgres
pgb_hba_rules: []                 # business hba rules for pgbouncer
pg_crontab: []                    # crontab entries for postgres dbsu
# global credentials, overwrite in global vars
pg_dbsu_password: ''              # dbsu password, empty string means no dbsu password by default
pg_replication_username: replicator
pg_replication_password: DBUser.Replicator
pg_admin_username: dbuser_dba
pg_admin_password: DBUser.DBA
pg_monitor_username: dbuser_monitor
pg_monitor_password: DBUser.Monitor

pg_users

Parameter Name: pg_users, Type: user[], Level: C

PostgreSQL business user list, needs to be defined at the PG cluster level. Default value: [] empty list.

Each array element is a user/role definition, for example:

- name: dbuser_meta               # required, `name` is the only required field for user definition
  password: DBUser.Meta           # optional, password, can be scram-sha-256 hash string or plaintext
  login: true                     # optional, can login by default
  superuser: false                # optional, default false, is superuser?
  createdb: false                 # optional, default false, can create database?
  createrole: false               # optional, default false, can create role?
  inherit: true                   # optional, by default, can this role use inherited privileges?
  replication: false              # optional, default false, can this role do replication?
  bypassrls: false                # optional, default false, can this role bypass row-level security?
  pgbouncer: true                 # optional, default false, add this user to pgbouncer user list? (production users using connection pool should explicitly set to true)
  connlimit: -1                   # optional, user connection limit, default -1 disables limit
  expire_in: 3650                 # optional, this role expires: calculated from creation + n days (higher priority than expire_at)
  expire_at: '2030-12-31'         # optional, when this role expires, use YYYY-MM-DD format string to specify a specific date (lower priority than expire_in)
  comment: pigsty admin user      # optional, description and comment string for this user/role
  roles: [dbrole_admin]           # optional, default roles are: dbrole_{admin,readonly,readwrite,offline}
  parameters: {}                  # optional, use `ALTER ROLE SET` for this role, configure role-level database parameters
  pool_mode: transaction          # optional, pgbouncer pool mode at user level, default transaction
  pool_connlimit: 100             # optional, user-level pool connection limit; omitted values inherit Pigsty's global default of 100
  search_path: public             # optional, key-value config parameter per postgresql docs (e.g., use pigsty as default search_path)

User-level pool quota is consistently defined by pool_connlimit (mapped to Pgbouncer max_user_connections).

pg_databases

Parameter Name: pg_databases, Type: database[], Level: C

PostgreSQL business database list, needs to be defined at the PG cluster level. Default value: [] empty list.

Each array element is a business database definition, for example:

- name: meta                      # required, `name` is the only required field for database definition
  baseline: cmdb.sql              # optional, database sql baseline file path (relative path in ansible search path, e.g., files/)
  pgbouncer: true                 # optional, add this database to pgbouncer database list? default true
  schemas: [pigsty]               # optional, additional schemas to create, array of schema name strings
  extensions:                     # optional, additional extensions to install: array of extension objects
    - { name: postgis , schema: public }  # can specify which schema to install extension into, or not (if not specified, installs to first schema in search_path)
    - { name: timescaledb }               # some extensions create and use fixed schemas, so no need to specify schema
  comment: pigsty meta database   # optional, description and comment for the database
  owner: postgres                 # optional, database owner, default is postgres
  template: template1             # optional, template to use, default is template1, target must be a template database
  encoding: UTF8                  # optional, database encoding, default UTF8 (must match template database)
  locale: C                       # optional, database locale setting, default C (must match template database)
  lc_collate: C                   # optional, database collate rule, default C (must match template database), no reason to change
  lc_ctype: C                     # optional, database ctype character set, default C (must match template database)
  tablespace: pg_default          # optional, default tablespace, default is 'pg_default'
  allowconn: true                 # optional, allow connections, default true. Explicitly set false to completely forbid connections
  revokeconn: false               # optional, revoke public connect privileges. default false, when true, CONNECT privilege revoked from users other than owner and admin
  register_datasource: true       # optional, register this database to grafana datasource? default true, explicitly false skips registration
  connlimit: -1                   # optional, database connection limit, default -1 means no limit, positive integer limits connections
  pool_auth_user: dbuser_meta     # optional, all connections to this pgbouncer database will authenticate using this user (useful when pgbouncer_auth_query enabled)
  pool_mode: transaction          # optional, database-level pgbouncer pooling mode, default transaction
  pool_size: 50                   # optional, database-level pgbouncer default pool size, default 50
  pool_reserve: 30                # optional, database-level pgbouncer pool reserve, default 30, max additional burst connections when default pool insufficient
  pool_size_min: 0                # optional, database-level pgbouncer pool minimum size, default 0
  pool_connlimit: 100             # optional, database-level max database connections, default 100

Since Pigsty v4.1.0, database pool fields are unified as pool_reserve and pool_connlimit; legacy aliases pool_size_reserve / pool_max_db_conn are converged.

In each database definition object, only name is a required field, all other fields are optional.

pg_services

Parameter Name: pg_services, Type: service[], Level: C

PostgreSQL service list, needs to be defined at the PG cluster level. Default value: [], empty list.

Used to define additional services at the database cluster level. Each object in the array defines a service. A complete service definition example:

- name: standby                   # required, service name, final svc name will use `pg_cluster` as prefix, e.g., pg-meta-standby
  port: 5435                      # required, exposed service port (as kubernetes service node port mode)
  ip: "*"                         # optional, IP address to bind service, default is all IP addresses
  selector: "[]"                  # required, service member selector, use JMESPath to filter inventory
  backup: "[? pg_role == `primary`]"  # optional, service member selector (backup), service is handled by these instances when default selector instances are all down
  dest: default                   # optional, target port, default|postgres|pgbouncer|<port_number>, default is 'default', Default means use pg_default_service_dest value to decide
  check: /sync                    # optional, health check URL path, default is /, here uses Patroni API: /sync, only sync standby and primary return 200 health status
  maxconn: 5000                   # optional, max frontend connections allowed, default 5000
  balance: roundrobin             # optional, haproxy load balancing algorithm (default roundrobin, other option: leastconn)
  options: 'inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100'

Note that this parameter is used to add additional services at the cluster level. If you want to globally define services that all PostgreSQL databases should provide, use the pg_default_services parameter.

pg_hba_rules

Parameter Name: pg_hba_rules, Type: hba[], Level: C

Client IP whitelist/blacklist rules for database cluster/instance. Default: [] empty list.

Array of objects, each object represents a rule. HBA rule object definition:

- title: allow intranet password access
  role: common
  rules:
    - host   all  all  10.0.0.0/8      md5
    - host   all  all  172.16.0.0/12   md5
    - host   all  all  192.168.0.0/16  md5
  • title: Rule title name, rendered as comment in HBA file.
  • rules: Rule array, each element is a standard HBA rule string.
  • role: Rule application scope, which instance roles will enable this rule?
    • common: Applies to all instances
    • primary, replica, offline: Only applies to instances with specific pg_role.
    • Special case: role: 'offline' rules apply to instances with pg_role : offline, and also to instances with pg_offline_query flag.

In addition to the native HBA rule definition above, Pigsty also provides a more convenient alias form:

- addr: 'intra'    # world|intra|infra|admin|local|localhost|cluster|<cidr>
  auth: 'pwd'      # trust|pwd|ssl|cert|deny|<official auth method>
  user: 'all'      # all|${dbsu}|${repl}|${admin}|${monitor}|<user>|<group>
  db: 'all'        # all|replication|....
  rules: []        # raw hba string precedence over above all
  title: allow intranet password access

pg_default_hba_rules is similar to this parameter, but it’s used to define global HBA rules, while this parameter is typically used to customize HBA rules for specific clusters/instances.

pgb_hba_rules

Parameter Name: pgb_hba_rules, Type: hba[], Level: C

Pgbouncer business HBA rules, default value: [], empty array.

This parameter is similar to pg_hba_rules, both are arrays of hba rule objects, the difference is that this parameter is for Pgbouncer.

pgb_default_hba_rules is similar to this parameter, but it’s used to define global connection pool HBA rules, while this parameter is typically used to customize HBA rules for specific connection pool clusters/instances.

pg_crontab

Parameter Name: pg_crontab, Type: string[], Level: C

Cron job list for the PostgreSQL database superuser (dbsu, default postgres), default value: [] empty array.

Each array element is a crontab entry line, using standard user crontab format: minute hour day month weekday command (no need to specify username).

pg_crontab:
  - '00 01 * * * /pg/bin/pg-backup full'      # Full backup at 1 AM daily
  - '00 13 * * * /pg/bin/pg-backup'           # Incremental backup at 1 PM daily

This parameter writes cron jobs to the postgres user’s personal crontab file:

  • EL systems: /var/spool/cron/postgres
  • Debian systems: /var/spool/cron/crontabs/postgres

Note: This parameter replaces the old practice of configuring postgres user tasks in node_crontab. Because node_crontab is written to /etc/crontab during NODE initialization, the postgres user may not exist yet, causing cron errors.

pg_replication_username

Parameter Name: pg_replication_username, Type: username, Level: G

PostgreSQL physical replication username, default is replicator, not recommended to change this parameter.

pg_replication_password

Parameter Name: pg_replication_password, Type: password, Level: G

PostgreSQL physical replication user password, default value: DBUser.Replicator.

Warning

Change this password in production environments.

pg_admin_username

Parameter Name: pg_admin_username, Type: username, Level: G

PostgreSQL / Pgbouncer admin name, default: dbuser_dba.

This is the globally used database administrator with database Superuser privileges and connection pool traffic management permissions. Please control its usage scope.

pg_admin_password

Parameter Name: pg_admin_password, Type: password, Level: G

PostgreSQL / Pgbouncer admin password, default: DBUser.DBA.

Warning

Change this password in production environments.

pg_monitor_username

Parameter Name: pg_monitor_username, Type: username, Level: G

PostgreSQL/Pgbouncer monitor username, default: dbuser_monitor.

This is a database/connection pool user for monitoring, not recommended to change this username.

However, if your existing database uses a different monitor user, you can use this parameter to specify the monitor username when defining monitoring targets.

pg_monitor_password

Parameter Name: pg_monitor_password, Type: password, Level: G

Password used by PostgreSQL/Pgbouncer monitor user, default: DBUser.Monitor.

Try to avoid using characters like @:/ that can be confused with URL delimiters in passwords to reduce unnecessary trouble.

Warning

Change this password in production environments.

pg_dbsu_password

Parameter Name: pg_dbsu_password, Type: password, Level: G/C

PostgreSQL pg_dbsu superuser password, default is empty string, meaning no password is set.

We don’t recommend configuring password login for dbsu as it increases the attack surface. The exception is: pg_mode = citus, in which case you need to configure a password for each shard cluster’s dbsu to allow connections within the shard cluster.

PG_INSTALL

This section is responsible for installing PostgreSQL and its extensions. If you want to install different major versions and extension plugins, just modify pg_version and pg_extensions. Note that not all extensions are available for all major versions.

pg_dbsu: postgres                 # os dbsu name, default is postgres, better not change it
pg_dbsu_uid: 26                   # os dbsu uid and gid, default is 26, for default postgres user and group
pg_dbsu_sudo: limit               # dbsu sudo privilege, none,limit,all,nopass. default is limit
pg_dbsu_home: /var/lib/pgsql      # postgresql home directory, default is `/var/lib/pgsql`
pg_dbsu_ssh_exchange: true        # exchange postgres dbsu ssh key among same pgsql cluster
pg_version: 18                    # postgres major version to be installed, default is 18
pg_bin_dir: /usr/pgsql/bin        # postgres binary dir, default is `/usr/pgsql/bin`
pg_log_dir: /pg/log/postgres      # postgres log dir, default is `/pg/log/postgres`
pg_packages:                      # pg packages to be installed, alias can be used
  - pgsql-main pgsql-common
pg_extensions: []                 # pg extensions to be installed, alias can be used

pg_dbsu

Parameter Name: pg_dbsu, Type: username, Level: C

OS dbsu username used by PostgreSQL, default is postgres, changing this username is not recommended.

However, in certain situations, you may need a username different from postgres, for example, when installing and configuring Greenplum / MatrixDB, you need to use gpadmin / mxadmin as the corresponding OS superuser.

pg_dbsu_uid

Parameter Name: pg_dbsu_uid, Type: int, Level: C

OS database superuser uid and gid, 26 is the default postgres user UID/GID from PGDG RPM.

For Debian/Ubuntu systems, there is no default value, and user 26 is often taken. Therefore, when Pigsty detects the installation environment is Debian-based and uid is 26, it will automatically use the replacement pg_dbsu_uid = 543.

pg_dbsu_sudo

Parameter Name: pg_dbsu_sudo, Type: enum, Level: C

Database superuser sudo privilege, can be none, limit, all, or nopass. Default is limit

  • none: No sudo privilege

  • limit: Limited sudo privilege for executing systemctl commands for database-related components (default option).

  • all: Full sudo privilege, requires password.

  • nopass: Full sudo privilege without password (not recommended).

  • Default value is limit, only allows executing sudo systemctl <start|stop|reload> <postgres|patroni|pgbouncer|...>.

pg_dbsu_home

Parameter Name: pg_dbsu_home, Type: path, Level: C

PostgreSQL home directory, default is /var/lib/pgsql, consistent with official pgdg RPM.

pg_dbsu_ssh_exchange

Parameter Name: pg_dbsu_ssh_exchange, Type: bool, Level: C

Whether to exchange OS dbsu ssh keys within the same PostgreSQL cluster?

Default is true, meaning database superusers in the same cluster can ssh to each other.

The exchange set comes from pg_cluster_members, which matches actual hosts with the same pg_cluster across the current inventory; it does not require an Ansible group named after the cluster. The execution-time -l limit still restricts playbook targets, so ensure it covers every member that needs configuration.

pg_version

Parameter Name: pg_version, Type: enum, Level: C

PostgreSQL major version to install, default is 18.

Note that PostgreSQL physical streaming replication cannot cross major versions, so it’s best not to configure this at the instance level.

You can use parameters in pg_packages and pg_extensions to install different packages and extensions for specific PG major versions.

pg_bin_dir

Parameter Name: pg_bin_dir, Type: path, Level: C

PostgreSQL binary directory, default is /usr/pgsql/bin.

The default value is a symlink manually created during installation, pointing to the specific installed Postgres version directory.

For example /usr/pgsql -> /usr/pgsql-15. On Ubuntu/Debian it points to /usr/lib/postgresql/15/bin.

For more details, see PGSQL File Structure.

pg_log_dir

Parameter Name: pg_log_dir, Type: path, Level: C

PostgreSQL log directory, default: /pg/log/postgres. The Vector log agent uses this variable to collect PostgreSQL logs.

Note that if the log directory pg_log_dir is prefixed with the data directory pg_data, it won’t be explicitly created (created automatically during data directory initialization).

pg_packages

Parameter Name: pg_packages, Type: string[], Level: C

PostgreSQL packages to install (RPM/DEB), this is an array of package names where elements can be space or comma-separated package aliases.

Pigsty v4 converges the default value to two aliases:

pg_packages:
  - pgsql-main pgsql-common
  • pgsql-main: Maps to PostgreSQL kernel, client, PL languages, and core extensions like pg_repack, wal2json, pgvector on the current platform.
  • pgsql-common: Maps to companion components required for running the database, such as Patroni, Pgbouncer, pgBackRest, pg_exporter, vip-manager, and other daemons.

Alias definitions can be found in pg_package_map under roles/node_id/vars/. Pigsty first resolves aliases based on OS and architecture, then replaces $v/${pg_version} with the actual major version pg_version, and finally installs the real packages. This shields package name differences between distributions.

If additional packages are needed (e.g., specific FDW or extensions), you can append aliases or real package names directly to pg_packages. But remember to keep pgsql-main pgsql-common, otherwise core components will be missing.

pg_extensions

Parameter Name: pg_extensions, Type: string[], Level: G/C

PostgreSQL extension packages to install (RPM/DEB), this is an array of extension package names or aliases.

Starting from v4, the default value is an empty list []. Pigsty no longer forces installation of large extensions, users can choose as needed to avoid extra disk and dependency usage.

To install extensions, fill in like this:

pg_extensions:
  - postgis timescaledb pgvector
  - pgsql-fdw     # use alias to install common FDWs at once

pg_package_map provides aliases that hide package-name differences between distributions. The following examples use single-extension aliases and category groups from the current v4.5 EL9 map; availability can differ by platform and PostgreSQL major version:

pg_extensions: # extensions to be installed on this cluster
  - timescaledb postgis pgvector pg_search pg_duckdb pg_repack wal2json
  # You can also select current category groups; install only the groups you need
  # - pgsql-time pgsql-gis pgsql-rag pgsql-fts pgsql-olap
  # - pgsql-feat pgsql-lang pgsql-type pgsql-util pgsql-func
  # - pgsql-admin pgsql-stat pgsql-sec pgsql-fdw pgsql-sim pgsql-etl

For the exact mapping, consult roles/node_id/vars/<os>.<arch>.yml for the target platform and the current extension catalog. pg_analytics and spat were removed from the v4.5 catalog and current mainstream-platform maps; do not copy them from older examples.

PG_BOOTSTRAP

Bootstrap PostgreSQL cluster with Patroni and set up 1:1 corresponding Pgbouncer connection pool.

It also initializes the database cluster with default roles, users, privileges, schemas, and extensions defined in PG_PROVISION.

The following parameters configure the PGSQL bootstrap stage. The internal variable pg_data always represents the /pg/data symlink and must not be overridden in inventory. Configure pg_fs_main to change the physical location of the primary data directory.

pg_fs_main: /data/postgres        # postgres main data directory, `/data/postgres` by default
pg_fs_backup: /data/backups       # postgres backup data directory, `/data/backups` by default
pg_storage_type: SSD              # storage type for pg main data, SSD,HDD, SSD by default
pg_dummy_filesize: 64MiB          # size of `/pg/dummy`, hold 64MB disk space for emergency use
pg_listen: '0.0.0.0'              # postgres/pgbouncer listen addresses, comma separated list
pg_port: 5432                     # postgres listen port, 5432 by default
pg_localhost: /var/run/postgresql # postgres unix socket dir for localhost connection
patroni_enabled: true             # if disabled, no postgres cluster will be created during init
patroni_mode: default             # patroni working mode: default,pause,remove
pg_namespace: /pg                 # top level key namespace in etcd, used by patroni & vip
patroni_port: 8008                # patroni listen port, 8008 by default
patroni_log_dir: /pg/log/patroni  # patroni log dir, `/pg/log/patroni` by default
patroni_ssl_enabled: false        # secure patroni RestAPI communications with SSL?
patroni_watchdog_mode: off        # patroni watchdog mode: automatic,required,off. off by default
patroni_username: postgres        # patroni restapi username, `postgres` by default
patroni_password: Patroni.API     # patroni restapi password, `Patroni.API` by default
pg_etcd_password: ''              # etcd password for this pg cluster, '' to use pg_cluster
pg_primary_db: postgres           # primary database name, used by citus,etc... ,postgres by default
pg_parameters: {}                 # extra parameters in postgresql.auto.conf
pg_files: []                      # extra files to be copied to postgres data directory (e.g. license)
pg_conf: oltp.yml                 # config template: oltp,olap,crit,tiny. `oltp.yml` by default
pg_max_conn: auto                 # postgres max connections, `auto` will use recommended value
pg_shared_buffer_ratio: 0.25      # postgres shared buffers ratio, 0.25 by default, 0.1~0.4
pg_io_method: worker              # io method for postgres, auto,sync,worker,io_uring, worker by default
pg_rto: norm                      # shared RTO mode: fast,norm,safe,wide
pg_rpo: 1048576                   # sampled failover-candidate lag threshold in bytes, `1MiB` by default
pg_libs: 'pg_stat_statements, auto_explain'  # preloaded libraries, `pg_stat_statements,auto_explain` by default
pg_delay: 0                       # replication apply delay for standby cluster leader
pg_checksum: true                 # enable data checksum for postgres cluster?
pg_pwd_enc: scram-sha-256         # passwords encryption algorithm: fixed to scram-sha-256
pg_encoding: UTF8                 # database cluster encoding, `UTF8` by default
pg_locale: C                      # database cluster local, `C` by default
pg_lc_collate: C                  # database cluster collate, `C` by default
pg_lc_ctype: C                    # database character type, `C` by default
#pgsodium_key: ""                 # pgsodium key, 64 hex digit, default to sha256(pg_cluster)
#pgsodium_getkey_script: ""       # pgsodium getkey script path, pgsodium_getkey by default

pg_data

Internal Variable: pg_data, Type: path

pg_data is an internal Pigsty variable, not a user configuration parameter. It always represents the PostgreSQL data-directory symlink at /pg/data.

Patroni templates, maintenance scripts, and cleanup workflows all rely on this symlink. Do not override or modify it in pigsty.yml. Configure pg_fs_main to change the underlying physical data location. See PGSQL File Structure for details.

pg_fs_main

Parameter Name: pg_fs_main, Type: path, Level: C

Mount point/file system path for PostgreSQL main data disk, default is /data/postgres.

Default value: /data/postgres, which will be used directly as the parent directory of PostgreSQL main data directory.

NVME SSD is recommended for PostgreSQL main data storage. Pigsty is optimized for SSD storage by default, but also supports HDD.

You can change pg_storage_type to HDD for HDD storage optimization.

pg_fs_backup

Parameter Name: pg_fs_backup, Type: path, Level: C

Mount point/file system path for PostgreSQL backup data disk, default is /data/backups.

If you’re using the default pgbackrest_method = local, it’s recommended to use a separate disk for backup storage.

The backup disk should be large enough to hold all backups, at least sufficient for 3 base backups + 2 days of WAL archives. Usually capacity isn’t a big issue since you can use cheap large HDDs as backup disks.

It’s recommended to use a separate disk for backup storage, otherwise Pigsty will fall back to the main data disk and consume main data disk capacity and IO.

pg_storage_type

Parameter Name: pg_storage_type, Type: enum, Level: C

Type of PostgreSQL data storage media: SSD or HDD, default is SSD.

Default value: SSD, which affects some tuning parameters like random_page_cost and effective_io_concurrency.

pg_dummy_filesize

Parameter Name: pg_dummy_filesize, Type: size, Level: C

Size of /pg/dummy, default is 64MiB, 64MB disk space for emergency use.

When disk is full, deleting the placeholder file can free some space for emergency use. Recommend at least 8GiB for production.

pg_listen

Parameter Name: pg_listen, Type: ip, Level: C

PostgreSQL / Pgbouncer listen address, default is 0.0.0.0 (all ipv4 addresses).

You can use placeholders in this variable, for example: '${ip},${lo}' or '${ip},${vip},${lo}':

  • ${ip}: Translates to inventory_hostname, which is the primary internal IP address defined in the inventory.
  • ${vip}: If pg_vip_enabled is enabled, will use the host part of pg_vip_address.
  • ${lo}: Will be replaced with 127.0.0.1

For production environments with high security requirements, it’s recommended to restrict listen IP addresses.

pg_port

Parameter Name: pg_port, Type: port, Level: C

Port that PostgreSQL server listens on, default is 5432.

pg_localhost

Parameter Name: pg_localhost, Type: path, Level: C

Unix socket directory for localhost PostgreSQL connection, default is /var/run/postgresql.

Unix socket directory for PostgreSQL and Pgbouncer local connections. pg_exporter and patroni will preferentially use Unix sockets to access PostgreSQL.

pg_namespace

Parameter Name: pg_namespace, Type: path, Level: C

Top-level namespace used in etcd, used by patroni and vip-manager, default is: /pg, not recommended to change.

patroni_enabled

Parameter Name: patroni_enabled, Type: bool, Level: C

Enable Patroni? Default is: true.

If disabled, no Postgres cluster will be created during initialization. Pigsty will skip the task of starting patroni, which can be used when trying to add some components to existing postgres instances.

patroni_mode

Parameter Name: patroni_mode, Type: enum, Level: C

Patroni working mode: default, pause, remove. Default: default.

  • default: Normal use of Patroni to bootstrap PostgreSQL cluster
  • pause: Similar to default, but enters maintenance mode after bootstrap
  • remove: Use Patroni to initialize cluster, then remove Patroni and use raw PostgreSQL.

patroni_port

Parameter Name: patroni_port, Type: port, Level: C

Patroni listen port, default is 8008, not recommended to change.

Patroni API server listens on this port for health checks and API requests.

patroni_log_dir

Parameter Name: patroni_log_dir, Type: path, Level: C

Patroni log directory, default is /pg/log/patroni, collected by Vector log agent.

patroni_ssl_enabled

Parameter Name: patroni_ssl_enabled, Type: bool, Level: G

Secure patroni RestAPI communications with SSL? Default is false.

This parameter is a global flag that can only be set before deployment. Because if SSL is enabled for patroni, you will have to use HTTPS instead of HTTP for health checks, fetching metrics, and calling APIs.

patroni_watchdog_mode

Parameter Name: patroni_watchdog_mode, Type: string, Level: C

Patroni watchdog mode: automatic, required, off, default is off.

In case of primary failure, Patroni can use watchdog to force shutdown old primary node to avoid split-brain.

  • off: Don’t use watchdog. No fencing at all (default behavior)
  • automatic: Enable watchdog if kernel has softdog module enabled and watchdog belongs to dbsu.
  • required: Force enable watchdog, refuse to start Patroni/PostgreSQL if softdog unavailable.

Default is off. You should not enable watchdog on Infra nodes. Critical systems where data consistency takes priority over availability, especially business clusters involving money, can consider enabling this option.

Note that if all your access traffic uses HAproxy health check service access, there is normally no split-brain risk.

patroni_username

Parameter Name: patroni_username, Type: username, Level: C

Patroni REST API username, default is postgres, used with patroni_password.

Patroni’s dangerous REST APIs (like restarting cluster) are protected by additional username/password. See Configure Cluster and Patroni RESTAPI for details.

patroni_password

Parameter Name: patroni_password, Type: password, Level: C

Patroni REST API password, default is Patroni.API.

Warning

Change this parameter in production environments.

pg_primary_db

Parameter Name: pg_primary_db, Type: string, Level: C

Specify the primary database name in the cluster, used for citus and other business databases, default is postgres.

For example, when using Patroni to manage HA Citus clusters, you must choose a “primary database”.

Additionally, the database name specified here will be displayed in the printed connection string after PGSQL module installation is complete.

pg_parameters

Parameter Name: pg_parameters, Type: dict, Level: G/C/I

Used to specify and manage configuration parameters in postgresql.auto.conf.

After all cluster instances are initialized, the pg_param task will write the key/value pairs from this dictionary sequentially to /pg/data/postgresql.auto.conf.

Note

Do not manually modify this configuration file or change cluster parameters with ALTER SYSTEM; the next configuration sync will overwrite those changes.

This variable has higher priority than cluster configuration in Patroni / DCS (i.e., higher priority than cluster configuration edited by Patroni edit-config), so it can typically be used to override cluster default parameters at instance level.

When your cluster members have different specifications (not recommended!), you can use this parameter for fine-grained configuration management of each instance.

pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary , pg_parameters: { shared_buffers: '5GB' } }
    10.10.10.12: { pg_seq: 2, pg_role: replica , pg_parameters: { shared_buffers: '4GB' } }
    10.10.10.13: { pg_seq: 3, pg_role: replica , pg_parameters: { shared_buffers: '3GB' } }

Note that some important cluster parameters (with requirements on primary/replica parameter values) are managed directly by Patroni via command line arguments, have highest priority, and cannot be overridden this way. For these parameters, you must use Patroni edit-config for management and configuration.

PostgreSQL parameters that must be consistent on primary and replicas (inconsistency will cause replica to fail to start!):

  • wal_level
  • max_connections
  • max_locks_per_transaction
  • max_worker_processes
  • max_prepared_transactions
  • track_commit_timestamp

Parameters that should preferably be consistent on primary and replicas (considering possibility of failover):

  • listen_addresses
  • port
  • cluster_name
  • hot_standby
  • wal_log_hints
  • max_wal_senders
  • max_replication_slots
  • wal_keep_segments
  • wal_keep_size

You can set non-existent parameters (e.g., GUCs from extensions, thus configuring “not yet existing” parameters that ALTER SYSTEM cannot modify), but modifying existing configuration to illegal values may cause PostgreSQL to fail to start, configure with caution!

pg_files

Parameter Name: pg_files, Type: path[], Level: C

Used to specify a list of files to be copied to the PGDATA directory, default is empty array: []

Files specified in this parameter will be copied to the {{ pg_data }} directory, mainly used to distribute license files required by special commercial PostgreSQL kernels.

Currently only PolarDB (Oracle compatible) kernel requires license files. For example, you can place the license.lic file in the files/ directory and specify in pg_files:

pg_files: [ license.lic ]

pg_conf

Parameter Name: pg_conf, Type: enum, Level: C

Configuration template: {oltp,olap,crit,tiny}.yml, default is oltp.yml.

  • tiny.yml: Optimized for small nodes, VMs, small demos (1-8 cores, 1-16GB)
  • oltp.yml: Optimized for OLTP workloads and latency-sensitive applications (4C8GB+) (default template)
  • olap.yml: Optimized for OLAP workloads and throughput (4C8G+)
  • crit.yml: Optimized for data consistency and critical applications (4C8G+)

Default is oltp.yml, but the configure script will set this to tiny.yml when current node is a small node.

You can have your own templates, just place them under templates/<mode>.yml and set this value to the template name to use.

pg_max_conn

Parameter Name: pg_max_conn, Type: int, Level: C

PostgreSQL server max connections. You can choose a value between 50 and 5000, or use auto for recommended value.

Default is auto, which sets max connections based on pg_conf and pg_default_service_dest.

  • tiny: 100
  • olap: 200
  • oltp: 200 (pgbouncer) / 1000 (postgres)
    • pg_default_service_dest = pgbouncer : 200
    • pg_default_service_dest = postgres : 1000
  • crit: 200 (pgbouncer) / 1000 (postgres)
    • pg_default_service_dest = pgbouncer : 200
    • pg_default_service_dest = postgres : 1000

Not recommended to set this value above 5000, otherwise you’ll need to manually increase haproxy service connection limits.

Pgbouncer’s transaction pool can mitigate excessive OLTP connection issues, so setting a large connection count is not recommended by default.

For OLAP scenarios, change pg_default_service_dest to postgres to bypass connection pooling.

pg_shared_buffer_ratio

Parameter Name: pg_shared_buffer_ratio, Type: float, Level: C

Postgres shared buffer memory ratio, default is 0.25, normal range is 0.1~0.4.

Default: 0.25, meaning 25% of node memory will be used as PostgreSQL’s shared buffer. If you want to enable huge pages for PostgreSQL, this value should be appropriately smaller than node_hugepage_ratio.

Setting this value above 0.4 (40%) is usually not a good idea, but may be useful in extreme cases.

Note that shared buffers are only part of PostgreSQL’s shared memory. To calculate total shared memory, use show shared_memory_size_in_huge_pages;.

pg_rto

Parameter Name: pg_rto, Type: enum, Level: C

Recovery Time Objective (RTO) mode controlling Patroni and HAProxy timeout parameters. The default is norm.

Pigsty provides four presets tuned for different network conditions and deployment scenarios:

Mode Scenario Network conditions Source target RTO Patroni TTL False-failover risk
fast Same rack or switch < 1 ms, highly stable < 30s 20s Higher
norm Same datacenter 1-5 ms, normal < 45s 30s Medium
safe Cross-datacenter 10-50 ms < 90s 60s Lower
wide Cross-region/continent 100-200 ms, public WAN < 150s 120s Lowest

A shorter RTO speeds recovery but increases the risk that network jitter is mistaken for failure. Choose a mode that matches your network conditions. See RTO Trade-offs for details.

The current template recognizes only these four string keys. Other values, including numbers, fall back to norm; they are not interpreted as seconds. To define another timeout combination, extend pg_rto_plan with a new key and use that key as pg_rto.

pg_rto: norm   # default, suitable for a single datacenter
pg_rto: safe   # recommended for cross-datacenter deployments
# pg_rto: 30   # invalid key; current templates fall back to norm

pg_rto_plan

Parameter Name: pg_rto_plan, Type: dict, Level: G

Dictionary of RTO presets defining Patroni HA and HAProxy health-check timeouts. The defaults contain four modes:

pg_rto_plan:  # [ttl, loop, retry, start, margin, inter, fastinter, downinter, rise, fall]
  fast: [ 20  ,5  ,5  ,15 ,5  ,'1s' ,'0.5s' ,'1s' ,3 ,3 ]  # rto < 30s
  norm: [ 30  ,5  ,10 ,25 ,5  ,'2s' ,'1s'   ,'2s' ,3 ,3 ]  # rto < 45s
  safe: [ 60  ,10 ,20 ,45 ,10 ,'3s' ,'1.5s' ,'3s' ,3 ,3 ]  # rto < 90s
  wide: [ 120 ,20 ,30 ,95 ,15 ,'4s' ,'2s'   ,'4s' ,3 ,3 ]  # rto < 150s

Each mode is an array of ten values controlling Patroni and HAProxy together:

Index Name Component Description
0 ttl Patroni Primary-lock TTL in seconds
1 loop_wait Patroni Main-loop sleep interval
2 retry_timeout Patroni DCS/PostgreSQL retry timeout
3 primary_start_timeout Patroni Time allowed for primary recovery
4 safety_margin Patroni Watchdog safety margin
5 inter HAProxy Health-check interval
6 fastinter HAProxy Fast interval during state transitions
7 downinter HAProxy Check interval while a server is down
8 rise HAProxy Consecutive successes required for UP
9 fall HAProxy Consecutive failures required for DOWN

Override this dictionary to customize an existing mode or add a new one:

pg_rto_plan:
  ultra: [ 10, 2, 3, 8, 2, '0.5s', '0.25s', '0.5s', 2, 2 ]  # aggressive; low-latency networks only

Caution: inappropriate timeout combinations can make the cluster unstable or cause frequent false failovers.

pg_rpo

Parameter Name: pg_rpo, Type: int, Level: C

Failover-candidate lag threshold in bytes, default: 1048576 (1 MiB). It is written to Patroni’s maximum_lag_on_failover and determines whether a replica is eligible for failover; it is not a hard upper bound on actual data loss. With asynchronous replication, worst-case loss also depends on write rate and when Patroni sampled the primary’s WAL position.

When the primary goes down and all replicas are lagging, you must make a difficult choice, trade-off between availability and consistency:

  • Promote a lagging replica and restore service as soon as possible, accepting possible data loss.
  • Wait for primary to come back online (may never happen), or manual intervention to avoid any data loss.

You can use the crit.yml conf template to ensure no data loss during failover, but this sacrifices some performance.

pg_libs

Parameter Name: pg_libs, Type: string, Level: C

Preloaded dynamic shared libraries, default is pg_stat_statements,auto_explain, two PostgreSQL built-in extensions that are strongly recommended to enable.

For existing clusters, you can directly configure cluster shared_preload_libraries parameter and apply.

If you want to use TimescaleDB or Citus extensions, you need to add timescaledb or citus to this list. timescaledb and citus should be placed at the front of this list, for example:

citus,timescaledb,pg_stat_statements,auto_explain

Other extensions requiring dynamic loading can also be added to this list, such as pg_cron, pgml, etc. Typically citus and timescaledb have highest priority and should be added to the front of the list.

pg_delay

Parameter Name: pg_delay, Type: interval, Level: I

Delayed standby replication delay, default: 0.

If this value is set to a positive value, the standby cluster leader will be delayed by this time before applying WAL changes. Setting to 1h means data in this cluster will always lag the original cluster by one hour.

See Delayed Standby Cluster for details.

pg_checksum

Parameter Name: pg_checksum, Type: bool, Level: C

Enable data checksum for PostgreSQL cluster? Default is true, enabled.

This parameter can only be set before PGSQL deployment (but you can enable it manually later).

Data checksums help detect disk corruption and hardware failures. This feature is enabled by default since Pigsty v3.5 to ensure data integrity.

pg_pwd_enc

Parameter Name: pg_pwd_enc, Type: enum, Level: C

Password encryption algorithm, fixed to scram-sha-256 since Pigsty v4.

All new users will use SCRAM credentials. md5 has been deprecated. For compatibility with old clients, upgrade to SCRAM in business connection pools or client drivers.

pg_encoding

Parameter Name: pg_encoding, Type: enum, Level: C

Database cluster encoding, default is UTF8.

Using other non-UTF8 encodings is not recommended.

pg_locale

Parameter Name: pg_locale, Type: enum, Level: C

Database cluster locale, default is C.

This parameter controls the database’s default Locale setting, affecting collation, character classification, and other behaviors. Using C or POSIX provides best performance and predictable sorting behavior.

If you need specific language localization support, you can set it to the corresponding Locale, such as en_US.UTF-8 or zh_CN.UTF-8. Note that Locale settings affect index sort order, so they cannot be changed after cluster initialization.

pg_lc_collate

Parameter Name: pg_lc_collate, Type: enum, Level: C

Database cluster collation, default is C.

Unless you know what you’re doing, modifying cluster-level collation settings is not recommended.

pg_lc_ctype

Parameter Name: pg_lc_ctype, Type: enum, Level: C

Database character set CTYPE, default is C.

Starting from Pigsty v3.5, to be consistent with pg_lc_collate, the default value changed to C.

pg_io_method

Parameter Name: pg_io_method, Type: enum, Level: C

PostgreSQL IO method, default is worker. Available options include:

  • auto: Automatically select based on operating system, uses io_uring on Debian-based systems or EL 10+, otherwise uses worker
  • sync: Use traditional synchronous IO method
  • worker: Use background worker processes to handle IO (default option)
  • io_uring: Use Linux’s io_uring asynchronous IO interface

Current tuning templates write this parameter only for PostgreSQL 18 and later, where it controls the asynchronous I/O execution method.

  • PostgreSQL 18’s actual GUC values are worker, io_uring, and sync; auto is Pigsty template-selection logic, not a PostgreSQL enum value.
  • PostgreSQL 18 defaults to worker, which uses background processes for asynchronous I/O.
  • If you’re using Debian 12/Ubuntu 22+ or EL 10+ systems and want optimal IO performance, consider setting this to io_uring.

Note that setting this value on systems that don’t support io_uring may cause PostgreSQL startup to fail, so auto or worker are safer choices.

pg_etcd_password

Parameter Name: pg_etcd_password, Type: password, Level: C

The password used by this PostgreSQL cluster in etcd, default is empty string ''.

If set to empty string, the pg_cluster parameter value will be used as the password (for Citus clusters, the pg_shard parameter value is used).

This password is used for authentication when Patroni connects to etcd and when vip-manager accesses etcd.

pgsodium_key

Parameter Name: pgsodium_key, Type: string, Level: C

The encryption master key for the pgsodium extension, consisting of 64 hexadecimal digits.

This parameter is not set by default. If not specified, Pigsty will automatically generate a deterministic key using the value of sha256(pg_cluster).

pgsodium is a PostgreSQL extension based on libsodium that provides encryption functions and transparent column encryption capabilities. If you need to use pgsodium’s encryption features, it’s recommended to explicitly specify a secure random key and keep it safe.

Example command to generate a random key:

openssl rand -hex 32   # Generate 64-digit hexadecimal key

pgsodium_getkey_script

Parameter Name: pgsodium_getkey_script, Type: path, Level: C

Path to the pgsodium key retrieval script, default uses the pgsodium_getkey script from Pigsty templates.

This script is used to retrieve pgsodium’s master key when PostgreSQL starts. The default script reads the key from environment variables or configuration files.

If you have custom key management requirements (such as using HashiCorp Vault, AWS KMS, etc.), you can provide a custom script path.

PG_PROVISION

If PG_BOOTSTRAP is about creating a new cluster, then PG_PROVISION is about creating default objects in the cluster, including:

pg_provision: true                # provision postgres cluster after bootstrap
pg_init: pg-init                  # init script for cluster template, default is `pg-init`
pg_default_roles:                 # default roles and users in postgres cluster
  - { name: dbrole_readonly  ,login: false ,comment: role for global read-only access     }
  - { name: dbrole_offline   ,login: false ,comment: role for restricted read-only access }
  - { name: dbrole_readwrite ,login: false ,roles: [dbrole_readonly] ,comment: role for global read-write access }
  - { name: dbrole_admin     ,login: false ,roles: [pg_monitor, dbrole_readwrite] ,comment: role for object creation }
  - { name: postgres     ,superuser: true  ,comment: system superuser }
  - { name: replicator ,replication: true  ,roles: [pg_monitor, dbrole_readonly] ,comment: system replicator }
  - { name: dbuser_dba   ,superuser: true  ,roles: [dbrole_admin]  ,pgbouncer: true ,pool_mode: session, pool_connlimit: 16 ,comment: pgsql admin user }
  - { name: dbuser_monitor ,roles: [pg_monitor, dbrole_readonly] ,pgbouncer: true ,parameters: {log_min_duration_statement: 1000 } ,pool_mode: session ,pool_connlimit: 8 ,comment: pgsql monitor user }
pg_default_privileges:            # default privileges when admin user creates objects
  - GRANT USAGE      ON SCHEMAS   TO dbrole_readonly
  - GRANT SELECT     ON TABLES    TO dbrole_readonly
  - GRANT SELECT     ON SEQUENCES TO dbrole_readonly
  - GRANT EXECUTE    ON FUNCTIONS TO dbrole_readonly
  - GRANT USAGE      ON SCHEMAS   TO dbrole_offline
  - GRANT SELECT     ON TABLES    TO dbrole_offline
  - GRANT SELECT     ON SEQUENCES TO dbrole_offline
  - GRANT EXECUTE    ON FUNCTIONS TO dbrole_offline
  - GRANT INSERT     ON TABLES    TO dbrole_readwrite
  - GRANT UPDATE     ON TABLES    TO dbrole_readwrite
  - GRANT DELETE     ON TABLES    TO dbrole_readwrite
  - GRANT USAGE      ON SEQUENCES TO dbrole_readwrite
  - GRANT UPDATE     ON SEQUENCES TO dbrole_readwrite
  - GRANT TRUNCATE   ON TABLES    TO dbrole_admin
  - GRANT REFERENCES ON TABLES    TO dbrole_admin
  - GRANT TRIGGER    ON TABLES    TO dbrole_admin
  - GRANT CREATE     ON SCHEMAS   TO dbrole_admin
pg_default_schemas: [ monitor ]   # default schemas
pg_default_extensions:            # default extensions
  - { name: pg_stat_statements ,schema: monitor }
  - { name: pgstattuple        ,schema: monitor }
  - { name: pg_buffercache     ,schema: monitor }
  - { name: pageinspect        ,schema: monitor }
  - { name: pg_prewarm         ,schema: monitor }
  - { name: pg_visibility      ,schema: monitor }
  - { name: pg_freespacemap    ,schema: monitor }
  - { name: postgres_fdw       ,schema: public  }
  - { name: file_fdw           ,schema: public  }
  - { name: btree_gist         ,schema: public  }
  - { name: btree_gin          ,schema: public  }
  - { name: pg_trgm            ,schema: public  }
  - { name: intagg             ,schema: public  }
  - { name: intarray           ,schema: public  }
  - { name: pg_repack }
pg_reload: true                   # reload config after HBA changes?
pg_default_hba_rules:             # postgres default HBA rules, ordered by `order`
  - {user: '${dbsu}'    ,db: all         ,addr: local     ,auth: ident ,title: 'dbsu access via local os user ident'  ,order: 100}
  - {user: '${dbsu}'    ,db: replication ,addr: local     ,auth: ident ,title: 'dbsu replication from local os ident' ,order: 150}
  - {user: '${repl}'    ,db: replication ,addr: localhost ,auth: pwd   ,title: 'replicator replication from localhost',order: 200}
  - {user: '${repl}'    ,db: replication ,addr: intra     ,auth: pwd   ,title: 'replicator replication from intranet' ,order: 250}
  - {user: '${repl}'    ,db: postgres    ,addr: intra     ,auth: pwd   ,title: 'replicator postgres db from intranet' ,order: 300}
  - {user: '${monitor}' ,db: all         ,addr: localhost ,auth: pwd   ,title: 'monitor from localhost with password' ,order: 350}
  - {user: '${monitor}' ,db: all         ,addr: infra     ,auth: pwd   ,title: 'monitor from infra host with password',order: 400}
  - {user: '${admin}'   ,db: all         ,addr: infra     ,auth: ssl   ,title: 'admin @ infra nodes with pwd & ssl'   ,order: 450}
  - {user: '${admin}'   ,db: all         ,addr: world     ,auth: ssl   ,title: 'admin @ everywhere with ssl & pwd'    ,order: 500}
  - {user: '+dbrole_readonly',db: all    ,addr: localhost ,auth: pwd   ,title: 'pgbouncer read/write via local socket',order: 550}
  - {user: '+dbrole_readonly',db: all    ,addr: intra     ,auth: pwd   ,title: 'read/write biz user via password'     ,order: 600}
  - {user: '+dbrole_offline' ,db: all    ,addr: intra     ,auth: pwd   ,title: 'allow etl offline tasks from intranet',order: 650}
pgb_default_hba_rules:            # pgbouncer default HBA rules, ordered by `order`
  - {user: '${dbsu}'    ,db: pgbouncer   ,addr: local     ,auth: peer  ,title: 'dbsu local admin access with os ident',order: 100}
  - {user: 'all'        ,db: all         ,addr: localhost ,auth: pwd   ,title: 'allow all user local access with pwd' ,order: 150}
  - {user: '${monitor}' ,db: pgbouncer   ,addr: intra     ,auth: pwd   ,title: 'monitor access via intranet with pwd' ,order: 200}
  - {user: '${monitor}' ,db: all         ,addr: world     ,auth: deny  ,title: 'reject all other monitor access addr' ,order: 250}
  - {user: '${admin}'   ,db: all         ,addr: intra     ,auth: pwd   ,title: 'admin access via intranet with pwd'   ,order: 300}
  - {user: '${admin}'   ,db: all         ,addr: world     ,auth: deny  ,title: 'reject all other admin access addr'   ,order: 350}
  - {user: 'all'        ,db: all         ,addr: intra     ,auth: pwd   ,title: 'allow all user intra access with pwd' ,order: 400}

pg_provision

Parameter Name: pg_provision, Type: bool, Level: C

Complete the PostgreSQL cluster provisioning work defined in this section after the cluster is bootstrapped. Default value is true.

If disabled, the PostgreSQL cluster will not be provisioned. For some special “PostgreSQL” clusters, such as Greenplum, you can disable this option to skip the provisioning phase.

pg_init

Parameter Name: pg_init, Type: string, Level: G/C

Location of the shell script for initializing database templates, default is pg-init. This script is copied to /pg/bin/pg-init and then executed.

This script is located at roles/pgsql/templates/pg-init

You can add your own logic to this script, or provide a new script in the templates/ directory and set pg_init to the new script name. When using a custom script, please preserve the existing initialization logic.

pg_default_roles

Parameter Name: pg_default_roles, Type: role[], Level: G/C

Default roles and users in Postgres cluster.

Pigsty has a built-in role system. See PGSQL Access Control: Role System for details.

pg_default_roles:                 # default roles and users in postgres cluster
  - { name: dbrole_readonly  ,login: false ,comment: role for global read-only access     }
  - { name: dbrole_offline   ,login: false ,comment: role for restricted read-only access }
  - { name: dbrole_readwrite ,login: false ,roles: [dbrole_readonly]               ,comment: role for global read-write access }
  - { name: dbrole_admin     ,login: false ,roles: [pg_monitor, dbrole_readwrite]  ,comment: role for object creation }
  - { name: postgres     ,superuser: true                                          ,comment: system superuser }
  - { name: replicator ,replication: true  ,roles: [pg_monitor, dbrole_readonly]   ,comment: system replicator }
  - { name: dbuser_dba   ,superuser: true  ,roles: [dbrole_admin]  ,pgbouncer: true ,pool_mode: session, pool_connlimit: 16 , comment: pgsql admin user }
  - { name: dbuser_monitor   ,roles: [pg_monitor, dbrole_readonly] ,pgbouncer: true ,parameters: {log_min_duration_statement: 1000 } ,pool_mode: session ,pool_connlimit: 8 ,comment: pgsql monitor user }

pg_default_privileges

Parameter Name: pg_default_privileges, Type: string[], Level: G/C

Default privileges (DEFAULT PRIVILEGE) settings in each database:

pg_default_privileges:            # default privileges when admin user creates objects
  - GRANT USAGE      ON SCHEMAS   TO dbrole_readonly
  - GRANT SELECT     ON TABLES    TO dbrole_readonly
  - GRANT SELECT     ON SEQUENCES TO dbrole_readonly
  - GRANT EXECUTE    ON FUNCTIONS TO dbrole_readonly
  - GRANT USAGE      ON SCHEMAS   TO dbrole_offline
  - GRANT SELECT     ON TABLES    TO dbrole_offline
  - GRANT SELECT     ON SEQUENCES TO dbrole_offline
  - GRANT EXECUTE    ON FUNCTIONS TO dbrole_offline
  - GRANT INSERT     ON TABLES    TO dbrole_readwrite
  - GRANT UPDATE     ON TABLES    TO dbrole_readwrite
  - GRANT DELETE     ON TABLES    TO dbrole_readwrite
  - GRANT USAGE      ON SEQUENCES TO dbrole_readwrite
  - GRANT UPDATE     ON SEQUENCES TO dbrole_readwrite
  - GRANT TRUNCATE   ON TABLES    TO dbrole_admin
  - GRANT REFERENCES ON TABLES    TO dbrole_admin
  - GRANT TRIGGER    ON TABLES    TO dbrole_admin
  - GRANT CREATE     ON SCHEMAS   TO dbrole_admin

Pigsty provides matching default privileges for the built-in role system. See PGSQL Access Control: Default Privileges for details.

pg_default_schemas

Parameter Name: pg_default_schemas, Type: string[], Level: G/C

Default schemas to create, default value is: [ monitor ]. This will create a monitor schema on all databases for placing various monitoring extensions, tables, views, and functions.

pg_default_extensions

Parameter Name: pg_default_extensions, Type: extension[], Level: G/C

List of extensions to be created and enabled by default in all databases, default value:

pg_default_extensions: # default extensions to be created
  - { name: pg_stat_statements ,schema: monitor }
  - { name: pgstattuple        ,schema: monitor }
  - { name: pg_buffercache     ,schema: monitor }
  - { name: pageinspect        ,schema: monitor }
  - { name: pg_prewarm         ,schema: monitor }
  - { name: pg_visibility      ,schema: monitor }
  - { name: pg_freespacemap    ,schema: monitor }
  - { name: postgres_fdw       ,schema: public  }
  - { name: file_fdw           ,schema: public  }
  - { name: btree_gist         ,schema: public  }
  - { name: btree_gin          ,schema: public  }
  - { name: pg_trgm            ,schema: public  }
  - { name: intagg             ,schema: public  }
  - { name: intarray           ,schema: public  }
  - { name: pg_repack }

The only third-party extension is pg_repack, which is important for database maintenance. All other extensions are built-in PostgreSQL Contrib extensions.

Monitoring-related extensions are installed in the monitor schema by default, which is created by pg_default_schemas.

pg_reload

Parameter Name: pg_reload, Type: bool, Level: A

Reload PostgreSQL after HBA changes, default value is true.

Set it to false to disable automatic configuration reload when you want to check before applying HBA changes.

pg_default_hba_rules

Parameter Name: pg_default_hba_rules, Type: hba[], Level: G/C

PostgreSQL host-based authentication rules, global default rules definition. Default value is:

pg_default_hba_rules:             # postgres default host-based authentication rules, ordered by `order`
  - {user: '${dbsu}'    ,db: all         ,addr: local     ,auth: ident ,title: 'dbsu access via local os user ident'  ,order: 100}
  - {user: '${dbsu}'    ,db: replication ,addr: local     ,auth: ident ,title: 'dbsu replication from local os ident' ,order: 150}
  - {user: '${repl}'    ,db: replication ,addr: localhost ,auth: pwd   ,title: 'replicator replication from localhost',order: 200}
  - {user: '${repl}'    ,db: replication ,addr: intra     ,auth: pwd   ,title: 'replicator replication from intranet' ,order: 250}
  - {user: '${repl}'    ,db: postgres    ,addr: intra     ,auth: pwd   ,title: 'replicator postgres db from intranet' ,order: 300}
  - {user: '${monitor}' ,db: all         ,addr: localhost ,auth: pwd   ,title: 'monitor from localhost with password' ,order: 350}
  - {user: '${monitor}' ,db: all         ,addr: infra     ,auth: pwd   ,title: 'monitor from infra host with password',order: 400}
  - {user: '${admin}'   ,db: all         ,addr: infra     ,auth: ssl   ,title: 'admin @ infra nodes with pwd & ssl'   ,order: 450}
  - {user: '${admin}'   ,db: all         ,addr: world     ,auth: ssl   ,title: 'admin @ everywhere with ssl & pwd'    ,order: 500}
  - {user: '+dbrole_readonly',db: all    ,addr: localhost ,auth: pwd   ,title: 'pgbouncer read/write via local socket',order: 550}
  - {user: '+dbrole_readonly',db: all    ,addr: intra     ,auth: pwd   ,title: 'read/write biz user via password'     ,order: 600}
  - {user: '+dbrole_offline' ,db: all    ,addr: intra     ,auth: pwd   ,title: 'allow etl offline tasks from intranet',order: 650}

The defaults target ordinary deployments on a trusted intranet; they are not a hardened baseline for public or regulated environments. The default +dbrole_offline rule has no role field and therefore applies to every instance. For instance isolation, copy the complete default list and set that rule to role: offline. See Authentication and Access Control: Offline Role and Instance Isolation.

This parameter is an array of HBA rule objects, identical in format to pg_hba_rules. It’s recommended to configure unified pg_default_hba_rules globally, and use pg_hba_rules for additional customization on specific clusters. Rules from both parameters are applied sequentially, with the latter having higher priority.

pgb_default_hba_rules

Parameter Name: pgb_default_hba_rules, Type: hba[], Level: G/C

Pgbouncer default host-based authentication rules, array of HBA rule objects.

Default value provides a fair security level for common scenarios. Check PGSQL Authentication for details.

pgb_default_hba_rules:            # pgbouncer default host-based authentication rules, ordered by `order`
  - {user: '${dbsu}'    ,db: pgbouncer   ,addr: local     ,auth: peer  ,title: 'dbsu local admin access with os ident',order: 100}
  - {user: 'all'        ,db: all         ,addr: localhost ,auth: pwd   ,title: 'allow all user local access with pwd' ,order: 150}
  - {user: '${monitor}' ,db: pgbouncer   ,addr: intra     ,auth: pwd   ,title: 'monitor access via intranet with pwd' ,order: 200}
  - {user: '${monitor}' ,db: all         ,addr: world     ,auth: deny  ,title: 'reject all other monitor access addr' ,order: 250}
  - {user: '${admin}'   ,db: all         ,addr: intra     ,auth: pwd   ,title: 'admin access via intranet with pwd'   ,order: 300}
  - {user: '${admin}'   ,db: all         ,addr: world     ,auth: deny  ,title: 'reject all other admin access addr'   ,order: 350}
  - {user: 'all'        ,db: all         ,addr: intra     ,auth: pwd   ,title: 'allow all user intra access with pwd' ,order: 400}

The default Pgbouncer HBA rules are simple:

  1. Allow login from localhost with password
  2. Allow login from intranet with password

Users can customize according to their own needs.

This parameter is identical in format to pgb_hba_rules. It’s recommended to configure unified pgb_default_hba_rules globally, and use pgb_hba_rules for additional customization on specific clusters. Rules from both parameters are applied sequentially, with the latter having higher priority.

PG_BACKUP

This section defines variables for pgBackRest, which is used for PGSQL Point-in-Time Recovery (PITR).

Check PGSQL Backup & PITR for detailed information.

pgbackrest_enabled: true          # enable pgBackRest on pgsql host?
pgbackrest_log_dir: /pg/log/pgbackrest # pgbackrest log dir, default is `/pg/log/pgbackrest`
pgbackrest_method: local          # pgbackrest repo method: local, minio, [user defined...]
pgbackrest_init_backup: true      # perform a full backup immediately after pgbackrest init?
pgbackrest_repo:                  # pgbackrest repo: https://pgbackrest.org/configuration.html#section-repository
  local:                          # default pgbackrest repo with local posix filesystem
    path: /pg/backup              # local backup directory, default is `/pg/backup`
    retention_full_type: count    # retain full backup by count
    retention_full: 2             # keep at most 3 full backups when using local filesystem repo, at least 2
  minio:                          # optional minio repo for pgbackrest
    type: s3                      # minio is s3-compatible, so use s3
    s3_endpoint: sss.pigsty       # minio endpoint domain, default is `sss.pigsty`
    s3_region: us-east-1          # minio region, default is us-east-1, not effective for minio
    s3_bucket: pgsql              # minio bucket name, default is `pgsql`
    s3_key: pgbackrest            # minio user access key for pgbackrest
    s3_key_secret: S3User.Backup  # minio user secret key for pgbackrest
    s3_uri_style: path            # use path style uri for minio, instead of host style
    path: /pgbackrest             # minio backup path, default is `/pgbackrest`
    storage_port: 9000            # minio port, default is 9000
    storage_ca_file: /etc/pki/ca.crt  # minio ca file path, default is `/etc/pki/ca.crt`
    block: y                      # enable block-level incremental backup (pgBackRest 2.46+)
    bundle: y                     # bundle small files into one file
    bundle_limit: 20MiB           # object storage file bundling threshold, default 20MiB
    bundle_size: 128MiB           # object storage file bundling target size, default 128MiB
    cipher_type: aes-256-cbc      # enable AES encryption for remote backup repo
    cipher_pass: pgBackRest       # AES encryption password, default is 'pgBackRest'
    retention_full_type: time     # retain full backup by time on minio repo
    retention_full: 14            # keep full backups from the past 14 days

pgbackrest_enabled

Parameter Name: pgbackrest_enabled, Type: bool, Level: C

Enable pgBackRest on PGSQL nodes? Default value is: true

When enabled, every node receives pgBackRest configuration. With the local filesystem repository (local), each member creates its own local stanza. Initial and scheduled backups run only on the current primary; pg-backup exits after its role check on a replica. A non-local shared stanza is initialized only on a primary without pg_upstream.

pgbackrest_log_dir

Parameter Name: pgbackrest_log_dir, Type: path, Level: C

pgBackRest log directory, default is /pg/log/pgbackrest. The Vector log agent references this parameter for log collection.

pgbackrest_method

Parameter Name: pgbackrest_method, Type: enum, Level: C

pgBackRest repository method: default options are local, minio, or other user-defined methods, default is local.

This parameter determines which repository to use for pgBackRest. All available repository methods are defined in pgbackrest_repo.

Pigsty uses the local backup repository by default, which creates a backup repository in the /pg/backup directory on the primary instance. The underlying storage path is specified by pg_fs_backup.

pgbackrest_init_backup

Parameter Name: pgbackrest_init_backup, Type: bool, Level: C

Perform a full backup immediately after pgBackRest initialization completes? Default is true.

The task attempts this only on the cluster primary when pg_upstream is not defined. It uses ignore_errors for backup failures, so enabling the parameter does not guarantee that a base backup exists. /etc/pgbackrest/initial.done is written only after the command succeeds. Verify the repository afterward with pig pb info (or pb info).

pgbackrest_repo

Parameter Name: pgbackrest_repo, Type: dict, Level: G/C

pgBackRest repository documentation: https://pgbackrest.org/configuration.html#section-repository

The default value contains local and minio as candidate definitions. pgbackrest_method selects one of them, and the v4.5.0 template renders only that selected entry as pgBackRest repo1; listing both keys is not a dual-repository backup configuration:

pgbackrest_repo:                  # pgbackrest repo: https://pgbackrest.org/configuration.html#section-repository
  local:                          # default pgbackrest repo with local posix filesystem
    path: /pg/backup              # local backup directory, default is `/pg/backup`
    retention_full_type: count    # retain full backup by count
    retention_full: 2             # keep at most 3 full backups when using local filesystem repo, at least 2
  minio:                          # optional minio repo for pgbackrest
    type: s3                      # minio is s3-compatible, so use s3
    s3_endpoint: sss.pigsty       # minio endpoint domain, default is `sss.pigsty`
    s3_region: us-east-1          # minio region, default is us-east-1, not effective for minio
    s3_bucket: pgsql              # minio bucket name, default is `pgsql`
    s3_key: pgbackrest            # minio user access key for pgbackrest
    s3_key_secret: S3User.Backup  # minio user secret key for pgbackrest
    s3_uri_style: path            # use path style uri for minio, instead of host style
    path: /pgbackrest             # minio backup path, default is `/pgbackrest`
    storage_port: 9000            # minio port, default is 9000
    storage_ca_file: /etc/pki/ca.crt  # minio ca file path, default is `/etc/pki/ca.crt`
    block: y                      # enable block-level incremental backup (pgBackRest 2.46+)
    bundle: y                     # bundle small files into one file
    bundle_limit: 20MiB           # object storage file bundling threshold, default 20MiB
    bundle_size: 128MiB           # object storage file bundling target size, default 128MiB
    cipher_type: aes-256-cbc      # enable AES encryption for remote backup repo
    cipher_pass: pgBackRest       # AES encryption password, default is 'pgBackRest'
    retention_full_type: time     # retain full backup by time on minio repo
    retention_full: 14            # keep full backups from the past 14 days

You can define new backup repositories, such as using AWS S3, GCP, or other cloud providers’ S3-compatible storage services.

Block Incremental Backup: Starting from pgBackRest 2.46, the block: y option enables block-level incremental backup. This means during incremental backups, pgBackRest only backs up changed data blocks instead of entire changed files, significantly reducing backup data volume and backup time. This feature is particularly useful for large databases, and it’s recommended to enable this option on object storage repositories.

PG_ACCESS

This section handles database access paths, including:

  • Deploy Pgbouncer connection pooler on each PGSQL node and set default behavior
  • Publish service ports through local or dedicated haproxy nodes
  • Bind optional L2 VIP and register DNS records
pgbouncer_enabled: true           # if disabled, pgbouncer will not be launched on pgsql host
pgbouncer_port: 6432              # pgbouncer listen port, 6432 by default
pgbouncer_log_dir: /pg/log/pgbouncer  # pgbouncer log dir, `/pg/log/pgbouncer` by default
pgbouncer_auth_query: false       # query postgres to retrieve unlisted business users?
pgbouncer_poolmode: transaction   # pooling mode: transaction,session,statement, transaction by default
pgbouncer_sslmode: disable        # pgbouncer client ssl mode, disable by default
pgbouncer_ignore_param: [ extra_float_digits, application_name, TimeZone, DateStyle, IntervalStyle, search_path ]
pg_weight: 100          #INSTANCE # relative load balance weight in service, 100 by default, 0-255
pg_service_provider: ''           # dedicate haproxy node group name, or empty string for local nodes by default
pg_default_service_dest: pgbouncer # default service destination if svc.dest='default'
pg_default_services:              # postgres default service definitions
  - { name: primary ,port: 5433 ,dest: default  ,check: /primary   ,selector: "[]" }
  - { name: replica ,port: 5434 ,dest: default  ,check: /read-only ,selector: "[]" , backup: "[? pg_role == `primary` || pg_role == `offline` ]" }
  - { name: default ,port: 5436 ,dest: postgres ,check: /primary   ,selector: "[]" }
  - { name: offline ,port: 5438 ,dest: postgres ,check: /replica   ,selector: "[? pg_role == `offline` || pg_offline_query ]" , backup: "[? pg_role == `replica` && !pg_offline_query]"}
pg_vip_enabled: false             # enable a l2 vip for pgsql primary? false by default
pg_vip_address: 127.0.0.1/24      # vip address in `<ipv4>/<mask>` format, require if vip is enabled
pg_vip_interface: auto            # vip network interface to listen, auto by default
pg_dns_suffix: ''                 # pgsql dns suffix, '' by default
pg_dns_target: auto               # auto, primary, vip, none, or ad hoc ip

pgbouncer_enabled

Parameter Name: pgbouncer_enabled, Type: bool, Level: C

Default value is true. If disabled, the Pgbouncer connection pooler will not be configured on PGSQL nodes.

pgbouncer_port

Parameter Name: pgbouncer_port, Type: port, Level: C

Pgbouncer listen port, default is 6432.

pgbouncer_log_dir

Parameter Name: pgbouncer_log_dir, Type: path, Level: C

Pgbouncer log directory, default is /pg/log/pgbouncer. The Vector log agent collects Pgbouncer logs based on this parameter.

pgbouncer_auth_query

Parameter Name: pgbouncer_auth_query, Type: bool, Level: C

Allow Pgbouncer to query PostgreSQL to allow users not explicitly listed to access PostgreSQL through the connection pool? Default value is false.

If enabled, pgbouncer users will authenticate against the postgres database using SELECT username, password FROM monitor.pgbouncer_auth($1). Otherwise, only business users with pgbouncer: true are allowed to connect to the Pgbouncer connection pool.

pgbouncer_poolmode

Parameter Name: pgbouncer_poolmode, Type: enum, Level: C

Pgbouncer connection pool pooling mode: transaction, session, statement, default is transaction.

  • session: Session-level pooling with best feature compatibility.
  • transaction: Transaction-level pooling with better performance (many small connections), may break some session-level features like NOTIFY/LISTEN, etc.
  • statements: Statement-level pooling for simple read-only queries.

If your application has feature compatibility issues, consider changing this parameter to session.

pgbouncer_sslmode

Parameter Name: pgbouncer_sslmode, Type: enum, Level: C

Pgbouncer client SSL mode, default is disable.

Note that enabling SSL may have a significant performance impact on your pgbouncer.

  • disable: Ignore if client requests TLS (default)
  • allow: Use TLS if client requests it. Use plain TCP if not. Does not verify client certificate.
  • prefer: Same as allow.
  • require: Client must use TLS. Reject client connection if not. Does not verify client certificate.
  • verify-ca: Client must use TLS with a valid client certificate.
  • verify-full: Same as verify-ca.

pgbouncer_ignore_param

Parameter Name: pgbouncer_ignore_param, Type: string[], Level: C

List of startup parameters ignored by PgBouncer, default value is:

[ extra_float_digits, application_name, TimeZone, DateStyle, IntervalStyle, search_path ]

These parameters are configured in the ignore_startup_parameters option in the PgBouncer configuration file. When clients set these parameters during connection, PgBouncer will not create new connections due to parameter mismatch in the connection pool.

This allows different clients to use the same connection pool even if they set different values for these parameters. This parameter was added in Pigsty v3.5.

pg_weight

Parameter Name: pg_weight, Type: int, Level: I

Relative load balancing weight in service, default is 100, range 0-255.

Default value: 100. You must define it in instance variables and reload service for it to take effect.

pg_service_provider

Parameter Name: pg_service_provider, Type: string, Level: G/C

Dedicated haproxy node group name, or empty string for local nodes by default.

If specified, PostgreSQL services will be registered to the dedicated haproxy node group instead of the current PGSQL cluster nodes.

Remember to allocate unique ports for each service on the dedicated haproxy nodes!

For example, if we define the following parameters on a 3-node pg-test cluster:

pg_service_provider: infra       # use load balancer on group `infra`
pg_default_services:             # alloc port 10001 and 10002 for pg-test primary/replica service
  - { name: primary ,port: 10001 ,dest: postgres  ,check: /primary   ,selector: "[]" }
  - { name: replica ,port: 10002 ,dest: postgres  ,check: /read-only ,selector: "[]" , backup: "[? pg_role == `primary` || pg_role == `offline` ]" }

pg_default_service_dest

Parameter Name: pg_default_service_dest, Type: enum, Level: G/C

When defining a service, if svc.dest='default', this parameter will be used as the default value.

Default value: pgbouncer, meaning the 5433 primary service and 5434 replica service will route traffic to pgbouncer by default.

If you don’t want to use pgbouncer, set it to postgres. Traffic will be routed directly to postgres.

pg_default_services

Parameter Name: pg_default_services, Type: service[], Level: G/C

Postgres default service definitions.

Default value is four default service definitions, as described in PGSQL Service.

pg_default_services:               # postgres default service definitions
  - { name: primary ,port: 5433 ,dest: default  ,check: /primary   ,selector: "[]" }
  - { name: replica ,port: 5434 ,dest: default  ,check: /read-only ,selector: "[]" , backup: "[? pg_role == `primary` || pg_role == `offline` ]" }
  - { name: default ,port: 5436 ,dest: postgres ,check: /primary   ,selector: "[]" }
  - { name: offline ,port: 5438 ,dest: postgres ,check: /replica   ,selector: "[? pg_role == `offline` || pg_offline_query ]" , backup: "[? pg_role == `replica` && !pg_offline_query]"}

pg_vip_enabled

Parameter Name: pg_vip_enabled, Type: bool, Level: C

Enable L2 VIP for PGSQL cluster? Default value is false, meaning no L2 VIP will be created.

When L2 VIP is enabled, a VIP will be bound to the cluster primary instance node, managed by vip-manager based on data in etcd.

L2 VIP can only be used within the same L2 network, which may impose additional constraints on your network topology.

pg_vip_address

Parameter Name: pg_vip_address, Type: cidr4, Level: C

VIP address in <ipv4>/<mask> format is required if VIP is enabled.

Default value: 127.0.0.1/24. This value consists of two parts: ipv4 and mask, separated by /.

pg_vip_interface

Parameter Name: pg_vip_interface, Type: string, Level: C/I

VIP network interface to listen on, auto by default. Pigsty detects the interface associated with the instance IP in the inventory.

For non-standard routing, policy routing, or other unusual network environments where auto-detection is unsuitable, explicitly override it in the instance variables:

pg-test:
    hosts:
        10.10.10.11: {pg_seq: 1, pg_role: replica ,pg_vip_interface: eth0 }
        10.10.10.12: {pg_seq: 2, pg_role: primary ,pg_vip_interface: eth1 }
        10.10.10.13: {pg_seq: 3, pg_role: replica ,pg_vip_interface: eth2 }
    vars:
      pg_vip_enabled: true          # enable L2 VIP for this cluster, binds to primary by default
      pg_vip_address: 10.10.10.3/24 # L2 network CIDR: 10.10.10.0/24, vip address: 10.10.10.3
      # pg_vip_interface: eth1      # if your nodes have a unified interface, you can define it here

pg_dns_suffix

Parameter Name: pg_dns_suffix, Type: string, Level: C

PostgreSQL DNS name suffix, default is empty string.

By default, the PostgreSQL cluster name is registered as a DNS domain in dnsmasq on Infra nodes for external resolution.

You can specify a domain suffix with this parameter, which will use {{ pg_cluster }}{{ pg_dns_suffix }} as the cluster DNS name.

For example, if you set pg_dns_suffix to .db.vip.company.tld, the pg-test cluster DNS name will be pg-test.db.vip.company.tld.

pg_dns_target

Parameter Name: pg_dns_target, Type: enum, Level: C

Could be: auto, primary, vip, none, or an ad hoc IP address, which will be the target IP address of cluster DNS record.

Default value: auto, which will bind to pg_vip_address if pg_vip_enabled, or fallback to cluster primary instance IP address.

  • vip: bind to pg_vip_address
  • primary: resolve to cluster primary instance IP address
  • auto: resolve to pg_vip_address if pg_vip_enabled, or fallback to cluster primary instance IP address
  • none: do not bind to any IP address
  • <ipv4>: bind to the given IP address

PG_MONITOR

The PG_MONITOR group parameters are used to monitor the status of PostgreSQL databases, Pgbouncer connection pools, and pgBackRest backup systems.

This parameter group defines three Exporter configurations: pg_exporter for monitoring PostgreSQL, pgbouncer_exporter for monitoring connection pools, and pgbackrest_exporter for monitoring backup status.

pg_exporter_enabled: true              # enable pg_exporter on pgsql host?
pg_exporter_config: pg_exporter.yml    # pg_exporter config file name
pg_exporter_cache_ttls: '1,10,60,300'  # pg_exporter collector ttl stages (seconds), default is '1,10,60,300'
pg_exporter_port: 9630                 # pg_exporter listen port, default is 9630
pg_exporter_params: 'sslmode=disable'  # extra url parameters for pg_exporter dsn
pg_exporter_url: ''                    # if specified, will override auto-generated pg dsn
pg_exporter_auto_discovery: true       # enable auto database discovery? enabled by default
pg_exporter_exclude_database: 'template0,template1,postgres' # csv list of databases not monitored during auto-discovery
pg_exporter_include_database: ''       # csv list of databases monitored during auto-discovery
pg_exporter_connect_timeout: 200       # pg_exporter connection timeout (ms), default is 200
pg_exporter_options: ''                # extra options to override pg_exporter
pgbouncer_exporter_enabled: true       # enable pgbouncer_exporter on pgsql host?
pgbouncer_exporter_port: 9631          # pgbouncer_exporter listen port, default is 9631
pgbouncer_exporter_url: ''             # if specified, will override auto-generated pgbouncer dsn
pgbouncer_exporter_options: ''         # extra options to override pgbouncer_exporter
pgbackrest_exporter_enabled: true      # enable pgbackrest_exporter on pgsql host?
pgbackrest_exporter_port: 9854         # pgbackrest_exporter listen port, default is 9854
pgbackrest_exporter_options: >-        # extra options to override pgbackrest_exporter
  --collect.interval=120
  --log.level=info

pg_exporter_enabled

Parameter Name: pg_exporter_enabled, Type: bool, Level: C

Enable pg_exporter on PGSQL nodes? Default value is: true.

PG Exporter is used to monitor PostgreSQL database instances. Set to false if you don’t want to install pg_exporter.

pg_exporter_config

Parameter Name: pg_exporter_config, Type: string, Level: C

pg_exporter’s collector configuration template name. The default is pg_exporter.yml. Pgbouncer Exporter uses its own fixed pgbouncer_exporter.yml template and is not affected by this parameter.

The default template is roles/pg_monitor/templates/pg_exporter.yml. To use a custom file, put it on Ansible’s template search path and specify its template name here.

pg_exporter_cache_ttls

Parameter Name: pg_exporter_cache_ttls, Type: string, Level: C

pg_exporter collector TTL stages (seconds), default is ‘1,10,60,300’.

Default value: 1,10,60,300, which will use different TTL values for different metric collectors: 1s, 10s, 60s, 300s.

PG Exporter has a built-in caching mechanism to avoid the improper impact of multiple Prometheus scrapes on the database. All metric collectors are divided into four categories by TTL:

ttl_fast: "{{ pg_exporter_cache_ttls.split(',')[0]|int }}"         # critical queries
ttl_norm: "{{ pg_exporter_cache_ttls.split(',')[1]|int }}"         # common queries
ttl_slow: "{{ pg_exporter_cache_ttls.split(',')[2]|int }}"         # slow queries (e.g table size)
ttl_slowest: "{{ pg_exporter_cache_ttls.split(',')[3]|int }}"      # ver slow queries (e.g bloat)

For example, with default configuration, liveness metrics are cached for at most 1s, most common metrics are cached for 10s (should match the monitoring scrape interval victoria_scrape_interval). A few slow-changing queries have 60s TTL, and very few high-overhead monitoring queries have 300s TTL.

pg_exporter_port

Parameter Name: pg_exporter_port, Type: port, Level: C

pg_exporter listen port, default value is: 9630

pg_exporter_params

Parameter Name: pg_exporter_params, Type: string, Level: C

Extra URL path parameters in the DSN used by pg_exporter.

Default value: sslmode=disable, which disables SSL for monitoring connections (since local unix sockets are used by default).

pg_exporter_url

Parameter Name: pg_exporter_url, Type: pgurl, Level: C

If specified, will override the auto-generated PostgreSQL DSN and use the specified DSN to connect to PostgreSQL. Default value is empty string.

If not specified, PG Exporter will use the following connection string to access PostgreSQL by default:

postgres://{{ pg_monitor_username }}:{{ pg_monitor_password }}@{{ pg_host }}:{{ pg_port }}/postgres{% if pg_exporter_params != '' %}?{{ pg_exporter_params }}{% endif %}

Use this parameter when you want to monitor a remote PostgreSQL instance, or need to use different monitoring user/password or configuration options.

pg_exporter_auto_discovery

Parameter Name: pg_exporter_auto_discovery, Type: bool, Level: C

Enable auto database discovery? Enabled by default: true.

By default, PG Exporter connects to the database specified in the DSN (default is the admin database postgres) to collect global metrics. If you want to collect metrics from all business databases, enable this option. PG Exporter will automatically discover all databases in the target PostgreSQL instance and collect database-level monitoring metrics from these databases.

pg_exporter_exclude_database

Parameter Name: pg_exporter_exclude_database, Type: string, Level: C

If database auto-discovery is enabled (enabled by default), databases in this parameter’s list will not be monitored. Default value is: template0,template1,postgres, meaning the admin database postgres and template databases are excluded from auto-monitoring.

As an exception, the database specified in the DSN is not affected by this parameter. For example, if PG Exporter connects to the postgres database, it will be monitored even if postgres is in this list.

pg_exporter_include_database

Parameter Name: pg_exporter_include_database, Type: string, Level: C

If database auto-discovery is enabled (enabled by default), only databases in this parameter’s list will be monitored. Default value is empty string, meaning this feature is not enabled.

The parameter format is a comma-separated list of database names, e.g., db1,db2,db3.

This parameter has higher priority than pg_exporter_exclude_database, acting as a whitelist mode. Use this parameter if you only want to monitor specific databases.

pg_exporter_connect_timeout

Parameter Name: pg_exporter_connect_timeout, Type: int, Level: C

pg_exporter connection timeout (milliseconds), default is 200 (in milliseconds).

How long will PG Exporter wait when trying to connect to a PostgreSQL database? Beyond this time, PG Exporter will give up the connection and report an error.

The default value of 200ms is sufficient for most scenarios (e.g., same availability zone monitoring), but if your monitored remote PostgreSQL is on another continent, you may need to increase this value to avoid connection timeouts.

pg_exporter_options

Parameter Name: pg_exporter_options, Type: arg, Level: C

Command line arguments passed to PG Exporter, default value is: "" empty string.

When using empty string, the default command arguments will be used:

{% if pg_exporter_options != '' %}
PG_EXPORTER_OPTS='--web.listen-address=:{{ pg_exporter_port }} {{ pg_exporter_options }}'
{% else %}
PG_EXPORTER_OPTS='--web.listen-address=:{{ pg_exporter_port }} --log.level=info'
{% endif %}

Note: Do not override the pg_exporter_port port configuration in this parameter.

pgbouncer_exporter_enabled

Parameter Name: pgbouncer_exporter_enabled, Type: bool, Level: C

Enable pgbouncer_exporter on PGSQL nodes? Default value is: true.

pgbouncer_exporter_port

Parameter Name: pgbouncer_exporter_port, Type: port, Level: C

pgbouncer_exporter listen port, default value is: 9631

pgbouncer_exporter_url

Parameter Name: pgbouncer_exporter_url, Type: pgurl, Level: C

If specified, will override the auto-generated pgbouncer DSN and use the specified DSN to connect to pgbouncer. Default value is empty string.

If not specified, Pgbouncer Exporter will use the following connection string to access Pgbouncer by default:

postgres://{{ pg_monitor_username }}:{{ pg_monitor_password }}@:{{ pgbouncer_port }}/pgbouncer?host={{ pg_localhost }}&sslmode=disable

Use this parameter when you want to monitor a remote Pgbouncer instance, or need to use different monitoring user/password or configuration options.

pgbouncer_exporter_options

Parameter Name: pgbouncer_exporter_options, Type: arg, Level: C

Command line arguments passed to Pgbouncer Exporter, default value is: "" empty string.

When using empty string, the default command arguments will be used:

{% if pgbouncer_exporter_options != '' %}
PG_EXPORTER_OPTS='--web.listen-address=:{{ pgbouncer_exporter_port }} {{ pgbouncer_exporter_options }}'
{% else %}
PG_EXPORTER_OPTS='--web.listen-address=:{{ pgbouncer_exporter_port }} --log.level=info'
{% endif %}

Note: Do not override the pgbouncer_exporter_port port configuration in this parameter.

pgbackrest_exporter_enabled

Parameter Name: pgbackrest_exporter_enabled, Type: bool, Level: C

Enable pgbackrest_exporter on PGSQL nodes? Default value is: true.

pgbackrest_exporter is used to monitor the status of the pgBackRest backup system, including key metrics such as backup size, time, type, and duration.

pgbackrest_exporter_port

Parameter Name: pgbackrest_exporter_port, Type: port, Level: C

pgbackrest_exporter listen port, default value is: 9854.

This port is registered in the VictoriaMetrics-compatible scrape targets for backup-related metrics.

pgbackrest_exporter_options

Parameter Name: pgbackrest_exporter_options, Type: arg, Level: C

Command-line arguments passed to pgbackrest_exporter. The default is:

pgbackrest_exporter_options: >-
  --collect.interval=120
  --log.level=info

This collects every 120 seconds at info log level. Setting the parameter replaces the default argument set as a whole.

PG_REMOVE

pgsql-rm.yml invokes the pg_remove role to safely remove PostgreSQL instances. This section’s parameters control cleanup behavior to avoid accidental deletion.

pg_rm_data: true                  # remove postgres data during remove? true by default
pg_rm_backup: true                # remove pgbackrest backup during primary remove? true by default
pg_rm_pkg: true                   # uninstall postgres packages during remove? true by default
pg_safeguard: false               # stop pg_remove running if pg_safeguard is enabled, false by default

pg_rm_data

Parameter Name: pg_rm_data, Type: bool, Level: G/C/A

Whether to clean up pg_data and symlinks when removing PGSQL instances, default is true.

This switch affects both pgsql-rm.yml and other scenarios that trigger pg_remove. Set to false to preserve the data directory for manual inspection or remounting.

pg_rm_backup

Parameter Name: pg_rm_backup, Type: bool, Level: G/C/A

Whether to also clean up the pgBackRest repository and configuration when removing the primary, default is true.

This parameter only applies to primary instances with pg_role=primary: pg_remove will first stop pgBackRest, delete the current cluster’s stanza, and remove data in pg_fs_backup when pgbackrest_method == 'local'. Standby clusters or upstream backups are not affected.

pg_rm_pkg

Parameter Name: pg_rm_pkg, Type: bool, Level: G/C/A

Whether to uninstall all packages installed by pg_packages when cleaning up PGSQL instances, default is true.

If you only want to temporarily stop and preserve binaries, set it to false. Otherwise, pg_remove will call the system package manager to completely uninstall PostgreSQL-related components.

pg_safeguard

Parameter Name: pg_safeguard, Type: bool, Level: G/C/A

Accidental deletion protection, default is false. When explicitly set to true, pg_remove will immediately terminate with a prompt, and will only continue after using -e pg_safeguard=false or disabling it in variables.

It’s recommended to enable this switch before batch cleanup in production environments, verify the commands and target nodes are correct, then disable it to avoid accidental deletion of instances.

8.12 - Playbook

How to manage PostgreSQL clusters with Ansible playbooks

Pigsty provides a series of playbooks for cluster provisioning, scaling, user/database management, monitoring, backup & recovery, and migration.

Playbook Function
pgsql.yml Initialize PostgreSQL cluster or add new replicas
pgsql-rm.yml Remove PostgreSQL cluster or specific instances
pgsql-user.yml Add new business user to existing PostgreSQL cluster
pgsql-db.yml Add new business database to existing PostgreSQL cluster
pgsql-monitor.yml Monitor remote PostgreSQL instances
pgsql-migration.yml Generate migration manual and scripts for existing PostgreSQL
pgsql-pitr.yml Perform Point-In-Time Recovery (PITR)

Safeguard

Be extra cautious when using PGSQL playbooks. Misuse of pgsql.yml and pgsql-rm.yml can lead to accidental database deletion!

  • Always add the -l parameter to limit the execution scope, and ensure you’re executing the right tasks on the right targets.
  • Limiting scope to a single cluster is recommended. Running pgsql.yml without parameters in production is a high-risk operation—think twice before proceeding.
  • Before removal, inspect pig pg list <cluster> and pig pb info, verify a recent backup, and have the operator enter the exact target.

To prevent accidental deletion, Pigsty’s PGSQL module provides a safeguard mechanism controlled by the pg_safeguard parameter. When pg_safeguard is set to true, the pgsql-rm.yml playbook will abort immediately, protecting your database cluster.

# Will abort execution, protecting data
./pgsql-rm.yml -l pg-test -e pg_safeguard=true

# Override the safeguard switch from the command line
./pgsql-rm.yml -l pg-test -e pg_safeguard=false

In addition to pg_safeguard, pgsql-rm.yml provides finer-grained control parameters:

Parameter Default Description
pg_safeguard false Safeguard switch; when true, playbook aborts
pg_rm_data true Whether to remove PostgreSQL data directory
pg_rm_backup true Whether to remove pgBackRest backup data (only when removing primary)
pg_rm_pkg true Whether to uninstall PostgreSQL packages

These parameters allow precise control over removal behavior:

# Remove the instance services, monitoring, and DCS registration, but keep its data directory
./pgsql-rm.yml -l pg-test -e pg_rm_data=false

# Remove cluster but keep backup data
./pgsql-rm.yml -l pg-test -e pg_rm_backup=false

# Remove cluster and uninstall packages
./pgsql-rm.yml -l pg-test -e pg_rm_pkg=true

pgsql.yml

The pgsql.yml playbook is used to initialize PostgreSQL clusters or add new replicas.

Here’s a demo of initializing a PostgreSQL cluster in the sandbox environment:

asciicast

Basic Usage

./pgsql.yml -l pg-meta            # Initialize cluster pg-meta
./pgsql.yml -l 10.10.10.13        # Initialize/add instance 10.10.10.13
./pgsql.yml -l pg-test -t pg_service  # Refresh services for cluster pg-test
./pgsql.yml -l pg-test -t pg_hba,pgbouncer_hba,pgbouncer_reload -e pg_reload=true  # Reload HBA rules

Wrapper Scripts

Pigsty provides convenient wrapper scripts to simplify common operations:

bin/pgsql-add pg-meta             # Initialize pgsql cluster pg-meta
bin/pgsql-add 10.10.10.10         # Initialize pgsql instance 10.10.10.10
bin/pgsql-add pg-test 10.10.10.13 # Add 10.10.10.13 to cluster pg-test (auto refresh services)
bin/pgsql-svc pg-test             # Refresh haproxy services for pg-test (use after membership changes)
bin/pgsql-hba pg-test             # Reload pg/pgb HBA rules for pg-test

Subtasks

This playbook contains the following subtasks:

# pg_install              : install postgres packages & extensions
#   - pg_dbsu             : setup postgres superuser
#     - pg_dbsu_create    : create dbsu user
#     - pg_dbsu_sudo      : configure dbsu sudo privileges
#     - pg_ssh            : exchange dbsu SSH keys
#   - pg_pkg              : install postgres packages
#     - pg_pre            : pre-installation tasks
#     - pg_ext            : install postgres extension packages
#     - pg_post           : post-installation tasks
#   - pg_link             : link pgsql version bin to /usr/pgsql
#   - pg_path             : add pgsql bin to system path
#   - pg_dir              : create postgres directories and setup FHS
#   - pg_bin              : sync /pg/bin scripts
#   - pg_alias            : configure pgsql/psql aliases
#   - pg_dummy            : create dummy placeholder file
#
# pg_bootstrap            : bootstrap postgres cluster
#   - pg_config           : generate postgres config
#     - pg_conf           : generate patroni config
#     - pg_key            : generate pgsodium key
#   - pg_cert             : issue certificates for postgres
#     - pg_cert_private   : check pg private key existence
#     - pg_cert_issue     : sign pg server certificate
#     - pg_cert_copy      : copy key & certs to pg node
#   - pg_launch           : launch patroni primary & replicas
#     - pg_watchdog       : grant watchdog permission to postgres
#     - pg_primary        : launch patroni/postgres primary
#     - pg_init           : init pg cluster with roles/templates
#     - pg_pass           : write .pgpass file to pg home
#     - pg_replica        : launch patroni/postgres replicas
#     - pg_hba            : generate pg HBA rules
#     - patroni_reload    : reload patroni config
#     - pg_patroni        : pause or remove patroni if necessary
#
# pg_provision            : provision postgres business users & databases
#   - pg_user             : provision postgres business users
#     - pg_user_config    : render create user SQL
#     - pg_user_create    : create user on postgres
#   - pg_db               : provision postgres business databases
#     - pg_db_drop        : drop database on postgres (state=absent/recreate)
#     - pg_db_config      : render create database SQL
#     - pg_db_create      : create database on postgres
#
# pg_backup               : init postgres PITR backup
#   - pgbackrest          : setup pgbackrest for backup
#     - pgbackrest_config : generate pgbackrest config
#     - pgbackrest_init   : init pgbackrest repo
#     - pgbackrest_backup : make initial backup after bootstrap
#
# pg_access               : init postgres service access layer
#   - pgbouncer           : deploy pgbouncer connection pooler
#     - pgbouncer_dir     : create pgbouncer directories
#     - pgbouncer_config  : generate pgbouncer config
#       - pgbouncer_hba   : generate pgbouncer HBA config
#       - pgbouncer_user  : generate pgbouncer userlist
#     - pgbouncer_launch  : launch pgbouncer service
#     - pgbouncer_reload  : reload pgbouncer config
#   - pg_vip              : bind VIP to primary with vip-manager
#     - pg_vip_config     : generate vip-manager config
#     - pg_vip_launch     : launch vip-manager to bind VIP
#   - pg_dns              : register DNS name to infra dnsmasq
#     - pg_dns_ins        : register pg instance name
#     - pg_dns_cls        : register pg cluster name
#   - pg_service          : expose pgsql service with haproxy
#     - pg_service_config : generate local haproxy config for pg services
#     - pg_service_reload : expose postgres services with haproxy
#
# pg_monitor              : setup pgsql monitoring and register to infra
#   - pg_exporter         : configure and launch pg_exporter
#   - pgbouncer_exporter  : configure and launch pgbouncer_exporter
#   - pgbackrest_exporter : configure and launch pgbackrest_exporter
#   - pg_register         : register pgsql to monitoring/logging/datasource
#     - add_metrics       : register pg as VictoriaMetrics monitoring target
#     - add_logs          : register pg as Vector log source
#     - add_ds            : register pg database as Grafana datasource

Related Administration Tasks

Notes

  • When running this playbook on a single replica, ensure the cluster primary is already initialized!
  • After scaling out, you need to Reload Service and Reload HBA. The wrapper script bin/pgsql-add handles these tasks automatically.

When scaling a cluster, if Patroni takes too long to bring up a replica, the Ansible playbook may abort due to timeout:

  • Typical error message: wait for postgres/patroni replica task runs for a long time before aborting
  • However, the replica creation process continues. For scenarios where replica creation takes more than a day, see FAQ: Replica creation failed.

pgsql-rm.yml

The pgsql-rm.yml playbook is used to remove PostgreSQL clusters or specific instances.

Here’s a demo of removing a PostgreSQL cluster in the sandbox environment:

asciicast

Basic Usage

./pgsql-rm.yml -l pg-test          # Remove cluster pg-test
./pgsql-rm.yml -l 10.10.10.13      # Remove instance 10.10.10.13

Command Line Arguments

This playbook supports the following command line arguments:

./pgsql-rm.yml -l pg-test          # Remove cluster pg-test
    -e pg_safeguard=false          # Safeguard switch, disabled by default; override when enabled
    -e pg_rm_data=true             # Whether to remove PostgreSQL data directory, default: remove
    -e pg_rm_backup=true           # Whether to remove pgBackRest backup (primary only), default: remove
    -e pg_rm_pkg=true              # Whether to uninstall PostgreSQL packages, default: uninstall

Wrapper Scripts

bin/pgsql-rm pg-meta               # Remove pgsql cluster pg-meta
bin/pgsql-rm pg-test 10.10.10.13   # Remove instance 10.10.10.13 from cluster pg-test

Subtasks

This playbook contains the following subtasks:

# pg_safeguard           : abort if pg_safeguard is enabled
#
# pg_monitor             : remove registration from monitoring system
#   - pg_deregister      : remove pg monitoring targets from infra
#     - rm_metrics       : remove targets from the VictoriaMetrics target directory
#     - rm_ds            : remove datasource from grafana
#     - rm_logs          : remove log targets from vector
#   - pg_exporter        : remove pg_exporter
#   - pgbouncer_exporter : remove pgbouncer_exporter
#   - pgbackrest_exporter: remove pgbackrest_exporter
#
# pg_access              : remove pg service access layer
#   - dns                : remove pg DNS records
#   - vip                : remove vip-manager
#   - pg_service         : remove pg service from haproxy
#   - pgbouncer          : remove pgbouncer connection middleware
#
# postgres               : remove postgres instances
#   - pg_replica         : remove all replicas
#   - pg_primary         : remove primary
#   - pg_meta            : remove metadata from etcd
#
# pg_backup              : remove backup repo (disable with pg_rm_backup=false)
# pg_data                : remove postgres data (disable with pg_rm_data=false)
# pg_pkg                 : uninstall pg packages (enabled by default; disable with pg_rm_pkg=false)
#   - pg_ext             : uninstall postgres extensions alone

Related Administration Tasks

Notes

  • Do not run this playbook on a primary that still has replicas—otherwise, remaining replicas will trigger automatic failover. Always remove all replicas first, then remove the primary. This is not a concern when removing the entire cluster at once.
  • Refresh cluster services after removing instances. When you remove a replica from a cluster, it remains in the load balancer configuration file. Since health checks will fail, the removed instance won’t affect cluster services. However, you should Reload Service at an appropriate time to ensure consistency between the production environment and configuration inventory.

pgsql-user.yml

The pgsql-user.yml playbook is used to add new business users to existing PostgreSQL clusters.

Basic Usage

./pgsql-user.yml -l pg-meta -e username=dbuser_meta

Wrapper Scripts

bin/pgsql-user pg-meta dbuser_meta  # Create user dbuser_meta on cluster pg-meta

Workflow

  1. Define user in the config inventory: all.children.<pg_cluster>.vars.pg_users[i]
  2. Execute playbook specifying cluster and username: pgsql-user.yml -l <pg_cluster> -e username=<name>

The playbook will:

  1. Generate user creation SQL at /pg/tmp/pg-user-{{ user.name }}.sql
  2. Execute user creation/update SQL on the cluster primary
  3. If pgbouncer_enabled: true, update /etc/pgbouncer/userlist.txt and useropts.txt
  4. Reload pgbouncer to apply configuration

User Definition Example

pg_users:
  - name: dbuser_meta               # Required, username is the only mandatory field
    password: DBUser.Meta           # Optional, can be scram-sha-256 hash or plaintext
    login: true                     # Optional, can login, default: true
    superuser: false                # Optional, is superuser, default: false
    createdb: false                 # Optional, can create database, default: false
    createrole: false               # Optional, can create role, default: false
    inherit: true                   # Optional, inherit privileges, default: true
    replication: false              # Optional, can replicate, default: false
    bypassrls: false                # Optional, bypass RLS, default: false
    pgbouncer: true                 # Optional, add to pgbouncer userlist, default: false
    connlimit: -1                   # Optional, connection limit, -1 means unlimited
    expire_in: 3650                 # Optional, expire in N days (overrides expire_at)
    expire_at: '2030-12-31'         # Optional, specify expiration date
    comment: pigsty admin user      # Optional, user comment
    roles: [dbrole_admin]           # Optional, roles to grant
    parameters: {}                  # Optional, role-level parameters
    pool_mode: transaction          # Optional, pgbouncer user-level pool mode
    pool_connlimit: 100             # Optional, user-level max connections; omitted values inherit global default 100

For details, see: Admin SOP: Create User

pgsql-db.yml

The pgsql-db.yml playbook is used to add new business databases to existing PostgreSQL clusters.

Basic Usage

./pgsql-db.yml -l pg-meta -e dbname=meta

Wrapper Scripts

bin/pgsql-db pg-meta meta  # Create database meta on cluster pg-meta

Workflow

  1. Define database in the config inventory: all.children.<pg_cluster>.vars.pg_databases[i]
  2. Execute playbook specifying cluster and database name: pgsql-db.yml -l <pg_cluster> -e dbname=<name>

The playbook will:

  1. Generate database creation SQL at /pg/tmp/pg-db-{{ database.name }}.sql
  2. Execute database creation/update SQL on the cluster primary
  3. If db.register_datasource is true, register database as Grafana datasource
  4. Update /etc/pgbouncer/database.txt and reload pgbouncer

Database Definition Example

pg_databases:
  - name: meta                      # Required, database name is the only mandatory field
    baseline: cmdb.sql              # Optional, database initialization SQL file path
    pgbouncer: true                 # Optional, add to pgbouncer, default: true
    schemas: [pigsty]               # Optional, additional schemas to create
    extensions:                     # Optional, extensions to install
      - { name: postgis, schema: public }
      - { name: timescaledb }
    comment: pigsty meta database   # Optional, database comment
    owner: postgres                 # Optional, database owner
    template: template1             # Optional, template database
    encoding: UTF8                  # Optional, character encoding
    locale: C                       # Optional, locale setting
    tablespace: pg_default          # Optional, default tablespace
    allowconn: true                 # Optional, allow connections
    revokeconn: false               # Optional, revoke public connect privilege
    register_datasource: true       # Optional, register as Grafana datasource
    connlimit: -1                   # Optional, connection limit
    pool_auth_user: dbuser_meta     # Optional, auth query user (with pgbouncer_auth_query)
    pool_mode: transaction          # Optional, pgbouncer pool mode
    pool_size: 50                   # Optional, pgbouncer default pool size
    pool_reserve: 30                # Optional, pgbouncer reserve pool size
    pool_size_min: 0                # Optional, pgbouncer minimum pool size
    pool_connlimit: 100             # Optional, pgbouncer max database connections

For details, see: Admin SOP: Create Database

pgsql-monitor.yml

The pgsql-monitor.yml playbook is used to bring remote PostgreSQL instances into Pigsty’s monitoring system.

Basic Usage

./pgsql-monitor.yml -e clsname=pg-foo  # Monitor remote cluster pg-foo

Wrapper Scripts

bin/pgmon-add pg-foo              # Monitor a remote pgsql cluster pg-foo
bin/pgmon-add pg-foo pg-bar       # Monitor multiple clusters simultaneously

Configuration

First, define pg_exporters in the infra group variables:

infra:
  hosts:
    10.10.10.10:
      pg_exporters:  # List all remote instances, assign unique unused local ports
        20001: { pg_cluster: pg-foo, pg_seq: 1, pg_host: 10.10.10.10 }
        20002: { pg_cluster: pg-foo, pg_seq: 2, pg_host: 10.10.10.11 }

Architecture Diagram

     ------ infra ------
     |                 |
     | victoria-metrics|            v---- pg-foo-1 ----v
     |       ^         |  metrics   |         ^        |
     |   pg_exporter <-|------------|----  postgres    |
     |   (port: 20001) |            | 10.10.10.10:5432 |
     |       ^         |            ^------------------^
     |       ^         |                      ^
     |       ^         |            v---- pg-foo-2 ----v
     |       ^         |  metrics   |         ^        |
     |   pg_exporter <-|------------|----  postgres    |
     |   (port: 20002) |            | 10.10.10.11:5433 |
     -------------------            ^------------------^

Configurable Parameters

pg_exporter_config: pg_exporter.yml    # pg_exporter config file name
pg_exporter_cache_ttls: '1,10,60,300'  # pg_exporter collector TTL stages
pg_exporter_port: 9630                 # pg_exporter listen port
pg_exporter_params: 'sslmode=disable'  # DSN extra URL parameters
pg_exporter_url: ''                    # Directly override auto-generated DSN
pg_exporter_auto_discovery: true       # Enable auto database discovery
pg_exporter_exclude_database: 'template0,template1,postgres'  # Databases to exclude
pg_exporter_include_database: ''       # Databases to include only
pg_exporter_connect_timeout: 200       # Connection timeout (milliseconds)
pg_monitor_username: dbuser_monitor    # Monitor username
pg_monitor_password: DBUser.Monitor    # Monitor password

Remote Database Setup

Remote PostgreSQL instances need a monitoring user:

CREATE USER dbuser_monitor;
COMMENT ON ROLE dbuser_monitor IS 'system monitor user';
ALTER USER dbuser_monitor PASSWORD 'DBUser.Monitor';
GRANT pg_monitor TO dbuser_monitor;
CREATE EXTENSION IF NOT EXISTS "pg_stat_statements" WITH SCHEMA "monitor";

Limitations

  • Only postgres metrics available
  • node, pgbouncer, patroni, haproxy metrics not available

For details, see: Admin SOP: Monitor RDS

pgsql-migration.yml

The pgsql-migration.yml playbook generates migration manuals and scripts for zero-downtime logical replication-based migration of existing PostgreSQL clusters.

Basic Usage

./pgsql-migration.yml -e@files/migration/pg-meta.yml

Workflow

  1. Define migration task configuration file (e.g., files/migration/pg-meta.yml)
  2. Execute playbook to generate migration manual and scripts
  3. Follow the manual to execute scripts step by step for migration

Migration Task Definition Example

# files/migration/pg-meta.yml
context_dir: ~/migration           # Migration manual and scripts output directory
src_cls: pg-meta                   # Source cluster name (required)
src_db: meta                       # Source database name (required)
src_ip: 10.10.10.10                # Source cluster primary IP (required)
dst_cls: pg-test                   # Target cluster name (required)
dst_db: test                       # Target database name (required)
dst_ip: 10.10.10.11                # Target cluster primary IP (required)

# Optional parameters
pg_dbsu: postgres
pg_replication_username: replicator
pg_replication_password: DBUser.Replicator
pg_admin_username: dbuser_dba
pg_admin_password: DBUser.DBA
pg_monitor_username: dbuser_monitor
pg_monitor_password: DBUser.Monitor

For details, see: Admin SOP: Migrate Cluster

pgsql-pitr.yml

The pgsql-pitr.yml playbook performs PostgreSQL Point-In-Time Recovery (PITR).

Basic Usage

# Recover to latest state (end of WAL archive stream)
./pgsql-pitr.yml -l pg-meta -e '{"pg_pitr": {}}'

# Recover to specific point in time
./pgsql-pitr.yml -l pg-meta -e '{"pg_pitr": {"time": "2025-07-13 10:00:00+00"}}'

# Recover to specific LSN
./pgsql-pitr.yml -l pg-meta -e '{"pg_pitr": {"lsn": "0/4001C80"}}'

# Recover to specific transaction ID
./pgsql-pitr.yml -l pg-meta -e '{"pg_pitr": {"xid": "250000"}}'

# Recover to named restore point
./pgsql-pitr.yml -l pg-meta -e '{"pg_pitr": {"name": "some_restore_point"}}'

# Recover from another cluster's backup
./pgsql-pitr.yml -l pg-test -e '{"pg_pitr": {"cluster": "pg-meta"}}'

PITR Task Parameters

pg_pitr:                           # Define PITR task
  cluster: "pg-meta"               # Source cluster name (for restoring from another cluster's backup)
  type: default                    # Recovery target type: default, time, xid, name, lsn, immediate
  time: "2025-01-01 10:00:00+00"   # Recovery target: point in time
  name: "some_restore_point"       # Recovery target: named restore point
  xid: "100000"                    # Recovery target: transaction ID
  lsn: "0/3000000"                 # Recovery target: log sequence number
  set: latest                      # Backup set to restore from, default: latest
  timeline: latest                 # Target timeline, can be integer, default: latest
  exclusive: false                 # Exclude target point, default: false
  action: pause                    # Post-recovery action: pause, promote, shutdown
  archive: true                    # Keep archive settings, default: true; set false for exploratory recovery
  backup: false                    # Backup existing data to /pg/data-backup before restore? default: false
  db_include: []                   # Include only these databases
  db_exclude: []                   # Exclude these databases
  link_map: {}                     # Tablespace link mapping
  process: 4                       # Parallel recovery processes, defaults to node_cpu
  repo: {}                         # Recovery source repo configuration
  data: /pg/data                   # Recovery data directory
  port: 5432                       # Recovery instance listen port

Subtasks

This playbook contains the following subtasks:

# down                 : stop HA and shutdown patroni and postgres
#   - pause            : pause patroni auto failover
#   - stop             : stop patroni and postgres services
#     - stop_patroni   : stop patroni service
#     - stop_postgres  : stop postgres service
#
# pitr                 : execute PITR recovery process
#   - config           : generate pgbackrest config and recovery script
#   - backup           : perform optional backup to original data
#   - restore          : run pgbackrest restore command
#   - recovery         : start postgres and complete recovery
#   - verify           : verify recovered cluster control data
#
# up                   : start postgres/patroni and restore HA
#   - etcd             : clean etcd metadata before startup
#   - start            : start patroni and postgres services
#     - start_postgres : start postgres service
#     - start_patroni  : start patroni service
#   - resume           : resume patroni auto failover

Recovery Target Types

Type Description Example
default Recover to end of WAL archive stream (latest state) {"pg_pitr": {}}
time Recover to specific point in time {"pg_pitr": {"time": "2025-07-13 10:00:00"}}
xid Recover to specific transaction ID {"pg_pitr": {"xid": "250000"}}
name Recover to named restore point {"pg_pitr": {"name": "before_ddl"}}
lsn Recover to specific LSN {"pg_pitr": {"lsn": "0/4001C80"}}
immediate Stop immediately after reaching consistent state {"pg_pitr": {"type": "immediate"}}

For details, see: Backup & Recovery Tutorial

8.13 - Extensions

Harness the synergistic power of PostgreSQL extensions

Pigsty provides 575 packaged extensions, covering 16 major categories including time-series, geospatial, vector, full-text search, analytics, and feature enhancements, ready to use out-of-the-box.

Using extensions in Pigsty involves four core steps: Download, Install, Config/Load, and Create.

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_databases:
      - name: meta
        extensions: [ postgis, timescaledb, vector ]   # Create: Create extensions in database
    pg_libs: 'timescaledb, pg_stat_statements, auto_explain' # Config: Preload extension libraries
    pg_extensions: [ postgis, timescaledb, pgvector ]  # Install: Install extension packages
Pigsty PostgreSQL extension ecosystem

8.13.1 - Quick Start

Four-step process overview for using extensions

Using extensions in Pigsty requires four steps: Download, Install, Config, and Create.

  1. Download: Download extension packages to the local repository (the default local repository only guarantees the base kernel and pgsql-main package set)
  2. Install: Install extension packages on cluster nodes
  3. Config: Some extensions need to be preloaded or configured with parameters
  4. Create: Execute CREATE EXTENSION in the database to create the extension

Declarative Configuration

Declare extensions in the Pigsty configuration manifest, and they will be automatically installed and created during cluster initialization:

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_databases:
      - name: meta
        extensions: [ postgis, timescaledb, vector ]   # Create extensions in database
    pg_libs: 'timescaledb, pg_stat_statements, auto_explain' # Preload extension libraries
    pg_extensions: [ postgis, timescaledb, pgvector ]  # Install extension packages

After executing ./pgsql.yml to initialize the cluster, the three extensions postgis, timescaledb, and vector will be available in the meta database.

Imperative Operations

For existing clusters, you can add extensions using command-line methods:

# 1. Install extension packages
./pgsql.yml -l pg-meta -t pg_extension -e '{"pg_extensions":["pgvector"]}'

# 2. Preload extension (if needed, requires restart after modification)
pg edit-config pg-meta --force -p shared_preload_libraries='timescaledb, pg_stat_statements, auto_explain'

# 3. Create extension in database
psql -d meta -c 'CREATE EXTENSION vector;'

You can also use the pig package manager to install extension packages, then run CREATE EXTENSION inside the database:

pig install pgvector                         # Install extension package
psql -d meta -c 'CREATE EXTENSION vector;'   # Create extension in database

Process Quick Reference

Step Parameter/Command Description
Download repo_extra_packages Specify extension packages to download to local repository
Install pg_extensions Specify extension packages to install on cluster
Config pg_libs Preload extensions to shared_preload_libraries
Create pg_databases.extensions Automatically execute CREATE EXTENSION in database

For detailed instructions, please refer to each subsection: Download, Install, Config, Create

8.13.2 - Introduction

Core concepts of PostgreSQL extensions and the Pigsty extension ecosystem

Extensions are the soul of PostgreSQL. Pigsty includes 575 pre-compiled, out-of-the-box extension plugins, fully unleashing PostgreSQL’s potential.

What are Extensions

PostgreSQL extensions are a modular mechanism that allows enhancing database functionality without modifying the core code. An extension typically consists of three parts:

  • Control file (.control): Required, contains extension metadata
  • SQL scripts (.sql): Optional, defines functions, types, operators, and other database objects
  • Dynamic library (.so): Optional, provides high-performance functionality implemented in C

Extensions can add to PostgreSQL: new data types, index methods, functions and operators, foreign data access, procedural languages, performance monitoring, security auditing, and more.

Core Extensions

Among the extensions included in Pigsty, the following are most representative:

Extension Description
PostGIS Geospatial data types and indexes, de facto GIS standard
TimescaleDB Time-series database with continuous aggregates, columnar storage, auto-compression
PGVector Vector data type with HNSW/IVFFlat indexes, essential for AI applications
Citus Distributed database with horizontal sharding capabilities
pg_duckdb Embedded DuckDB analytical engine for OLAP acceleration
pg_search ParadeDB search extension, providing BM25 and full-text search capabilities
Apache AGE Graph database supporting OpenCypher query language
pg_graphql Native GraphQL query support

Most extensions can coexist and even be combined, creating synergistic effects far greater than the sum of their parts.

Extension Categories

Pigsty organizes extensions into 16 categories:

Category Alias Description Typical Extensions
Time-series time Time-series data processing timescaledb, pg_cron, periods
Geospatial gis Geospatial data postgis, h3, pgrouting
Vector rag Vector retrieval and AI pgvector, vchord, pg_vectorize
Search fts Full-text search pgroonga, zhparser, pg_bigm
Analytics olap OLAP and analytics pg_duckdb, pg_mooncake, citus
Feature feat Feature enhancements age, pg_graphql, hll, rum
Language lang Procedural languages plpython3u, pljava, plv8
Type type Data types hstore, ltree, ip4r
Utility util Utility tools http, pg_net, pgjwt
Function func Function libraries pg_uuidv7, topn, tdigest
Admin admin Operations management pg_repack, pg_squeeze, pgagent
Stat stat Monitoring statistics pg_stat_statements, pg_qualstats, auto_explain
Security sec Security auditing pgaudit, pgsodium, pg_tde
FDW fdw Foreign data access postgres_fdw, mysql_fdw, oracle_fdw
Compatibility sim Database compatibility orafce, babelfish
ETL etl Data synchronization pglogical, wal2json, decoderbufs

You can batch install an entire category of extensions using category aliases, for example: pg_extensions: [ pgsql-gis, pgsql-rag ].

Predefined Extension Stacks

Pigsty provides several predefined extension stacks for convenient scenario-based selection:

Stack Included Extensions
gis-stack postgis, pgrouting, pointcloud, h3, q3c, ogr_fdw
rag-stack pgvector, vchord, pgvectorscale, pg_similarity, pg_tiktoken
fts-stack pgroonga, pg_bigm, zhparser, hunspell
olap-stack pg_duckdb, pg_mooncake, timescaledb, pg_partman, plproxy
feat-stack age, hll, rum, pg_graphql, pg_jsonschema, jsquery
stat-stack pg_show_plans, pg_stat_kcache, pg_qualstats, pg_wait_sampling
supa-stack pg_graphql, pg_jsonschema, wrappers, pgvector, pgsodium, vault

Simply use these names in pg_extensions to install the entire stack.

Extension Resources

8.13.3 - Packages

Extension package aliases and category naming conventions

Pigsty uses a package alias mechanism to simplify extension installation and management.

Package Alias Mechanism

Managing extensions involves multiple layers of name mapping:

Layer Example pgvector Example postgis
Extension Name vector postgis, postgis_topology, …
Package Alias pgvector postgis
RPM Package Name pgvector_18 postgis36_18*
DEB Package Name postgresql-18-pgvector postgresql-18-postgis-3*

Pigsty provides a package alias abstraction layer, so users don’t need to worry about specific RPM/DEB package names:

pg_extensions: [ pgvector, postgis, timescaledb ]  # Use package aliases

Pigsty automatically translates to the correct package names based on the operating system and PostgreSQL version.

Note

CREATE EXTENSION uses the extension name (for example, vector), not the package alias (pgvector).

Category Aliases

All extensions are organized into 16 categories, which can be batch installed using category aliases:

# Use generic category aliases (auto-adapt to current PG version)
pg_extensions: [ pgsql-gis, pgsql-rag, pgsql-fts ]

# Or use version-specific category aliases
pg_extensions: [ pg18-gis, pg18-rag, pg18-fts ]

Except for the olap category, all category extensions can be installed simultaneously. Within the olap category, there are conflicts: pg_duckdb and pg_mooncake are mutually exclusive.

Category List

Category Description Typical Extensions
time Time-series timescaledb, pg_cron, periods
gis Geospatial postgis, h3, pgrouting
rag Vector/RAG pgvector, pgml, vchord
fts Full-text Search pg_trgm, zhparser, pgroonga
olap Analytics citus, pg_duckdb, pg_mooncake
feat Feature age, pg_graphql, rum
lang Language plpython3u, pljava, plv8
type Data Type hstore, ltree, citext
util Utility http, pg_net, pgjwt
func Function pgcrypto, uuid-ossp, pg_uuidv7
admin Admin pg_repack, pgagent, pg_squeeze
stat Statistics pg_stat_statements, pg_qualstats, auto_explain
sec Security pgaudit, pgcrypto, pgsodium
fdw Foreign Data Wrapper postgres_fdw, mysql_fdw, oracle_fdw
sim Compatibility orafce, babelfishpg_tds
etl Data/ETL pglogical, wal2json, decoderbufs

Browse Extension Catalog

You can browse detailed information about all available extensions on the Pigsty Extension Catalog website, including:

  • Extension name, description, version
  • Supported PostgreSQL versions
  • Supported OS distributions
  • Installation methods, preloading requirements
  • License, source repository

8.13.4 - Download

Download extension packages from software repositories to local

Before installing extensions, ensure that extension packages are downloaded to the local repository or available from upstream.

Default Behavior

Pigsty downloads the base PostgreSQL 18 kernel packages to the local software repository by default. The default extra download set is repo_extra_packages_default: [ pgsql-main ], which includes the PostgreSQL kernel, client, procedural languages, and basic extension packages such as pg_repack, wal2json, and pgvector.

If you need other extensions from the 575-extension catalog, explicitly add them to repo_extra_packages. Pigsty does not download every extension to the local repository by default.

Benefits of using a local repository:

  • Accelerated installation, avoiding repeated downloads
  • Reduced network traffic consumption
  • Improved delivery reliability
  • Ensured version consistency

Download New Extensions

To download additional extensions, add them to repo_extra_packages and rebuild the repository:

all:
  vars:
    repo_extra_packages: [ pgvector, postgis, timescaledb, pg_duckdb ]
# Re-download packages to local repository
./infra.yml -t repo_build

# Refresh package source cache on all nodes
./node.yml -t node_repo

Using Upstream Repositories

You can also install directly from internet upstream repositories without pre-downloading:

# Add upstream software sources on nodes
./node.yml -t node_repo -e node_repo_modules=node,pgsql

This approach is suitable for:

  • Quick testing of latest versions
  • Installing rare extensions
  • Environments with good network conditions

But may face:

  • Network instability affecting installation
  • Version inconsistency risks

Extension Sources

Extension packages come from two main sources:

Repository Description
PGDG PostgreSQL official repository, providing core extensions
Pigsty Pigsty supplementary repository, providing additional extensions

The Pigsty repository only includes extensions not present in the PGDG repository. Once an extension enters the PGDG repository, the Pigsty repository will remove it or keep it consistent.

Repository URLs:

For detailed repository configuration, see Extension Repository.

8.13.5 - Install

Install extension packages on cluster nodes

Pigsty uses the operating system’s package manager (yum/apt) to install extension packages.

Related Parameters

Two parameters are used to specify extensions to install:

Parameter Purpose Default Behavior
pg_packages Global common packages Ensure present (no upgrade)
pg_extensions Cluster-specific extensions Install latest version

pg_packages is typically used to specify base components needed by all clusters (PostgreSQL kernel, Patroni, pgBouncer, etc.) and essential extensions.

pg_extensions is used to specify extensions needed by specific clusters.

pg_packages:                           # Global base packages
  - pgsql-main pgsql-common
pg_extensions:                         # Cluster extensions
  - postgis timescaledb pgvector

Install During Cluster Initialization

Declare extensions in cluster configuration, and they will be automatically installed during initialization:

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_extensions: [ postgis, timescaledb, pgvector, pg_duckdb ]

When executing ./pgsql.yml to initialize the cluster, extensions will be automatically installed.

Install Extensions on Existing Cluster

For initialized clusters, there are multiple ways to install extensions:

Using Pigsty Playbook 

# Install using playbook after modifying configuration
./pgsql.yml -l pg-meta -t pg_extension

# Or specify extensions directly on command line
./pgsql.yml -l pg-meta -t pg_extension -e '{"pg_extensions":["pg_duckdb"]}'

Using pig Package Manager 

# Install extension using pig
pig install pg_duckdb

# Batch install
ansible pg-meta -b -a 'pig install pg_duckdb pgvector'

Using Package Manager Directly 

# EL systems
sudo yum install -y pg_duckdb_18*

# Debian/Ubuntu systems
sudo apt install -y postgresql-18-pg-duckdb

Using Package Aliases

Pigsty supports using standardized package aliases, automatically translating to package names for the corresponding PG version:

pg_extensions:
  - pgvector           # Auto-translates to pgvector_18* (EL) or postgresql-18-pgvector (Debian)
  - postgis            # Auto-translates to postgis36_18* (EL) or postgresql-18-postgis-3* (Debian)
  - pgsql-gis          # Category alias, installs entire GIS category of extensions

You can also use raw package names directly:

pg_extensions:
  - pgvector_18*                    # EL system raw package name
  - postgresql-18-pgvector          # Debian system raw package name

For package alias definitions, see:

Verify Installation

After installation, verify in the database:

-- Check installed extensions
SELECT * FROM pg_available_extensions WHERE name = 'vector';

-- Check if extension files exist
\dx

8.13.6 - Config

Preload extension libraries and configure extension parameters

Some extensions require preloading dynamic libraries or configuring parameters before use. This section describes how to configure extensions.

Preload Extensions

Most extensions can be enabled directly with CREATE EXTENSION after installation, but some extensions using PostgreSQL’s Hook mechanism require preloading.

Preloading is specified via the shared_preload_libraries parameter and requires a database restart to take effect.

Extensions Requiring Preload

Common extensions that require preloading:

Extension Description
timescaledb Time-series database extension, must be placed first
citus Distributed database extension, must be placed first
pg_stat_statements SQL statement statistics, enabled by default in Pigsty
auto_explain Automatically log slow query execution plans, enabled by default in Pigsty
pg_cron Scheduled task scheduling
pg_net Asynchronous HTTP requests
pg_tle Trusted language extensions
pgaudit Audit logging
pg_stat_kcache Kernel statistics
pg_squeeze Online table space reclamation
pgml PostgresML machine learning

For the complete list, see the Extension Catalog (marked with LOAD).

Preload Order

The loading order of extensions in shared_preload_libraries is important:

  • timescaledb and citus must be placed first
  • If using both, citus should come before timescaledb
  • Statistics extensions should come after pg_stat_statements to use the same query_id
pg_libs: 'citus, timescaledb, pg_stat_statements, auto_explain'

Configure During Cluster Initialization

When creating a new cluster, use the pg_libs parameter to specify preloaded extensions:

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_libs: 'timescaledb, pg_stat_statements, auto_explain'
    pg_extensions: [ timescaledb, postgis, pgvector ]

The value of pg_libs will be written to shared_preload_libraries during cluster initialization.

Default Value

The default value of pg_libs is pg_stat_statements, auto_explain. These two Contrib extensions provide basic observability:

  • pg_stat_statements: Track execution statistics of all SQL statements
  • auto_explain: Automatically log execution plans for slow queries

Modify Configuration on Existing Cluster

For initialized clusters, use patronictl to modify shared_preload_libraries:

# Add timescaledb to preload libraries
pg edit-config pg-meta --force -p shared_preload_libraries='timescaledb, pg_stat_statements, auto_explain'

# Restart cluster to apply configuration
pg restart pg-meta

You can also directly modify postgresql.conf or use ALTER SYSTEM:

ALTER SYSTEM SET shared_preload_libraries = 'timescaledb, pg_stat_statements, auto_explain';

A PostgreSQL service restart is required after modification.

Extension Parameter Configuration

Many extensions have configurable parameters that can be set in the following locations:

During Cluster Initialization

Use the pg_parameters parameter to specify:

pg-meta:
  vars:
    pg_cluster: pg-meta
    pg_libs: 'pg_cron, pg_stat_statements, auto_explain'
    pg_parameters:
      cron.database_name: postgres           # Database used by pg_cron
      pg_stat_statements.track: all          # Track all statements
      auto_explain.log_min_duration: 1000    # Log queries exceeding 1 second

Runtime Modification

Use ALTER SYSTEM or patronictl:

-- Modify parameter
ALTER SYSTEM SET pg_stat_statements.track = 'all';

-- Reload configuration
SELECT pg_reload_conf();
# Modify using patronictl
pg edit-config pg-meta --force -p 'pg_stat_statements.track=all'

Important Notes

  1. Preload errors prevent startup: If an extension in shared_preload_libraries doesn’t exist or fails to load, PostgreSQL will not start. Ensure extensions are properly installed before adding to preload.

  2. Modification requires restart: Changes to shared_preload_libraries require restarting the PostgreSQL service to take effect.

  3. Partial functionality available: Some extensions can be partially used without preloading, but full functionality requires preloading.

  4. View current configuration: Use the following command to view current preload libraries:

SHOW shared_preload_libraries;

8.13.7 - Create

Create and enable extensions in databases

After installing extension packages, you need to execute CREATE EXTENSION in the database to use extension features.

View Available Extensions

After installing extension packages, you can view available extensions:

-- View all available extensions
SELECT * FROM pg_available_extensions;

-- View specific extension
SELECT * FROM pg_available_extensions WHERE name = 'vector';

-- View enabled extensions
SELECT * FROM pg_extension;

Create Extensions

Use CREATE EXTENSION to enable extensions in the database:

-- Create extension
CREATE EXTENSION vector;

-- Create extension in specific schema
CREATE EXTENSION postgis SCHEMA public;

-- Automatically install dependent extensions
CREATE EXTENSION postgis_topology CASCADE;

-- Create if not exists
CREATE EXTENSION IF NOT EXISTS vector;
Note

CREATE EXTENSION uses the extension name (for example, vector), not the package alias (pgvector).

Create During Cluster Initialization

Declare extensions in pg_databases, and they will be automatically created during cluster initialization:

pg-meta:
  vars:
    pg_cluster: pg-meta
    pg_databases:
      - name: meta
        extensions:
          - { name: vector }                         # Use default schema
          - { name: postgis, schema: public }        # Specify schema
          - { name: pg_stat_statements, schema: monitor }

Pigsty will automatically execute CREATE EXTENSION after database creation.

Extensions Requiring Preload

Some extensions must be added to shared_preload_libraries and restarted before creation:

pg-meta:
  vars:
    pg_cluster: pg-meta
    pg_libs: 'timescaledb, pg_stat_statements, auto_explain'
    pg_databases:
      - name: meta
        extensions:
          - { name: timescaledb }  # Requires preload

If you try to create without preloading, you will receive an error message.

Common extensions requiring preload: timescaledb, citus, pg_cron, pg_net, pgaudit, etc. See Configure Extensions.

Extension Dependencies

Some extensions depend on other extensions and need to be created in order:

-- postgis_topology depends on postgis
CREATE EXTENSION postgis;
CREATE EXTENSION postgis_topology;

-- Or use CASCADE to automatically install dependencies
CREATE EXTENSION postgis_topology CASCADE;

Extensions Not Requiring Creation

A few extensions don’t provide SQL interfaces and don’t need CREATE EXTENSION:

Extension Description
wal2json Logical decoding plugin, used directly in replication slots
decoderbufs Logical decoding plugin
decoder_raw Logical decoding plugin

These extensions can be used immediately after installation, for example:

-- Create logical replication slot using wal2json
SELECT * FROM pg_create_logical_replication_slot('test_slot', 'wal2json');

View Extension Information 

-- View extension details
\dx+ vector

-- View objects contained in extension
SELECT * FROM pg_extension_config_dump('vector');

-- View extension version
SELECT extversion FROM pg_extension WHERE extname = 'vector';

8.13.8 - Update

Upgrade PostgreSQL extension versions

Extension updates involve two levels: package updates (operating system level) and extension object updates (database level).

Update Packages

Use package managers to update extension packages:

# EL systems
sudo yum update pgvector_18*

# Debian/Ubuntu systems
sudo apt update && sudo apt upgrade postgresql-18-pgvector

Batch update using Pigsty:

# Update extension packages for specified cluster
./pgsql.yml -l pg-meta -t pg_extension -e '{"pg_extensions":["pgvector"]}'

# Using pig package manager
pig update pgvector

Update Extension Objects

After package updates, extension objects in the database may need to be synchronized.

View Updatable Extensions 

-- View installed extensions and their versions
SELECT name, default_version, installed_version
FROM pg_available_extensions
WHERE installed_version IS NOT NULL;

-- View upgradable extensions
SELECT name, installed_version, default_version
FROM pg_available_extensions
WHERE installed_version IS NOT NULL
  AND installed_version <> default_version;

Execute Extension Update 

-- Update to latest version
ALTER EXTENSION pgvector UPDATE;

-- Update to specific version
ALTER EXTENSION pgvector UPDATE TO '0.8.0';

View Update Paths 

-- View available upgrade paths for extension
SELECT * FROM pg_extension_update_paths('pgvector');

Important Notes

  1. Backup first: Backup the database before updating extensions, especially for extensions involving data type changes.

  2. Check compatibility: Some extension major version upgrades may be incompatible. Consult the extension’s upgrade documentation.

  3. Preloaded extensions: If updating a preloaded extension (like timescaledb), a database restart may be required after the update.

  4. Dependencies: If other extensions depend on the updated extension, update them in dependency order.

  5. Replication environments: In master-slave replication environments, test updates on slaves first, then update the master after confirmation.

Common Issues

Update Failure

If ALTER EXTENSION UPDATE fails, it may be because:

  • No available upgrade path
  • Extension is in use
  • Insufficient permissions
-- View extension dependencies
SELECT * FROM pg_depend WHERE refobjid = (SELECT oid FROM pg_extension WHERE extname = 'pgvector');

Rollback Update

PostgreSQL extensions typically don’t support direct rollback. To rollback:

  1. Restore from backup
  2. Or: Uninstall new version extension, install old version package, recreate extension

8.13.9 - Remove

Uninstall PostgreSQL extensions

Removing extensions involves two levels: dropping extension objects (database level) and uninstalling packages (operating system level).

Drop Extension Objects

Use DROP EXTENSION to remove extensions from the database:

-- Drop extension
DROP EXTENSION pgvector;

-- If there are dependent objects, cascade delete is required
DROP EXTENSION pgvector CASCADE;

Warning: CASCADE will drop all objects that depend on this extension (tables, functions, views, etc.). Use with caution.

Check Extension Dependencies

It’s recommended to check dependencies before dropping:

-- View objects that depend on an extension
SELECT
    classid::regclass,
    objid,
    deptype
FROM pg_depend
WHERE refobjid = (SELECT oid FROM pg_extension WHERE extname = 'pgvector');

-- View tables using extension types
SELECT
    c.relname AS table_name,
    a.attname AS column_name,
    t.typname AS type_name
FROM pg_attribute a
JOIN pg_class c ON a.attrelid = c.oid
JOIN pg_type t ON a.atttypid = t.oid
WHERE t.typname = 'vector';

Remove Preload

If the extension is in shared_preload_libraries, it must be removed from the preload list after dropping:

# Modify shared_preload_libraries, remove extension
pg edit-config pg-meta --force -p shared_preload_libraries='pg_stat_statements, auto_explain'

# Restart to apply configuration
pg restart pg-meta

Uninstall Packages

After dropping the extension from the database, you can optionally uninstall the package:

# EL systems
sudo yum remove pgvector_18*

# Debian/Ubuntu systems
sudo apt remove postgresql-18-pgvector

# Using pig package manager
pig remove pgvector

Typically keeping the package doesn’t cause issues. Only uninstall when you need to free disk space or resolve conflicts.

Important Notes

  1. Data loss risk: Using CASCADE will drop dependent objects, potentially causing data loss.

  2. Application compatibility: Ensure applications no longer use the extension’s functionality before dropping.

  3. Preload order: If dropping a preloaded extension, be sure to also remove it from shared_preload_libraries, otherwise the database may fail to start.

  4. Master-slave environments: In replication environments, DROP EXTENSION automatically replicates to slaves.

Operation Sequence

Complete extension removal workflow:

# 1. Check dependencies
psql -d mydb -c "SELECT * FROM pg_depend WHERE refobjid = (SELECT oid FROM pg_extension WHERE extname = 'pgvector');"

# 2. Drop extension from database
psql -d mydb -c "DROP EXTENSION pgvector;"

# 3. If it's a preloaded extension, remove from shared_preload_libraries
pg edit-config pg-meta --force -p shared_preload_libraries='pg_stat_statements, auto_explain'

# 4. Restart database (if preload configuration was modified)
pg restart pg-meta

# 5. Optional: Uninstall package
sudo yum remove pgvector_18*

8.13.10 - Default Extensions

PostgreSQL extensions installed by default in Pigsty

Pigsty installs and enables some core extensions by default when initializing PostgreSQL clusters.

Default Installed Extensions

Extensions installed by default via pg_packages:

Extension Description
pg_repack Handle table bloat online, important maintenance tool
wal2json Logical decoding outputs JSON format changes, commonly used in CDC scenarios
pgvector Vector data type and indexes, installed with pgsql-main by default

The default value of pg_extensions is an empty array []. Declare additional extensions as needed, for example:

Extension Description
postgis Geospatial database extension
timescaledb Time-series database extension
pgvector Vector data type and indexes

Default Enabled Extensions

Extensions enabled by default in all databases via pg_default_extensions:

Extension Schema Description
pg_stat_statements monitor SQL statement execution statistics
pgstattuple monitor Tuple-level statistics
pg_buffercache monitor Buffer cache inspection
pageinspect monitor Page-level inspection
pg_prewarm monitor Relation prewarming
pg_visibility monitor Visibility map inspection
pg_freespacemap monitor Free space map inspection
postgres_fdw public PostgreSQL foreign data wrapper
file_fdw public File foreign data wrapper
btree_gist public B-tree GiST operator classes
btree_gin public B-tree GIN operator classes
pg_trgm public Trigram matching
intagg public Integer aggregator
intarray public Integer array functions
pg_repack repack Online table reorganization

These extensions provide basic monitoring, operations, and feature enhancement capabilities.

Default Preloaded Extensions

Extensions preloaded by default into shared_preload_libraries via pg_libs:

Extension Description
pg_stat_statements Track execution statistics of all SQL statements
auto_explain Automatically log execution plans for slow queries

These two extensions provide basic observability and are strongly recommended to keep.

Customize Default Extensions

You can customize default installed and enabled extensions by modifying configuration parameters:

all:
  vars:
    # Modify default extension packages
    pg_packages:
      - pgsql-main pgsql-common
      - pg_repack_$v* wal2json_$v*

    # Modify default installed extensions
    pg_extensions: [ postgis, timescaledb, pgvector ]

    # Modify default preloaded extensions
    pg_libs: 'timescaledb, pg_stat_statements, auto_explain'

    # Modify default enabled extensions
    pg_default_extensions:
      - { name: pg_stat_statements, schema: monitor }
      - { name: pg_repack }
      # ... add more

For detailed extension usage, please refer to:

8.13.11 - Repository

Pigsty extension software repository configuration

Pigsty provides supplementary extension repositories, offering additional extension packages on top of the PGDG official repository.

YUM Repository

Applicable to EL 8/9/10 and compatible systems (RHEL, Rocky, AlmaLinux, CentOS, etc.).

Add Repository 

# Add GPG public key
curl -fsSL https://repo.pigsty.io/key | sudo tee /etc/pki/rpm-gpg/RPM-GPG-KEY-pigsty >/dev/null

# Add repository configuration
curl -fsSL https://repo.pigsty.io/yum/repo | sudo tee /etc/yum.repos.d/pigsty.repo >/dev/null

# Refresh cache
sudo yum makecache

China Mainland Mirror 

curl -fsSL https://repo.pigsty.cc/key | sudo tee /etc/pki/rpm-gpg/RPM-GPG-KEY-pigsty >/dev/null
curl -fsSL https://repo.pigsty.cc/yum/repo | sudo tee /etc/yum.repos.d/pigsty.repo >/dev/null

Repository URLs

APT Repository

Applicable to Debian 12/13 and Ubuntu 22.04/24.04/26.04 and compatible systems.

Add Repository 

# Add GPG public key
curl -fsSL https://repo.pigsty.io/key | sudo gpg --dearmor -o /etc/apt/keyrings/pigsty.gpg

# Get distribution codename and add repository
distro_codename=$(lsb_release -cs)
sudo tee /etc/apt/sources.list.d/pigsty.list > /dev/null <<EOF
deb [signed-by=/etc/apt/keyrings/pigsty.gpg] https://repo.pigsty.io/apt/infra generic main
deb [signed-by=/etc/apt/keyrings/pigsty.gpg] https://repo.pigsty.io/apt/pgsql/${distro_codename} ${distro_codename} main
EOF

# Refresh cache
sudo apt update

China Mainland Mirror 

curl -fsSL https://repo.pigsty.cc/key | sudo gpg --dearmor -o /etc/apt/keyrings/pigsty.gpg

distro_codename=$(lsb_release -cs)
sudo tee /etc/apt/sources.list.d/pigsty.list > /dev/null <<EOF
deb [signed-by=/etc/apt/keyrings/pigsty.gpg] https://repo.pigsty.cc/apt/infra generic main
deb [signed-by=/etc/apt/keyrings/pigsty.gpg] https://repo.pigsty.cc/apt/pgsql/${distro_codename} ${distro_codename} main
EOF

Repository URLs

GPG Signature

All packages are signed with GPG:

  • Fingerprint: 9592A7BC7A682E7333376E09E7935D8DB9BD8B20
  • Short ID: B9BD8B20

Repository Policy

The Pigsty repository follows these principles:

  1. Supplementary: Only includes extensions not present in the PGDG repository
  2. Consistency: Once an extension enters the PGDG repository, the Pigsty repository will remove it or keep it consistent
  3. Compatibility: Supports multiple major versions of PostgreSQL 14-18
  4. Multi-platform: Supports x86_64 and aarch64 architectures

Related Resources

8.14 - PG Kernels

How to use PostgreSQL kernel forks in Pigsty, such as Citus, Babelfish, IvorySQL, PolarDB, and more.

In Pigsty, you can replace the native PostgreSQL kernel with different PostgreSQL “flavors” to unlock specialized capabilities.

Pigsty supports multiple PostgreSQL kernels and compatibility branches so you can get compatibility layers, multi-master replication, graph queries, MPP warehousing, transparent encryption, and more inside one operational framework.

One thing to keep in mind is that not every kernel has the same delivery depth in Pigsty: PostgreSQL, Citus, Babelfish, IvorySQL, PolarDB, AgensGraph, and pgEdge already have relatively clear templates and configuration paths; Cloudberry and Greenplum, by contrast, are more often managed through gpsql mode, and their MPP initialization plus scale-out operations are still better handled with upstream tooling.

Kernel Key Feature Description
PostgreSQL Native kernel, full extension set Vanilla PostgreSQL with 575 extensions
Supabase Backend as a Service PostgreSQL-based BaaS, Firebase alternative
Citus Horizontal scaling Distributed PostgreSQL via native extension
Babelfish SQL Server compatible SQL Server wire-protocol compatibility (PG17/18)
IvorySQL Oracle compatible Oracle syntax and PL/SQL compatibility
OpenHalo MySQL compatible MySQL wire-protocol compatibility
Percona Transparent data encryption Percona distribution with pg_tde
DocumentDB MongoDB migration DocumentDB + FerretDB wire compatibility
OrioleDB OLTP optimization Zheap, no bloat, S3 storage
PolarDB Aurora-style RAC RAC, China-local compliance scenario
Cloudberry Open-source MPP warehouse Cloudberry integrated through gpsql mode
AgensGraph Property graph + Cypher Graph query capability inside PostgreSQL
pgEdge Spock multi-master replication Distributed PostgreSQL for edge scenarios
PostgreSQL forks and compatible kernels

Versions

Kernel Debian / Ubuntu EL
PostgreSQL / Citus PostgreSQL 18.6 (Ubuntu 18.6-1.pgdg26.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 15.2.0-16ubuntu1) 15.2.0, 64-bit PostgreSQL 18.6 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 14.3.1 20251022 (Red Hat 14.3.1-4), 64-bit
IvorySQL PostgreSQL 18.4 (IvorySQL 5.4) on x86_64-pc-linux-gnu, compiled by gcc (GCC) 9.5.0, 64-bit PostgreSQL 18.4 (IvorySQL 5.4) on x86_64-pc-linux-gnu, compiled by gcc (GCC) 9.5.0, 64-bit
Babelfish Babelfish 17.7 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 15.2.0-16ubuntu1) 15.2.0, 64-bit Babelfish 17.7 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 14.3.1 20251022 (Red Hat 14.3.1-4), 64-bit
PolarDB PostgreSQL 17.10 (PolarDB 17.10.1.0 build accf02e2) on x86_64-linux-gnu PostgreSQL 17.10 (PolarDB 17.10.1.0 build accf02e2) on x86_64-linux-gnu
Percona PostgreSQL 18.4 - Percona Server for PostgreSQL 18.4.1 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 15.2.0-16ubuntu1) 15.2.0, 64-bit PostgreSQL 18.4 - Percona Server for PostgreSQL 18.4.1 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 14.3.1 20250617 (Red Hat 14.3.1-2), 64-bit
OrioleDB OrioleDB 18.4 (OrioleDB 1.8-beta16) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 15.2.0-16ubuntu1) 15.2.0, 64-bit OrioleDB 18.4 (OrioleDB 1.8-beta16) on x86_64-pc-linux-gnu, compiled by gcc (GCC) 14.3.1 20251022 (Red Hat 14.3.1-4), 64-bit
OpenHalo openHalo 14.18 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 15.2.0-16ubuntu1) 15.2.0, 64-bit openHalo 14.18 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 14.3.1 20251022 (Red Hat 14.3.1-4), 64-bit
DocumentDB PostgreSQL 18.6 (Ubuntu 18.6-1.pgdg26.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 15.2.0-16ubuntu1) 15.2.0, 64-bit PostgreSQL 18.6 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 14.3.1 20251022 (Red Hat 14.3.1-4), 64-bit
AgensGraph PostgreSQL 17.10 (AgensGraph 2.17.0) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 15.2.0-16ubuntu1) 15.2.0, 64-bit PostgreSQL 17.10 (AgensGraph 2.17.0) on x86_64-pc-linux-gnu, compiled by gcc (GCC) 14.3.1 20251022 (Red Hat 14.3.1-4), 64-bit
pgEdge PostgreSQL 18.4 (pgEdge 5.0.10) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 15.2.0-16ubuntu1) 15.2.0, 64-bit PostgreSQL 18.4 (pgEdge 5.0.10) on x86_64-pc-linux-gnu, compiled by gcc (GCC) 14.3.1 20251022 (Red Hat 14.3.1-4), 64-bit
Cloudberry PostgreSQL 14.4 (Apache Cloudberry 2.0.0-incubating build 1) on aarch64-unknown-linux-gnu, compiled by gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-11), 64-bit

8.14.1 - PostgreSQL

Vanilla PostgreSQL kernel with 575 extensions

PostgreSQL is the world’s most advanced and popular open-source database.

Pigsty installs PostgreSQL 18 by default, supports PostgreSQL 14 ~ 18, and provides 575 PG extensions.

Quick Start

Install Pigsty using the pgsql configuration template.

./configure -c pgsql     # Use postgres kernel
./deploy.yml             # Deploy the Pigsty core chain with native PostgreSQL

Most configuration templates use PostgreSQL kernel by default, for example:

  • meta : Default, postgres with core extensions (vector, postgis, timescale)
  • rich : postgres with all extensions installed
  • slim : postgres only, no monitoring infrastructure
  • ha/full : 4-node sandbox for HA demonstration
  • pgsql : minimal postgres kernel configuration example

Configuration

Vanilla PostgreSQL kernel requires no special adjustments:

pg-meta:
  hosts:
    10.10.10.10: { pg_seq: 1, pg_role: primary }
  vars:
    pg_cluster: pg-meta
    pg_users:
      - { name: dbuser_meta ,password: DBUser.Meta   ,pgbouncer: true ,roles: [dbrole_admin   ] ,comment: pigsty admin user }
      - { name: dbuser_view ,password: DBUser.Viewer ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer  }
    pg_databases:
      - { name: meta, baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty] ,extensions: [ vector ]}
    pg_hba_rules:
      - { user: dbuser_view , db: all ,addr: infra ,auth: pwd ,title: 'allow grafana dashboard access cmdb from infra nodes' }
    pg_crontab: [ '00 01 * * * /pg/bin/pg-backup full' ] # Full backup at 1 AM daily
    pg_packages: [ pgsql-main, pgsql-common ]   # pg kernel and common utilities
    #pg_extensions: [ pg18-time ,pg18-gis ,pg18-rag ,pg18-fts ,pg18-olap ,pg18-feat ,pg18-lang ,pg18-type ,pg18-util ,pg18-func ,pg18-admin ,pg18-stat ,pg18-sec ,pg18-fdw ,pg18-sim ,pg18-etl]

Version Selection

To use a different PostgreSQL major version, you can configure it using the -v parameter:

./configure -c pgsql            # Default is postgresql 18, no need to specify explicitly
./configure -c pgsql -v 18      # Explicitly use postgresql 18
./configure -c pgsql -v 17      # Use postgresql 17
./configure -c pgsql -v 16      # Use postgresql 16
./configure -c pgsql -v 15      # Use postgresql 15
./configure -c pgsql -v 14      # Use postgresql 14

If a PostgreSQL cluster is already installed, you need to uninstall it before installing a new version:

./pgsql-rm.yml -l pg-meta # Uninstall cluster pg-meta

Extension Ecosystem

Pigsty provides a rich extension ecosystem for PostgreSQL. See the Extension Catalog for details.

8.14.2 - Supabase

How to self-host Supabase with Pigsty, deploy an open-source Firebase alternative with a complete backend stack in one click.

Supabase — Build in a weekend, Scale to millions

Supabase is an open-source Firebase alternative that wraps PostgreSQL and provides authentication, out-of-the-box APIs, edge functions, real-time subscriptions, object storage, and vector embedding capabilities. This is a low-code all-in-one backend platform that lets you skip most backend development work, requiring only database design and frontend knowledge to quickly ship products!

Supabase’s motto is: “Build in a weekend, Scale to millions”. Indeed, Supabase is extremely cost-effective at small to micro scales (4c8g), like a cyber bodhisattva. — But when you really scale to millions of users — you should seriously consider self-hosting Supabase — whether for functionality, performance, or cost considerations.

Pigsty provides you with a complete one-click self-hosting solution for Supabase. Self-hosted Supabase enjoys full PostgreSQL monitoring, IaC, PITR, and high availability, and compared to Supabase cloud services, it provides up to 575 out-of-the-box PostgreSQL extensions and can more fully utilize the performance and cost advantages of modern hardware.

For the complete self-hosting tutorial, please refer to: Supabase Self-Hosting Guide

Supabase

Quick Start

Pigsty’s default supabase.yml configuration template defines a single-node Supabase.

First, use Pigsty’s standard installation process to install the Silo and PostgreSQL instances required for Supabase:

 curl -fsSL https://repo.pigsty.io/get | bash
./bootstrap          # Environment check, install dependencies
./configure -c supabase  # Important: modify passwords and other key info in config!
./deploy.yml         # Install Pigsty, deploy PGSQL and MINIO!

Before deploying Supabase, please modify the Supabase parameters in the pigsty.yml config file according to your actual situation (mainly passwords!)

Then, run docker.yml and app.yml to complete the remaining work and deploy Supabase containers:

./docker.yml       # Install Docker module
./app.yml          # Start Supabase stateless components!

For users in China, please configure appropriate Docker mirror sites or proxy servers to bypass GFW to pull DockerHub images. For professional subscriptions, we provide the ability to offline install Pigsty and Supabase without internet access.

Pigsty exposes web services through Nginx on the admin node/INFRA node by default. You can add DNS resolution for supa.pigsty pointing to this node locally, then access https://supa.pigsty through a browser to enter the Supabase Studio management interface.

Default username and password: supabase / pigsty

demo/supabase.cast

8.14.3 - Babelfish

Use Babelfish (PG17/18) in Pigsty to provide SQL Server protocol/T-SQL compatibility.

Babelfish is a PostgreSQL-based SQL Server compatibility layer, open-sourced by AWS.

Overview

Pigsty lets you deploy Babelfish in mssql mode and provide, on top of PostgreSQL:

  • SQL Server wire protocol compatibility (TDS, 1433)
  • T-SQL compatibility
  • Unified integration with Pigsty capabilities (HA, backup, monitoring, IaC)

In Pigsty v4, Babelfish supports PostgreSQL 17/18. The default template uses pg_version: 17, and Babelfish is part of Pigsty’s standard delivery path with support for all Linux platforms.

Current Behavior

Compared with older Babelfish/PG15 docs, current behavior is:

  • Supported kernel majors are PG17/18; the template defaults to PG17 (pg_version: 17)
  • Default package group: babelfish + pgsql-common + sqlcmd
  • Mainstream platform coverage:
    • OS: el8, el9, el10, d12, d13, u22, u24, u26
    • Arch: x86_64, aarch64
  • mssql template no longer requires an extra mssql repo module (defaults to node,infra,pgsql)

Older docs may still contain deprecated naming. Pigsty now consistently uses Babelfish and babelfish aliases.

Quick Start

Use the built-in Pigsty template:

./configure -c mssql [-v 17/18]
./deploy.yml

After deployment, connect directly with SQL Server clients:

sqlcmd -S <ip>,1433 -U dbuser_mssql -P DBUser.MSSQL -d mssql

Key Configuration

Core parameters in the mssql template:

pg_mode: mssql
pg_version: 17  # optional: 18
pg_packages: [ babelfish, pgsql-common, sqlcmd ]
pg_libs: 'babelfishpg_tds, pg_stat_statements, auto_explain'

pg_databases:
  - name: mssql
    baseline: mssql.sql
    extensions:
      - { name: uuid-ossp }
      - { name: babelfishpg_common }
      - { name: babelfishpg_tsql }
      - { name: babelfishpg_tds }
      - { name: babelfishpg_money }
      - { name: pg_hint_plan }
      - { name: system_stats }
      - { name: tds_fdw }
    parameters: { 'babelfishpg_tsql.migration_mode': 'multi-db' }

pg_hba_rules:
  - { user: dbuser_mssql, db: mssql, addr: intra, auth: md5, order: 525 }

pg_default_services:
  - { name: primary, port: 5433, dest: 1433 }
  - { name: replica, port: 5434, dest: 1433 }

Connectivity and Ports

Babelfish clusters expose two protocol endpoints:

  • PostgreSQL protocol: 5432
  • SQL Server protocol (TDS): 1433

With Pigsty service abstraction you can also use:

  • 5433: fixed route to primary 1433
  • 5434: route to readable node 1433
# Primary write access
sqlcmd -S <any-node-ip>,5433 -U dbuser_mssql -P DBUser.MSSQL

# Read replica query
sqlcmd -S <any-node-ip>,5434 -U dbuser_mssql -P DBUser.MSSQL

Notes

  • Babelfish auth rules must use md5 instead of default scram-sha-256.
  • Default migration mode is multi-db; switch with babelfishpg_tsql.migration_mode if needed.
  • Not all native PostgreSQL extensions are directly usable on Babelfish kernels; validate package availability and compatibility first.
  • Tighten HBA and network exposure for production; do not keep demo-level open rules.

Related Docs

Available Extensions

The Babelfish kernel has 55 available extensions. After removing bundled PG Contrib extensions, the following extra extensions remain:

Extension Version Description
babelfishpg_common 5.4.0 Transact SQL Datatype Support
babelfishpg_money 1.1.0 babelfishpg_money
babelfishpg_tds 1.0.0 TDS protocol extension
babelfishpg_tsql 5.4.0 Transact SQL compatibility

8.14.4 - Percona

Percona Postgres distribution with TDE transparent encryption support

Percona Postgres is a patched Postgres kernel with pg_tde (Transparent Data Encryption) extension.

Starting with v4.4.0, Pigsty packages Percona PostgreSQL under the private /usr/pgtde-$v; v4.5.0 keeps this layout prefix (/usr/pgtde-18 for PostgreSQL 18). The pgtde package alias installs both the kernel package and its contrib package, including pg_tde, PostGIS, pgvector, wal2json, pg_repack, pgaudit, and pg_stat_monitor.

Quick Start

Use Pigsty’s standard installation process with the pgtde configuration template.

curl -fsSL https://repo.pigsty.io/get | bash; cd ~/pigsty;
./configure -c pgtde     # Use percona postgres kernel
./deploy.yml             # Deploy the Pigsty core chain with Percona PostgreSQL

Configuration

The following parameters need to be adjusted to deploy a Percona cluster:

pg-meta:
  hosts:
    10.10.10.10: { pg_seq: 1, pg_role: primary }
  vars:
    pg_mode: pgtde
    pg_cluster: pg-meta
    pg_users:
      - { name: dbuser_meta ,password: DBUser.Meta   ,pgbouncer: true ,roles: [dbrole_admin   ] ,comment: pgsql admin user }
      - { name: dbuser_view ,password: DBUser.Viewer ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer  }
    pg_databases:
      - name: meta
        baseline: cmdb.sql
        comment: pigsty tde database
        schemas: [pigsty]
        extensions: [ vector, postgis, pg_tde ,pgaudit, { name: pg_stat_monitor, schema: monitor } ]
    pg_hba_rules:
      - { user: dbuser_view , db: all ,addr: infra ,auth: pwd ,title: 'allow grafana dashboard access cmdb from infra nodes' }
    pg_crontab: [ '00 01 * * * /pg/bin/pg-backup full' ] # Full backup at 1 AM daily

    # Percona PostgreSQL TDE kernel settings
    pg_packages: [ pgtde, pgsql-common ]
    pg_libs: 'pg_tde, pgaudit, pg_stat_statements, pg_stat_monitor, auto_explain'

The pgtde packages are delivered by Pigsty’s pgsql repository module. The legacy percona module is not required by this template.

Available Extensions

The Percona Postgres kernel has 73 available extensions. After removing bundled PG Contrib extensions, the following extra extensions remain:

Extension Version Description
address_standardizer 3.5.7 Used to parse an address into constituent elements. Generally used to support geocoding address normalization step.
address_standardizer_data_us 3.5.7 Address Standardizer US dataset example
pg_repack 1.5.3 Reorganize tables in PostgreSQL databases with minimal locks
pg_stat_monitor 2.3.2 The pg_stat_monitor is a PostgreSQL Query Performance Monitoring tool, based on PostgreSQL contrib module pg_stat_statements. pg_stat_monitor provides aggregated statistics, client information, plan details including plan, and histogram information.
pg_tde 2.2.1 pg_tde access method
pgaudit 18.0 provides auditing functionality
postgis 3.5.7 PostGIS geometry and geography spatial types and functions
postgis_raster 3.5.7 PostGIS raster types and functions
postgis_sfcgal 3.5.7 PostGIS SFCGAL functions
postgis_tiger_geocoder 3.5.7 PostGIS tiger geocoder and reverse geocoder
postgis_topology 3.5.7 PostGIS topology spatial types and functions
set_user 4.2.0 similar to SET ROLE but with added logging
vector 0.8.3 vector data type and ivfflat and hnsw access methods

Key Features

  • Transparent Data Encryption: Provides data-at-rest encryption using the pg_tde extension
  • PostgreSQL 18 Compatible: Based on the Percona PostgreSQL 18 package set
  • Enterprise Extensions: Includes enterprise-grade features like pgaudit, pg_stat_monitor
  • Complete Ecosystem: Supports popular extensions like pgvector, PostGIS

Note: Currently in stable stage - thoroughly evaluate before production use.

8.14.5 - openHalo

MySQL-compatible Postgres 14 branch

OpenHalo is an open-source PostgreSQL kernel that provides MySQL wire-protocol compatibility.

openHalo is based on PostgreSQL 14.18 and provides wire-level compatibility with MySQL 5.7.32-log / 8.0. Pigsty delivers it through pg_mode: mysql and the openhalo package alias.

Pigsty supports OpenHalo deployment on all supported Linux platforms.

Quick Start

Use Pigsty’s standard installation flow with the mysql template.

curl -fsSL https://repo.pigsty.io/get | bash; cd ~/pigsty;
./configure -c mysql    # Use MySQL (openHalo) template
./deploy.yml            # Install (change passwords in pigsty.yml before production use)

Cluster Config 

pg-meta:
  hosts:
    10.10.10.10: { pg_seq: 1, pg_role: primary }
  vars:
    pg_cluster: pg-meta
    pg_users:
      - {name: dbuser_meta ,password: DBUser.Meta   ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: pigsty admin user }
      - {name: dbuser_view ,password: DBUser.Viewer ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer for meta database }
    pg_databases:
      - {name: postgres, extensions: [ aux_mysql ]} # mysql-compatible database
      - {name: meta ,baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty]}
    pg_hba_rules:
      - {user: dbuser_view , db: all ,addr: infra ,auth: pwd ,title: 'allow grafana dashboard access cmdb from infra nodes'}
    pg_crontab: [ '00 01 * * * /pg/bin/pg-backup full' ] # full backup at 1 AM daily

    # OpenHalo specific settings
    pg_mode: mysql
    pg_version: 14
    pg_packages: [ openhalo, pgsql-common ]

OpenHalo provides a dedicated extension, aux_mysql, which includes functions and types needed for MySQL compatibility. Enable it in the postgres database to get full compatibility behavior.

  • aux_mysql 1.5: MySQL Supplementary Extension
  • /usr/halo-14/share/postgresql/extension/aux_mysql.control
  • $libdir/mysm, mysm.so

Usage

For MySQL access, connections still use the postgres database. The MySQL “database” concept maps to PostgreSQL “schema”. So use mysql maps to the mysql schema in the postgres database.

MySQL usernames/passwords are the same PostgreSQL credentials.

Client Access

OpenHalo listens on port 3306 for MySQL wire protocol clients.

Pigsty’s conf/mysql installs a MySQL client by default.

mysql -h 127.0.0.1 -u dbuser_dba

At present, OpenHalo upstream reports Navicat works normally on this port, while IntelliJ DataGrip may fail.

Compatibility Parameters

Pigsty defaults database_compat_mode to mysql. You can further tune compatibility behavior with settings like:

mysql.listener_on = true                        # enable MySQL listener; restart required
mysql.port = 3306                               # second_port for MySQL mode; restart required
mysql.halo_mysql_version = '5.7.32-log'         # restart required
mysql.ci_collation = true                       # restart required
mysql.explicit_defaults_for_timestamp = false   # restart required
mysql.auto_rollback_tx_on_error = false         # restart required

Patch Notes

The OpenHalo kernel packaged by Pigsty is based on HaloTech-Co-Ltd/openHalo with small adjustments:

  • Restore default database name from halo0root to postgres
  • Remove 1.0. prefix in the default version string, keeping 14.18
  • Adjust default config to enable MySQL compatibility and listen on 3306

Pigsty does not provide warranty coverage for OpenHalo kernel behavior. Kernel-specific issues should be addressed with the upstream vendor.

Warning: This kernel is currently in beta1 stage; evaluate risks carefully before production use.

Available Extensions

The OpenHalo kernel has 59 available extensions. After removing bundled PG Contrib extensions, the following extra extensions remain:

Extension Version Description
aux_mysql 1.5 MySQL Supplementary Extension
hstore_plpython2u 1.0 transform between hstore and plpython2u
hstore_plpythonu 1.0 transform between hstore and plpythonu
jsonb_plpython2u 1.0 transform between jsonb and plpython2u
jsonb_plpythonu 1.0 transform between jsonb and plpythonu
ltree_plpython2u 1.0 transform between ltree and plpython2u
ltree_plpythonu 1.0 transform between ltree and plpythonu

8.14.6 - OrioleDB

Next-generation OLTP engine for PostgreSQL

OrioleDB is a PostgreSQL storage engine extension that claims to provide 4x OLTP performance, no xid wraparound and table bloat issues, and “cloud-native” (data stored in S3) capabilities.

Pigsty ships OrioleDB as a patched PostgreSQL kernel plus the OrioleDB extension.

You can run OrioleDB as an RDS using Pigsty. Current packages support PostgreSQL 16, 17, and 18 on supported Linux platforms. pg_mode still uses oriole for the /usr/oriole-$v install path, while the orioledb package alias resolves to versioned kernel packages such as orioledb-16, orioledb-17, and orioledb-18. The current Pigsty package line is OrioleDB 1.8 beta16.

Quick Start

Follow Pigsty’s standard installation process using the oriole configuration template.

curl -fsSL https://repo.pigsty.io/get | bash; cd ~/pigsty;
./configure -c oriole    # Use OrioleDB configuration template
./deploy.yml             # Install Pigsty with OrioleDB

For production deployment, ensure you modify the password parameters in the pigsty.yml configuration before running the install playbook.

Configuration 

pg-meta:
  hosts:
    10.10.10.10: { pg_seq: 1, pg_role: primary }
  vars:
    pg_cluster: pg-meta
    pg_users:
      - {name: dbuser_meta ,password: DBUser.Meta   ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: pigsty admin user }
      - {name: dbuser_view ,password: DBUser.Viewer ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer for meta database }
    pg_databases:
      - {name: meta ,baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty], extensions: [orioledb]}
    pg_hba_rules:
      - {user: dbuser_view , db: all ,addr: infra ,auth: pwd ,title: 'allow grafana dashboard access cmdb from infra nodes'}
    pg_crontab: [ '00 01 * * * /pg/bin/pg-backup full' ] # Full backup at 1 AM daily

    # OrioleDB specific settings
    pg_mode: oriole                                         # oriole compatibility mode
    pg_version: 18                                          # OrioleDB supports PG 16, 17, and 18
    pg_packages: [ orioledb, pgsql-common ]                 # Install OrioleDB kernel
    pg_libs: 'orioledb, pg_stat_statements, auto_explain'   # Load OrioleDB extension

Usage

To use OrioleDB, install the orioledb package alias. Pigsty resolves it to the selected PG16, PG17, or PG18 OrioleDB kernel package according to pg_version.

Initialize TPC-B-like tables with pgbench using 100 warehouses:

pgbench -is 100 meta
pgbench -nv -P1 -c10 -S -T1000 meta
pgbench -nv -P1 -c50 -S -T1000 meta
pgbench -nv -P1 -c10    -T1000 meta
pgbench -nv -P1 -c50    -T1000 meta

Next, you can rebuild these tables using the orioledb storage engine and observe the performance difference:

-- Create OrioleDB tables
CREATE TABLE pgbench_accounts_o (LIKE pgbench_accounts INCLUDING ALL) USING orioledb;
CREATE TABLE pgbench_branches_o (LIKE pgbench_branches INCLUDING ALL) USING orioledb;
CREATE TABLE pgbench_history_o (LIKE pgbench_history INCLUDING ALL) USING orioledb;
CREATE TABLE pgbench_tellers_o (LIKE pgbench_tellers INCLUDING ALL) USING orioledb;

-- Copy data from regular tables to OrioleDB tables
INSERT INTO pgbench_accounts_o SELECT * FROM pgbench_accounts;
INSERT INTO pgbench_branches_o SELECT * FROM pgbench_branches;
INSERT INTO pgbench_history_o SELECT  * FROM pgbench_history;
INSERT INTO pgbench_tellers_o SELECT * FROM pgbench_tellers;

-- Drop original tables and rename OrioleDB tables
DROP TABLE pgbench_accounts, pgbench_branches, pgbench_history, pgbench_tellers;
ALTER TABLE pgbench_accounts_o RENAME TO pgbench_accounts;
ALTER TABLE pgbench_branches_o RENAME TO pgbench_branches;
ALTER TABLE pgbench_history_o RENAME TO pgbench_history;
ALTER TABLE pgbench_tellers_o RENAME TO pgbench_tellers;

Key Features

  • No XID Wraparound: Eliminates transaction ID wraparound maintenance
  • No Table Bloat: Advanced storage management prevents table bloat
  • Cloud Storage: Native support for S3-compatible object storage
  • OLTP Optimized: Designed for transactional workloads
  • Improved Performance: Better space utilization and query performance

Note: Currently in Beta stage - thoroughly evaluate before production use.

Available Extensions

The OrioleDB kernel has 53 available extensions. After removing bundled PG Contrib extensions, the following extra extensions remain:

Extension Version Description
orioledb 1.8 OrioleDB – the next generation transactional engine

8.14.7 - Cloudberry

Use the Cloudberry open-source MPP data warehouse kernel in Pigsty and manage nodes, monitoring, and configuration through gpsql mode.

Cloudberry is an open-source MPP data warehouse kernel derived from the Greenplum ecosystem, suitable for large-scale parallel analytics workloads.

Overview

In Pigsty, Cloudberry uses gpsql mode and shares the same identity model, monitoring logic, and directory conventions as Greenplum / MatrixDB.

  • Kernel package: cloudberry
  • Mode identifier: pg_mode: gpsql
  • Role flag: gp_role: master | segment
  • Current repo version: Cloudberry 2.1.0
  • Current main package version: DEB 2.1.0-2PIGSTY, RPM 2.1.0-3PIGSTY
  • Default binary directory: /usr/cloudberry

The important boundary is this: Pigsty currently focuses on package delivery, node management, monitoring onboarding, access control, and configuration orchestration for Cloudberry. For MPP cluster initialization, scale-out, rebalance, and other upstream-specific operational actions, you should still use the official Cloudberry toolchain.

The current Pigsty repository provides cloudberry, cloudberry-backup, and cloudberry-pxf packages for both DEB and RPM platforms.

Installation

There is no standalone cloudberry one-click template yet. The more common workflow is:

  1. Enroll the target nodes into Pigsty.
  2. Install the cloudberry kernel package.
  3. Describe the coordinator / segment topology with gpsql mode.
  4. Use Pigsty to unify monitoring, accounts, access control, and backup integration.

If you only need to install the kernel package on a node:

./node.yml -t node_repo    -e '{"node_repo_modules":"local,node,pgsql"}'
./node.yml -t node_install -e '{"node_packages":["cloudberry"]}'

If you are onboarding an existing Cloudberry cluster, it is usually better to keep the original initialization workflow and add Pigsty inventory plus monitoring configuration incrementally.

Configuration

Cloudberry uses gpsql mode rather than a dedicated cloudberry mode. Compared with vanilla PostgreSQL, you at least need to care about the extra identity parameters pg_shard and gp_role; if you want to label shard groups explicitly, you can also add pg_group.

Here is a minimal readable topology example:

all:
  children:
    cb-mdw:
      hosts:
        10.10.10.10: { pg_seq: 1, pg_role: primary }
      vars:
        pg_cluster: cb-mdw
        pg_mode: gpsql
        pg_shard: cb
        gp_role: master
        pg_packages: [ cloudberry, pgsql-common ]

    cb-sdw:
      hosts:
        10.10.10.11:
          nodename: cb-sdw-1
          pg_instances:
            6000: { pg_cluster: cb-seg1, pg_seq: 1, pg_role: primary, pg_exporter_port: 9633 }
        10.10.10.12:
          nodename: cb-sdw-2
          pg_instances:
            6000: { pg_cluster: cb-seg2, pg_seq: 1, pg_role: primary, pg_exporter_port: 9633 }
      vars:
        pg_cluster: cb-sdw
        pg_mode: gpsql
        pg_shard: cb
        gp_role: segment
        pg_preflight_skip: true
        pg_packages: [ cloudberry, pgsql-common ]
        pg_exporter_config: pg_exporter_basic.yml
        pg_exporter_params: 'options=-c%20gp_role%3Dutility&sslmode=disable'

Two details are easy to miss:

  • gp_role: master is for the coordinator / master node, and business access usually lands there.
  • gp_role: segment nodes usually need pg_exporter to connect in utility mode for monitoring.

Client Access

For application and BI access, Cloudberry still exposes the PostgreSQL wire protocol, so most PostgreSQL-compatible clients, drivers, and BI tools can connect without special handling.

But keep the following in mind:

  • Applications and analytics queries should connect to the master / coordinator, not directly to segment nodes.
  • Segment nodes are better treated as data/compute shards and monitoring targets.
  • If you want a unified access endpoint, you can still use Pigsty’s HAProxy / PgBouncer / DNS service abstractions.

Extensions and Ecosystem

Cloudberry comes from the PostgreSQL ecosystem, but it is not simply “vanilla PostgreSQL plus a few extensions”. For the extension packages already available in Pigsty, it is better to think in two categories:

  • Pure SQL objects or components with weak ABI coupling are usually easier to adapt.
  • Extensions that depend on PGXS or the kernel C ABI often need separate validation or even recompilation against the Cloudberry version and toolchain.

If your workload depends on postgis, vector extensions, FDWs, auditing, or custom C extensions, validate them on the target Cloudberry version first rather than copying a vanilla PostgreSQL extension list unchanged.

Notes

  • Cloudberry currently has no dedicated Pigsty template, so you should model it manually with gpsql mode.
  • The current delivery focus is packages, configuration, and monitoring; it does not replace the official Cloudberry MPP initialization and scale-out toolchain.
  • Because this is an MPP distributed kernel, vanilla PostgreSQL operational assumptions do not automatically transfer to every Patroni / PgBouncer / PgBackRest node role.
  • If you need horizontal PostgreSQL scaling rather than a full MPP warehouse, Citus is usually the better first choice.

Related Docs

8.14.8 - AgensGraph

Use the AgensGraph (PG17) graph database kernel in Pigsty to get property graph and Cypher/SQL hybrid query capabilities within the PostgreSQL ecosystem.

AgensGraph is a property graph database kernel built on PostgreSQL, supporting openCypher queries and mixed Cypher/SQL workflows.

Overview

Pigsty integrates AgensGraph through pg_mode: agens while preserving most of the standard PostgreSQL operational model.

  • Kernel package: agensgraph
  • Mode identifier: pg_mode: agens
  • Current template version: AgensGraph 2.17.0
  • Current version string: PostgreSQL 17.10 (AgensGraph 2.17.0)
  • Built-in template: agens
  • Typical use cases: graph relationship analysis, path queries, knowledge graphs, and risk/association analysis layered onto relational data

From the client side, AgensGraph still speaks the PostgreSQL wire protocol, so normal PostgreSQL clients, drivers, and connection pools can connect directly. The real difference from vanilla PostgreSQL is not how you connect, but that the database now contains graph objects, Cypher syntax, and the agtype data type.

Installation

Use the built-in Pigsty template:

./configure -c agens
./deploy.yml

The agens template automatically enables pg_mode: agens and installs the agensgraph kernel package. After deployment, verify the kernel version:

psql -d meta -c "SELECT version();"

Configuration

Key configuration for AgensGraph in Pigsty:

all:
  vars:
    node_repo_modules: node,infra,pgsql
    pg_version: 17

  children:
    pg-meta:
      vars:
        pg_mode: agens
        pg_packages: [ agensgraph, pgsql-common ]

AgensGraph does not require a special preload stack like pgEdge or Babelfish, so most standard Pigsty patterns for HA, backup, monitoring, access control, and IaC remain unchanged. If your workload is dominated by graph traversal and complex path queries, focus on work_mem, shared_buffers, and planner cost settings instead of assuming default OLTP habits will fit.

Usage

After connecting to the database, the usual first step is to create a graph and set graph_path:

CREATE GRAPH g;
SET graph_path = g;

Create labels, vertices, and edges:

CREATE VLABEL person;
CREATE ELABEL knows;

CREATE (:person {name: 'Jack'});
CREATE (:person {name: 'Emily'})-[:knows]->(:person {name: 'Tom'});

Run graph queries and updates:

MATCH (:person {name: 'Emily'})-[:knows]->(v:person)
RETURN v.name;

MATCH (v:person {name: 'Jack'})
SET v.age = '24';

To call Cypher from within SQL, use the cypher() function:

SELECT *
FROM cypher('g', $$ MATCH (v:person) RETURN v.name $$) AS (name agtype);

In real projects, the more common pattern is to mix “relational tables + graph labels + Cypher queries”: transactions, privileges, and backup workflows still follow PostgreSQL, while graph analysis logic lives in AgensGraph graph objects and the cypher() interface.

Notes

  • AgensGraph is currently fixed to the PG17-compatible line, so do not assume PG18 extension availability will carry over.
  • The default agens template is single-node for quick validation; production deployments should extend to an HA topology.
  • Not all third-party PostgreSQL extensions are guaranteed to work on the AgensGraph kernel; verify compatibility first.
  • Graph objects and relational objects can coexist in the same database, but in production it is usually better to define clear database or naming conventions so they do not become tangled together.
  • Tune memory and cost parameters based on your graph model scale; do not blindly use defaults.
  • For compatibility or semantic issues with the AgensGraph kernel, consult the official manual and upstream issues first.

Related Docs

Available Extensions

The AgensGraph kernel has 60 available extensions. After removing bundled PG Contrib extensions, the following extra extensions remain:

Extension Version Description
meta 1.0 Utility functions for agensgraph

8.14.9 - pgEdge

Use the pgEdge (PG15-18) kernel in Pigsty to build distributed PostgreSQL for edge scenarios on top of Spock multi-master logical replication.

pgEdge is a distributed PostgreSQL distribution for edge scenarios, built on Spock multi-master logical replication.

Overview

Pigsty integrates pgEdge through pg_mode: pgedge and delivers it through the standard PostgreSQL cluster workflow:

  • pgedge: a PG15, PG16, PG17, and PG18 compatible kernel; the template defaults to PG18
  • spock: Active-active multi-master logical replication
  • snowflake: Distributed unique sequences
  • lolor: Large object logical replication compatibility layer

The current Pigsty repository ships versioned pgEdge kernel packages for pgedge-15, pgedge-16, pgedge-17, and pgedge-18; the template defaults to pg_version: 18. The spock, snowflake, and lolor control files, SQL files, and shared libraries are bundled in the pgedge-$v kernel package, so they are no longer listed as separate pg_extensions packages to install. From the client side, pgEdge is still PostgreSQL wire compatible, so psql, JDBC/ODBC, DBeaver, and similar tools work as usual.

The delivery model in Pigsty is: validate the kernel on a single node first, then expand to a multi-node replication topology. The template handles the kernel, extensions, monitoring, backup, and access control out of the box, but the actual multi-master topology still needs to be designed around your workload consistency and conflict strategy.

Installation

Use the built-in Pigsty template:

./configure -c pgedge
./deploy.yml

The template pre-installs spock, snowflake, and lolor in the meta database. After deployment, verify the kernel and extensions:

psql -d meta -c "SELECT version();"
psql -d meta -c "SELECT extname, extversion FROM pg_extension WHERE extname IN ('spock','snowflake','lolor') ORDER BY 1;"

For the full template and parameters, see: pgedge config template.

Configuration

Key parameters in the pgedge template (matching conf/pgedge.yml):

pg_mode: pgedge
pg_version: 18
pg_packages: [ pgedge, pgsql-common ]
pg_libs: 'spock, lolor, pg_stat_statements, auto_explain'
pg_databases:
  - { name: meta ,baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty] ,extensions: [spock, snowflake, lolor] }

If you plan to grow into a multi-node multi-master topology, it is better to configure logical replication capacity and snowflake.node explicitly:

pg_parameters:
  wal_level: logical
  max_replication_slots: 16
  max_wal_senders: 16
  'snowflake.node': 1

snowflake.node must be unique on every writable node, otherwise distributed IDs will collide.

Usage

The common workflow in Pigsty is still: validate the kernel on a single node first, then expand into a multi-node Spock replication topology.

If you need these capabilities in a business database as well, create the extensions first:

CREATE EXTENSION IF NOT EXISTS spock;
CREATE EXTENSION IF NOT EXISTS snowflake;
CREATE EXTENSION IF NOT EXISTS lolor;

Then use the Spock SQL API or the pgEdge CLI to create nodes, replication sets, and subscriptions. If your schema already uses serial or identity, plan the snowflake sequence migration before enabling multi-master writes, otherwise cross-node primary key collisions are likely.

Notes

  • Replication in pgEdge is organized per database, not as an instance-wide “turn everything into multi-master” switch.
  • Replicated tables should have a PRIMARY KEY or an appropriate REPLICA IDENTITY.
  • UNLOGGED and TEMPORARY tables do not participate in Spock logical replication.
  • Spock configuration and operations typically require superuser privileges, so production deployments should define privilege boundaries clearly.
  • If your workload depends on large object replication, use lolor explicitly rather than assuming native large objects will replicate correctly.
  • Cross-region multi-master is not a checkbox feature. Network latency, conflict handling, and the write model all need to be evaluated first.

Related Docs

Available Extensions

The pgEdge kernel has 63 available extensions. After removing bundled PG Contrib extensions, the following extra extensions remain:

Extension Version Description
lolor 1.2.2 Large Objects support for logical replication
snowflake 2.5.0 Snowflake style IDs for PostgreSQL
spock 5.0.10 PostgreSQL Logical Replication

8.14.10 - DocumentDB

DocumentDB and FerretDB provide MongoDB wire-protocol compatibility

DocumentDB is an open-source PostgreSQL document database extension maintained by Microsoft. FerretDB is a stateless protocol translation proxy built on top of it. Together, they expose a MongoDB wire-compatible endpoint from a standard PostgreSQL kernel: applications using MongoDB drivers can connect directly, while requests are translated into PostgreSQL operations.

Unlike other kernel variants, this is not a standalone PostgreSQL fork. The data layer runs native PostgreSQL 16–18 and is managed by the standard PGSQL module. Persistence, transactions, high availability, backup, monitoring, and access control remain PostgreSQL responsibilities; FerretDB is deployed as a Pigsty Docker APP and handles only protocol translation.

Pigsty is a FerretDB community partner, provides binary packages for FerretDB and the DocumentDB extensions, and delivers the complete stack out of the box through the mongo configuration template.

Quick Start

Use Pigsty’s standard installation flow with the mongo configuration template:

curl -fsSL https://repo.pigsty.io/get | bash; cd ~/pigsty;
./configure -c mongo    # Use the Mongo (DocumentDB + FerretDB) configuration template
./deploy.yml            # Install; change passwords in pigsty.yml first for production
./docker.yml -l pg-meta # Install Docker on the pg-meta node
./app.yml -l pg-meta    # Deploy the FerretDB Docker APP

FerretDB listens on the local loopback address at port 27017 by default. Connect with mongosh or any MongoDB-compatible client:

mongosh 'mongodb://mongod:[email protected]:27017/'

Configuration

Source: pigsty/conf/mongo.yml. See the Mongo configuration template documentation for the complete template.

The key PostgreSQL settings are the documentdb extension and its preload libraries, plus the backend superuser used by FerretDB:

pg-meta:
  hosts:
    10.10.10.10: { pg_seq: 1, pg_role: primary }
  vars:
    pg_cluster: pg-meta
    pg_users:
      - { name: mongod ,password: DBUser.Mongo ,superuser: true ,comment: FerretDB backend user }
    pg_databases:
      - { name: postgres, extensions: [ documentdb, postgis, vector, pg_cron, rum ]}
    pg_extensions: [ documentdb, postgis, pgvector, pg_cron, rum ]
    pg_libs: 'pg_documentdb, pg_documentdb_core, pg_documentdb_extended_rum, pg_cron, pg_stat_statements, auto_explain'

FerretDB is deployed as a Docker APP. Its settings are ordinary overrides under apps.ferretdb.conf, and the container connects to the local primary service on port 5436 through host.docker.internal:

docker_enabled: true
app: ferretdb
apps:
  ferretdb:
    conf:
      FERRETDB_IMAGE: ghcr.io/ferretdb/ferretdb:2.7.0
      FERRETDB_POSTGRESQL_URL: 'postgres://mongod:[email protected]:5436/postgres?pool_min_conns=1&pool_max_conns=20'
      FERRETDB_BIND_ADDR: 127.0.0.1
      FERRETDB_PORT: 27017
      FERRETDB_AUTH: true
      FERRETDB_TELEMETRY: disabled

High Availability

Because FerretDB is fully stateless, its HA topology follows the standard PostgreSQL cluster pattern. The template retains a commented three-node pg-mongo example. Each node runs a FerretDB container bound to local port 27018, and HAProxy aggregates them behind the floating endpoint 10.10.10.4:27017 (mongo.pigsty).

Patroni and etcd continue to manage PostgreSQL failover. The Mongo endpoint automatically recovers after the primary switches.

Notes

  • FerretDB enables authentication by default (FERRETDB_AUTH: true) but does not yet implement MongoDB authorization roles. PostgreSQL users and HBA rules remain the actual security boundary.
  • Client-side MongoDB TLS is disabled by default, and the Mongo endpoint is not exposed to the network. Change FERRETDB_BIND_ADDR only when remote access is required.
  • The backend cluster uses standard PostgreSQL parameters, playbooks, and dashboards; there is no independent FERRET module or mongo_* parameter group.
  • Repeat an authenticated CRUD smoke test after upgrading FerretDB or DocumentDB.

8.14.11 - Citus

Deploy native high-availability Citus horizontally sharded clusters with Pigsty, seamlessly scaling PostgreSQL across multiple shards and accelerating OLTP/OLAP queries.

Pigsty natively supports Citus. This is a distributed horizontal scaling extension based on the native PostgreSQL kernel.

Citus

Installation

Citus is a PostgreSQL extension plugin that can be installed and enabled on a native PostgreSQL cluster following the standard plugin installation process.

./pgsql.yml -t pg_extension -e '{"pg_extensions":["citus"]}'

Configuration

To define a citus cluster, you need to specify the following parameters:

  • pg_mode must be set to citus instead of the default pgsql
  • You must define the shard name pg_shard and shard number pg_group on each shard cluster
  • You must define pg_primary_db to specify the database managed by Patroni
  • If you want to use postgres from pg_dbsu instead of the default pg_admin_username to execute admin commands, then pg_dbsu_password must be set to a non-empty plaintext password

Additionally, you need extra hba rules to allow SSL access from localhost and other data nodes.

You can define each Citus cluster as a separate group, like standard PostgreSQL clusters. The current complete template is conf/ha/citus.yml:

all:
  children:
    pg-citus0: # citus shard 0
      hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
      vars: { pg_cluster: pg-citus0 , pg_group: 0 }
    pg-citus1: # citus shard 1
      hosts: { 10.10.10.11: { pg_seq: 1, pg_role: primary } }
      vars: { pg_cluster: pg-citus1 , pg_group: 1 }
    pg-citus2: # citus shard 2
      hosts: { 10.10.10.12: { pg_seq: 1, pg_role: primary } }
      vars: { pg_cluster: pg-citus2 , pg_group: 2 }
    pg-citus3: # citus shard 3
      hosts:
        10.10.10.13: { pg_seq: 1, pg_role: primary }
        10.10.10.14: { pg_seq: 2, pg_role: replica }
      vars: { pg_cluster: pg-citus3 , pg_group: 3 }
  vars:                               # Global parameters for all Citus clusters
    pg_mode: citus                    # pgsql cluster mode must be set to: citus
    pg_shard: pg-citus                # citus horizontal shard name: pg-citus
    pg_primary_db: meta               # citus database name: meta
    pg_dbsu_password: DBUser.Postgres # If using dbsu, you need to configure a password for it
    pg_users: [ { name: dbuser_meta ,password: DBUser.Meta ,pgbouncer: true ,roles: [ dbrole_admin ] } ]
    pg_databases: [ { name: meta ,extensions: [ { name: citus }, { name: postgis }, { name: timescaledb } ] } ]
    pg_hba_rules:
      - { user: 'all' ,db: all  ,addr: 127.0.0.1/32 ,auth: ssl ,title: 'all user ssl access from localhost' }
      - { user: 'all' ,db: all  ,addr: intra        ,auth: ssl ,title: 'all user ssl access from intranet'  }

You can also specify identity parameters for all Citus cluster members within a single group, as shown in conf/ha/citus.yml:

#==========================================================#
# pg-citus: 10 node citus cluster (5 x primary-replica pair)
#==========================================================#
pg-citus: # citus group
  hosts:
    10.10.10.50: { pg_group: 0, pg_cluster: pg-citus0 ,pg_vip_address: 10.10.10.60/24 ,pg_seq: 0, pg_role: primary }
    10.10.10.51: { pg_group: 0, pg_cluster: pg-citus0 ,pg_vip_address: 10.10.10.60/24 ,pg_seq: 1, pg_role: replica }
    10.10.10.52: { pg_group: 1, pg_cluster: pg-citus1 ,pg_vip_address: 10.10.10.61/24 ,pg_seq: 0, pg_role: primary }
    10.10.10.53: { pg_group: 1, pg_cluster: pg-citus1 ,pg_vip_address: 10.10.10.61/24 ,pg_seq: 1, pg_role: replica }
    10.10.10.54: { pg_group: 2, pg_cluster: pg-citus2 ,pg_vip_address: 10.10.10.62/24 ,pg_seq: 0, pg_role: primary }
    10.10.10.55: { pg_group: 2, pg_cluster: pg-citus2 ,pg_vip_address: 10.10.10.62/24 ,pg_seq: 1, pg_role: replica }
    10.10.10.56: { pg_group: 3, pg_cluster: pg-citus3 ,pg_vip_address: 10.10.10.63/24 ,pg_seq: 0, pg_role: primary }
    10.10.10.57: { pg_group: 3, pg_cluster: pg-citus3 ,pg_vip_address: 10.10.10.63/24 ,pg_seq: 1, pg_role: replica }
    10.10.10.58: { pg_group: 4, pg_cluster: pg-citus4 ,pg_vip_address: 10.10.10.64/24 ,pg_seq: 0, pg_role: primary }
    10.10.10.59: { pg_group: 4, pg_cluster: pg-citus4 ,pg_vip_address: 10.10.10.64/24 ,pg_seq: 1, pg_role: replica }
  vars:
    pg_mode: citus                    # pgsql cluster mode: citus
    pg_shard: pg-citus                # citus shard name: pg-citus
    pg_primary_db: test               # primary database used by citus
    pg_dbsu_password: DBUser.Postgres # all dbsu password access for citus cluster
    pg_vip_enabled: true
    pg_vip_interface: auto
    pg_extensions: [ 'citus postgis timescaledb pgvector' ]
    pg_libs: 'citus, timescaledb, pg_stat_statements, auto_explain' # citus will be added by patroni automatically
    pg_users: [ { name: test ,password: test ,pgbouncer: true ,roles: [ dbrole_admin ] } ]
    pg_databases: [ { name: test ,owner: test ,extensions: [ { name: citus }, { name: postgis } ] } ]
    pg_hba_rules:
      - { user: 'all' ,db: all  ,addr: 10.10.10.0/24 ,auth: trust ,title: 'trust citus cluster members'        }
      - { user: 'all' ,db: all  ,addr: 127.0.0.1/32  ,auth: ssl   ,title: 'all user ssl access from localhost' }
      - { user: 'all' ,db: all  ,addr: intra         ,auth: ssl   ,title: 'all user ssl access from intranet'  }

Usage

You can access any node just like accessing a regular cluster:

pgbench -i postgres://test:test@pg-citus0/test
pgbench -nv -P1 -T1000 -c 2 postgres://test:test@pg-citus0/test

By default, changes you make to one Shard only occur on that cluster and are not synchronized to other Shards.

If you want to distribute writes across all Shards, you can use the API functions provided by Citus to mark tables as:

  • Distributed tables (automatic partitioning, requires specifying partition key)
  • Reference tables (full replication: does not require specifying partition key)

Starting from Citus 11.2, any Citus database node can play the role of coordinator, meaning any primary node can write:

psql -h pg-citus0 -d test -c "SELECT create_distributed_table('pgbench_accounts', 'aid'); SELECT truncate_local_data_after_distributing_table('public.pgbench_accounts');"
psql -h pg-citus0 -d test -c "SELECT create_reference_table('pgbench_branches')         ; SELECT truncate_local_data_after_distributing_table('public.pgbench_branches');"
psql -h pg-citus0 -d test -c "SELECT create_reference_table('pgbench_history')          ; SELECT truncate_local_data_after_distributing_table('public.pgbench_history');"
psql -h pg-citus0 -d test -c "SELECT create_reference_table('pgbench_tellers')          ; SELECT truncate_local_data_after_distributing_table('public.pgbench_tellers');"

After distributing the tables, you can also access them on other nodes:

psql -h pg-citus1 -d test -c '\dt+'

For example, a full table scan will show that the execution plan has become a distributed plan:

vagrant@meta-1:~$ psql -h pg-citus3 -d test -c 'explain select * from pgbench_accounts'
                                               QUERY PLAN
---------------------------------------------------------------------------------------------------------
 Custom Scan (Citus Adaptive)  (cost=0.00..0.00 rows=100000 width=352)
   Task Count: 32
   Tasks Shown: One of 32
   ->  Task
         Node: host=10.10.10.52 port=5432 dbname=test
         ->  Seq Scan on pgbench_accounts_102008 pgbench_accounts  (cost=0.00..81.66 rows=3066 width=97)
(6 rows)

You can initiate writes from several different primary nodes:

pgbench -nv -P1 -T1000 -c 2 postgres://test:test@pg-citus1/test
pgbench -nv -P1 -T1000 -c 2 postgres://test:test@pg-citus2/test
pgbench -nv -P1 -T1000 -c 2 postgres://test:test@pg-citus3/test
pgbench -nv -P1 -T1000 -c 2 postgres://test:test@pg-citus4/test

When a node fails, the native high availability support provided by Patroni will promote the standby node and automatically take over.

test=# select * from  pg_dist_node;
 nodeid | groupid |  nodename   | nodeport | noderack | hasmetadata | isactive | noderole | nodecluster | metadatasynced | shouldhaveshards
--------+---------+-------------+----------+----------+-------------+----------+----------+-------------+----------------+------------------
      1 |       0 | 10.10.10.51 |     5432 | default  | t           | t        | primary  | default     | t              | f
      2 |       2 | 10.10.10.54 |     5432 | default  | t           | t        | primary  | default     | t              | t
      5 |       1 | 10.10.10.52 |     5432 | default  | t           | t        | primary  | default     | t              | t
      3 |       4 | 10.10.10.58 |     5432 | default  | t           | t        | primary  | default     | t              | t
      4 |       3 | 10.10.10.56 |     5432 | default  | t           | t        | primary  | default     | t              | t

8.14.12 - IvorySQL

Use HighGo’s open-source IvorySQL kernel to achieve Oracle syntax/PLSQL compatibility based on PostgreSQL clusters.

IvorySQL is an open-source PostgreSQL kernel fork that aims to provide “Oracle compatibility” based on PG.

Overview

The Pigsty PGSQL repository directly provides IvorySQL 5.4 packages compatible with PostgreSQL 18.4 across the currently supported EL, Debian, Ubuntu, and dual-architecture platforms. Online installation uses Pigsty’s pgsql repository; the Professional Edition also provides offline delivery for the corresponding platforms.

IvorySQL

Pigsty’s ivorysql package alias points to IvorySQL 5, compatible with PostgreSQL 18. Real package names are mapped by platform variables under roles/node_id/vars/; for example, EL uses ivorysql5, while Debian/Ubuntu uses ivorysql-5.

The last IvorySQL version supporting EL7 was 3.3, corresponding to PostgreSQL 16.3; the last version based on PostgreSQL 17 is IvorySQL 4.4

Installation

Install with Pigsty’s built-in ivory configuration template:

./configure -c ivory
./deploy.yml

Configuration

The following parameters need to be configured for IvorySQL database clusters:

#----------------------------------#
# Ivory SQL Configuration
#----------------------------------#
node_repo_modules: node,infra,pgsql       # use Pigsty node/infra/pgsql repos
pg_mode: ivory                    # IvorySQL Oracle Compatible Mode
pg_packages: [ ivorysql, pgsql-common ]
pg_libs: 'liboracle_parser, pg_stat_statements, auto_explain'
pg_extensions: [ ]                # do not install any vanilla postgresql extensions

When using Oracle compatibility mode, you need to dynamically load the liboracle_parser extension plugin.

Client Access

IvorySQL 5 is equivalent to PostgreSQL 18, and any client tool compatible with the PostgreSQL wire protocol can access IvorySQL clusters.

Available Extensions

The IvorySQL kernel has 95 available extensions. After removing bundled PG Contrib extensions, the following extra extensions remain:

Extension Version Description
address_standardizer 3.5.4 Used to parse an address into constituent elements. Generally used to support geocoding address normalization step.
address_standardizer_data_us 3.5.4 Address Standardizer US dataset example
age 1.7.0 AGE database extension
ddlx 0.31 DDL eXtractor functions
gb18030_2022 1.0 support gb18030 2022 with extension
http 1.7 HTTP client for PostgreSQL, allows web page retrieval inside the database.
ivorysql_ora 1.0 Oracle Compatible extenison on Postgres Database
ora_btree_gin 1.0 support for indexing oracle datatypes in GIN
ora_btree_gist 1.0 support for oracle indexing common datatypes in GiST
pg_bigm 1.2 text similarity measurement and index searching based on bigrams
pg_cron 1.6 Job scheduler for PostgreSQL
pg_curl 2.4 PostgreSQL cURL allows most curl actions, including data transfer with URL syntax via HTTP, HTTPS, FTP, FTPS, GOPHER, TFTP, SCP, SFTP, SMB, TELNET, DICT, LDAP, LDAPS, FILE, IMAP, SMTP, POP3, RTSP and RTMP
pg_get_functiondef 1.0 Get function’s definition
pg_hint_plan 1.8.0 optimizer hints for PostgreSQL
pg_jieba 1.1.1 a parser for full-text search of Chinese
pg_partman 5.3.1 Extension to manage partitioned tables by time or ID
pg_show_plans 2.1 show query plans of all currently running SQL statements
pg_stat_monitor 2.3 The pg_stat_monitor is a PostgreSQL Query Performance Monitoring tool, based on PostgreSQL contrib module pg_stat_statements. pg_stat_monitor provides aggregated statistics, client information, plan details including plan, and histogram information.
pg_textsearch 0.1.0 Full-text search with BM25 ranking
pgagent 4.2 A PostgreSQL job scheduler
pgaudit 18.0 provides auditing functionality
pgroonga 4.0.4 Super fast and all languages supported full text search index based on Groonga
pgroonga_database 4.0.4 PGroonga database management module
pgrouting 3.8.0 pgRouting Extension
plisql 1.0 PL/iSQL procedural language
plpgsql_check 2.8 extended check for plpgsql functions
postgis 3.5.4 PostGIS geometry and geography spatial types and functions
postgis_raster 3.5.4 PostGIS raster types and functions
postgis_sfcgal 3.5.4 PostGIS SFCGAL functions
postgis_tiger_geocoder 3.5.4 PostGIS tiger geocoder and reverse geocoder
postgis_topology 3.5.4 PostGIS topology spatial types and functions
redis_fdw 1.0 Foreign data wrapper for querying a Redis server
system_stats 3.0 EnterpriseDB system statistics for PostgreSQL
vector 0.8.1 vector data type and ivfflat and hnsw access methods
zhparser 2.3 a parser for full-text search of Chinese

Please note that Pigsty does not assume any warranty responsibility for using the IvorySQL kernel. Any issues or requirements encountered when using this kernel should be addressed with the original vendor.

8.14.13 - PolarDB PG

Using Alibaba Cloud’s open-source PolarDB for PostgreSQL kernel to provide domestic innovation qualification support, with Oracle RAC-like user experience.

Overview

Pigsty allows you to create PostgreSQL clusters with “domestic innovation qualification” credentials using PolarDB!

PolarDB for PostgreSQL now uses PostgreSQL 17 as its base. The polar template, default path, and extension notes in Pigsty have all been updated to PG17. Any client tool compatible with the PostgreSQL wire protocol can access PolarDB clusters.

Pigsty’s PGSQL repository provides PolarDB PG open-source installation packages, but they are not downloaded to the local software repository during Pigsty installation.

PolarDB for PostgreSQL

Installation

Use the built-in Pigsty template:

./configure -c polar
./deploy.yml

Change Summary

Starting with Pigsty v4.4, the PolarDB PG kernel uses packages built and maintained by Pigsty. The main changes are:

Item Old docs / old default Current
Kernel baseline PostgreSQL 15 PostgreSQL 17
Default PolarDB path /u01/polardb_pg /usr/polar-17
Supported architectures x86_64 x86_64, aarch64
Available extensions Old docs said 61 pg_available_extensions returns 93; 34 remain after filtering contrib
Replication user requirement replicator must be SUPERUSER unchanged

Configuration

The following parameters need special configuration for PolarDB database clusters:

#----------------------------------#
# PGSQL & PolarDB
#----------------------------------#
pg_version: 17
pg_mode: polar
pg_packages: [ polardb, pgsql-common ]
pg_exporter_exclude_database: 'template0,template1,postgres,polardb_admin'
pg_default_roles:
  - { name: dbrole_readonly  ,login: false ,comment: role for global read-only access     }
  - { name: dbrole_offline   ,login: false ,comment: role for restricted read-only access }
  - { name: dbrole_readwrite ,login: false ,roles: [dbrole_readonly] ,comment: role for global read-write access }
  - { name: dbrole_admin     ,login: false ,roles: [pg_monitor, dbrole_readwrite] ,comment: role for object creation }
  - { name: postgres     ,superuser: true  ,comment: system superuser }
  - { name: replicator   ,superuser: true  ,replication: true ,roles: [pg_monitor, dbrole_readonly] ,comment: system replicator } # <- superuser is required for replication
  - { name: dbuser_dba   ,superuser: true  ,roles: [dbrole_admin]  ,pgbouncer: true ,pool_mode: session, pool_connlimit: 16 ,comment: pgsql admin user }
  - { name: dbuser_monitor ,roles: [pg_monitor] ,pgbouncer: true ,parameters: {log_min_duration_statement: 1000 } ,pool_mode: session ,pool_connlimit: 8 ,comment: pgsql monitor user }

The default installation directory for the polar kernel has moved to /usr/polar-17. One important difference is that PolarDB PG still requires the replicator replication user to be SUPERUSER, unlike vanilla PostgreSQL.

Available Extensions

The PolarDB PG kernel has 93 available extensions. After removing bundled PG Contrib extensions, the following extra extensions remain:

Extension Version Description
hll 2.18 type for storing hyperloglog data
ip4r 2.4
log_fdw 1.4 foreign-data wrapper for Postgres log file access
pase 0.0.1 ant ai similarity search
pg_bigm 1.2 text similarity measurement and index searching based on bigrams
pg_cron 1.5 Job scheduler for PostgreSQL
pg_cron_preload 1.0 polardb pg extend catalog
pg_hint_plan 1.7.0 optimizer hints for PostgreSQL
pg_jieba 1.1.0 a parser for full-text search of Chinese
pg_partman 5.2.4 Extension to manage partitioned tables by time or ID
pg_profile 4.10 PostgreSQL load profile repository and report builder
pg_repack 1.5.1-1 Reorganize tables in PostgreSQL databases with minimal locks
pg_similarity 1.0 support similarity queries
pg_squeeze 1.9 A tool to remove unused space from a relation.
pg_stat_kcache 2.3.0 Kernel statistics gathering
pgaudit 17.1 provides auditing functionality
pgtap 1.3.3 Unit testing for PostgreSQL
pldbgapi 1.1 server-side support for debugging PL/pgSQL functions
polar_advisor 1.1 polar_advisor
polar_feature_utils 1.0 PolarDB feature utilization
polar_io_stat 1.0 polar io stat in multi dimension
polar_monitor 1.3 monitor functions for PolarDB
polar_monitor_preload 1.0 examine the polardb information
polar_parameter_manager 1.2 Extension to select parameters for manger.
polar_proxy_utils 1.0 Extension to provide operations about proxy.
polar_resource_manager 1.0 a background process that forcibly frees user session process memory
polar_smgrperf 1.0 smgr perf test extension
polar_tde_utils 1.0 Internal extension for TDE
polar_vfs 1.0 polar virtual file system for different storage
polar_worker 1.1 polar_worker
prefix 1.2.0 Prefix Range module for PostgreSQL
roaringbitmap 0.5 support for Roaring Bitmaps
sequential_uuids 1.0.3 generator of sequential UUIDs
varbitx 1.1 varbit functions pack

8.14.14 - PolarDB Oracle

Using Alibaba Cloud’s commercial PolarDB for Oracle kernel (closed source, PG14, only available in special enterprise edition customization)

Pigsty allows you to create PolarDB for Oracle clusters with “domestic innovation qualification” credentials using PolarDB!

According to the Security and Reliability Evaluation Results Announcement (No. 1, 2023), Appendix 3, Centralized Database. PolarDB v2.0 is an autonomous, controllable, secure, and reliable domestic innovation database.

PolarDB for Oracle is an Oracle-compatible version developed based on PolarDB for PostgreSQL. Both share the same kernel, distinguished by the --compatibility-mode parameter.

We collaborate with the Alibaba Cloud kernel team to provide a complete database solution based on PolarDB v2.0 kernel and Pigsty. Please contact sales for inquiries, or purchase on Alibaba Cloud Marketplace.

The PolarDB for Oracle kernel is currently only available on EL7 (CentOS 7) systems.

PolarDB for Oracle

Extensions

Currently, the PolarDB 2.0 (Oracle compatible) kernel comes with the following 188 extension plugins:

name default_version comment
cube 1.5 data type for multidimensional cubes
ip4r 2.4 NULL
adminpack 2.1 administrative functions for PostgreSQL
dict_xsyn 1.0 text search dictionary template for extended synonym processing
amcheck 1.4 functions for verifying relation integrity
autoinc 1.0 functions for autoincrementing fields
hstore 1.8 data type for storing sets of (key, value) pairs
bloom 1.0 bloom access method - signature file based index
earthdistance 1.1 calculate great-circle distances on the surface of the Earth
hstore_plperl 1.0 transform between hstore and plperl
bool_plperl 1.0 transform between bool and plperl
file_fdw 1.0 foreign-data wrapper for flat file access
bool_plperlu 1.0 transform between bool and plperlu
fuzzystrmatch 1.1 determine similarities and distance between strings
hstore_plperlu 1.0 transform between hstore and plperlu
btree_gin 1.3 support for indexing common datatypes in GIN
hstore_plpython2u 1.0 transform between hstore and plpython2u
btree_gist 1.6 support for indexing common datatypes in GiST
hll 2.17 type for storing hyperloglog data
hstore_plpython3u 1.0 transform between hstore and plpython3u
citext 1.6 data type for case-insensitive character strings
hstore_plpythonu 1.0 transform between hstore and plpythonu
hypopg 1.3.1 Hypothetical indexes for PostgreSQL
insert_username 1.0 functions for tracking who changed a table
dblink 1.2 connect to other PostgreSQL databases from within a database
decoderbufs 0.1.0 Logical decoding plugin that delivers WAL stream changes using a Protocol Buffer format
intagg 1.1 integer aggregator and enumerator (obsolete)
dict_int 1.0 text search dictionary template for integers
intarray 1.5 functions, operators, and index support for 1-D arrays of integers
isn 1.2 data types for international product numbering standards
jsonb_plperl 1.0 transform between jsonb and plperl
jsonb_plperlu 1.0 transform between jsonb and plperlu
jsonb_plpython2u 1.0 transform between jsonb and plpython2u
jsonb_plpython3u 1.0 transform between jsonb and plpython3u
jsonb_plpythonu 1.0 transform between jsonb and plpythonu
lo 1.1 Large Object maintenance
log_fdw 1.0 foreign-data wrapper for csvlog
ltree 1.2 data type for hierarchical tree-like structures
ltree_plpython2u 1.0 transform between ltree and plpython2u
ltree_plpython3u 1.0 transform between ltree and plpython3u
ltree_plpythonu 1.0 transform between ltree and plpythonu
moddatetime 1.0 functions for tracking last modification time
old_snapshot 1.0 utilities in support of old_snapshot_threshold
oracle_fdw 1.2 foreign data wrapper for Oracle access
oss_fdw 1.1 foreign-data wrapper for OSS access
pageinspect 2.1 inspect the contents of database pages at a low level
pase 0.0.1 ant ai similarity search
pg_bigm 1.2 text similarity measurement and index searching based on bigrams
pg_freespacemap 1.2 examine the free space map (FSM)
pg_hint_plan 1.4 controls execution plan with hinting phrases in comment of special form
pg_buffercache 1.5 examine the shared buffer cache
pg_prewarm 1.2 prewarm relation data
pg_repack 1.4.8-1 Reorganize tables in PostgreSQL databases with minimal locks
pg_sphere 1.0 spherical objects with useful functions, operators and index support
pg_cron 1.5 Job scheduler for PostgreSQL
pg_jieba 1.1.0 a parser for full-text search of Chinese
pg_stat_kcache 2.2.1 Kernel statistics gathering
pg_stat_statements 1.9 track planning and execution statistics of all SQL statements executed
pg_surgery 1.0 extension to perform surgery on a damaged relation
pg_trgm 1.6 text similarity measurement and index searching based on trigrams
pg_visibility 1.2 examine the visibility map (VM) and page-level visibility info
pg_wait_sampling 1.1 sampling based statistics of wait events
pgaudit 1.6.2 provides auditing functionality
pgcrypto 1.3 cryptographic functions
pgrowlocks 1.2 show row-level locking information
pgstattuple 1.5 show tuple-level statistics
pgtap 1.2.0 Unit testing for PostgreSQL
pldbgapi 1.1 server-side support for debugging PL/pgSQL functions
plperl 1.0 PL/Perl procedural language
plperlu 1.0 PL/PerlU untrusted procedural language
plpgsql 1.0 PL/pgSQL procedural language
plpython2u 1.0 PL/Python2U untrusted procedural language
plpythonu 1.0 PL/PythonU untrusted procedural language
plsql 1.0 Oracle compatible PL/SQL procedural language
pltcl 1.0 PL/Tcl procedural language
pltclu 1.0 PL/TclU untrusted procedural language
polar_bfile 1.0 The BFILE data type enables access to binary file LOBs that are stored in file systems outside Database
polar_bpe 1.0 polar_bpe
polar_builtin_cast 1.1 Internal extension for builtin casts
polar_builtin_funcs 2.0 implement polar builtin functions
polar_builtin_type 1.5 polar_builtin_type for PolarDB
polar_builtin_view 1.5 polar_builtin_view
polar_catalog 1.2 polardb pg extend catalog
polar_channel 1.0 polar_channel
polar_constraint 1.0 polar_constraint
polar_csn 1.0 polar_csn
polar_dba_views 1.0 polar_dba_views
polar_dbms_alert 1.2 implement polar_dbms_alert - supports asynchronous notification of database events.
polar_dbms_application_info 1.0 implement polar_dbms_application_info - record names of executing modules or transactions in the database.
polar_dbms_pipe 1.1 implements polar_dbms_pipe - package lets two or more sessions in the same instance communicate.
polar_dbms_aq 1.2 implement dbms_aq - provides an interface to Advanced Queuing.
polar_dbms_lob 1.3 implement dbms_lob - provides subprograms to operate on BLOBs, CLOBs, and NCLOBs.
polar_dbms_output 1.2 implement polar_dbms_output - enables you to send messages from stored procedures.
polar_dbms_lock 1.0 implement polar_dbms_lock - provides an interface to Oracle Lock Management services.
polar_dbms_aqadm 1.3 polar_dbms_aqadm - procedures to manage Advanced Queuing configuration and administration information.
polar_dbms_assert 1.0 implement polar_dbms_assert - provide an interface to validate properties of the input value.
polar_dbms_metadata 1.0 implement polar_dbms_metadata - provides a way for you to retrieve metadata from the database dictionary.
polar_dbms_random 1.0 implement polar_dbms_random - a built-in random number generator, not intended for cryptography
polar_dbms_crypto 1.1 implement dbms_crypto - provides an interface to encrypt and decrypt stored data.
polar_dbms_redact 1.0 implement polar_dbms_redact - provides an interface to mask data from queries by an application.
polar_dbms_debug 1.1 server-side support for debugging PL/SQL functions
polar_dbms_job 1.0 polar_dbms_job
polar_dbms_mview 1.1 implement polar_dbms_mview - enables to refresh materialized views.
polar_dbms_job_preload 1.0 polar_dbms_job_preload
polar_dbms_obfuscation_toolkit 1.1 implement polar_dbms_obfuscation_toolkit - enables an application to get data md5.
polar_dbms_rls 1.1 implement polar_dbms_rls - a fine-grained access control administrative built-in package
polar_multi_toast_utils 1.0 polar_multi_toast_utils
polar_dbms_session 1.2 implement polar_dbms_session - support to set preferences and security levels.
polar_odciconst 1.0 implement ODCIConst - Provide some built-in constants in Oracle.
polar_dbms_sql 1.2 implement polar_dbms_sql - provides an interface to execute dynamic SQL.
polar_osfs_toolkit 1.0 osfs library tools and functions extension
polar_dbms_stats 14.0 stabilize plans by fixing statistics
polar_monitor 1.5 monitor functions for PolarDB
polar_osfs_utils 1.0 osfs library utils extension
polar_dbms_utility 1.3 implement polar_dbms_utility - provides various utility subprograms.
polar_parameter_check 1.0 kernel extension for parameter validation
polar_dbms_xmldom 1.0 implement dbms_xmldom and dbms_xmlparser - support standard DOM interface and xml parser object
polar_parameter_manager 1.1 Extension to select parameters for manager.
polar_faults 1.0.0 simulate some database faults for end user or testing system.
polar_monitor_preload 1.1 examine the polardb information
polar_proxy_utils 1.0 Extension to provide operations about proxy.
polar_feature_utils 1.2 PolarDB feature utilization
polar_global_awr 1.0 PolarDB Global AWR Report
polar_publication 1.0 support polardb pg logical replication
polar_global_cache 1.0 polar_global_cache
polar_px 1.0 Parallel Execution extension
polar_serverless 1.0 polar serverless extension
polar_resource_manager 1.0 a background process that forcibly frees user session process memory
polar_sys_context 1.1 implement polar_sys_context - returns the value of parameter associated with the context namespace at the current instant.
polar_gpc 1.3 polar_gpc
polar_tde_utils 1.0 Internal extension for TDE
polar_gtt 1.1 polar_gtt
polar_utl_encode 1.2 implement polar_utl_encode - provides functions that encode RAW data into a standard encoded format
polar_htap 1.1 extension for PolarDB HTAP
polar_htap_db 1.0 extension for PolarDB HTAP database level operation
polar_io_stat 1.0 polar io stat in multi dimension
polar_utl_file 1.0 implement utl_file - support PL/SQL programs can read and write operating system text files
polar_ivm 1.0 polar_ivm
polar_sql_mapping 1.2 Record error sqls and mapping them to correct one
polar_stat_sql 1.0 Kernel statistics gathering, and sql plan nodes information gathering
tds_fdw 2.0.2 Foreign data wrapper for querying a TDS database (Sybase or Microsoft SQL Server)
xml2 1.1 XPath querying and XSLT
polar_upgrade_catalogs 1.1 Upgrade catalogs for old version instance
polar_utl_i18n 1.1 polar_utl_i18n
polar_utl_raw 1.0 implement utl_raw - provides SQL functions for manipulating RAW datatypes.
timescaledb 2.9.2 Enables scalable inserts and complex queries for time-series data
polar_vfs 1.0 polar virtual file system for different storage
polar_worker 1.0 polar_worker
postgres_fdw 1.1 foreign-data wrapper for remote PostgreSQL servers
refint 1.0 functions for implementing referential integrity (obsolete)
roaringbitmap 0.5 support for Roaring Bitmaps
tsm_system_time 1.0 TABLESAMPLE method which accepts time in milliseconds as a limit
vector 0.5.0 vector data type and ivfflat and hnsw access methods
rum 1.3 RUM index access method
unaccent 1.1 text search dictionary that removes accents
seg 1.4 data type for representing line segments or floating-point intervals
sequential_uuids 1.0.2 generator of sequential UUIDs
uuid-ossp 1.1 generate universally unique identifiers (UUIDs)
smlar 1.0 compute similarity of any one-dimensional arrays
varbitx 1.1 varbit functions pack
sslinfo 1.2 information about SSL certificates
tablefunc 1.0 functions that manipulate whole tables, including crosstab
tcn 1.0 Triggered change notifications
zhparser 1.0 a parser for full-text search of Chinese
address_standardizer 3.3.2 Ganos PostGIS address standardizer
address_standardizer_data_us 3.3.2 Ganos PostGIS address standardizer data us
ganos_fdw 6.0 Ganos Spatial FDW extension for POLARDB
ganos_geometry 6.0 Ganos geometry lite extension for POLARDB
ganos_geometry_pyramid 6.0 Ganos Geometry Pyramid extension for POLARDB
ganos_geometry_sfcgal 6.0 Ganos geometry lite sfcgal extension for POLARDB
ganos_geomgrid 6.0 Ganos geometry grid extension for POLARDB
ganos_importer 6.0 Ganos Spatial importer extension for POLARDB
ganos_networking 6.0 Ganos networking
ganos_pointcloud 6.0 Ganos pointcloud extension For POLARDB
ganos_pointcloud_geometry 6.0 Ganos_pointcloud LIDAR data and ganos_geometry data for POLARDB
ganos_raster 6.0 Ganos raster extension for POLARDB
ganos_scene 6.0 Ganos scene extension for POLARDB
ganos_sfmesh 6.0 Ganos surface mesh extension for POLARDB
ganos_spatialref 6.0 Ganos spatial reference extension for POLARDB
ganos_trajectory 6.0 Ganos trajectory extension for POLARDB
ganos_vomesh 6.0 Ganos volume mesh extension for POLARDB
postgis_tiger_geocoder 3.3.2 Ganos PostGIS tiger geocoder
postgis_topology 3.3.2 Ganos PostGIS topology

8.14.15 - PostgresML

How to deploy PostgresML with Pigsty: ML, training, inference, Embedding, RAG inside DB.

PostgresML is a PostgreSQL extension that supports the latest large language models (LLM), vector operations, classical machine learning, and traditional Postgres application workloads.

PostgresML (pgml) is a PostgreSQL extension written in Rust. You can run standalone Docker images, but this documentation is not a docker-compose template introduction, for reference only.

PostgresML officially supports Ubuntu 22.04, but we also maintain RPM versions for EL 8/9, if you don’t need CUDA and NVIDIA-related features.

You need internet access on database nodes to download Python dependencies from PyPI and models from HuggingFace.

PostgresML is Deprecated

Because the company behind it has ceased operations.

Configuration

PostgresML is an extension written in Rust. Pigsty maintains prebuilt packages for PG14-17 on EL8/EL9 and Debian/Ubuntu platforms.

Creating a New Cluster

PostgresML 2.10.0 is available for PostgreSQL 14-17. The example below uses PG17; if you use PG14-16, change pg_version to the corresponding major version.

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_users:
      - {name: dbuser_meta     ,password: DBUser.Meta     ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: pigsty admin user }
      - {name: dbuser_view     ,password: DBUser.Viewer   ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer for meta database }
    pg_databases:
      - { name: meta ,baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty] ,extensions: [{name: postgis, schema: public}, {name: timescaledb}]}
    pg_hba_rules:
      - {user: dbuser_view , db: all ,addr: infra ,auth: pwd ,title: 'allow grafana dashboard access cmdb from infra nodes'}
    pg_version: 17
    pg_libs: 'pgml, pg_stat_statements, auto_explain'
    pg_extensions: [ pgml, pgvector, wal2json, pg_repack ]

Pigsty resolves the pgml package alias to the platform-specific package name: pgml_$v on EL and postgresql-$v-pgml on Debian/Ubuntu. You also need to add pgml to pg_libs.

Enabling on an Existing Cluster

To enable pgml on an existing cluster, you can install it using Ansible’s package module:

ansible pg-meta -m package -b -a 'name=pgml_17'
# ansible el8,el9 -m package -b -a 'name=pgml_17'              # EL 8/9
# ansible u22,u24 -m package -b -a 'name=postgresql-17-pgml'   # Debian/Ubuntu

Python Dependencies

You also need to install PostgresML’s Python dependencies on cluster nodes. Official tutorial: Installation Guide

Install Python and PIP

Ensure python3, pip, and venv are installed:

# Ubuntu 22.04 (python3.10), need to install pip and venv using apt
sudo apt install -y python3 python3-pip python3-venv

For EL 8 / EL9 and compatible distributions, you can use python3.11:

# EL 8/9, can upgrade the default pip and virtualenv
sudo yum install -y python3.11 python3.11-pip       # install latest python3.11
python3.11 -m pip install --upgrade pip virtualenv  # use python3.11 on EL8 / EL9
Using PyPI Mirrors

For users in mainland China, we recommend using Tsinghua University’s PyPI mirror.

pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple    # set global mirror (recommended)
pip install -i https://pypi.tuna.tsinghua.edu.cn/simple some-package        # use for single installation

Install Dependencies

Create a Python virtual environment and use pip to install dependencies from requirements.txt and requirements-xformers.txt.

If you’re using EL 8/9, replace python3 with python3.11 in the following commands.

su - postgres;                          # create virtual environment as database superuser
mkdir -p /data/pgml; cd /data/pgml;     # create virtual environment directory
python3    -m venv /data/pgml           # create virtual environment directory (Ubuntu 22.04)
source /data/pgml/bin/activate          # activate virtual environment

# write Python dependencies and install with pip
cat > /data/pgml/requirements.txt <<EOF
accelerate==0.22.0
auto-gptq==0.4.2
bitsandbytes==0.41.1
catboost==1.2
ctransformers==0.2.27
datasets==2.14.5
deepspeed==0.10.3
huggingface-hub==0.17.1
InstructorEmbedding==1.0.1
lightgbm==4.1.0
orjson==3.9.7
pandas==2.1.0
rich==13.5.2
rouge==1.0.1
sacrebleu==2.3.1
sacremoses==0.0.53
scikit-learn==1.3.0
sentencepiece==0.1.99
sentence-transformers==2.2.2
tokenizers==0.13.3
torch==2.0.1
torchaudio==2.0.2
torchvision==0.15.2
tqdm==4.66.1
transformers==4.33.1
xgboost==2.0.0
langchain==0.0.287
einops==0.6.1
pynvml==11.5.0
EOF

# install dependencies using pip in the virtual environment
python3 -m pip install -r /data/pgml/requirements.txt
python3 -m pip install xformers==0.0.21 --no-dependencies

# additionally, 3 Python packages need to be installed globally using sudo!
sudo python3 -m pip install xgboost lightgbm scikit-learn

Enable PostgresML

After installing the pgml extension and Python dependencies on all cluster nodes, you can enable pgml on the PostgreSQL cluster.

Use the patronictl command to configure the cluster, add pgml to shared_preload_libraries, and specify your virtual environment directory in pgml.venv:

shared_preload_libraries: pgml, timescaledb, pg_stat_statements, auto_explain
pgml.venv: '/data/pgml'

Then restart the database cluster and create the extension using SQL commands:

CREATE EXTENSION vector;        -- also recommend installing pgvector!
CREATE EXTENSION pgml;          -- create PostgresML in the current database
SELECT pgml.version();          -- print PostgresML version information

If everything is normal, you should see output similar to the following:

# create extension pgml;
INFO:  Python version: 3.11.2 (main, Oct  5 2023, 16:06:03) [GCC 8.5.0 20210514 (Red Hat 8.5.0-18)]
INFO:  Scikit-learn 1.3.0, XGBoost 2.0.0, LightGBM 4.1.0, NumPy 1.26.1
CREATE EXTENSION

# SELECT pgml.version(); -- print PostgresML version information
 version
---------
 2.7.8

Done! For more details, please refer to the official PostgresML documentation: https://postgresml.org/docs/guides/use-cases/

8.14.16 - Greenplum

Deploy/Monitor Greenplum clusters with Pigsty, build Massively Parallel Processing (MPP) PostgreSQL data warehouse clusters!

Pigsty supports deploying Greenplum clusters and its derivative distribution YMatrixDB, and provides the capability to integrate existing Greenplum deployments into Pigsty monitoring.

Overview

Greenplum / YMatrix cluster deployment capabilities are only available in the professional/enterprise editions and are not currently open source.

Installation

Pigsty provides installation packages for Greenplum 6 (@el7) and Greenplum 7 (@el8). Open source users can install and configure them manually.

# EL 7 Only (Greenplum6)
./node.yml -t node_install  -e '{"node_repo_modules":"pgsql","node_packages":["open-source-greenplum-db-6"]}'

# EL 8 Only (Greenplum7)
./node.yml -t node_install  -e '{"node_repo_modules":"pgsql","node_packages":["open-source-greenplum-db-7"]}'

Configuration

To define a Greenplum cluster, you need to use pg_mode = gpsql and additional identity parameters pg_shard and gp_role.

#================================================================#
#                        GPSQL Clusters                          #
#================================================================#

#----------------------------------#
# cluster: mx-mdw (gp master)
#----------------------------------#
mx-mdw:
  hosts:
    10.10.10.10: { pg_seq: 1, pg_role: primary , nodename: mx-mdw-1 }
  vars:
    gp_role: master          # this cluster is used as greenplum master
    pg_shard: mx             # pgsql sharding name & gpsql deployment name
    pg_cluster: mx-mdw       # this master cluster name is mx-mdw
    pg_databases:
      - { name: matrixmgr , extensions: [ { name: matrixdbts } ] }
      - { name: meta }
    pg_users:
      - { name: meta , password: DBUser.Meta , pgbouncer: true }
      - { name: dbuser_monitor , password: DBUser.Monitor , roles: [ dbrole_readonly ], superuser: true }

    pgbouncer_enabled: true                # enable pgbouncer for greenplum master
    pgbouncer_exporter_enabled: false      # enable pgbouncer_exporter for greenplum master
    pg_exporter_params: 'host=127.0.0.1&sslmode=disable'  # use 127.0.0.1 as local monitor host

#----------------------------------#
# cluster: mx-sdw (gp master)
#----------------------------------#
mx-sdw:
  hosts:
    10.10.10.11:
      nodename: mx-sdw-1        # greenplum segment node
      pg_instances:             # greenplum segment instances
        6000: { pg_cluster: mx-seg1, pg_seq: 1, pg_role: primary , pg_exporter_port: 9633 }
        6001: { pg_cluster: mx-seg2, pg_seq: 2, pg_role: replica , pg_exporter_port: 9634 }
    10.10.10.12:
      nodename: mx-sdw-2
      pg_instances:
        6000: { pg_cluster: mx-seg2, pg_seq: 1, pg_role: primary , pg_exporter_port: 9633  }
        6001: { pg_cluster: mx-seg3, pg_seq: 2, pg_role: replica , pg_exporter_port: 9634  }
    10.10.10.13:
      nodename: mx-sdw-3
      pg_instances:
        6000: { pg_cluster: mx-seg3, pg_seq: 1, pg_role: primary , pg_exporter_port: 9633 }
        6001: { pg_cluster: mx-seg1, pg_seq: 2, pg_role: replica , pg_exporter_port: 9634 }
  vars:
    gp_role: segment               # these are nodes for gp segments
    pg_shard: mx                   # pgsql sharding name & gpsql deployment name
    pg_cluster: mx-sdw             # these segment clusters name is mx-sdw
    pg_preflight_skip: true        # skip preflight check (since pg_seq & pg_role & pg_cluster not exists)
    pg_exporter_config: pg_exporter_basic.yml                             # use basic config to avoid segment server crash
    pg_exporter_params: 'options=-c%20gp_role%3Dutility&sslmode=disable'  # use gp_role = utility to connect to segments

Additionally, PG Exporter requires extra connection parameters to connect to Greenplum Segment instances for metric collection.

8.14.17 - Neon

Use Neon’s open-source Serverless PostgreSQL kernel to build flexible, scale-to-zero, forkable PG services.

Neon adopts a storage and compute separation architecture, providing seamless autoscaling, scale to zero, and unique database branching capabilities.

Neon official website: https://neon.tech/

Neon binaries are currently too large to include in the open-source package set. This support path remains in pilot stage; contact Pigsty sales if you need it.

8.15 - Param Templates

Use Pigsty’s built-in Patroni config templates or customize your own

Pigsty provides four preset Patroni/PostgreSQL config templates optimized for different workloads:

Template CPU Cores Use Case Characteristics
/docs/pgsql/template/oltp.yml 4-128C OLTP transactions High concurrency, low latency
/docs/pgsql/template/olap.yml 4-128C OLAP analytics Large queries, high parallelism
/docs/pgsql/template/crit.yml 4-128C Consistency-first Consistency-first, detailed auditing
/docs/pgsql/template/tiny.yml 1-3C Tiny instances Resource-constrained envs

Use pg_conf to select a template; default is /docs/pgsql/template/oltp.yml.

The database tuning template pg_conf should be paired with the OS tuning template node_tune.

All four standard templates set wal_level to logical. PostgreSQL 18.6 adds the output_plugin_libraries security allowlist; Pigsty permits the built-in pgoutput and test_decoding plugins plus wal2json, which is installed by the default pgsql-main package set. To use another logical-decoding output plugin, review its code and privilege boundary, then add its exact library name through pg_parameters. Patroni filters the template setting on older PostgreSQL versions that do not support it.

Usage

Set pg_conf in your cluster definition. It’s recommended to set node_tune accordingly for OS-level tuning:

pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica }
  vars:
    pg_cluster: pg-test
    pg_conf: oltp.yml    # PostgreSQL config template (default)
    node_tune: oltp      # OS tuning template (default)

For critical financial workloads, use /docs/pgsql/template/crit.yml:

pg-finance:
  hosts:
    10.10.10.21: { pg_seq: 1, pg_role: primary }
    10.10.10.22: { pg_seq: 2, pg_role: replica }
    10.10.10.23: { pg_seq: 3, pg_role: replica }
  vars:
    pg_cluster: pg-finance
    pg_conf: crit.yml    # PostgreSQL critical template
    node_tune: crit      # OS critical tuning

For low-spec VMs or dev environments, use /docs/pgsql/template/tiny.yml:

pg-dev:
  hosts:
    10.10.10.31: { pg_seq: 1, pg_role: primary }
  vars:
    pg_cluster: pg-dev
    pg_conf: tiny.yml    # PostgreSQL tiny template
    node_tune: tiny      # OS tiny tuning

Comparison

The four templates differ significantly in key parameters:

Connections & Memory

Parameter OLTP OLAP CRIT TINY
max_connections 500/1000 500 500/1000 250
work_mem range 64MB-1GB 64MB-8GB 64MB-1GB 16MB-256MB
maintenance_work_mem 25% shmem 50% shmem 25% shmem 25% shmem
max_locks_per_transaction 1-2x maxconn 2-4x maxconn 1-2x maxconn 1-2x maxconn

Parallel Query

Parameter OLTP OLAP CRIT TINY
max_worker_processes max(cpu+16, 24) max(cpu+20, 28) max(cpu+16, 24) max(cpu+12, 20)
max_parallel_workers 50% cpu 80% cpu 50% cpu 50% cpu
max_parallel_workers_per_gather 20% cpu (max 8) 50% cpu 0 (off) 0 (off)
parallel_setup_cost 2000 1000 2000 1000
parallel_tuple_cost 0.2 0.1 0.2 0.1

Sync Replication

Parameter OLTP OLAP CRIT TINY
synchronous_mode depends pg_rpo depends pg_rpo forced on depends pg_rpo
data_checksums optional optional forced on optional

Vacuum Config

Parameter OLTP OLAP CRIT TINY
vacuum_cost_delay 20ms 10ms 20ms 20ms
vacuum_cost_limit 2000 10000 2000 2000
autovacuum_max_workers 3 3 3 2

Timeout & Security

Parameter OLTP OLAP CRIT TINY
idle_in_transaction_session_timeout 10min off 1min 10min
log_min_duration_statement 100ms 1000ms 100ms 100ms
default_statistics_target 400 1000 400 200
track_activity_query_size 8KB 8KB 32KB 8KB
log_connections auth auth full default

IO Config (PG18)

Parameter OLTP OLAP CRIT TINY
io_workers 25% cpu (4-16) 50% cpu (4-32) 25% cpu (4-8) 3
temp_file_limit 1/20 disk, max 100GB 1/5 disk, max 400GB 1/20 disk, max 100GB 1/20 disk, max 100GB

Selection Guide

  • OLTP Template: Default choice for most transaction processing. Ideal for e-commerce, social, gaming apps.

  • OLAP Template: For data warehouses, BI reports, ETL. Allows large queries, high parallelism, relaxed timeouts.

  • CRIT Template: For financial transactions, core accounting with strict consistency/security requirements. Forced sync replication, checksums, full audit.

  • TINY Template: For dev/test environments, resource-constrained VMs, Raspberry Pi. Minimizes resource usage, disables parallel queries.

Custom Templates

Create custom templates based on existing ones. Templates are in roles/pgsql/templates/:

roles/pgsql/templates/
├── oltp.yml    # OLTP template (default)
├── olap.yml    # OLAP template
├── crit.yml    # CRIT critical template
└── tiny.yml    # TINY micro template

Steps to create a custom template:

  1. Copy an existing template as base
  2. Modify parameters as needed
  3. Place in roles/pgsql/templates/
  4. Reference via pg_conf

Example:

cp roles/pgsql/templates/oltp.yml roles/pgsql/templates/myapp.yml
# Edit myapp.yml as needed

Then use in your cluster:

pg-myapp:
  vars:
    pg_conf: myapp.yml

Templates use Jinja2 syntax; parameters are dynamically computed based on node resources (CPU, memory, disk).

Tuning Strategy

For technical details on template parameter optimization, see Tuning Strategy:

  • Memory tuning (shared buffers, work mem, max connections)
  • CPU tuning (parallel query worker config)
  • Storage tuning (WAL size, temp file limits)
  • Manual parameter adjustment

Related Parameters

  • pg_conf: PostgreSQL config template
  • node_tune: OS tuning template, should match pg_conf
  • pg_rto: Recovery time objective, affects failover timeout
  • pg_rpo: Candidate-replica lag threshold; setting it to 0 enables synchronous replication in the general templates
  • pg_max_conn: Override template max connections
  • pg_shared_buffer_ratio: Shared buffer memory ratio
  • pg_storage_type: Storage type, affects IO params

8.15.1 - Parameter Optimization Policy

Learn the parameter optimization strategies Pigsty uses for the 4 different PostgreSQL workload scenarios.

Pigsty provides four scenario-based parameter templates by default, which can be specified and used through the pg_conf parameter.

  • tiny.yml: Optimized for small nodes, VMs, and demos (the template is labeled for 1-3 cores)
  • oltp.yml: Optimized for OLTP workloads and latency-sensitive applications (4C8GB+) (default template)
  • olap.yml: Optimized for OLAP workloads and throughput (4C8G+)
  • crit.yml: Optimized for data consistency and critical applications (4C8G+)

Pigsty adopts different parameter optimization strategies for these four default scenarios, as shown below:

Memory Parameter Tuning

Pigsty automatically detects the system’s memory size and uses it as the basis for setting the maximum number of connections and memory-related parameters.

  • pg_max_conn: PostgreSQL maximum connections, auto will use recommended values for different scenarios
  • pg_shared_buffer_ratio: Shared buffer memory ratio, default is 0.25

By default, Pigsty uses 25% of memory as PostgreSQL shared buffers. The rest is shared among connections, work_mem, background processes, and the operating-system cache.

By default, if the user has not set a pg_max_conn maximum connections value, Pigsty will use defaults according to the following rules:

  • oltp: 500 (pgbouncer) / 1000 (postgres)
  • crit: 500 (pgbouncer) / 1000 (postgres)
  • tiny: 250
  • olap: 500

For OLTP and CRIT templates, if the service is not pointing to the pgbouncer connection pool but directly connects to the postgres database, the maximum connections will be doubled to 1000.

After determining the maximum connections, work_mem is calculated from shared memory size / maximum connections and limited to the range of 64MB ~ 1GB.

{% raw %}
{% if pg_max_conn != 'auto' and pg_max_conn|int >= 20 %}{% set pg_max_connections = pg_max_conn|int %}{% else %}{% if pg_default_service_dest|default('postgres') == 'pgbouncer' %}{% set pg_max_connections = 500 %}{% else %}{% set pg_max_connections = 1000 %}{% endif %}{% endif %}
{% set pg_max_prepared_transactions = pg_max_connections if 'citus' in pg_libs else 0 %}
{% set pg_max_locks_per_transaction = (2 * pg_max_connections)|int if 'citus' in pg_libs or 'timescaledb' in pg_libs else pg_max_connections %}
{% set pg_shared_buffers = (node_mem_mb|int * pg_shared_buffer_ratio|float) | round(0, 'ceil') | int %}
{% set pg_maintenance_mem = (pg_shared_buffers|int * 0.25)|round(0, 'ceil')|int %}
{% set pg_effective_cache_size = node_mem_mb|int - pg_shared_buffers|int  %}
{% set pg_workmem =  ([ ([ (pg_shared_buffers / pg_max_connections)|round(0,'floor')|int , 64 ])|max|int , 1024])|min|int %}
{% endraw %}

CPU Parameter Tuning

In PostgreSQL, there are 4 important parameters related to parallel queries. Pigsty automatically optimizes parameters based on the current system’s CPU cores. The templates first calculate a parallel/extension worker budget and then add another eight reserved slots when writing max_worker_processes. The final GUC is therefore eight higher than the intermediate variable defined near the top of each template.

OLTP Setting Logic Range Limits
max_worker_processes max(CPU + 8, 16) + 8 max(CPU + 16, 24)
max_parallel_workers max(ceil(50% CPU), 2) 1/2 CPU rounded up, minimum 2
max_parallel_maintenance_workers max(ceil(33% CPU), 2) 1/3 CPU rounded up, minimum 2
max_parallel_workers_per_gather min(max(ceil(20% CPU), 2),8) 1/5 CPU rounded down, minimum 2, max 8
OLAP Setting Logic Range Limits
max_worker_processes max(CPU + 12, 20) + 8 max(CPU + 20, 28)
max_parallel_workers max(ceil(80% CPU, 2)) 4/5 CPU rounded up, minimum 2
max_parallel_maintenance_workers max(ceil(33% CPU), 2) 1/3 CPU rounded up, minimum 2
max_parallel_workers_per_gather max(floor(50% CPU), 2) 1/2 CPU rounded up, minimum 2
CRIT Setting Logic Range Limits
max_worker_processes max(CPU + 8, 16) + 8 max(CPU + 16, 24)
max_parallel_workers max(ceil(50% CPU), 2) 1/2 CPU rounded up, minimum 2
max_parallel_maintenance_workers max(ceil(33% CPU), 2) 1/3 CPU rounded up, minimum 2
max_parallel_workers_per_gather 0, enable as needed
TINY Setting Logic Range Limits
max_worker_processes max(CPU + 4, 12) + 8 max(CPU + 12, 20)
max_parallel_workers max(floor(50% CPU), 1) 50% CPU rounded down, minimum 1
max_parallel_maintenance_workers max(floor(33% CPU), 1) 33% CPU rounded down, minimum 1
max_parallel_workers_per_gather 0 Disables parallel gather per query

Note that the CRIT and TINY templates disable parallel queries by setting max_parallel_workers_per_gather = 0. Users can enable parallel queries as needed by setting this parameter.

Both OLTP and CRIT templates additionally set the following parameters, doubling the parallel query cost to reduce the tendency to use parallel queries.

parallel_setup_cost: 2000           # double from 100 to increase parallel cost
parallel_tuple_cost: 0.2            # double from 0.1 to increase parallel cost
min_parallel_table_scan_size: 32MB  # 4x default 8MB, prefer non-parallel scan
min_parallel_index_scan_size: 2MB   # 4x default 512kB, prefer non-parallel scan

Note that adjustments to the max_worker_processes parameter only take effect after a restart. Additionally, when a replica’s configuration value for this parameter is higher than the primary’s, the replica will fail to start. This parameter must be adjusted through Patroni configuration management, which ensures consistent primary-replica configuration and prevents new replicas from failing to start during failover.

Storage Space Parameters

Pigsty automatically detects the total space of the disk where the /data/postgres main data directory is located and uses it as the basis for specifying the following parameters:

{% raw %}
{% set pg_size_twentieth = ([([(node_fs_bytes|int / 21474836480)|round(0, 'ceil')|int, 1])|max, 100])|min %}
min_wal_size: {{ ([pg_size_twentieth, 200])|min }}GB                  # 1/20 disk size, max 200GB
max_wal_size: {{ ([pg_size_twentieth * 4, 2000])|min }}GB             # 2/10 disk size, max 2000GB
max_slot_wal_keep_size: {{ ([pg_size_twentieth * 6, 3000])|min }}GB   # 3/10 disk size, max 3000GB
temp_file_limit: {{ ([pg_size_twentieth, 200])|min }}GB               # 1/20 of disk size, max 200GB
{% endraw %}
  • pg_size_twentieth is one twentieth of disk capacity rounded up, clamped to 1-100GB.
  • Therefore, in the three standard templates, the effective cap for temp_file_limit and min_wal_size is 100GB.
  • The effective cap for max_wal_size is 400GB.
  • The effective cap for max_slot_wal_keep_size is 600GB.

The OLAP template sets temp_file_limit to pg_size_twentieth × 4, for an effective cap of 400GB. Existing 200GB/2TB/3TB comments at the ends of template lines do not account for the 100GB cap already applied to pg_size_twentieth; the rendered expression is authoritative.

Manual Parameter Tuning

In addition to using Pigsty’s automatically configured parameters, you can also manually tune PostgreSQL parameters.

Use the pg edit-config <cluster> command to interactively edit cluster configuration:

pg edit-config pg-meta

Or use the -p parameter to directly set parameters:

pg edit-config -p log_min_duration_statement=1000 pg-meta
pg edit-config --force -p shared_preload_libraries='timescaledb, pg_cron, pg_stat_statements, auto_explain' pg-meta

You can also use the Patroni REST API to modify configuration:

curl -u 'postgres:Patroni.API' \
    -d '{"postgresql":{"parameters": {"log_min_duration_statement":200}}}' \
    -s -X PATCH http://10.10.10.10:8008/config | jq .

8.15.2 - OLTP Template

PostgreSQL config template optimized for online transaction processing workloads

oltp.yml is Pigsty’s default config template, optimized for online transaction processing (OLTP). Designed for 4-128 core CPUs with high concurrency, low latency, and high throughput.

Pair with node_tune = oltp for OS-level tuning.

Use Cases

OLTP template is ideal for:

  • E-commerce: Order processing, inventory, user transactions
  • Social apps: User feeds, messaging, following relationships
  • Gaming backends: Player data, leaderboards, game state
  • SaaS applications: Multi-tenant business systems
  • Web apps: CRUD-intensive workloads

Workload characteristics:

  • Many short transactions (millisecond-level)
  • High concurrent connections (hundreds to thousands)
  • Read/write ratio typically 7:3 to 9:1
  • Latency-sensitive, requires fast response
  • High data consistency requirements

Usage

oltp.yml is the default template, no explicit specification needed:

pg-oltp:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica }
  vars:
    pg_cluster: pg-oltp
    # pg_conf: oltp.yml  # PostgreSQL config template (default)
    # node_tune: oltp    # OS tuning template (default)

Or explicitly specify:

pg-oltp:
  vars:
    pg_conf: oltp.yml    # PostgreSQL config template
    node_tune: oltp      # OS tuning template

Parameter Details

Connection Management 

max_connections: 500/1000   # depends on pgbouncer usage
superuser_reserved_connections: 10
  • When pg_default_service_dest is pgbouncer, max_connections is set to 500
  • When traffic connects directly to PostgreSQL, max_connections is set to 1000
  • Override via pg_max_conn parameter

Memory Config

OLTP template memory allocation strategy:

Parameter Formula Description
shared_buffers mem × pg_shared_buffer_ratio Default ratio 0.25
maintenance_work_mem shared_buffers × 25% For VACUUM, CREATE INDEX
work_mem 64MB - 1GB Based on shared_buffers/max_connections
effective_cache_size total mem - shared_buffers Estimated cache memory

work_mem calculation:

work_mem = min(max(shared_buffers / max_connections, 64MB), 1GB)

Ensures each connection has sufficient sort/hash memory without over-allocation.

Parallel Query

OLTP template moderately limits parallel queries to prevent resource contention:

max_worker_processes: max(cpu + 16, 24)
max_parallel_workers: 50% × cpu (min 2)
max_parallel_workers_per_gather: 20% × cpu (2-8)
max_parallel_maintenance_workers: 33% × cpu (min 2)

Parallel cost estimates are increased to favor serial execution:

parallel_setup_cost: 2000      # 2x default (1000)
parallel_tuple_cost: 0.2       # 2x default (0.1)
min_parallel_table_scan_size: 32MB   # 4x default (8MB), prefer non-parallel scan
min_parallel_index_scan_size: 2MB    # 4x default (512kB), prefer non-parallel scan

WAL Config 

min_wal_size: disk/20 (effective max 100GB)
max_wal_size: disk/5 (effective max 400GB)
max_slot_wal_keep_size: disk×3/10 (effective max 600GB)
wal_buffers: 16MB
wal_writer_delay: 20ms
wal_writer_flush_after: 1MB
commit_delay: 20
commit_siblings: 10
checkpoint_timeout: 15min
checkpoint_completion_target: 0.80

Balances data safety and write performance.

Vacuum Config 

vacuum_cost_delay: 20ms         # sleep after each vacuum round
vacuum_cost_limit: 2000         # cost limit per vacuum round
autovacuum_max_workers: 3
autovacuum_naptime: 1min
autovacuum_vacuum_scale_factor: 0.08    # 8% table change triggers vacuum
autovacuum_analyze_scale_factor: 0.04   # 4% table change triggers analyze
autovacuum_freeze_max_age: 1000000000

Conservative vacuum settings avoid impacting online transaction performance.

Query Optimization 

random_page_cost: 1.1           # SSD optimized
effective_io_concurrency: 200   # SSD concurrent IO
default_statistics_target: 400  # Statistics precision

Enables planner to generate better query plans.

Logging & Monitoring 

log_min_duration_statement: 100         # log queries > 100ms
log_statement: ddl                      # log DDL statements
log_checkpoints: on
log_lock_waits: on
log_temp_files: 1024                    # log temp files > 1MB
log_autovacuum_min_duration: 1s
track_io_timing: on
track_functions: all
track_activity_query_size: 8192

Client Timeouts 

deadlock_timeout: 50ms
idle_in_transaction_session_timeout: 10min

10-minute idle transaction timeout prevents zombie transactions holding locks.

Extension Config 

shared_preload_libraries: 'pg_stat_statements, auto_explain'

# auto_explain
auto_explain.log_min_duration: 1s
auto_explain.log_analyze: on
auto_explain.log_verbose: on
auto_explain.log_timing: on
auto_explain.log_nested_statements: true

# pg_stat_statements
pg_stat_statements.max: 10000
pg_stat_statements.track: all
pg_stat_statements.track_utility: off
pg_stat_statements.track_planning: off

Template Comparison

Feature OLTP OLAP CRIT
max_connections 500-1000 500 500-1000
work_mem 64MB-1GB 64MB-8GB 64MB-1GB
Parallel query Moderate limit Aggressive Disabled
Vacuum intensity Conservative Aggressive Conservative
Txn timeout 10min Disabled 1min
Slow query threshold 100ms 1000ms 100ms

Why OLTP over OLAP?

  • Queries are mostly simple point/range lookups
  • Transaction response time requires milliseconds
  • High concurrent connections
  • No complex analytical queries

Why OLTP over CRIT?

  • Small probability of data loss acceptable (async replication)
  • Complete audit logs not required
  • Better write performance desired

Performance Tuning Tips

Connection Pooling

For high concurrency, use PgBouncer connection pool:

pg-oltp:
  vars:
    pg_default_service_dest: pgbouncer  # default
    pgbouncer_poolmode: transaction     # transaction-level pooling

Read Separation

Use read replicas to share read load:

pg-oltp:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica }
    10.10.10.13: { pg_seq: 3, pg_role: replica }

Monitoring Metrics

Focus on these metrics:

  • Connections: Active/waiting connection counts
  • Transaction rate: TPS, commit/rollback ratio
  • Response time: Query latency percentiles (p50/p95/p99)
  • Lock waits: Lock wait time, deadlock counts
  • Replication lag: Replica delay time and bytes

References

8.15.3 - OLAP Template

PostgreSQL config template optimized for online analytical processing workloads

olap.yml is optimized for online analytical processing (OLAP). Designed for 4-128 core CPUs with support for large queries, high parallelism, relaxed timeouts, and aggressive vacuum.

Pair with node_tune = olap for OS-level tuning.

Use Cases

OLAP template is ideal for:

  • Data warehouses: Historical data storage, multidimensional analysis
  • BI reports: Complex report queries, dashboard data sources
  • ETL processing: Data extraction, transformation, loading
  • Data analysis: Ad-hoc queries, data exploration
  • HTAP mixed workloads: Analytical replicas

Workload characteristics:

  • Complex queries (seconds to minutes)
  • Low concurrent connections (tens to hundreds)
  • Read-intensive, writes typically batch operations
  • Throughput-sensitive, tolerates higher latency
  • Scans large data volumes

Usage

Specify pg_conf = olap.yml in cluster definition:

pg-olap:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica }
  vars:
    pg_cluster: pg-olap
    pg_conf: olap.yml    # PostgreSQL analytics template
    node_tune: olap      # OS analytics tuning

Use olap.yml template for dedicated offline replicas:

pg-mixed:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica }
    10.10.10.13: { pg_seq: 3, pg_role: offline, pg_conf: olap.yml }  # offline analytics replica
  vars:
    pg_cluster: pg-mixed
    pg_conf: oltp.yml    # primary and online replicas use OLTP
    node_tune: oltp      # OS OLTP tuning

Parameter Details

Connection Management 

max_connections: 500
superuser_reserved_connections: 10

OLAP scenarios typically don’t need many connections; 500 is sufficient for most analytical workloads.

Memory Config

OLAP template uses more aggressive memory allocation:

Parameter Formula Description
shared_buffers mem × pg_shared_buffer_ratio Default ratio 0.25
maintenance_work_mem shared_buffers × 50% Faster index creation and VACUUM
work_mem 64MB - 8GB Larger sort/hash memory
effective_cache_size total mem - shared_buffers Estimated cache memory

work_mem calculation (differs from OLTP):

work_mem = min(max(shared_buffers / max_connections, 64MB), 8GB)

Larger work_mem allows bigger sort and hash operations in memory, avoiding disk spill.

Locks & Transactions 

max_locks_per_transaction: 2-4x maxconn   # OLTP: 1-2x

OLAP queries may involve more tables (partitions, many JOINs), requiring more lock slots.

Parallel Query

OLAP template aggressively enables parallel queries:

max_worker_processes: max(cpu + 20, 28)      # OLTP: max(cpu + 16, 24)
max_parallel_workers: 80% × cpu (min 2)      # OLTP: 50%
max_parallel_workers_per_gather: 50% × cpu   # OLTP: 20% (max 8)
max_parallel_maintenance_workers: 33% × cpu

Parallel cost estimates use defaults to favor parallel plans:

# parallel_setup_cost: 1000    # default, not doubled
# parallel_tuple_cost: 0.1     # default, not doubled

Partition-wise optimization enabled:

enable_partitionwise_join: on       # smart partition JOIN
enable_partitionwise_aggregate: on  # smart partition aggregation

IO Config (PG18) 

io_workers: 50% × cpu (4-32)    # OLTP: 25% (4-16)

More IO workers support parallel large table scans.

WAL Config 

min_wal_size: disk/20 (effective max 100GB)
max_wal_size: disk/5 (effective max 400GB)
max_slot_wal_keep_size: disk×3/10 (effective max 600GB)
temp_file_limit: disk/5 (effective max 400GB)   # OLTP: disk/20, effective max 100GB

Larger temp_file_limit allows bigger intermediate results to spill to disk.

Vacuum Config

OLAP template uses aggressive vacuum settings:

vacuum_cost_delay: 10ms         # OLTP: 20ms, faster vacuum
vacuum_cost_limit: 10000        # OLTP: 2000, more work per round
autovacuum_max_workers: 3
autovacuum_naptime: 1min
autovacuum_vacuum_scale_factor: 0.08
autovacuum_analyze_scale_factor: 0.04

Analytical databases often have bulk writes requiring aggressive vacuum to reclaim space.

Query Optimization 

random_page_cost: 1.1
effective_io_concurrency: 200
default_statistics_target: 1000    # OLTP: 400, more precise stats

Higher default_statistics_target provides more accurate query plans, crucial for complex analytics.

Logging & Monitoring 

log_min_duration_statement: 1000    # OLTP: 100ms, relaxed threshold
log_statement: ddl
log_checkpoints: on
log_lock_waits: on
log_temp_files: 1024
log_autovacuum_min_duration: 1s
track_io_timing: on
track_cost_delay_timing: on         # PG18+, track vacuum cost delay
track_functions: all
track_activity_query_size: 8192

Client Timeouts 

deadlock_timeout: 50ms
idle_in_transaction_session_timeout: 0   # OLTP: 10min, disabled

Analytical queries may need to hold transactions for extended periods, so idle timeout is disabled.

Key Differences from OLTP

Parameter OLAP OLTP Reason
max_connections 500 500-1000 Fewer analytical connections
work_mem limit 8GB 1GB Support larger in-memory sorts
maintenance_work_mem 50% buffer 25% buffer Faster index creation
max_locks_per_transaction 2-4x 1-2x More tables in queries
max_parallel_workers 80% cpu 50% cpu Aggressive parallelism
max_parallel_workers_per_gather 50% cpu 20% cpu Aggressive parallelism
parallel_setup_cost 1000 2000 Default, encourages parallel
parallel_tuple_cost 0.1 0.2 Default, encourages parallel
enable_partitionwise_join on off Partition optimization
enable_partitionwise_aggregate on off Partition optimization
vacuum_cost_delay 10ms 20ms Aggressive vacuum
vacuum_cost_limit 10000 2000 Aggressive vacuum
temp_file_limit 1/5 disk 1/20 disk Allow larger temp files
io_workers 50% cpu 25% cpu More parallel IO
log_min_duration_statement 1000ms 100ms Relaxed slow query threshold
default_statistics_target 1000 400 More precise stats
idle_in_transaction_session_timeout Disabled 10min Allow long transactions

Performance Tuning Tips

With TimescaleDB

OLAP template works great with TimescaleDB:

pg-timeseries:
  vars:
    pg_conf: olap.yml
    pg_libs: 'timescaledb, pg_stat_statements, auto_explain'
    pg_extensions:
      - timescaledb

With pg_duckdb

For ultimate analytical performance, combine with pg_duckdb:

pg-analytics:
  vars:
    pg_conf: olap.yml
    pg_libs: 'pg_duckdb, pg_stat_statements, auto_explain'

Columnar Storage

Consider columnar storage extensions:

pg_extensions:
  - citus_columnar  # or pg_mooncake

Resource Isolation

For mixed workloads, isolate analytics to dedicated replicas:

pg-mixed:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }               # OLTP writes
    10.10.10.12: { pg_seq: 2, pg_role: replica }               # OLTP reads
    10.10.10.13: { pg_seq: 3, pg_role: offline }               # OLAP analytics
  vars:
    pg_cluster: pg-mixed

Monitoring Metrics

Focus on these metrics:

  • Query time: Long query execution time distribution
  • Parallelism: Parallel worker utilization
  • Temp files: Temp file size and count
  • Disk IO: Sequential and index scan IO volume
  • Cache hit ratio: shared_buffers and OS cache hit rates

References

8.15.4 - CRIT Template

PostgreSQL parameter template for consistency-first workloads, with strict synchronous replication, data checksums, and detailed connection logging.

crit.yml targets transactional workloads with elevated consistency and audit requirements. It forces data checksums and Patroni strict synchronous mode, adds connection logging, and adjusts selected WAL, timeout, and parallel-query parameters.

The template increases write latency and may block writes when no synchronous replica is available. Before use, confirm consistency objectives, failure domains, client commit settings, and availability requirements.

Also evaluate node_tune: crit, although host tuning and database parameters can be selected independently.

Usage 

pg-critical:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica }
    10.10.10.13: { pg_seq: 3, pg_role: replica }
  vars:
    pg_cluster: pg-critical
    pg_conf: crit.yml
    node_tune: crit

A three-node topology leaves room to select another synchronous replica after one node fails. Continued write availability still depends on remaining node state, DCS, network, and synchronous-replica selection. Exercise failures on the target topology.

Strict Synchronous Replication

CRIT does not derive synchronous mode from pg_rpo. It enables these settings unconditionally:

synchronous_mode: true
synchronous_mode_strict: true

synchronous_mode_strict prevents Patroni from falling back to asynchronous replication when no synchronous replica is available. The primary therefore blocks writes that require synchronous acknowledgment.

The mode targets preservation of acknowledged transactions when:

  • the session has not lowered synchronous_commit to local, off, or another asynchronous level;
  • a synchronous replica acknowledges WAL during commit;
  • failover selects only an eligible node containing the required WAL.

RPO must therefore be validated against client parameters, replication state, and the failure model; it cannot be inferred from the template name alone.

To require acknowledgment from multiple synchronous replicas, change Patroni dynamic configuration:

pg edit-config pg-critical
synchronous_node_count: 2

A higher synchronous-replica count imposes stricter conditions for accepting writes.

Data Checksums

CRIT initialization always includes:

initdb:
  - data-checksums

This overrides a disabled pg_checksum setting and enables page checksums for a new cluster. Checksums detect page damage after write; they do not detect logical errors or every memory error.

Connection and Query Logging

CRIT logs DDL, statements taking longer than 100 ms, and disconnection events:

log_statement: ddl
log_min_duration_statement: 100
log_disconnections: 'on'

PostgreSQL 18 and later use:

log_connections: 'receipt,authentication,authorization'

Earlier versions use log_connections: on. These records support connection auditing but are not fine-grained SQL audit logs. Enable pgaudit separately to record object reads and writes, roles, or statement classes.

track_activity_query_size is set to 32 KiB to retain longer active-query text. Logs may contain SQL and business data; restrict access and set an appropriate retention period.

Watchdog

CRIT changes Patroni watchdog from disabled to automatic:

watchdog:
  mode: automatic
  device: /dev/watchdog

automatic activates only when the system has a usable watchdog device. If fencing must be mandatory, verify hardware, virtualization support, and device permissions before setting required explicitly. A bad configuration can prevent primary startup or disrupt failover.

Key Parameter Differences

Parameter CRIT OLTP Default Effect
synchronous_mode Always enabled Derived from pg_rpo Consistency first
synchronous_mode_strict true General template behavior Blocks writes without a synchronous replica
data-checksums Always enabled Controlled by pg_checksum Page-damage detection
max_parallel_workers_per_gather 0 Calculated from CPU Reduces parallel-query variability
wal_writer_delay 10ms 20ms Processes WAL more frequently
wal_writer_flush_after 0 1MB Changes WAL flush behavior
idle_replication_slot_timeout 3d 7d Removes idle replication slots sooner
idle_in_transaction_session_timeout 1min 10min Terminates idle transactions sooner
track_activity_query_size 32KiB 8KiB Retains longer query text
log_connections Detailed connection events PostgreSQL 18 logs authorization by default Adds connection-audit detail
log_disconnections on off Records disconnections

CRIT also disables parallel gather for individual queries and adjusts parallel costs, autovacuum, WAL, and statistics parameters. The active values for a release are defined in roles/pgsql/templates/crit.yml.

Preloaded Extensions

CRIT generates shared_preload_libraries from pg_libs. The role default sets:

pg_libs: 'pg_stat_statements, auto_explain'

Selecting crit.yml alone does not load passwordcheck. Configure it explicitly when password-complexity checks are required:

pg_libs: '$libdir/passwordcheck, pg_stat_statements, auto_explain'

ha/safe includes this override. To use pgaudit, also add it to pg_libs and configure the audit scope:

pg_libs: '$libdir/passwordcheck, pg_stat_statements, auto_explain, pgaudit'
pg_parameters:
  pgaudit.log: 'ddl, role, write'

Performance and Availability Impact

  • Synchronous commit waits for a synchronous replica; write latency includes at least replica network and WAL durability time.
  • Strict synchronous mode blocks writes when no synchronous replica is available.
  • Disabling parallel gather can reduce throughput for large queries, but also reduces resource variability from parallel execution.
  • More detailed logging and statistics consume additional I/O, CPU, and storage.
  • A shorter idle-transaction timeout may terminate application sessions that hold a transaction open without executing statements.

The impact depends on hardware, network, queries, and client behavior. Test with the actual workload instead of relying on a fixed latency or throughput percentage.

Launch Checklist

  • Deploy at least one usable synchronous replica and verify write behavior during node failure
  • Check whether applications change synchronous_commit
  • Select watchdog automatic or required according to availability requirements
  • Verify collection, access control, and retention for connection logs
  • Configure pg_libs and extension parameters explicitly when password checks or SQL auditing are required
  • Test write latency, throughput, and idle-transaction timeouts with the production workload
  • Exercise primary, synchronous-replica, DCS, and network-partition failures

Related Documentation

8.15.5 - TINY Template

PostgreSQL config template optimized for micro instances and resource-constrained environments

tiny.yml is optimized for micro instances and resource-constrained environments. Designed for 1-3 core CPUs with minimal resource usage, conservative memory allocation, and disabled parallel queries.

Pair with node_tune = tiny for OS-level tuning.

Use Cases

TINY template is ideal for:

  • Dev/test: Local development, CI/CD testing
  • Low-spec VMs: 1-2 core CPU, 1-4GB RAM cloud instances
  • Edge computing: Raspberry Pi, embedded devices
  • Demos: Quick Pigsty experience
  • Personal projects: Resource-limited blogs, small apps

Resource constraints:

  • 1-3 CPU cores
  • 1-8 GB RAM
  • Limited disk space
  • May share resources with other services

Usage

Specify pg_conf = tiny.yml in cluster definition:

pg-dev:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
  vars:
    pg_cluster: pg-dev
    pg_conf: tiny.yml    # PostgreSQL micro instance template
    node_tune: tiny      # OS micro instance tuning

Single-node development:

pg-local:
  hosts:
    127.0.0.1: { pg_seq: 1, pg_role: primary }
  vars:
    pg_cluster: pg-local
    pg_conf: tiny.yml    # PostgreSQL micro instance template
    node_tune: tiny      # OS micro instance tuning

Parameter Details

Connection Management 

max_connections: 250   # OLTP: 500-1000, reduced connection overhead
superuser_reserved_connections: 10

Micro instances don’t need many concurrent connections; 250 is sufficient for dev/test.

Memory Config

TINY template uses conservative memory allocation:

Parameter Formula Description
shared_buffers mem × pg_shared_buffer_ratio Default ratio 0.25
maintenance_work_mem shared_buffers × 25% For VACUUM, CREATE INDEX
work_mem 16MB - 256MB Smaller sort/hash memory
effective_cache_size total mem - shared_buffers Estimated cache memory

work_mem calculation (differs from OLTP):

work_mem = min(max(shared_buffers / max_connections, 16MB), 256MB)

Smaller work_mem limit (256MB vs OLTP’s 1GB) prevents memory exhaustion.

Parallel Query (Fully Disabled)

TINY template completely disables parallel queries:

max_worker_processes: max(cpu + 12, 20)     # OLTP: max(cpu + 16, 24)
max_parallel_workers: 50% × cpu (min 1)      # OLTP: 50% (min 2)
max_parallel_workers_per_gather: 0           # parallel queries disabled
max_parallel_maintenance_workers: 33% × cpu (min 1)

max_parallel_workers_per_gather: 0 ensures queries won’t spawn parallel workers, avoiding resource contention on low-core systems.

IO Config (PG18) 

io_workers: 3   # fixed value, OLTP: 25% cpu (4-16)

Fixed low IO worker count suitable for resource-constrained environments.

Vacuum Config 

vacuum_cost_delay: 20ms
vacuum_cost_limit: 2000
autovacuum_max_workers: 2          # OLTP: 3, one fewer worker
autovacuum_naptime: 1min
# autovacuum_vacuum_scale_factor uses default
# autovacuum_analyze_scale_factor uses default

Fewer autovacuum workers reduce background resource usage.

Query Optimization 

random_page_cost: 1.1
effective_io_concurrency: 200
default_statistics_target: 200     # OLTP: 400, lower precision saves space

Lower default_statistics_target reduces pg_statistic table size.

Logging Config 

log_min_duration_statement: 100    # same as OLTP
log_statement: ddl
log_checkpoints: on
log_lock_waits: on
log_temp_files: 1024
# log_connections uses default (no extra logging)

TINY template doesn’t enable extra connection logging to reduce log volume.

Client Timeouts 

deadlock_timeout: 50ms
idle_in_transaction_session_timeout: 10min   # same as OLTP

Extension Config 

shared_preload_libraries: 'pg_stat_statements, auto_explain'

pg_stat_statements.max: 2500      # OLTP: 10000, reduced memory usage
pg_stat_statements.track: all
pg_stat_statements.track_utility: off
pg_stat_statements.track_planning: off

pg_stat_statements.max reduced from 10000 to 2500, saving ~75% memory.

Key Differences from OLTP

Parameter TINY OLTP Reason
max_connections 250 500-1000 Reduce connection overhead
work_mem limit 256MB 1GB Prevent memory exhaustion
max_worker_processes max(cpu+12, 20) max(cpu+16, 24) Fewer background processes
max_parallel_workers_per_gather 0 20% cpu Disable parallel queries
autovacuum_max_workers 2 3 Reduce background load
default_statistics_target 200 400 Save space
pg_stat_statements.max 2500 10000 Reduce memory usage
io_workers 3 25% cpu Fixed low value

Resource Estimates

TINY template resource usage by configuration:

1 Core 1GB RAM 

shared_buffers: ~256MB
work_mem: ~16MB
maintenance_work_mem: ~64MB
max_connections: 250
max_worker_processes: 20

PostgreSQL process memory: ~400-600MB

2 Core 4GB RAM 

shared_buffers: ~1GB
work_mem: ~32MB
maintenance_work_mem: ~256MB
max_connections: 250
max_worker_processes: 20

PostgreSQL process memory: ~1.5-2GB

4 Core 8GB RAM

Consider using OLTP template instead:

pg-small:
  vars:
    pg_conf: oltp.yml   # 4C8G can use OLTP template

Performance Tuning Tips

Further Resource Reduction

For extremely constrained resources:

pg_parameters:
  max_connections: 100           # further reduce
  shared_buffers: 128MB          # further reduce
  maintenance_work_mem: 32MB
  work_mem: 8MB

Disable Unnecessary Extensions 

pg_libs: 'pg_stat_statements'    # keep only essential extensions

Disable Unnecessary Features 

pg_parameters:
  track_io_timing: off           # disable IO timing tracking
  track_functions: none          # disable function tracking

Use External Connection Pool

Even on micro instances, PgBouncer significantly improves concurrency:

pg-tiny:
  vars:
    pg_conf: tiny.yml
    pg_default_service_dest: pgbouncer
    pgbouncer_poolmode: transaction

Cloud Platform Recommendations

AWS

  • t3.micro: 1 vCPU, 1GB RAM - suitable for TINY
  • t3.small: 2 vCPU, 2GB RAM - suitable for TINY
  • t3.medium: 2 vCPU, 4GB RAM - consider OLTP

Alibaba Cloud

  • ecs.t6-c1m1.small: 1 vCPU, 1GB RAM - suitable for TINY
  • ecs.t6-c1m2.small: 1 vCPU, 2GB RAM - suitable for TINY
  • ecs.t6-c1m4.small: 1 vCPU, 4GB RAM - suitable for TINY

Tencent Cloud

  • SA2.SMALL1: 1 vCPU, 1GB RAM - suitable for TINY
  • SA2.SMALL2: 1 vCPU, 2GB RAM - suitable for TINY
  • SA2.SMALL4: 1 vCPU, 4GB RAM - suitable for TINY

Edge Device Deployment

Raspberry Pi 4 

pg-pi:
  hosts:
    192.168.1.100: { pg_seq: 1, pg_role: primary }
  vars:
    pg_cluster: pg-pi
    pg_conf: tiny.yml       # PostgreSQL micro instance template
    node_tune: tiny         # OS micro instance tuning
    pg_storage_type: SSD    # SSD storage recommended

Docker Container 

pg-docker:
  hosts:
    172.17.0.2: { pg_seq: 1, pg_role: primary }
  vars:
    pg_cluster: pg-docker
    pg_conf: tiny.yml       # PostgreSQL micro instance template
    node_tune: tiny         # OS micro instance tuning

Upgrading to OLTP

When your application grows and needs more resources, easily upgrade to OLTP template:

  1. Upgrade VM specs (4 core 8GB+)
  2. Modify cluster config:
pg-growing:
  vars:
    pg_conf: oltp.yml    # change from tiny.yml to oltp.yml
    node_tune: oltp      # change from tiny to oltp
  1. Reconfigure cluster or redeploy

References

8.16 - FAQ

Frequently asked questions about PostgreSQL

Why can’t my current user use the pg admin alias?

Starting from Pigsty v4.0, permissions to manage global Patroni / PostgreSQL clusters using the pg admin alias have been tightened to the admin group (admin) on admin nodes.

The admin user (dba) created by the node.yml playbook has this permission by default. If your current user wants this permission, you need to explicitly add them to the admin group:

sudo usermod -aG admin <username>

PGSQL Init Fails: Fail to wait for postgres/patroni primary

There are multiple possible causes for this error. You need to check Ansible, Systemd / Patroni / PostgreSQL logs to find the real cause.

  • Possibility 1: Cluster config error - find and fix the incorrect config items.
  • Possibility 2: A cluster with the same name exists, or the previous same-named cluster primary was improperly removed.
  • Possibility 3: Residual metadata from a same-named cluster remains in DCS. First inspect the exact keyspace with etcdctl get --prefix /pg/<cls>/; only after confirming the backup and full cluster name should you use etcdctl del --prefix /pg/<cls>/. The trailing / is the namespace boundary and must not be omitted, or a cluster whose name merely starts with the same text can also match. This is destructive; prefer the controlled decommissioning workflow.
  • Possibility 4: Your PostgreSQL or node-related RPM pkgs were not successfully installed.
  • Possibility 5: Your Watchdog kernel module was not properly enabled/loaded.
  • Possibility 6: The locale you specified during database init doesn’t exist (e.g., used en_US.UTF8 but English language pack or Locale support wasn’t installed).
  • If you encounter other causes, please submit an Issue or ask the community for help.

PGSQL Init Fails: Fail to wait for postgres/patroni replica

There are several possible causes:

Immediate failure: Usually due to config errors, network issues, corrupted DCS metadata, etc. You must check /pg/log to find the actual cause.

Failure after a while: This might be due to source instance data corruption. See PGSQL FAQ: How to create a replica when data is corrupted?

Timeout after a long time: If the wait for postgres replica task takes 30 minutes or longer and fails due to timeout, this is common for large clusters (e.g., 1TB+, may take hours to create a replica).

In this case, the underlying replica creation process is still ongoing. You can use pg list <cls> to check cluster status and wait for the replica to catch up with the primary. Then use the following command to continue with remaining tasks and complete the full replica init:

./pgsql.yml -t pg_hba,pg_reload,pg_backup,pgbouncer,pg_vip,pg_dns,pg_service,pg_exporter,pg_register -l <problematic_replica>

PGSQL Init Fails: ABORT due to pg_safeguard enabled

This means the PostgreSQL instance being cleaned has the deletion safeguard enabled. Disable pg_safeguard to remove the Postgres instance.

If the deletion safeguard pg_safeguard is enabled, you cannot remove running PGSQL instances using bin/pgsql-rm or the pgsql-rm.yml playbook.

To disable pg_safeguard, you can set pg_safeguard to false in the config inventory, or use the command param -e pg_safeguard=false when executing the playbook.

./pgsql-rm.yml -e pg_safeguard=false -l <cls_to_remove>

How to Ensure No Data Loss During Failover?

Use the crit.yml param template, set pg_rpo to 0, or config the cluster for sync commit mode.

Consider using Sync Standby and Quorum Commit to ensure zero data loss during failover.

For more details, see the intro in Security Considerations - Availability.

How to Rescue When Disk is Full?

If the disk is full and even Shell commands cannot execute, rm -rf /pg/dummy can release some emergency space.

By default, pg_dummy_filesize is set to 64MB. In prod envs, it’s recommended to increase it to 8GB or larger.

It will be placed at /pg/dummy path on the PGSQL main data disk. You can delete this file to free up some emergency space:

At least it will allow you to run some shell scripts on that node to further reclaim other space (e.g., logs/WAL, stale data, WAL archives and backups).

How to Create a Replica When Cluster Data is Corrupted?

Pigsty sets the clonefrom: true tag in the patroni config of all instances, marking the instance as available for creating replicas.

If an instance has corrupted data files causing errors when creating new replicas, you can set clonefrom: false to avoid pulling data from the corrupted instance. Here’s how:

$ vi /etc/patroni/patroni.yml

tags:
  nofailover: false
  clonefrom: true      # ----------> change to false
  noloadbalance: false
  nosync: false
  version:  '18'
  spec: '4C.8G.50G'
  conf: 'oltp.yml'

$ systemctl reload patroni    # Reload Patroni config

What is the Perf Overhead of PostgreSQL Monitoring?

A regular PostgreSQL instance scrape takes about 200ms. The scrape interval defaults to 10 seconds, which is almost negligible for a prod multi-core database instance.

Note that Pigsty enables in-database object monitoring by default, so if your database has hundreds of thousands of table/index objects, scraping may increase to several seconds.

You can modify Prometheus’s scrape frequency. Please ensure: the scrape cycle should be significantly longer than the duration of a single scrape.

How to Monitor an Existing PostgreSQL Instance?

Detailed monitoring config instructions are provided in PGSQL Monitor.

How to Manually Remove PostgreSQL Monitoring Targets? 

./pgsql-rm.yml -t rm_metrics -l <cls>     # Remove all instances of cluster 'cls' from victoria
bin/pgmon-rm <ins>     # Remove a single instance 'ins' monitoring object from Victoria, especially suitable for removing added external instances

8.17 - Misc

Miscellaneous Topics

8.17.1 - Service / Access

Separate read and write operations, route traffic correctly, and deliver PostgreSQL cluster capabilities reliably.

Separate read and write operations, route traffic correctly, and deliver PostgreSQL cluster capabilities reliably.

Service is an abstraction: it is the form in which database clusters provide capabilities to the outside world and encapsulates the details of the underlying cluster.

Services are critical for stable access in production environments and show their value when high availability clusters automatically fail over. Single-node users typically don’t need to worry about this concept.

Single-Node Users

The concept of “service” is for production environments. Personal users/single-node clusters can simply access the database directly using instance name/IP address.

For example, Pigsty’s default single-node pg-meta.meta database can be connected directly using three different users:

psql postgres://dbuser_dba:[email protected]/meta     # Connect directly with DBA superuser
psql postgres://dbuser_meta:[email protected]/meta   # Connect with default business admin user
psql postgres://dbuser_view:DBUser.Viewer@pg-meta/meta     # Connect with default read-only user via instance domain name

Service Overview

In real-world production environments, we use replication-based primary-replica database clusters. In a cluster, there is one and only one instance as the leader (primary) that can accept writes. Other instances (replicas) continuously fetch change logs from the cluster leader and stay consistent with it. At the same time, replicas can also handle read-only requests, significantly reducing the load on the primary in read-heavy scenarios. Therefore, separating write requests and read-only requests to the cluster is a very common practice.

In addition, for production environments with high-frequency short connections, we also pool requests through a connection pool middleware (Pgbouncer) to reduce the overhead of creating connections and backend processes. But for scenarios such as ETL and change execution, we need to bypass the connection pool and access the database directly. At the same time, high-availability clusters will experience failover when failures occur, and failover will cause changes to the cluster’s leader. Therefore, high-availability database solutions require that write traffic can automatically adapt to changes in the cluster’s leader. These different access requirements (read-write separation, pooling and direct connection, automatic failover adaptation) ultimately abstract the concept of Service.

Typically, database clusters must provide this most basic service:

  • Read-Write Service (primary): Can read and write to the database

For production database clusters, at least these two services should be provided:

  • Read-Write Service (primary): Write data: can only be carried by the primary.
  • Read-Only Service (replica): Read data: can be carried by replicas, or by the primary if there are no replicas

In addition, depending on specific business scenarios, there may be other services, such as:

  • Default Direct Service (default): Allows (admin) users to access the database directly, bypassing the connection pool
  • Offline Replica Service (offline): Dedicated replicas that do not handle online read-only traffic, used for ETL and analytical queries
  • Standby Replica Service (standby): Read-only service without replication lag, handled by sync standby/primary for read-only queries
  • Delayed Replica Service (delayed): Access old data from the same cluster at a previous point in time, handled by delayed replica

Default Services

Pigsty provides four different services by default for each PostgreSQL database cluster. Here are the default services and their definitions:

Service Port Description
primary 5433 Production read-write, connects to primary connection pool (6432)
replica 5434 Production read-only, connects to replica connection pool (6432)
default 5436 Admin, ETL writes, direct access to primary (5432)
offline 5438 OLAP, ETL, personal users, interactive queries

Taking the default pg-meta cluster as an example, it provides four default services:

psql postgres://dbuser_meta:DBUser.Meta@pg-meta:5433/meta   # pg-meta-primary : production read-write via primary pgbouncer(6432)
psql postgres://dbuser_meta:DBUser.Meta@pg-meta:5434/meta   # pg-meta-replica : production read-only via replica pgbouncer(6432)
psql postgres://dbuser_dba:DBUser.DBA@pg-meta:5436/meta     # pg-meta-default : direct connection via primary postgres(5432)
psql postgres://dbuser_stats:DBUser.Stats@pg-meta:5438/meta # pg-meta-offline : direct connection via offline postgres(5432)

You can see how these four services work from the sample cluster architecture diagram:

pigsty-ha.png

The actual DNS target of pg-meta is controlled by pg_dns_target. The default auto points to the L2 VIP when VIP is enabled; otherwise it points to the inventory primary’s IP. VIP is not enabled by default. See Accessing Services.

Service Implementation

In Pigsty, services are implemented using haproxy on nodes, differentiated by different ports on host nodes.

Haproxy is enabled by default on each node managed by Pigsty to expose services, and database nodes are no exception. Although nodes in a cluster have primary-replica distinctions from the database perspective, from the service perspective, each node is the same: This means that even if you access a replica node, as long as you use the correct service port, you can still use the primary’s read-write service. This design can hide complexity: so as long as you can access any instance on a PostgreSQL cluster, you can completely access all services.

This design is similar to NodePort services in Kubernetes. Similarly, in Pigsty, each service includes the following two core elements:

  1. Access endpoints exposed through NodePort (port number, where to access?)
  2. Target instances selected through Selectors (instance list, who carries the load?)

Pigsty’s service delivery boundary stops at the cluster’s HAProxy, and users can access these load balancers in various ways. See Accessing Services.

All services are declared through configuration files. For example, the PostgreSQL default services are defined by the pg_default_services parameter:

pg_default_services:
- { name: primary ,port: 5433 ,dest: default  ,check: /primary   ,selector: "[]" }
- { name: replica ,port: 5434 ,dest: default  ,check: /read-only ,selector: "[]" , backup: "[? pg_role == `primary` || pg_role == `offline` ]" }
- { name: default ,port: 5436 ,dest: postgres ,check: /primary   ,selector: "[]" }
- { name: offline ,port: 5438 ,dest: postgres ,check: /replica   ,selector: "[? pg_role == `offline` || pg_offline_query ]" , backup: "[? pg_role == `replica` && !pg_offline_query]"}

You can also define additional services in pg_services. Both pg_default_services and pg_services are arrays of service definition objects.

Defining Services

Pigsty allows you to define your own services:

  • pg_default_services: Services uniformly exposed by all PostgreSQL clusters, four by default.
  • pg_services: Additional PostgreSQL services, can be defined at global or cluster level as needed.
  • haproxy_services: Directly customize HAProxy service content, can be used for accessing other components

For PostgreSQL clusters, you typically only need to focus on the first two. Each service definition generates a new configuration file in the configuration directory of all related HAProxy instances: /etc/haproxy/conf.d/<pg_cluster>-<service>.cfg Here’s a custom service example standby: when you want to provide a read-only service without replication lag, you can add this record to pg_services:

- name: standby                   # Required, service name, final svc name uses `pg_cluster` as prefix, e.g.: pg-meta-standby
  port: 5435                      # Required, exposed service port (as kubernetes service node port mode)
  ip: "*"                         # Optional, IP address the service binds to, all IP addresses by default
  selector: "[]"                  # Required, service member selector, uses JMESPath to filter configuration manifest
  backup: "[? pg_role == `primary`]"  # Optional, service member selector (backup), instances selected here only carry the service when all default selector instances are down
  dest: default                   # Optional, target port, default|postgres|pgbouncer|<port_number>, defaults to 'default', meaning pg_default_service_dest decides
  check: /sync                    # Optional, health check URL path, defaults to /, here uses Patroni API: /sync, only sync standby and primary return 200 healthy status code
  maxconn: 5000                   # Optional, maximum number of allowed frontend connections, defaults to 5000
  balance: roundrobin             # Optional, haproxy load balancing algorithm (defaults to roundrobin, other options: leastconn)
  options: 'inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100'

The above service definition is rendered as the HAProxy configuration file /etc/haproxy/conf.d/pg-test-standby.cfg on the sample three-node pg-test cluster:

#---------------------------------------------------------------------
# service: pg-test-standby @ 10.10.10.11:5435
#---------------------------------------------------------------------
# service instances 10.10.10.11, 10.10.10.13, 10.10.10.12
# service backups   10.10.10.11
listen pg-test-standby
    bind *:5435            # <--- Binds port 5435 on all IP addresses
    mode tcp               # <--- Load balancer works on TCP protocol
    maxconn 5000           # <--- Maximum connections 5000, can be increased as needed
    balance roundrobin     # <--- Load balancing algorithm is rr round-robin, can also use leastconn
    option httpchk         # <--- Enable HTTP health check
    option http-keep-alive # <--- Keep HTTP connection
    http-check send meth OPTIONS uri /sync   # <---- Here uses /sync, Patroni health check API, only sync standby and primary return 200 healthy status code
    http-check expect status 200             # <---- Health check return code 200 means normal
    default-server inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100
    # servers: selector "[]" includes all three pg-test instances as pg-test-standby backends; /sync admits only the primary and synchronous standby.
    server pg-test-1 10.10.10.11:6432 check port 8008 weight 100 backup  # <----- Only primary satisfies condition pg_role == `primary`, selected by backup selector
    server pg-test-3 10.10.10.13:6432 check port 8008 weight 100         #        Therefore serves as service fallback instance: normally doesn't handle requests, only handles read-only requests when all other replicas fail, thus maximally avoiding read-write service being affected by read-only service
    server pg-test-2 10.10.10.12:6432 check port 8008 weight 100         #

Here, all three instances of the pg-test cluster are selected by selector: "[]", rendered into the backend server list of the pg-test-standby service. But due to the /sync health check, Patroni Rest API only returns healthy HTTP 200 status code on the primary and sync standby, so only the primary and sync standby can actually handle requests. Additionally, the primary satisfies the condition pg_role == primary, is selected by the backup selector, and is marked as a backup server, only used when no other instances (i.e., sync standby) can meet the demand.

Primary Service

The Primary service is perhaps the most critical service in production environments. It provides read-write capability to the database cluster on port 5433. The service definition is as follows:

- { name: primary ,port: 5433 ,dest: default  ,check: /primary   ,selector: "[]" }
  • The selector parameter selector: "[]" means all cluster members will be included in the Primary service
  • But only the primary can pass the health check (check: /primary) and actually carry Primary service traffic.
  • The destination parameter dest: default means the Primary service destination is affected by the pg_default_service_dest parameter
  • The default value default of dest will be replaced by the value of pg_default_service_dest, which defaults to pgbouncer.
  • By default, the Primary service destination is the connection pool on the primary, which is the port specified by pgbouncer_port, defaulting to 6432

If the value of pg_default_service_dest is postgres, then the primary service destination will bypass the connection pool and use the PostgreSQL database port directly (pg_port, default 5432). This parameter is very useful for scenarios that don’t want to use a connection pool.

Example: haproxy configuration for pg-test-primary
listen pg-test-primary
    bind *:5433         # <--- primary service defaults to port 5433
    mode tcp
    maxconn 5000
    balance roundrobin
    option httpchk
    option http-keep-alive
    http-check send meth OPTIONS uri /primary # <--- primary service defaults to Patroni RestAPI /primary health check
    http-check expect status 200
    default-server inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100
    # servers
    server pg-test-1 10.10.10.11:6432 check port 8008 weight 100
    server pg-test-3 10.10.10.13:6432 check port 8008 weight 100
    server pg-test-2 10.10.10.12:6432 check port 8008 weight 100

Patroni’s high availability mechanism ensures that at most one instance’s /primary health check is true at any time, so the Primary service will always route traffic to the primary instance.

One benefit of using the Primary service instead of direct database connection is that if the cluster has a split-brain situation for some reason (e.g., kill -9 killing the primary Patroni without watchdog), Haproxy can still avoid split-brain in this case, because it will only distribute traffic when Patroni is alive and returns primary status.

Replica Service

The Replica service is second only to the Primary service in importance in production environments. It provides read-only capability to the database cluster on port 5434. The service definition is as follows:

- { name: replica ,port: 5434 ,dest: default  ,check: /read-only ,selector: "[]" , backup: "[? pg_role == `primary` || pg_role == `offline` ]" }
  • The selector parameter selector: "[]" means all cluster members will be included in the Replica service
  • All instances can pass the health check (check: /read-only) and carry Replica service traffic.
  • Backup selector: [? pg_role == 'primary' || pg_role == 'offline' ] marks the primary and offline replicas as backup servers.
  • Only when all normal replicas are down will the Replica service be carried by the primary or offline replicas.
  • The destination parameter dest: default means the Replica service destination is also affected by the pg_default_service_dest parameter
  • The default value default of dest will be replaced by the value of pg_default_service_dest, which defaults to pgbouncer, same as the Primary service
  • By default, the Replica service destination is the connection pool on the replicas, which is the port specified by pgbouncer_port, defaulting to 6432
Example: haproxy configuration for pg-test-replica
listen pg-test-replica
    bind *:5434
    mode tcp
    maxconn 5000
    balance roundrobin
    option httpchk
    option http-keep-alive
    http-check send meth OPTIONS uri /read-only
    http-check expect status 200
    default-server inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100
    # servers
    server pg-test-1 10.10.10.11:6432 check port 8008 weight 100 backup
    server pg-test-3 10.10.10.13:6432 check port 8008 weight 100
    server pg-test-2 10.10.10.12:6432 check port 8008 weight 100

The Replica service is very flexible: if there are surviving dedicated Replica instances, it will prioritize using these instances to handle read-only requests. Only when all replica instances are down will the primary handle read-only requests. For the common one-primary-one-replica two-node cluster, this means: use the replica as long as it’s alive, use the primary when the replica is down.

Additionally, unless all dedicated read-only instances are down, the Replica service will not use dedicated Offline instances, thus avoiding mixing online fast queries and offline slow queries together, interfering with each other.

Default Service

The Default service provides services on port 5436. It is a variant of the Primary service.

The Default service always bypasses the connection pool and connects directly to PostgreSQL on the primary. This is useful for admin connections, ETL writes, CDC data change capture, etc.

- { name: default ,port: 5436 ,dest: postgres ,check: /primary   ,selector: "[]" }

If pg_default_service_dest is changed to postgres, then the Default service is completely equivalent to the Primary service except for port and name. In this case, you can consider removing Default from default services.

Example: haproxy configuration for pg-test-default
listen pg-test-default
    bind *:5436         # <--- Except for listening port/target port and service name, other configurations are exactly the same as primary service
    mode tcp
    maxconn 5000
    balance roundrobin
    option httpchk
    option http-keep-alive
    http-check send meth OPTIONS uri /primary
    http-check expect status 200
    default-server inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100
    # servers
    server pg-test-1 10.10.10.11:5432 check port 8008 weight 100
    server pg-test-3 10.10.10.13:5432 check port 8008 weight 100
    server pg-test-2 10.10.10.12:5432 check port 8008 weight 100

Offline Service

The Offline service provides services on port 5438. It bypasses the connection pool to directly access the PostgreSQL database, typically used for slow queries, analytical queries, ETL reads, and personal interactive queries:

- { name: offline ,port: 5438 ,dest: postgres ,check: /replica   ,selector: "[? pg_role == `offline` || pg_offline_query ]" , backup: "[? pg_role == `replica` && !pg_offline_query]"}

The Offline service routes traffic directly to dedicated offline replicas, or normal read-only instances with the pg_offline_query flag.

  • The selector parameter filters two types of instances from the cluster: offline replicas with pg_role = offline, or normal read-only instances with pg_offline_query = true
  • The main difference between dedicated offline replicas and flagged normal replicas is: the former does not handle Replica service requests by default, avoiding mixing fast and slow requests together, while the latter does by default.
  • The backup selector parameter filters one type of instance from the cluster: normal replicas without offline flag. This means if offline instances or flagged normal replicas fail, other normal replicas can be used to carry the Offline service.
  • The health check /replica only returns 200 for replicas, the primary returns an error, so the Offline service will never distribute traffic to the primary instance, even if only this primary is left in the cluster.
  • At the same time, the primary instance is neither selected by the selector nor by the backup selector, so it will never carry the Offline service. Therefore, the Offline service can always avoid user access to the primary, thus avoiding impact on the primary.
Example: haproxy configuration for pg-test-offline
listen pg-test-offline
    bind *:5438
    mode tcp
    maxconn 5000
    balance roundrobin
    option httpchk
    option http-keep-alive
    http-check send meth OPTIONS uri /replica
    http-check expect status 200
    default-server inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100
    # servers
    server pg-test-3 10.10.10.13:5432 check port 8008 weight 100
    server pg-test-2 10.10.10.12:5432 check port 8008 weight 100 backup

The Offline service provides limited read-only service, typically used for two types of queries: interactive queries (personal users), slow queries and long transactions (analytics/ETL).

The Offline service requires extra care. HAProxy’s /replica health check automatically rejects the new primary after a switchover, but selector uses static pg_role / pg_offline_query labels from the inventory. In a one-primary-one-replica cluster where only the replica serves Offline queries, a switchover may temporarily leave no eligible backend. Reloading an unchanged inventory does not add the old primary to the Offline backend list. First update the inventory labels (or pg_offline_query) to match the new plan and then reload services, or switch the primary back.

If your business model is relatively simple, you can consider removing the Default service and Offline service, and use the Primary service and Replica service to connect directly to the database.

Reload Services

Reload services when cluster membership changes, service definitions or static selector labels change, or relative weights are adjusted. Normal Primary/Replica switchover is handled by Patroni health checks and does not require a separate reload.

bin/pgsql-svc <cls> [ip...]         # Reload services for lb cluster or lb instance
# ./pgsql.yml -t pg_service         # Actual ansible task for reloading services

Accessing Services

Pigsty’s service delivery boundary stops at the cluster’s HAProxy. Users can access these load balancers in various ways.

The typical approach is to use DNS or VIP access, binding them to all or any number of load balancers in the cluster.

pigsty-access.jpg

You can use different host & port combinations, which provide PostgreSQL services in different ways.

Host

Type Example Description
Cluster Domain pg-test Access via cluster domain name (resolved by dnsmasq @ infra node)
Cluster VIP Address 10.10.10.3 Access via L2 VIP address managed by vip-manager, bound to primary node
Instance Hostname pg-test-1 Access via any instance hostname (resolved by dnsmasq @ infra node)
Instance IP Address 10.10.10.11 Access any instance’s IP address

Port

Pigsty uses different ports to distinguish pg services

Port Service Type Description
5432 postgres Database Direct access to postgres server
6432 pgbouncer Middleware Access postgres via connection pool middleware
5433 primary Service Access primary pgbouncer (or postgres)
5434 replica Service Access replica pgbouncer (or postgres)
5436 default Service Access primary postgres
5438 offline Service Access offline postgres

Combinations

# Access via cluster domain name (the examples below assume L2 VIP is enabled; without it, auto targets the inventory primary IP)
postgres://test@pg-test:5432/test # DNS -> L2 VIP -> Primary direct connection
postgres://test@pg-test:6432/test # DNS -> L2 VIP -> Primary connection pool -> Primary
postgres://test@pg-test:5433/test # DNS -> L2 VIP -> HAProxy -> Primary connection pool -> Primary
postgres://test@pg-test:5434/test # DNS -> L2 VIP -> HAProxy -> Replica connection pool -> Replica
postgres://dbuser_dba@pg-test:5436/test # DNS -> L2 VIP -> HAProxy -> Primary direct connection (for admin)
postgres://dbuser_stats@pg-test:5438/test # DNS -> L2 VIP -> HAProxy -> Offline direct connection (for ETL/personal queries)

# Direct access via cluster VIP
postgres://[email protected]:5432/test # L2 VIP -> Primary direct access
postgres://[email protected]:6432/test # L2 VIP -> Primary connection pool -> Primary
postgres://[email protected]:5433/test # L2 VIP -> HAProxy -> Primary connection pool -> Primary
postgres://[email protected]:5434/test # L2 VIP -> HAProxy -> Replica connection pool -> Replica
postgres://[email protected]:5436/test # L2 VIP -> HAProxy -> Primary direct connection (for admin)
postgres://[email protected]:5438/test # L2 VIP -> HAProxy -> Offline direct connection (for ETL/personal queries)

# Specify any cluster instance name directly
postgres://test@pg-test-1:5432/test # DNS -> Database instance direct connection (single instance access)
postgres://test@pg-test-1:6432/test # DNS -> Connection pool -> Database
postgres://test@pg-test-1:5433/test # DNS -> HAProxy -> Connection pool -> Database read/write
postgres://test@pg-test-1:5434/test # DNS -> HAProxy -> Connection pool -> Database read-only
postgres://dbuser_dba@pg-test-1:5436/test # DNS -> HAProxy -> Database direct connection
postgres://dbuser_stats@pg-test-1:5438/test # DNS -> HAProxy -> Database offline read/write

# Specify any cluster instance IP directly
postgres://[email protected]:5432/test # Database instance direct connection (direct instance specification, no automatic traffic distribution)
postgres://[email protected]:6432/test # Connection pool -> Database
postgres://[email protected]:5433/test # HAProxy -> Connection pool -> Database read/write
postgres://[email protected]:5434/test # HAProxy -> Connection pool -> Database read-only
postgres://[email protected]:5436/test # HAProxy -> Database direct connection
postgres://[email protected]:5438/test # HAProxy -> Database offline read/write

# Smart client: automatic read-write separation
postgres://[email protected]:6432,10.10.10.12:6432,10.10.10.13:6432/test?target_session_attrs=primary
postgres://[email protected]:6432,10.10.10.12:6432,10.10.10.13:6432/test?target_session_attrs=prefer-standby

Overriding Services

You can override default service configuration in multiple ways. A common requirement is to have Primary service and Replica service bypass the Pgbouncer connection pool and access the PostgreSQL database directly.

To achieve this, you can change pg_default_service_dest to postgres, so all services with svc.dest='default' in their service definitions will use postgres instead of the default pgbouncer as the target.

If you have already pointed Primary service to PostgreSQL, then default service becomes redundant and can be considered for removal.

If you don’t need to distinguish between personal interactive queries and analytical/ETL slow queries, you can consider removing Offline service from the default service list pg_default_services.

If you don’t need read-only replicas to share online read-only traffic, you can also remove Replica service from the default service list.

Delegating Services

Pigsty exposes PostgreSQL services through haproxy on nodes. All haproxy instances in the entire cluster are configured with the same service definitions.

However, you can delegate pg services to specific node groups (e.g., dedicated haproxy load balancer cluster) instead of haproxy on PostgreSQL cluster members.

To do this, you need to override the default service definitions using pg_default_services and set pg_service_provider to the proxy group name.

For example, this configuration will expose the pg cluster’s primary service on the proxy haproxy node group on port 10013.

pg_service_provider: proxy       # Use load balancer from `proxy` group on port 10013
pg_default_services:  [{ name: primary ,port: 10013 ,dest: postgres  ,check: /primary   ,selector: "[]" }]

Users need to ensure that the port for each delegated service is unique in the proxy cluster.

An example of using a dedicated load balancer cluster is provided in the 20-node production environment simulation sandbox: conf/ha/simu.yml

8.17.2 - Access Control

Entry points for PostgreSQL access-control concepts, configuration, and administration.

Pigsty access-control documentation is organized by purpose:

dbrole_offline provides independent read-only object privileges; it does not restrict instance scope automatically. To allow it only on offline instances, set role: offline explicitly on the corresponding HBA rule and verify the generated pg_hba.conf on both online and offline instances.

8.17.3 - User / Role

Users/roles refer to logical objects within a database cluster created using the SQL commands CREATE USER/ROLE.

In this context, users refer to logical objects within a database cluster created using the SQL commands CREATE USER/ROLE.

In PostgreSQL, users belong directly to the database cluster rather than to a specific database. Therefore, when creating business databases and business users, you should follow the principle of “users first, then databases.”

Defining Users

Pigsty defines roles and users in database clusters through two configuration parameters:

  • pg_default_roles: Defines globally unified roles and users
  • pg_users: Defines business users and roles at the database cluster level

The former defines roles and users shared across the entire environment, while the latter defines business roles and users specific to individual clusters. Both have the same format and are arrays of user definition objects.

You can define multiple users/roles, and they will be created sequentially—first global, then cluster-level, and finally in array order—so later users can belong to roles defined earlier.

Here is the business user definition for the default cluster pg-meta in the Pigsty demo environment:

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_users:
      - {name: dbuser_meta     ,password: DBUser.Meta     ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: pigsty admin user }
      - {name: dbuser_view     ,password: DBUser.Viewer   ,pgbouncer: true ,roles: [dbrole_readonly] ,comment: read-only viewer for meta database }
      - {name: dbuser_grafana  ,password: DBUser.Grafana  ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for grafana database    }
      - {name: dbuser_bytebase ,password: DBUser.Bytebase ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for bytebase database   }
      - {name: dbuser_kong     ,password: DBUser.Kong     ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for kong api gateway    }
      - {name: dbuser_gitea    ,password: DBUser.Gitea    ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for gitea service       }
      - {name: dbuser_wiki     ,password: DBUser.Wiki     ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for wiki.js service     }
      - {name: dbuser_noco     ,password: DBUser.Noco     ,pgbouncer: true ,roles: [dbrole_admin]    ,comment: admin user for nocodb service      }

Each user/role definition is an object that may include the following fields. Using dbuser_meta as an example:

- name: dbuser_meta               # Required, `name` is the only mandatory field in user definition
  password: DBUser.Meta           # Optional, password can be scram-sha-256 hash string or plaintext
  login: true                     # Optional, can login by default
  superuser: false                # Optional, default is false, is this a superuser?
  createdb: false                 # Optional, default is false, can create databases?
  createrole: false               # Optional, default is false, can create roles?
  inherit: true                   # Optional, by default this role can use inherited privileges?
  replication: false              # Optional, default is false, can this role perform replication?
  bypassrls: false                # Optional, default is false, can this role bypass row-level security?
  pgbouncer: true                 # Optional, default is false, add this user to pgbouncer user list? (production users using connection pool should explicitly set to true)
  connlimit: -1                   # Optional, user connection limit, default -1 disables limit
  expire_in: 3650                 # Optional, this role expires: calculated from creation + n days (higher priority than expire_at)
  expire_at: '2030-12-31'         # Optional, when this role expires, use YYYY-MM-DD format string to specify a date (lower priority than expire_in)
  comment: pigsty admin user      # Optional, description and comment string for this user/role
  roles: [dbrole_admin]           # Optional, default roles are: dbrole_{admin,readonly,readwrite,offline}
  parameters: {}                  # Optional, use `ALTER ROLE SET` to configure role-level database parameters for this role
  pool_mode: transaction          # Optional, pgbouncer pool mode defaulting to transaction, user level
  pool_connlimit: 100             # Optional, user-level max pool connections; omitted inherits global default 100
  search_path: public             # Optional, key-value configuration parameters per postgresql documentation (e.g., use pigsty as default search_path)
  • The only required field is name, which should be a valid and unique username in the PostgreSQL cluster.
  • Roles don’t need a password, but for loginable business users, a password is usually required.
  • password can be plaintext or scram-sha-256 / md5 hash string; please avoid using plaintext passwords.
  • Users/roles are created one by one in array order, so ensure roles/groups are defined before their members.
  • login, superuser, createdb, createrole, inherit, replication, bypassrls are boolean flags.
  • pgbouncer is disabled by default: to add business users to the pgbouncer user list, you should explicitly set it to true.

ACL System

Pigsty provides a built-in access control / ACL model. Assign its default business roles to users as required:

  • dbrole_readwrite: Role with global read-write access (production accounts primarily used by business should have database read-write privileges)
  • dbrole_readonly: Role with global read-only access (if other businesses need read-only access, use this role)
  • dbrole_admin: Role with DDL privileges (business administrators, scenarios requiring table creation in applications)
  • dbrole_offline: Independent read-only role for ad hoc queries, ETL, and analytics; restrict its instance scope explicitly through HBA

If you want to redesign your own ACL system, consider customizing the following parameters and templates:

Creating Users

Users and roles defined in pg_default_roles and pg_users are automatically created one by one during the cluster initialization PROVISION phase. If you want to create users on an existing cluster, you can use the bin/pgsql-user tool. Add the new user/role definition to all.children.<cls>.pg_users and use the following method to create the user:

bin/pgsql-user <cls> <username>    # pgsql-user.yml -l <cls> -e username=<username>

Unlike databases, the user creation playbook is always idempotent. When the target user already exists, Pigsty will modify the target user’s attributes to match the configuration. So running it repeatedly on existing clusters is usually not a problem.

Please Use Playbooks to Create Users

We don’t recommend manually creating new business users, especially when you want the user to use the default pgbouncer connection pool: unless you’re willing to manually maintain the user list in Pgbouncer and keep it consistent with PostgreSQL. When creating new users with bin/pgsql-user tool or pgsql-user.yml playbook, the user will also be added to the Pgbouncer Users list.

Modifying Users

The method for modifying PostgreSQL user attributes is the same as Creating Users.

First, adjust your user definition, modify the attributes that need adjustment, then execute the following command to apply:

bin/pgsql-user <cls> <username>    # pgsql-user.yml -l <cls> -e username=<username>

Note that modifying users will not delete users, but modify user attributes through the ALTER USER command; it also won’t revoke user privileges and groups, and will use the GRANT command to grant new roles.

Pgbouncer Users

Pgbouncer is enabled by default and serves as a connection pool middleware, with its users managed by default.

Pigsty adds all users in pg_users that explicitly have the pgbouncer: true flag to the pgbouncer user list.

Users in the Pgbouncer connection pool are listed in /etc/pgbouncer/userlist.txt:

"postgres" ""
"dbuser_wiki" "SCRAM-SHA-256$4096:+77dyhrPeFDT/TptHs7/7Q==$KeatuohpKIYzHPCt/tqBu85vI11o9mar/by0hHYM2W8=:X9gig4JtjoS8Y/o1vQsIX/gY1Fns8ynTXkbWOjUfbRQ="
"dbuser_view" "SCRAM-SHA-256$4096:DFoZHU/DXsHL8MJ8regdEw==$gx9sUGgpVpdSM4o6A2R9PKAUkAsRPLhLoBDLBUYtKS0=:MujSgKe6rxcIUMv4GnyXJmV0YNbf39uFRZv724+X1FE="
"dbuser_monitor" "SCRAM-SHA-256$4096:fwU97ZMO/KR0ScHO5+UuBg==$CrNsmGrx1DkIGrtrD1Wjexb/aygzqQdirTO1oBZROPY=:L8+dJ+fqlMQh7y4PmVR/gbAOvYWOr+KINjeMZ8LlFww="
"dbuser_meta" "SCRAM-SHA-256$4096:leB2RQPcw1OIiRnPnOMUEg==$eyC+NIMKeoTxshJu314+BmbMFpCcspzI3UFZ1RYfNyU=:fJgXcykVPvOfro2MWNkl5q38oz21nSl1dTtM65uYR1Q="
"dbuser_kong" "SCRAM-SHA-256$4096:bK8sLXIieMwFDz67/0dqXQ==$P/tCRgyKx9MC9LH3ErnKsnlOqgNd/nn2RyvThyiK6e4=:CDM8QZNHBdPf97ztusgnE7olaKDNHBN0WeAbP/nzu5A="
"dbuser_grafana" "SCRAM-SHA-256$4096:HjLdGaGmeIAGdWyn2gDt/Q==$jgoyOB8ugoce+Wqjr0EwFf8NaIEMtiTuQTg1iEJs9BM=:ed4HUFqLyB4YpRr+y25FBT7KnlFDnan6JPVT9imxzA4="
"dbuser_gitea" "SCRAM-SHA-256$4096:l1DBGCc4dtircZ8O8Fbzkw==$tpmGwgLuWPDog8IEKdsaDGtiPAxD16z09slvu+rHE74=:pYuFOSDuWSofpD9OZhG7oWvyAR0PQjJBffgHZLpLHds="
"dbuser_dba" "SCRAM-SHA-256$4096:zH8niABU7xmtblVUo2QFew==$Zj7/pq+ICZx7fDcXikiN7GLqkKFA+X5NsvAX6CMshF0=:pqevR2WpizjRecPIQjMZOm+Ap+x0kgPL2Iv5zHZs0+g="
"dbuser_bytebase" "SCRAM-SHA-256$4096:OMoTM9Zf8QcCCMD0svK5gg==$kMchqbf4iLK1U67pVOfGrERa/fY818AwqfBPhsTShNQ=:6HqWteN+AadrUnrgC0byr5A72noqnPugItQjOLFw0Wk="

User-level connection pool parameters are maintained in a separate file: /etc/pgbouncer/useropts.txt, for example:

dbuser_dba                  = pool_mode=session max_user_connections=16
dbuser_monitor              = pool_mode=session max_user_connections=8

When you create a database, the Pgbouncer database list definition file will be refreshed and take effect through online configuration reload, without affecting existing connections.

Pgbouncer runs with the same dbsu as PostgreSQL, which defaults to the postgres operating system user. You can use the pgb alias to access pgbouncer management functions using the dbsu.

The connection pool user configuration files userlist.txt and useropts.txt are automatically refreshed when you create users, and take effect through online configuration reload, normally without affecting existing connections.

Note that the pgbouncer_auth_query parameter allows you to use dynamic queries to complete connection pool user authentication—this is a compromise when you don’t want to manage users in the connection pool.

8.17.4 - Database

Database refers to the logical object created using the SQL command CREATE DATABASE within a database cluster.

In this context, Database refers to the logical object created using the SQL command CREATE DATABASE within a database cluster.

A PostgreSQL server can serve multiple databases simultaneously. In Pigsty, you can define the required databases in the cluster configuration.

Pigsty will modify and customize the default template database template1, creating default schemas, installing default extensions, and configuring default privileges. Newly created databases will inherit these settings from template1 by default.

By default, all business databases will be added to the Pgbouncer connection pool in a 1:1 manner; pg_exporter will use an auto-discovery mechanism to find all business databases and monitor objects within them.

Define Database

Business databases are defined in the database cluster parameter pg_databases, which is an array of database definition objects. Databases in the array are created sequentially according to the definition order, so later defined databases can use previously defined databases as templates.

Below is the database definition for the default pg-meta cluster in the Pigsty demo environment:

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_databases:
      - { name: meta ,baseline: cmdb.sql ,comment: pigsty meta database ,schemas: [pigsty] ,extensions: [{name: postgis, schema: public}, {name: timescaledb}]}
      - { name: grafana  ,owner: dbuser_grafana  ,revokeconn: true ,comment: grafana primary database }
      - { name: bytebase ,owner: dbuser_bytebase ,revokeconn: true ,comment: bytebase primary database }
      - { name: kong     ,owner: dbuser_kong     ,revokeconn: true ,comment: kong the api gateway database }
      - { name: gitea    ,owner: dbuser_gitea    ,revokeconn: true ,comment: gitea meta database }
      - { name: wiki     ,owner: dbuser_wiki     ,revokeconn: true ,comment: wiki meta database }
      - { name: noco     ,owner: dbuser_noco     ,revokeconn: true ,comment: nocodb database }

Each database definition is an object that may include the following fields, using the meta database as an example:

- name: meta                      # REQUIRED, `name` is the only mandatory field of a database definition
  baseline: cmdb.sql              # optional, database sql baseline path (relative path among ansible search path, e.g. files/)
  pgbouncer: true                 # optional, add this database to pgbouncer database list? true by default
  schemas: [pigsty]               # optional, additional schemas to be created, array of schema names
  extensions:                     # optional, additional extensions to be installed: array of extension objects
    - { name: postgis , schema: public }  # can specify which schema to install the extension in, or leave it unspecified (will install in the first schema of search_path)
    - { name: timescaledb }               # for example, some extensions create and use fixed schemas, so no schema specification is needed.
  comment: pigsty meta database   # optional, comment string for this database
  owner: postgres                 # optional, database owner, postgres by default
  template: template1             # optional, which template to use, template1 by default, target must be a template database
  encoding: UTF8                  # optional, database encoding, UTF8 by default (MUST same as template database)
  locale: C                       # optional, database locale, C by default (MUST same as template database)
  lc_collate: C                   # optional, database collate, C by default (MUST same as template database), no reason not to recommend changing.
  lc_ctype: C                     # optional, database ctype, C by default (MUST same as template database)
  tablespace: pg_default          # optional, default tablespace, 'pg_default' by default
  allowconn: true                 # optional, allow connection, true by default. false will disable connect at all
  revokeconn: false               # optional, revoke public connection privilege. false by default, when set to true, CONNECT privilege will be revoked from users other than owner and admin
  register_datasource: true       # optional, register this database to grafana datasources? true by default, explicitly set to false to skip registration
  connlimit: -1                   # optional, database connection limit, default -1 disable limit, set to positive integer will limit connections
  pool_auth_user: dbuser_meta     # optional, all connections to this pgbouncer database will be authenticated using this user (only useful when pgbouncer_auth_query is enabled)
  pool_mode: transaction          # optional, pgbouncer pool mode at database level, default transaction
  pool_size: 50                   # optional, pgbouncer pool size at database level, default 50
  pool_reserve: 30                # optional, pgbouncer reserve pool at database level, default 30
  pool_size_min: 0                # optional, pgbouncer pool size min at database level, default 0
  pool_connlimit: 100             # optional, max database connections at database level, default 100

The only required field is name, which should be a valid and unique database name in the current PostgreSQL cluster, other parameters have reasonable defaults.

  • name: Database name, required.
  • baseline: SQL file path (Ansible search path, usually in files), used to initialize database content.
  • owner: Database owner, default is postgres
  • template: Template used when creating the database, default is template1
  • encoding: Database default character encoding, default is UTF8, default is consistent with the instance. It is recommended not to configure and modify.
  • locale: Database default locale, default is C, it is recommended not to configure, keep consistent with the instance.
  • lc_collate: Database default locale string collation, default is same as instance setting, it is recommended not to modify, must be consistent with template database. It is strongly recommended not to configure, or configure to C.
  • lc_ctype: Database default LOCALE, default is same as instance setting, it is recommended not to modify or set, must be consistent with template database. It is recommended to configure to C or en_US.UTF8.
  • allowconn: Whether to allow connection to the database, default is true, not recommended to modify.
  • revokeconn: Whether to revoke connection privilege to the database? Default is false. If true, PUBLIC CONNECT privilege on the database will be revoked. Only default users (dbsu|monitor|admin|replicator|owner) can connect. In addition, admin|owner will have GRANT OPTION, can grant connection privileges to other users.
  • tablespace: Tablespace associated with the database, default is pg_default.
  • connlimit: Database connection limit, default is -1, meaning no limit.
  • extensions: Object array, each object defines an extension in the database, and the schema in which it is installed.
  • parameters: KV object, each KV defines a parameter that needs to be modified for the database through ALTER DATABASE.
  • pgbouncer: Boolean option, whether to add this database to Pgbouncer. All databases will be added to Pgbouncer list unless explicitly specified as pgbouncer: false.
  • comment: Database comment information.
  • pool_auth_user: When pgbouncer_auth_query is enabled, all connections to this pgbouncer database will use the user specified here to execute authentication queries. You need to use a user with access to the pg_shadow table.
  • pool_mode: Database level pgbouncer pool mode, default is transaction, i.e., transaction pooling. If left empty, will use pgbouncer_poolmode parameter as default value.
  • pool_size: Database-level Pgbouncer default pool size, default 50.
  • pool_reserve: Database-level Pgbouncer reserve pool, default 30; when the regular pool is exhausted, at most this many burst connections can be added.
  • pool_size_min: Database level pgbouncer pool size min, default is 0
  • pool_connlimit: Database level pgbouncer connection pool max database connections, default is 100

Newly created databases are forked from the template1 database by default. This template database will be customized during the PG_PROVISION phase: configured with extensions, schemas, and default privileges, so newly created databases will also inherit these configurations unless you explicitly use another database as a template.

For database access privileges, see Access Control: Database Isolation.

Create Database

Databases defined in pg_databases will be automatically created during cluster initialization. If you wish to create database on an existing cluster, you can use the bin/pgsql-db wrapper script. Add new database definition to all.children.<cls>.pg_databases, and create that database with the following command:

bin/pgsql-db <cls> <dbname>    # pgsql-db.yml -l <cls> -e dbname=<dbname>

Here are some considerations when creating a new database:

The create database playbook is idempotent by default, however when you use baseline scripts, it may not be: in this case, it’s usually not recommended to re-run this on existing databases unless you’re sure the provided baseline SQL is also idempotent.

We don’t recommend manually creating new databases, especially when you’re using the default pgbouncer connection pool: unless you’re willing to manually maintain the Pgbouncer database list and keep it consistent with PostgreSQL. When creating new databases using the pgsql-db tool or pgsql-db.yml playbook, this database will also be added to the Pgbouncer Database list.

If your database definition has a non-trivial owner (default is dbsu postgres), make sure the owner user exists before creating the database. Best practice is always to create users before creating databases.

Pgbouncer Database

Pigsty will configure and enable a Pgbouncer connection pool for PostgreSQL instances in a 1:1 manner by default, communicating via /var/run/postgresql Unix Socket.

Connection pools can optimize short connection performance, reduce concurrency contention, avoid overwhelming the database with too many connections, and provide additional flexibility during database migration.

Pigsty adds all databases in pg_databases to pgbouncer’s database list by default. You can disable pgbouncer connection pool support for a specific database by explicitly setting pgbouncer: false in the database definition.

The Pgbouncer database list is defined in /etc/pgbouncer/database.txt, and connection pool parameters from the database definition are reflected here:

meta                        = host=/var/run/postgresql mode=session
grafana                     = host=/var/run/postgresql mode=transaction
bytebase                    = host=/var/run/postgresql auth_user=dbuser_meta
kong                        = host=/var/run/postgresql pool_size=32 reserve_pool=64
gitea                       = host=/var/run/postgresql min_pool_size=10
wiki                        = host=/var/run/postgresql
noco                        = host=/var/run/postgresql
mongo                       = host=/var/run/postgresql

When you create databases, the Pgbouncer database list definition file will be refreshed and take effect through online configuration reload, normally without affecting existing connections.

Pgbouncer runs with the same dbsu as PostgreSQL, defaulting to the postgres os user. You can use the pgb alias to access pgbouncer management functions using dbsu.

To route a managed Pgbouncer database to another node, edit /etc/pgbouncer/database.txt, then reload the configuration and rebuild existing server connections:

# Change only mydb's backend target
sed -i -E '/^mydb[[:space:]]*=/ s#host=[^[:space:]]+#host=10.10.10.12#' /etc/pgbouncer/database.txt
pgb -c "RELOAD;"
pgb -c "RECONNECT mydb;"
pgb -c "WAIT_CLOSE mydb;"

The current pgb-route helper edits only /etc/pgbouncer/pgbouncer.ini, while Pigsty-managed database routes live in the included database.txt; it therefore does not change those managed routes.

8.17.5 - Authentication / HBA

Detailed explanation of Host-Based Authentication (HBA) in Pigsty.

Detailed explanation of Host-Based Authentication (HBA) in Pigsty.

Authentication is the foundation of Access Control and Default Privileges. PostgreSQL supports several authentication methods.

Here we mainly introduce HBA: Host Based Authentication. HBA rules define which users can access which databases from which locations and in which ways.

Client Authentication

To connect to a PostgreSQL database, users must first be authenticated (password is used by default).

You can provide the password in the connection string (not secure), or pass it using the PGPASSWORD environment variable or .pgpass file. Refer to the psql documentation and PostgreSQL Connection Strings for more details.

psql 'host=<host> port=<port> dbname=<dbname> user=<username> password=<password>'
psql postgres://<username>:<password>@<host>:<port>/<dbname>
PGPASSWORD=<password>; psql -U <username> -h <host> -p <port> -d <dbname>

For example, to connect to Pigsty’s default meta database, you can use the following connection strings:

psql 'host=10.10.10.10 port=5432 dbname=meta user=dbuser_dba password=DBUser.DBA'
psql postgres://dbuser_dba:[email protected]:5432/meta
PGPASSWORD=DBUser.DBA; psql -U dbuser_dba -h 10.10.10.10 -p 5432 -d meta

By default, Pigsty enables server-side SSL encryption but does not verify client SSL certificates. To connect using client SSL certificates, you can provide client parameters using the PGSSLCERT and PGSSLKEY environment variables or sslkey and sslcert parameters.

psql 'postgres://dbuser_dba:[email protected]:5432/meta?sslkey=/path/to/dbuser_dba.key&sslcert=/path/to/dbuser_dba.crt'

Client certificates (CN = username) can be signed using the local CA with the cert.yml playbook.

Defining HBA

In Pigsty, there are four parameters related to HBA rules:

These are all arrays of HBA rule objects. Each HBA rule is an object in one of the following two forms:

1. Raw Form

The raw form of HBA is almost identical to the PostgreSQL pg_hba.conf format:

- title: allow intranet password access
  role: common
  rules:
    - host   all  all  10.0.0.0/8      md5
    - host   all  all  172.16.0.0/12   md5
    - host   all  all  192.168.0.0/16  md5

In this form, the rules field is an array of strings, where each line is a raw HBA rule. The title field is rendered as a comment explaining what the rules below do.

The role field specifies which instance roles the rule applies to. When an instance’s pg_role matches the role, the HBA rule will be added to that instance’s HBA.

  • HBA rules with role: common will be added to all instances.
  • HBA rules with role: primary will only be added to primary instances.
  • HBA rules with role: replica will only be added to replica instances.
  • HBA rules with role: offline will be added to offline instances (pg_role = offline or pg_offline_query = true)

2. Alias Form

The alias form allows you to maintain HBA rules in a simpler, clearer, and more convenient way: it replaces the rules field with addr, auth, user, and db fields. The title and role fields still apply.

- addr: 'intra'    # world|intra|infra|admin|local|localhost|cluster|<cidr>
  auth: 'pwd'      # trust|pwd|ssl|cert|deny|<official auth method>
  user: 'all'      # all|${dbsu}|${repl}|${admin}|${monitor}|<user>|<group>
  db: 'all'        # all|replication|....
  rules: []        # raw hba string precedence over above all
  title: allow intranet password access
  • addr: where - Which IP address ranges are affected by this rule?
    • world: All IP addresses
    • intra: All intranet IP address ranges: '10.0.0.0/8', '172.16.0.0/12', '192.168.0.0/16'
    • infra: IP addresses of Infra nodes
    • admin: IP addresses of admin_ip management nodes
    • local: Local Unix Socket
    • localhost: Local Unix Socket and TCP 127.0.0.1/32 loopback address
    • cluster: IP addresses of all members in the same PostgreSQL cluster
    • <cidr>: A specific CIDR address block or IP address
  • auth: how - What authentication method does this rule specify?
    • deny: Deny access
    • trust: Trust directly, no authentication required
    • pwd: Password authentication, uses md5 or scram-sha-256 authentication based on the pg_pwd_enc parameter
    • sha/scram-sha-256: Force use of scram-sha-256 password authentication.
    • md5: md5 password authentication, but can also be compatible with scram-sha-256 authentication, not recommended.
    • ssl: On top of password authentication pwd, require SSL to be enabled
    • ssl-md5: On top of password authentication md5, require SSL to be enabled
    • ssl-sha: On top of password authentication sha, require SSL to be enabled
    • os/ident: Use ident authentication with the operating system user identity
    • peer: Use peer authentication method, similar to os ident
    • cert: Use client SSL certificate-based authentication, certificate CN is the username
  • user: who: Which users are affected by this rule?
  • db: which: Which databases are affected by this rule?
    • all: All databases
    • replication: Allow replication connections (not specifying a specific database)
    • A specific database

3. Definition Location

Typically, global HBA is defined in all.vars. To modify the global default HBA rules, copy them from conf/ha/full.yml into all.vars and edit them.

Cluster-specific HBA rules are defined in the database cluster-level configuration:

Here are some examples of cluster HBA rule definitions:

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_hba_rules:
      - { user: dbuser_view ,db: all    ,addr: infra        ,auth: pwd  ,title: 'Allow dbuser_view password access to all databases from infrastructure nodes'}
      - { user: all         ,db: all    ,addr: 100.0.0.0/8  ,auth: pwd  ,title: 'Allow all users password access to all databases from K8S network'          }
      - { user: '${admin}'  ,db: world  ,addr: 0.0.0.0/0    ,auth: cert ,title: 'Allow admin user to login from anywhere with client certificate'       }

Reloading HBA

HBA is a static rule configuration file that needs to be reloaded to take effect after modification. The default HBA rule set typically doesn’t need to be reloaded because it doesn’t involve Role or cluster members.

If your HBA design uses specific instance role restrictions or cluster member restrictions, then when cluster instance members change (add/remove/failover), some HBA rules’ effective conditions/scope change, and you typically also need to reload HBA to reflect the latest changes.

To reload postgres/pgbouncer hba rules:

bin/pgsql-hba <cls>                 # Reload hba rules for cluster `<cls>`
bin/pgsql-hba <cls> ip1 ip2...      # Reload hba rules for specific instances

The underlying Ansible playbook commands actually executed are:

./pgsql.yml -l <cls> -e pg_reload=true -t pg_hba,pg_reload
./pgsql.yml -l <cls> -e pg_reload=true -t pgbouncer_hba,pgbouncer_reload

Default HBA

Pigsty has a default set of HBA rules that are secure enough for most scenarios. These rules use the alias form, so they are basically self-explanatory.

pg_default_hba_rules:             # postgres global default HBA rules 
  - {user: '${dbsu}'    ,db: all         ,addr: local     ,auth: ident ,title: 'dbsu access via local os user ident'  }
  - {user: '${dbsu}'    ,db: replication ,addr: local     ,auth: ident ,title: 'dbsu replication from local os ident' }
  - {user: '${repl}'    ,db: replication ,addr: localhost ,auth: pwd   ,title: 'replicator replication from localhost'}
  - {user: '${repl}'    ,db: replication ,addr: intra     ,auth: pwd   ,title: 'replicator replication from intranet' }
  - {user: '${repl}'    ,db: postgres    ,addr: intra     ,auth: pwd   ,title: 'replicator postgres db from intranet' }
  - {user: '${monitor}' ,db: all         ,addr: localhost ,auth: pwd   ,title: 'monitor from localhost with password' }
  - {user: '${monitor}' ,db: all         ,addr: infra     ,auth: pwd   ,title: 'monitor from infra host with password'}
  - {user: '${admin}'   ,db: all         ,addr: infra     ,auth: ssl   ,title: 'admin @ infra nodes with pwd & ssl'   }
  - {user: '${admin}'   ,db: all         ,addr: world     ,auth: ssl   ,title: 'admin @ everywhere with ssl & pwd'   }
  - {user: '+dbrole_readonly',db: all    ,addr: localhost ,auth: pwd   ,title: 'pgbouncer read/write via local socket'}
  - {user: '+dbrole_readonly',db: all    ,addr: intra     ,auth: pwd   ,title: 'read/write biz user via password'     }
  - {user: '+dbrole_offline' ,db: all    ,addr: intra     ,auth: pwd   ,title: 'allow etl offline tasks from intranet'}
pgb_default_hba_rules:            # pgbouncer global default HBA rules 
  - {user: '${dbsu}'    ,db: pgbouncer   ,addr: local     ,auth: peer  ,title: 'dbsu local admin access with os ident'}
  - {user: 'all'        ,db: all         ,addr: localhost ,auth: pwd   ,title: 'allow all user local access with pwd' }
  - {user: '${monitor}' ,db: pgbouncer   ,addr: intra     ,auth: pwd   ,title: 'monitor access via intranet with pwd' }
  - {user: '${monitor}' ,db: all         ,addr: world     ,auth: deny  ,title: 'reject all other monitor access addr' }
  - {user: '${admin}'   ,db: all         ,addr: intra     ,auth: pwd   ,title: 'admin access via intranet with pwd'   }
  - {user: '${admin}'   ,db: all         ,addr: world     ,auth: deny  ,title: 'reject all other admin access addr'   }
  - {user: 'all'        ,db: all         ,addr: intra     ,auth: pwd   ,title: 'allow all user intra access with pwd' }
Example: Rendered pg_hba.conf
#==============================================================#
# File      :   pg_hba.conf
# Desc      :   Postgres HBA Rules for pg-meta-1 [primary]
# Time      :   2023-01-11 15:19
# Host      :   pg-meta-1 @ 10.10.10.10:5432
# Path      :   /pg/data/pg_hba.conf
# Note      :   ANSIBLE MANAGED, DO NOT CHANGE!
# Author    :   Ruohang Feng ([email protected])
# License   :   Apache-2.0
#==============================================================#

# addr alias
# local     : /var/run/postgresql
# admin     : 10.10.10.10
# infra     : 10.10.10.10
# intra     : 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16

# user alias
# dbsu    :  postgres
# repl    :  replicator
# monitor :  dbuser_monitor
# admin   :  dbuser_dba

# dbsu access via local os user ident [default]
local    all                postgres                              ident

# dbsu replication from local os ident [default]
local    replication        postgres                              ident

# replicator replication from localhost [default]
local    replication        replicator                            scram-sha-256
host     replication        replicator         127.0.0.1/32       scram-sha-256

# replicator replication from intranet [default]
host     replication        replicator         10.0.0.0/8         scram-sha-256
host     replication        replicator         172.16.0.0/12      scram-sha-256
host     replication        replicator         192.168.0.0/16     scram-sha-256

# replicator postgres db from intranet [default]
host     postgres           replicator         10.0.0.0/8         scram-sha-256
host     postgres           replicator         172.16.0.0/12      scram-sha-256
host     postgres           replicator         192.168.0.0/16     scram-sha-256

# monitor from localhost with password [default]
local    all                dbuser_monitor                        scram-sha-256
host     all                dbuser_monitor     127.0.0.1/32       scram-sha-256

# monitor from infra host with password [default]
host     all                dbuser_monitor     10.10.10.10/32     scram-sha-256

# admin @ infra nodes with pwd & ssl [default]
hostssl  all                dbuser_dba         10.10.10.10/32     scram-sha-256

# admin @ everywhere with ssl & pwd [default]
hostssl  all                dbuser_dba         0.0.0.0/0          scram-sha-256

# pgbouncer read/write via local socket [default]
local    all                +dbrole_readonly                      scram-sha-256
host     all                +dbrole_readonly   127.0.0.1/32       scram-sha-256

# read/write biz user via password [default]
host     all                +dbrole_readonly   10.0.0.0/8         scram-sha-256
host     all                +dbrole_readonly   172.16.0.0/12      scram-sha-256
host     all                +dbrole_readonly   192.168.0.0/16     scram-sha-256

# allow etl offline tasks from intranet [default]
host     all                +dbrole_offline    10.0.0.0/8         scram-sha-256
host     all                +dbrole_offline    172.16.0.0/12      scram-sha-256
host     all                +dbrole_offline    192.168.0.0/16     scram-sha-256

# allow application database intranet access [common] [DISABLED]
#host    kong            dbuser_kong         10.0.0.0/8          md5
#host    bytebase        dbuser_bytebase     10.0.0.0/8          md5
#host    grafana         dbuser_grafana      10.0.0.0/8          md5
Example: Rendered pgb_hba.conf
#==============================================================#
# File      :   pgb_hba.conf
# Desc      :   Pgbouncer HBA Rules for pg-meta-1 [primary]
# Time      :   2023-01-11 15:28
# Host      :   pg-meta-1 @ 10.10.10.10:5432
# Path      :   /etc/pgbouncer/pgb_hba.conf
# Note      :   ANSIBLE MANAGED, DO NOT CHANGE!
# Author    :   Ruohang Feng ([email protected])
# License   :   Apache-2.0
#==============================================================#

# PGBOUNCER HBA RULES FOR pg-meta-1 @ 10.10.10.10:6432
# ansible managed: 2023-01-11 14:30:58

# addr alias
# local     : /var/run/postgresql
# admin     : 10.10.10.10
# infra     : 10.10.10.10
# intra     : 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16

# user alias
# dbsu    :  postgres
# repl    :  replicator
# monitor :  dbuser_monitor
# admin   :  dbuser_dba

# dbsu local admin access with os ident [default]
local    pgbouncer          postgres                              peer

# allow all user local access with pwd [default]
local    all                all                                   scram-sha-256
host     all                all                127.0.0.1/32       scram-sha-256

# monitor access via intranet with pwd [default]
host     pgbouncer          dbuser_monitor     10.0.0.0/8         scram-sha-256
host     pgbouncer          dbuser_monitor     172.16.0.0/12      scram-sha-256
host     pgbouncer          dbuser_monitor     192.168.0.0/16     scram-sha-256

# reject all other monitor access addr [default]
host     all                dbuser_monitor     0.0.0.0/0          reject

# admin access via intranet with pwd [default]
host     all                dbuser_dba         10.0.0.0/8         scram-sha-256
host     all                dbuser_dba         172.16.0.0/12      scram-sha-256
host     all                dbuser_dba         192.168.0.0/16     scram-sha-256

# reject all other admin access addr [default]
host     all                dbuser_dba         0.0.0.0/0          reject

# allow all user intra access with pwd [default]
host     all                all                10.0.0.0/8         scram-sha-256
host     all                all                172.16.0.0/12      scram-sha-256
host     all                all                192.168.0.0/16     scram-sha-256

Security Hardening

For scenarios requiring higher security, use the hardened conf/ha/safe.yml configuration template, which uses the following default HBA rule set:

pg_default_hba_rules:             # postgres host-based auth rules by default
  - {user: '${dbsu}'    ,db: all         ,addr: local     ,auth: ident ,title: 'dbsu access via local os user ident'  }
  - {user: '${dbsu}'    ,db: replication ,addr: local     ,auth: ident ,title: 'dbsu replication from local os ident' }
  - {user: '${repl}'    ,db: replication ,addr: localhost ,auth: ssl   ,title: 'replicator replication from localhost'}
  - {user: '${repl}'    ,db: replication ,addr: intra     ,auth: ssl   ,title: 'replicator replication from intranet' }
  - {user: '${repl}'    ,db: postgres    ,addr: intra     ,auth: ssl   ,title: 'replicator postgres db from intranet' }
  - {user: '${monitor}' ,db: all         ,addr: localhost ,auth: pwd   ,title: 'monitor from localhost with password' }
  - {user: '${monitor}' ,db: all         ,addr: infra     ,auth: ssl   ,title: 'monitor from infra host with password'}
  - {user: '${admin}'   ,db: all         ,addr: infra     ,auth: ssl   ,title: 'admin @ infra nodes with pwd & ssl'   }
  - {user: '${admin}'   ,db: all         ,addr: world     ,auth: cert  ,title: 'admin @ everywhere with ssl & cert'   }
  - {user: '+dbrole_readonly',db: all    ,addr: localhost ,auth: ssl   ,title: 'pgbouncer read/write via local socket'}
  - {user: '+dbrole_readonly',db: all    ,addr: intra     ,auth: ssl   ,title: 'read/write biz user via password'     }
  - {user: '+dbrole_offline' ,db: all    ,addr: intra     ,auth: ssl   ,title: 'allow etl offline tasks from intranet'}
pgb_default_hba_rules:            # pgbouncer host-based authentication rules
  - {user: '${dbsu}'    ,db: pgbouncer   ,addr: local     ,auth: peer  ,title: 'dbsu local admin access with os ident'}
  - {user: 'all'        ,db: all         ,addr: localhost ,auth: pwd   ,title: 'allow all user local access with pwd' }
  - {user: '${monitor}' ,db: pgbouncer   ,addr: intra     ,auth: ssl   ,title: 'monitor access via intranet with pwd' }
  - {user: '${monitor}' ,db: all         ,addr: world     ,auth: deny  ,title: 'reject all other monitor access addr' }
  - {user: '${admin}'   ,db: all         ,addr: intra     ,auth: ssl   ,title: 'admin access via intranet with pwd'   }
  - {user: '${admin}'   ,db: all         ,addr: world     ,auth: deny  ,title: 'reject all other admin access addr'   }
  - {user: 'all'        ,db: all         ,addr: intra     ,auth: ssl   ,title: 'allow all user intra access with pwd' }

See Authentication for methods and default boundaries, and Security Considerations for production hardening.

9 - Module: INFRA

Optional standalone infrastructure that provides NTP, DNS, observability and other foundational services for PostgreSQL.

Configuration | Administration | Playbooks | Monitoring | Parameters

Overview

Every Pigsty deployment includes a set of infrastructure components that provide services for managed nodes and database clusters:

Component Port Description
Nginx 80/443 Web service portal, local repo, and unified entry point
Grafana 3000 Visualization platform for monitoring dashboards and data apps
VictoriaMetrics 8428 Time-series database with VMUI, compatible with Prometheus API
VictoriaLogs 9428 Centralized log database, receives structured logs from Vector
VictoriaTraces 10428 Tracing and event storage for slow SQL / request tracing
VMAlert 8880 Alert rule evaluator, triggers alerts based on VictoriaMetrics metrics
AlertManager 9059 Alert aggregation and dispatch, receives notifications from VMAlert
BlackboxExporter 9115 ICMP/TCP/HTTP blackbox probing
DNSMASQ 53 DNS server for internal domain resolution
Chronyd 123 NTP time server
PostgreSQL 5432 CMDB and default database
Ansible - Runs playbooks, orchestrates all infrastructure

In Pigsty, the PGSQL module uses some services on INFRA nodes, specifically:

  • Database cluster/host node domains depend on DNSMASQ on INFRA nodes for resolution.
  • Installing software on database nodes uses the local yum/apt repo hosted by Nginx on INFRA nodes.
  • Database cluster/node monitoring metrics are scraped and stored by VictoriaMetrics on INFRA nodes, accessible via VMUI / PromQL.
  • Database and node runtime logs are collected by Vector and pushed to VictoriaLogs on INFRA, searchable in Grafana.
  • VMAlert evaluates alert rules based on metrics in VictoriaMetrics and forwards events to Alertmanager.
  • Users initiate management of database nodes from Infra/Admin nodes using Ansible or other tools:
    • Execute cluster creation, scaling, instance/cluster recycling
    • Create business users, databases, modify services, HBA changes;
    • Execute log collection, garbage cleanup, backup, inspections, etc.
  • Database nodes sync time from the NTP server on INFRA/ADMIN nodes by default
  • If no dedicated cluster exists, the HA component Patroni uses etcd on INFRA nodes as the HA DCS.
  • If no dedicated cluster exists, the backup component pgbackrest uses MinIO on INFRA nodes as an optional centralized backup repository.

Nginx

Nginx is the access entry point for all WebUI services in Pigsty, using port 80 on the admin node by default.

Many infrastructure components with WebUI are exposed through Nginx, such as Grafana, VictoriaMetrics (VMUI), AlertManager, and HAProxy traffic management pages. Additionally, static file resources like yum/apt repos are served through Nginx.

Nginx exposes built-in Web services through subpaths under i.pigsty by default. It can also route access requests to corresponding upstream components based on domain names according to infra_portal configuration. If you use other domains or public domains, you can modify them here:

infra_portal:  # domain names and upstream servers
  home         : { domain: i.pigsty }
  grafana      : { domain: g.pigsty ,endpoint: "${admin_ip}:3000" , websocket: true }
  prometheus   : { domain: p.pigsty ,endpoint: "${admin_ip}:8428" }   # VMUI
  alertmanager : { domain: a.pigsty ,endpoint: "${admin_ip}:9059" }
  blackbox     : { endpoint: "${admin_ip}:9115" }
  vmalert      : { endpoint: "${admin_ip}:8880" }
  #logs         : { domain: logs.pigsty ,endpoint: "${admin_ip}:9428" }
  #minio        : { domain: sss.pigsty  ,endpoint: "${admin_ip}:9001" ,scheme: https ,websocket: true }

Pigsty strongly recommends using domain names to access Pigsty UI systems rather than direct IP+port access, for these reasons:

  • Using domains makes it easy to enable HTTPS traffic encryption, consolidate access to Nginx, audit all requests, and conveniently integrate authentication mechanisms.
  • Some components only listen on 127.0.0.1 by default, so they can only be accessed through Nginx proxy.
  • Domain names are easier to remember and provide additional configuration flexibility.

If you don’t have available internet domains or local DNS resolution, you can add local static resolution records in /etc/hosts (MacOS/Linux) or C:\Windows\System32\drivers\etc\hosts (Windows).

Nginx configuration parameters are at: Configuration: INFRA - NGINX

Local Software Repository

Pigsty creates a local software repository during installation to accelerate subsequent software installation.

This repository is served by Nginx, located by default at /www/pigsty, accessible via http://i.pigsty/pigsty.

A Pigsty offline bundle is a compressed prebuilt RPM/APT repository directory. The current source uses SOW to create repositories. If /www/pigsty/repo_complete exists, Pigsty treats the local repository as complete and skips upstream downloads. This file contains SHA-256 checksums; it is not merely an empty marker.

The repo definition file is at /www/pigsty.repo, accessible by default via http://${admin_ip}/pigsty.repo

curl -L http://i.pigsty/pigsty.repo -o /etc/yum.repos.d/pigsty.repo

You can also use the file local repo directly without Nginx:

[pigsty-local]
name=Pigsty local $releasever - $basearch
baseurl=file:///www/pigsty/
enabled=1
gpgcheck=0

Local repository configuration parameters are at: Configuration: INFRA - REPO

Victoria Observability Suite

Pigsty v4 uses the VictoriaMetrics family to replace Prometheus/Loki, providing unified monitoring, logging, and tracing capabilities:

  • VictoriaMetrics listens on port 8428 by default, accessible via https://i.pigsty/vmetrics/ for VMUI, compatible with Prometheus API. You can also configure a dedicated domain in infra_portal.
  • VMAlert evaluates alert rules in /infra/rules/*.yml, listens on port 8880, and sends alert events to Alertmanager.
  • VictoriaLogs listens on port 9428, supports the https://i.pigsty/vlogs/ query interface. All nodes run Vector by default, pushing structured system logs, PostgreSQL logs, etc. to VictoriaLogs.
  • VictoriaTraces listens on port 10428 for slow SQL / Trace collection, Grafana accesses it as a Jaeger datasource.
  • Alertmanager listens on port 9059, accessible via https://i.pigsty/alertmgr/ for managing alert notifications. If a.pigsty is configured in infra_portal, it can also be accessed through a dedicated domain. After configuring SMTP, Webhook, etc., it can push messages.
  • Blackbox Exporter listens on port 9115 by default for Ping/TCP/HTTP probing, accessible via https://i.pigsty/blackbox/.

For more information, see: Configuration: INFRA - VICTORIA and Configuration: INFRA - PROMETHEUS.

Grafana

Grafana is the core of Pigsty’s WebUI, listening on port 3000 by default. It can be accessed through https://i.pigsty/ui/ or directly via IP:3000; if g.pigsty is configured in infra_portal, it can also be accessed through a dedicated domain.

Pigsty comes with preconfigured datasources for VictoriaMetrics / Logs / Traces (vmetrics-*, vlogs-*, vtraces-*), and numerous dashboards with URL-based navigation for quick problem location.

Grafana can also be used as a general low-code visualization platform, so Pigsty installs plugins like ECharts and victoriametrics-datasource by default for building monitoring dashboards or inspection reports.

Grafana configuration parameters are at: Configuration: INFRA - GRAFANA.

Ansible

Pigsty installs Ansible on the meta node by default. Ansible is a popular operations tool with declarative configuration style and idempotent playbook design that greatly reduces system maintenance complexity.

DNSMASQ

DNSMASQ provides DNS resolution services within the environment. Domain names from other modules are registered with the DNSMASQ service on INFRA nodes.

DNS records are placed by default in the /etc/dnsmasq.d/pigsty/ directory on all INFRA nodes.

DNSMASQ configuration parameters are at: Configuration: INFRA - DNS

Chronyd

NTP service synchronizes time across all nodes in the environment (optional)

NTP configuration parameters are at: Configuration: NODES - NTP

Configuration

To install the INFRA module on a node, first add it to the infra group in the config inventory and assign an instance number infra_seq

# Configure single INFRA node
infra: { hosts: { 10.10.10.10: { infra_seq: 1 } }}

# Configure two INFRA nodes
infra:
  hosts:
    10.10.10.10: { infra_seq: 1 }
    10.10.10.11: { infra_seq: 2 }

Then use the infra.yml playbook to initialize the INFRA module on the nodes.

Administration

Here are some administration tasks related to the INFRA module:

Install/Uninstall Infra Module 

./infra.yml     # Install INFRA module on infra group
./infra-rm.yml  # Full removal, including data and packages

infra-rm.yml has no deletion safeguard. Without tags, it removes infra_data, nginx_data, nginx_home (default: /www), and /var/lib/grafana. If you only need to stop services or deregister targets, use tags. See Playbooks for the complete removal scope.

Manage Local Software Repository

You can use the following playbook subtasks to manage the local yum repo on Infra nodes:

./infra.yml -t repo              # Create local repo from internet or offline package

./infra.yml -t repo_dir          # Create local repo directory
./infra.yml -t repo_check        # Check if local repo already exists
./infra.yml -t repo_prepare      # If exists, use existing local repo
./infra.yml -t repo_build        # If not exists, build local repo from upstream
./infra.yml     -t repo_upstream     # Handle upstream repo files in /etc/yum.repos.d
./infra.yml     -t repo_remove       # If repo_remove == true, delete existing repo files
./infra.yml     -t repo_add          # Add upstream repo files to /etc/yum.repos.d (or /etc/apt/sources.list.d)
./infra.yml     -t repo_url_pkg      # Download packages from internet defined by repo_url_packages
./infra.yml     -t repo_cache        # Create upstream repo metadata cache with yum makecache / apt update
./infra.yml     -t repo_boot_pkg     # Install SOW and dnf/yum download utilities
./infra.yml     -t repo_pkg          # Download packages & dependencies from upstream repos
./infra.yml     -t repo_create       # Atomically create RPM/APT metadata with sow create --pigsty
./infra.yml     -t repo_use          # Add newly built repo to /etc/yum.repos.d | /etc/apt/sources.list.d
./infra.yml -t repo_nginx        # If no nginx serving, start nginx as web server

The most commonly used commands are:

./infra.yml     -t repo_upstream     # Add upstream repos defined in repo_upstream to INFRA nodes
./infra.yml     -t repo_pkg          # Download packages and dependencies from upstream repos
./infra.yml     -t repo_create       # Create/update local RPM/APT repository metadata with SOW

Manage Infrastructure Components

You can use the following playbook subtasks to manage various infrastructure components on Infra nodes:

./infra.yml -t infra           # Configure infrastructure
./infra.yml -t infra_user      # Setup infra OS user group
./infra.yml -t infra_dir       # Create infrastructure data, config, and runtime directories
./infra.yml -t infra_env       # Configure admin environment: env_patroni, env_pg, env_pgadmin, env_etcd, env_pglog, env_var
./infra.yml -t infra_pkg       # Install software packages required by INFRA: infra_packages
./infra.yml -t infra_cert      # Issue certificates for infra components
./infra.yml -t dns             # Configure DNSMasq: dns_config, dns_record, dns_launch
./infra.yml -t nginx           # Configure Nginx: nginx_config, nginx_cert, nginx_static, nginx_launch, nginx_certbot, nginx_reload, nginx_exporter
./infra.yml -t victoria        # Configure VictoriaMetrics/Logs/Traces: vmetrics|vlogs|vtraces|vmalert
./infra.yml -t alertmanager    # Configure AlertManager: alertmanager_config, alertmanager_launch
./infra.yml -t blackbox        # Configure Blackbox Exporter: blackbox_config, blackbox_launch
./infra.yml -t grafana         # Configure Grafana: grafana_clean, grafana_config, grafana_plugin, grafana_launch, grafana_provision
./infra.yml -t infra_register  # Register infra components to VictoriaMetrics / Grafana

Other commonly used tasks include:

./infra.yml -t nginx_index                        # Re-render Nginx homepage content
./infra.yml -t nginx_config,nginx_reload          # Re-render Nginx portal config, expose new upstream services
./infra.yml -t vmetrics_config,vmetrics_launch    # Regenerate VictoriaMetrics main config and restart service
./infra.yml -t vlogs_config,vlogs_launch          # Re-render VictoriaLogs config
./infra.yml -t vmetrics_clean                     # Clean VictoriaMetrics storage data directory
./infra.yml -t grafana_plugin                     # Download Grafana plugins from internet

Playbooks

Pigsty provides three playbooks related to the INFRA module:

  • infra.yml: Initialize pigsty infrastructure on infra nodes
  • infra-rm.yml: Remove infrastructure components from infra nodes
  • deploy.yml: Deploy the NODE, INFRA, ETCD, MINIO, and PGSQL core chain in one pass

infra.yml

The INFRA module playbook infra.yml initializes pigsty infrastructure on INFRA nodes

Executing this playbook completes the following tasks

  • Configure meta node directories and environment variables
  • Download and build a local software repository to accelerate subsequent installation. (If using offline package, skip download phase)
  • Add the current meta node as a regular node under Pigsty management
  • Deploy infrastructure components including VictoriaMetrics/Logs/Traces, VMAlert, Grafana, Alertmanager, Blackbox Exporter, etc.

This playbook executes on INFRA nodes by default

  • Pigsty uses the current node executing this playbook as Pigsty’s INFRA node and ADMIN node by default.
  • During configuration, Pigsty marks the current node as Infra/Admin node and replaces the placeholder IP 10.10.10.10 in config templates with the current node’s primary IP address.
  • Besides initiating management and hosting infrastructure, this node is no different from a regular managed node.
  • In single-node installation, ETCD is also installed on this node to provide DCS service

Notes about this playbook

  • This is an idempotent playbook; repeated execution will wipe infrastructure components on meta nodes.
  • To preserve historical monitoring data, first set vmetrics_clean, vlogs_clean, vtraces_clean to false.
  • When offline repo /www/pigsty/repo_complete exists, this playbook skips downloading software from internet. Full execution takes about 5-8 minutes depending on machine configuration.
  • Downloading directly from upstream internet sources without offline package may take 10-20 minutes depending on your network conditions.

asciicast

infra-rm.yml

The INFRA module playbook infra-rm.yml removes pigsty infrastructure from INFRA nodes

Common subtasks include:

./infra-rm.yml               # Full removal: deregister, stop, remove config/environment/data, and uninstall packages
./infra-rm.yml -t deregister # Only deregister monitoring targets, datasources, and log collection
./infra-rm.yml -t service    # Stop infrastructure services on INFRA
./infra-rm.yml -t data       # Remove INFRA data
./infra-rm.yml -t package    # Uninstall packages

Full execution has no deletion safeguard and removes infra_data, nginx_data, nginx_home (default: /www), and /var/lib/grafana. Back up any data you need before running it.

deploy.yml

The INFRA module playbook deploy.yml deploys the NODE, INFRA, ETCD, MINIO, and PGSQL core chain on all nodes in one pass. Optional modules such as Docker, Redis, Kafka, native MySQL, JUICE, and VIBE require their own playbooks.

This playbook is described in more detail in Playbook: One-Time Installation.

Monitoring

Pigsty Home: Pigsty monitoring system homepage

Pigsty Home Dashboard

pigsty.jpg

INFRA Overview: Pigsty infrastructure self-monitoring overview

INFRA Overview Dashboard

infra-overview.jpg

Nginx Instance: Nginx metrics and logs

Nginx Overview Dashboard

nginx-overview.jpg

Grafana Instance: Grafana metrics and logs

Grafana Overview Dashboard

grafana-overview.jpg

VictoriaMetrics Instance: VictoriaMetrics scraping, querying, and storage metrics

VMAlert Instance: Alert rule evaluation and queue status

Alertmanager Instance: Alert aggregation, notification pipelines, and Silences

VictoriaLogs Instance: Log ingestion rate, query load, and index hits

VictoriaTraces Instance: Trace/KV storage and Jaeger interface

Logs Instance: Node log search based on Vector + VictoriaLogs

Logs Instance Dashboard

logs-instance.jpg

CMDB Overview: CMDB visualization

CMDB Overview Dashboard

cmdb-overview.jpg

ETCD Overview: etcd metrics and logs

ETCD Overview Dashboard

etcd-overview.jpg

Parameters

The INFRA module has the following 10 parameter groups.

  • META: Pigsty metadata
  • CA: Self-signed PKI/CA infrastructure
  • INFRA_ID: Infrastructure portal, Nginx domains
  • REPO: Local software repository
  • INFRA_PACKAGE: Infrastructure software packages
  • NGINX: Nginx web server
  • DNS: DNSMASQ domain server
  • VICTORIA: VictoriaMetrics / Logs / Traces suite
  • PROMETHEUS: Alertmanager and Blackbox Exporter
  • GRAFANA: Grafana observability suite
Parameter Overview

For the latest default values, types, and hierarchy, please refer to the Parameter Reference to stay consistent with the Pigsty version.

9.1 - Configuration

How to configure INFRA nodes? Customize Nginx, local repo, DNS, NTP, monitoring components.

Configuration Guide

INFRA = primarily monitoring infrastructure, optional for PostgreSQL databases.

Unless manually configured to depend on DNS/NTP services on INFRA nodes, INFRA module failures typically don’t affect PG cluster operations.

Single INFRA node suffices for most scenarios. Prod env recommends 2-3 INFRA nodes for HA.

For better resource utilization, ETCD module (required by PG HA) can share nodes with INFRA module.

Using more than 3 INFRA nodes provides little additional benefit, but more ETCD nodes (e.g., 5) can improve DCS availability.

Configuration Examples

Add node IPs to infra group in config inventory, assign INFRA instance number infra_seq.

Default single INFRA node config:

all:
  children:
    infra: { hosts: { 10.10.10.10: { infra_seq: 1 } }}

By default, 10.10.10.10 placeholder replaced with current node’s primary IP during config.

Use infra.yml playbook to init INFRA module on nodes.

More Nodes

Two INFRA nodes config:

all:
  children:
    infra:
      hosts:
        10.10.10.10: { infra_seq: 1 }
        10.10.10.11: { infra_seq: 2 }

Three INFRA nodes config (with params):

all:
  children:
    infra:
      hosts:
        10.10.10.10: { infra_seq: 1 }
        10.10.10.11: { infra_seq: 2, repo_enabled: false }
        10.10.10.12: { infra_seq: 3, repo_enabled: false }
      vars:
        grafana_clean: false
        vmetrics_clean: false
        vlogs_clean: false
        vtraces_clean: false

INFRA High Availability

Most INFRA module components = “stateless/identical state”. For HA, focus on “load balancing”.

HA achievable via Keepalived L2 VIP or HAProxy L4 load balancing. L2 VIP recommended for L2-reachable networks.

Config example:

infra:
  hosts:
    10.10.10.10: { infra_seq: 1 }
    10.10.10.11: { infra_seq: 2 }
    10.10.10.12: { infra_seq: 3 }
  vars:
    vip_enabled: true
    vip_vrid: 128
    vip_address: 10.10.10.8
    vip_interface: eth1

    infra_portal:
      home         : { domain: i.pigsty }
      grafana      : { domain: g.pigsty ,endpoint: "10.10.10.8:3000" , websocket: true }
      prometheus   : { domain: p.pigsty ,endpoint: "10.10.10.8:8428" }
      alertmanager : { domain: a.pigsty ,endpoint: "10.10.10.8:9059" }
      blackbox     : { endpoint: "10.10.10.8:9115" }
      vmalert      : { endpoint: "10.10.10.8:8880" }

Set VIP-related params and modify service endpoints in infra_portal.

Nginx Configuration

See Nginx Parameter Config and Tutorial: Nginx.

Local Repo Configuration

See Repo Parameter Config.

DNS Configuration

See DNS Parameter Config and Tutorial: DNS.

NTP Configuration

See NTP Parameter Config.

9.2 - Parameters

INFRA module provides 10 sections with 70+ configurable parameters

The INFRA module is responsible for deploying Pigsty’s infrastructure components: local software repository, Nginx, DNSMasq, VictoriaMetrics, VictoriaLogs, Grafana, Alertmanager, Blackbox Exporter, and other monitoring and alerting infrastructure.

Pigsty v4.x uses VictoriaMetrics to replace Prometheus and VictoriaLogs to replace Loki, providing a superior observability solution.

Section Description
META Pigsty metadata: version, admin IP, region, language, proxy
CA Self-signed CA certificate management
INFRA_ID Infrastructure node identity and service portal
REPO Local software repository configuration
INFRA_PACKAGE Infrastructure node package installation
NGINX Nginx web server and reverse proxy configuration
DNS DNSMasq DNS server configuration
VICTORIA VictoriaMetrics/Logs/Traces observability stack
PROMETHEUS Alertmanager and Blackbox Exporter
GRAFANA Grafana visualization platform configuration

Parameter Overview

META parameters define Pigsty metadata, including version string, admin node IP, repository mirror region, default language, and proxy settings.

Parameter Type Level Description
version string G Pigsty version string
admin_ip ip G Admin node IP address
region enum G Upstream mirror region: default,china,europe
language enum G Default language: en or zh
proxy_env dict G Global proxy environment variables

CA parameters configure Pigsty’s self-signed CA certificate management, including CA creation, CA name, and certificate validity.

Parameter Type Level Description
ca_create bool G Allow creation if the CA private key is missing? Default true
ca_cn string G CA CN name, fixed as pigsty-ca
cert_validity interval G Certificate validity, default 20 years

INFRA_ID parameters define infrastructure node identity, including node sequence number, service portal configuration, and data directory.

Parameter Type Level Description
infra_seq int I Infrastructure node sequence, REQUIRED
infra_portal dict G Infrastructure services exposed via Nginx portal
infra_data path G Infrastructure data directory, default /data/infra
infra_services service[] G Built-in home navigation entries
infra_extra_services service[] G Additional home navigation entries, default []

REPO parameters configure the local software repository, including repository enable switch, directory paths, upstream source definitions, and packages to download.

Parameter Type Level Description
repo_enabled bool G/I Create local repo on this infra node?
repo_home path G Repo home directory, default /www
repo_name string G Repo name, default pigsty
repo_endpoint url G Repo access endpoint: domain or ip:port
repo_remove bool G/A Remove existing upstream repo definitions?
repo_modules string G/A Enabled upstream repo modules, comma separated
repo_upstream upstream[] G Upstream repo definitions
repo_packages string[] G Packages to download from upstream
repo_extra_packages string[] G/C/I Extra packages to download
repo_url_packages string[] G Extra packages downloaded via URL

INFRA_PACKAGE parameters define RPM/DEB packages to install on infrastructure nodes.

Parameter Type Level Description
infra_packages string[] G Packages to install on infra nodes

NGINX parameters configure Nginx web server and reverse proxy, including enable switch, ports, SSL mode, certificates, and basic authentication.

Parameter Type Level Description
nginx_enabled bool G/I Enable Nginx on this infra node?
nginx_clean bool G/A Clean existing Nginx config during init?
nginx_exporter_enabled bool G/I Enable nginx_exporter on this infra node?
nginx_exporter_port port G nginx_exporter listen port, default 9113
nginx_sslmode enum G Nginx SSL mode: disable,enable,enforce
nginx_cert_validity duration G Nginx self-signed cert validity, default 397d
nginx_home path G Nginx content dir, default /www, symlink to nginx_data
nginx_data path G Nginx actual data dir, default /data/nginx
nginx_users dict G Nginx basic auth users: username-password dict
nginx_port port G Nginx listen port, default 80
nginx_ssl_port port G Nginx SSL listen port, default 443
certbot_sign bool G/A Sign cert with certbot?
certbot_email string G/A Certbot notification email address
certbot_options string G/A Certbot extra command line options

DNS parameters configure DNSMasq DNS server, including enable switch, listen port, and dynamic DNS records.

Parameter Type Level Description
dns_enabled bool G/I Setup dnsmasq on this infra node?
dns_port port G DNS server listen port, default 53
dns_records string[] G Dynamic DNS records resolved by dnsmasq

VICTORIA parameters configure the VictoriaMetrics/Logs/Traces observability stack, including enable switches, ports, and data retention policies.

Parameter Type Level Description
vmetrics_enabled bool G/I Enable VictoriaMetrics on this infra node?
vmetrics_clean bool G/A Clean VictoriaMetrics data during init?
vmetrics_port port G VictoriaMetrics listen port, default 8428
vmetrics_scrape_interval interval G Global scrape interval, default 10s
vmetrics_scrape_timeout interval G Global scrape timeout, default 8s
vmetrics_options arg G VictoriaMetrics extra CLI options
vlogs_enabled bool G/I Enable VictoriaLogs on this infra node?
vlogs_clean bool G/A Clean VictoriaLogs data during init?
vlogs_port port G VictoriaLogs listen port, default 9428
vlogs_options arg G VictoriaLogs extra CLI options
vtraces_enabled bool G/I Enable VictoriaTraces on this infra node?
vtraces_clean bool G/A Clean VictoriaTraces data during init?
vtraces_port port G VictoriaTraces listen port, default 10428
vtraces_options arg G VictoriaTraces extra CLI options
vmalert_enabled bool G/I Enable VMAlert on this infra node?
vmalert_port port G VMAlert listen port, default 8880
vmalert_options arg G VMAlert extra CLI options

PROMETHEUS parameters configure Alertmanager and Blackbox Exporter, providing alert management and network probing capabilities.

Parameter Type Level Description
blackbox_enabled bool G/I Setup blackbox_exporter on this infra node?
blackbox_port port G blackbox_exporter listen port, default 9115
blackbox_options arg G blackbox_exporter extra CLI options
alertmanager_enabled bool G/I Setup alertmanager on this infra node?
alertmanager_port port G AlertManager listen port, default 9059
alertmanager_options arg G alertmanager extra CLI options
exporter_metrics_path path G Exporter metrics path, default /metrics

GRAFANA parameters configure the Grafana visualization platform, including enable switch, port, admin credentials, and data source configuration.

Parameter Type Level Description
grafana_enabled bool G/I Enable Grafana on this infra node?
grafana_port port G Grafana listen port, default 3000
grafana_clean bool G/A Clean Grafana data during init?
grafana_admin_username username G Grafana admin username, default admin
grafana_admin_password password G Grafana admin password, default pigsty
grafana_auth_proxy bool G Enable Grafana auth proxy?
grafana_pgurl url G External PostgreSQL URL for Grafana persistence
grafana_view_password password G Grafana metadb PG datasource password

META

This section defines Pigsty deployment metadata: version string, admin node IP address, repository mirror region, default language, and HTTP(S) proxy for downloading packages.

version: v4.5.0                   # pigsty version string
admin_ip: 10.10.10.10             # admin node ip address
region: default                   # upstream mirror region: default,china,europe
language: en                      # default language: en or zh
proxy_env:                        # global proxy env when downloading packages
  no_proxy: "localhost,127.0.0.1,10.0.0.0/8,192.168.0.0/16,*.pigsty,*.aliyun.com,mirrors.*,*.myqcloud.com,*.tsinghua.edu.cn"
  # http_proxy:  # set your proxy here: e.g http://user:[email protected]
  # https_proxy: # set your proxy here: e.g http://user:[email protected]
  # all_proxy:   # set your proxy here: e.g http://user:[email protected]

version

name: version, type: string, level: G

Pigsty version string. The current source default is v4.5.0.

Pigsty uses this version string internally for feature control and content rendering. Do not modify this parameter arbitrarily.

Pigsty uses semantic versioning, and the version string typically starts with the character v, e.g., v4.5.0.

admin_ip

name: admin_ip, type: ip, level: G

Admin node IP address, default is the placeholder IP address: 10.10.10.10

The node specified by this parameter will be treated as the admin node, typically pointing to the first node where Pigsty is installed, i.e., the control node.

The default value 10.10.10.10 is a placeholder that will be replaced with the actual admin node IP address during configure.

Many parameters reference this parameter, such as:

In these parameters, the string ${admin_ip} will be replaced with the actual value of admin_ip. Using this mechanism, you can specify different admin nodes for different nodes.

region

name: region, type: enum, level: G

Upstream mirror region, available options: default, china, europe, default is default

If a region other than default is set, and there’s a corresponding entry in repo_upstream with a matching baseurl, it will be used instead of the default baseurl.

For example, if your region is set to china, Pigsty will attempt to use Chinese mirror sites to accelerate downloads. If an upstream repository doesn’t have a corresponding China region mirror, the default upstream mirror site will be used instead. Additionally, URLs defined in repo_url_packages will be replaced from repo.pigsty.io to repo.pigsty.cc to use domestic mirrors.

language

name: language, type: enum, level: G

Default language setting, options are en (English) or zh (Chinese), default is en.

This parameter affects the language preference of some Pigsty-generated configurations and content, such as the initial language setting of Grafana dashboards.

If you are a Chinese user, it is recommended to set this parameter to zh for a better Chinese support experience.

proxy_env

name: proxy_env, type: dict, level: G

Global proxy environment variables used when downloading packages, default value specifies no_proxy, which is the list of addresses that should not use a proxy:

proxy_env:
  no_proxy: "localhost,127.0.0.1,10.0.0.0/8,192.168.0.0/16,*.pigsty,*.aliyun.com,mirrors.*,*.myqcloud.com,*.tsinghua.edu.cn"
  #http_proxy: 'http://username:[email protected]'
  #https_proxy: 'http://username:[email protected]'
  #all_proxy: 'http://username:[email protected]'

When installing from the Internet in mainland China, certain packages may be blocked. You can use a proxy to solve this problem.

Note that if the Docker module is used, the proxy server configuration here will also be written to the Docker Daemon configuration file.

Note that if the -x parameter is specified during ./configure, the proxy configuration information in the current environment will be automatically filled into the generated pigsty.yaml file.

CA

Pigsty uses self-signed CA certificates to support advanced security features such as HTTPS access, PostgreSQL SSL connections, etc.

ca_create: true                   # allow creation if CA private key is missing? default true
ca_cn: pigsty-ca                  # CA CN name, fixed as pigsty-ca
cert_validity: 7300d              # certificate validity, default 20 years

ca_create

name: ca_create, type: bool, level: G

Allow CA creation when the private key is missing? The default is true.

When set to true, Pigsty creates a new CA private key if files/pki/ca/ca.key is absent. If ca.crt is absent, it uses the existing or newly created private key to issue the CA certificate.

If you already have a CA public-private key pair, you can copy them to the files/pki/ca directory:

  • files/pki/ca/ca.crt: CA public key certificate
  • files/pki/ca/ca.key: CA private key file

Pigsty reuses an existing CA key pair. If the private key is absent and this parameter is false, deployment stops with an error. If only ca.crt is missing, Pigsty still reissues it with the existing private key. Always back up and restore the matching ca.key and ca.crt together to avoid a certificate/key mismatch.

Be sure to retain and backup the newly generated CA private key file during deployment, as it is crucial for issuing new certificates later.

Note

Pigsty v3.x used the ca_method parameter (create, recreate, or copy); v4.x simplifies it to the boolean ca_create.

ca_cn

name: ca_cn, type: string, level: G

CA CN (Common Name), fixed as pigsty-ca, not recommended to modify.

You can use the following command to view the Pigsty CA certificate details on a node:

openssl x509 -text -in /etc/pki/ca.crt

cert_validity

name: cert_validity, type: interval, level: G

Certificate validity period for issued certificates, default is 20 years, sufficient for most scenarios. Default value: 7300d

This parameter affects the validity of all certificates issued by the Pigsty CA, including:

  • PostgreSQL server certificates
  • Patroni API certificates
  • etcd server/client certificates
  • Other internal service certificates

Note: The validity of HTTPS certificates used by Nginx is controlled separately by nginx_cert_validity, because modern browsers have stricter requirements for website certificate validity (maximum 397 days).

INFRA_ID

Infrastructure identity and portal definition.

#infra_seq: 1                     # infra node sequence, REQUIRED identity parameter
infra_portal:                     # infrastructure services exposed via Nginx portal
  home : { domain: i.pigsty }     # default home server definition
infra_data: /data/infra           # infrastructure default data directory
infra_services: [...]             # built-in home navigation entries
infra_extra_services: []          # additional home navigation entries

infra_seq

name: infra_seq, type: int, level: I

Infrastructure node sequence number, REQUIRED identity parameter that must be explicitly specified on infrastructure nodes, so no default value is provided.

This parameter is used to uniquely identify each node in multi-infrastructure node deployments, typically using positive integers starting from 1.

Example configuration:

infra:
  hosts:
    10.10.10.10: { infra_seq: 1 }
    10.10.10.11: { infra_seq: 2 }

infra_portal

name: infra_portal, type: dict, level: G

Infrastructure services exposed via Nginx portal. The v4.x default value is very concise:

infra_portal:
  home : { domain: i.pigsty }     # default home server definition

Pigsty will automatically configure the corresponding reverse proxies based on the actually enabled components. Users typically only need to define the home domain name.

Each record consists of a Key and a Value dictionary, where name is the key representing the component name, and the value is an object that can configure the following parameters:

  • name: REQUIRED, specifies the name of the Nginx server
    • Default record: home is a fixed name, please do not modify it.
    • Used as part of the Nginx configuration file name, corresponding to: /etc/nginx/conf.d/<name>.conf
    • Nginx servers without a domain field will not generate configuration files but will be used as references.
  • domain: OPTIONAL, when the service needs to be exposed via Nginx, this is a REQUIRED field specifying the domain name to use
    • In Pigsty self-signed Nginx HTTPS certificates, the domain will be added to the SAN field of the Nginx SSL certificate
    • Pigsty web page cross-references will use the default domain name here
  • endpoint: Usually used as an alternative to path, specifies the upstream server address. Setting endpoint indicates this is a reverse proxy server
    • ${admin_ip} can be used as a placeholder in the configuration and will be dynamically replaced with admin_ip during deployment
    • Default reverse proxy servers use endpoint.conf as the configuration template
    • Reverse proxy servers can also configure websocket and schema parameters
  • path: Usually used as an alternative to endpoint, specifies the local file server path. Setting path indicates this is a local web server
    • Local web servers use path.conf as the configuration template
    • Local web servers can also configure the index parameter to enable file index pages
  • certbot: Certbot certificate name; if configured, Certbot will be used to apply for certificates
    • If multiple servers specify the same certbot, Pigsty will merge certificate applications; the final certificate name will be this certbot value
  • cert: Certificate file path; if configured, will override the default certificate path
  • key: Certificate key file path; if configured, will override the default certificate key path
  • websocket: Whether to enable WebSocket support
    • Only reverse proxy servers can configure this parameter; if enabled, upstream WebSocket connections will be allowed
  • schema: Protocol used by the upstream server; if configured, will override the default protocol
    • Default is http; if configured as https, it will force HTTPS connections to the upstream server
  • index: Whether to enable file index pages
    • Only local web servers can configure this parameter; if enabled, autoindex configuration will be enabled to automatically generate directory index pages
  • log: Nginx log file path
    • If specified, access logs will be written to this file; otherwise, the default log file will be used based on server type
    • Reverse proxy servers use /var/log/nginx/<name>.log as the default log file path
    • Local web servers use the default Access log
  • conf: Nginx configuration file path
    • Explicitly specifies the configuration template file to use, located in roles/infra/templates/nginx or templates/nginx directory
    • If this parameter is not specified, the default configuration template will be used
  • config: Nginx configuration code block
    • Configuration text directly injected into the Nginx Server configuration block
  • enforce_https: Redirect HTTP to HTTPS
    • Global configuration can be specified via nginx_sslmode: enforce
    • This configuration does not affect the default home server, which will always listen on both ports 80 and 443 to ensure compatibility

infra_data

name: infra_data, type: path, level: G

Infrastructure data directory, default value is /data/infra.

This directory is used to store data files for infrastructure components, including:

  • VictoriaMetrics time series database data
  • VictoriaLogs log data
  • VictoriaTraces trace data
  • Other infrastructure component persistent data

It is recommended to place this directory on a separate data disk for easier management and expansion.

infra_services

name: infra_services, type: service[], level: G

Built-in navigation entries on the Pigsty home page. Current defaults include Metrics, Logs, Traces, Monitor Targets, Alert Rules, Alert Manager, CA Certificate, Software Repo, and Explain Visualizer.

Each item can use name, url, desc, and icon, plus the Chinese display fields name_cn and desc_cn. This parameter replaces the entire default list; use infra_extra_services if you only want to append entries.

infra_extra_services

name: infra_extra_services, type: service[], level: G

Navigation entries appended to infra_services. The default is [], and each item uses the same structure as infra_services. For example:

infra_extra_services:
  - { name: My Service, url: 'https://example.com', desc: 'External Service', icon: 'database' }

REPO

This section is about local software repository configuration. Pigsty enables a local software repository (APT/YUM) on infrastructure nodes by default.

During initialization, Pigsty downloads all packages and their dependencies (specified by repo_packages) from the Internet upstream repository (specified by repo_upstream) to {{ nginx_home }} / {{ repo_name }} (default /www/pigsty). The total size of all software and dependencies is approximately 1GB.

The current package candidate is SOW 0.3.0, and the source uses SOW to generate RPM/APT metadata. After a successful build, repo_complete is both a SHA-256 manifest and a completion marker. When it is present, Pigsty skips downloading and rebuilding by default and uses the existing repository. Force a rebuild with ./infra.yml -t repo_build -e repo_build=true.

Both repo_create and cache_create invoke sow create --pigsty directly, with no fallback to createrepo_c or dpkg-scanpackages. If an older offline bundle or local repository does not contain SOW, refresh the media or install SOW from the Pigsty INFRA repository first.

If some packages download too slowly, you can set a download proxy using the proxy_env configuration to complete the initial download, or directly download the pre-packaged offline package, which is essentially a local software repository built on the same operating system.

repo_enabled: true                # create local repo on this infra node?
repo_home: /www                   # repo home directory, default /www
repo_name: pigsty                 # repo name, default pigsty
repo_endpoint: http://${admin_ip}:80 # repo access endpoint
repo_remove: true                 # remove existing upstream repo definitions
repo_modules: infra,node,pgsql    # enabled upstream repo modules
#repo_upstream: []                # upstream repo definitions (inherited from OS variables)
#repo_packages: []                # packages to download (inherited from OS variables)
#repo_extra_packages: []          # extra packages to download
repo_url_packages: []             # extra packages downloaded via URL

repo_enabled

name: repo_enabled, type: bool, level: G/I

Create a local software repository on this infrastructure node? Default is true, meaning all Infra nodes will set up a local software repository.

If you have multiple infrastructure nodes, you can keep only 1-2 nodes as software repositories; other nodes can set this parameter to false to avoid duplicate software download builds.

repo_home

name: repo_home, type: path, level: G

Local software repository home directory, defaults to Nginx’s root directory: /www.

On a fresh installation, if this path does not exist, the role creates a symlink to nginx_data. Existing directories and symlinks are preserved unchanged. Modification is generally discouraged; if required, keep it consistent with nginx_home.

repo_name

name: repo_name, type: string, level: G

Local repository name, default is pigsty. Changing this repository name is not recommended.

The final repository path is {{ repo_home }}/{{ repo_name }}, defaulting to /www/pigsty.

repo_endpoint

name: repo_endpoint, type: url, level: G

Endpoint used by other nodes to access this repository, default value: http://${admin_ip}:80.

Pigsty starts Nginx on infrastructure nodes at ports 80/443 by default, providing local software repository (static files) service.

If you modify nginx_port or nginx_ssl_port, or use a different infrastructure node from the control node, adjust this parameter accordingly.

If you use a domain name, you can add resolution in node_default_etc_hosts, node_etc_hosts, or dns_records.

repo_remove

name: repo_remove, type: bool, level: G/A

Remove existing upstream repository definitions when building the local repository? Default value: true.

When this parameter is enabled, all existing repository files in /etc/yum.repos.d will be moved and backed up to /etc/yum.repos.d/backup. On Debian systems, /etc/apt/sources.list and /etc/apt/sources.list.d are removed and backed up to /etc/apt/backup.

Since existing OS sources have uncontrollable content, using Pigsty-validated upstream software sources can improve the success rate and speed of downloading packages from the Internet.

In certain situations (e.g., your OS is some EL/Deb compatible variant that uses private sources for many packages), you may need to keep existing upstream repository definitions. In such cases, set this parameter to false.

repo_modules

name: repo_modules, type: string, level: G/A

Which upstream repository modules will be added to the local software source, default value: infra,node,pgsql

When Pigsty attempts to add upstream repositories, it filters entries in repo_upstream based on this parameter’s value. Only entries whose module field matches this parameter’s value will be added to the local software source.

During the build stage, infra is automatically added to the effective module list so SOW can be installed. This bootstrap dependency is restored even if a user overrides repo_modules without infra.

Modules are comma-separated. Available module lists can be found in the repo_upstream definitions; common modules include:

  • local: Local Pigsty repository
  • infra: Infrastructure packages (Nginx, Docker, etc.)
  • node: OS base packages
  • pgsql: PostgreSQL-related packages
  • extra: Extra PostgreSQL extensions
  • docker: Docker-related
  • redis: Redis-related
  • mongo: MongoDB-related
  • mysql: MySQL-related
  • etc…

repo_upstream

name: repo_upstream, type: upstream[], level: G

Where to download upstream packages when building the local repository? This parameter has no default value. If not explicitly specified by the user in the configuration file, it will be loaded from the repo_upstream_default variable defined in roles/node_id/vars based on the current node’s OS family.

Pigsty provides complete upstream repository definitions for currently supported OS versions (EL 8/9/10, Debian 12/13, Ubuntu 22/24/26), including:

  • OS base repositories (BaseOS, AppStream, EPEL, etc.)
  • PostgreSQL official PGDG repository
  • Pigsty extension repository
  • Various third-party software repositories (Docker, Nginx, Grafana, etc.)

Each upstream repository definition contains the following fields:

- name: pigsty-pgsql              # repository name
  description: 'Pigsty PGSQL'     # repository description
  module: pgsql                   # module it belongs to
  releases: [8,9,10]              # supported OS versions
  arch: [x86_64, aarch64]         # supported CPU architectures
  baseurl:                        # repository URL, configured by region
    default: 'https://repo.pigsty.io/yum/pgsql/el$releasever.$basearch'
    china: 'https://repo.pigsty.cc/yum/pgsql/el$releasever.$basearch'
  # meta: { module_hotfixes: 1 } # enable only when explicitly replacing an EL module stream

RPM upstream repositories retain native DNF module filtering by default. Set meta.module_hotfixes only on a repository that must actually replace an EL module stream. Pigsty’s aggregated local repository itself is consumed with module_hotfixes=1, but Pigsty no longer generates fake modules.yaml / ModuleMD metadata.

Users typically don’t need to modify this parameter unless they have special repository requirements. For detailed repository definitions, refer to the configuration files for corresponding operating systems in the roles/node_id/vars/ directory.

repo_packages

name: repo_packages, type: string[], level: G

String array type, where each line is a space-separated list of software packages, specifying packages (and their dependencies) to download using repotrack or apt download.

This parameter has no default value, meaning its default state is undefined. If not explicitly defined, Pigsty will load the default from the repo_packages_default variable defined in roles/node_id/vars:

[ node-bootstrap, infra-package, infra-addons, node-package1, node-package2, node-package3, pgsql-utility, extra-modules ]

Each element in this parameter will be translated according to the package_map in the above files, based on the specific OS distro major version. For example, on EL systems it translates to:

node-bootstrap:          "ansible python3 python3-requests python3-jmespath python3-cryptography dnf-utils sow sshpass"
infra-package:           "nginx dnsmasq etcd haproxy vip-manager node-exporter keepalived-exporter pg-exporter pgbackrest-exporter redis-exporter redis valkey silo mcli sow pig"
infra-addons:            "grafana grafana-plugins grafana-victoriametrics-ds grafana-victorialogs-ds victoria-metrics victoria-logs victoria-traces vlogscli vmutils vector alertmanager"

As a convention, repo_packages typically includes packages unrelated to the PostgreSQL major version (such as Infra, Node, and PGDG Common parts), while PostgreSQL major version-related packages (kernel, extensions) are usually specified in repo_extra_packages to facilitate switching PG major versions.

repo_extra_packages

name: repo_extra_packages, type: string[], level: G/C/I

Used to specify additional packages to download without modifying repo_packages (typically PG major version-related packages), default value is an empty list.

If not explicitly defined, Pigsty will load the default from the repo_extra_packages_default variable defined in roles/node_id/vars:

[ pgsql-main ]

Elements in this parameter undergo package name translation, where $v will be replaced with pg_version, i.e., the current PG major version (default 18).

The pgsql-main here translates on EL systems to:

postgresql$v postgresql$v-server postgresql$v-libs postgresql$v-contrib postgresql$v-plperl postgresql$v-plpython3 postgresql$v-pltcl postgresql$v-llvmjit pg_repack_$v* wal2json_$v* pgvector_$v*

Users can typically specify PostgreSQL major version-related packages here without affecting the other PG version-independent packages defined in repo_packages.

repo_url_packages

name: repo_url_packages, type: object[] | string[], level: G

Packages downloaded directly from the Internet using URLs, default is an empty array: []

You can use URL strings directly as array elements in this parameter, or use object structures to explicitly specify URLs and filenames.

Note that this parameter is affected by the region variable. If you’re in mainland China, Pigsty will automatically replace URLs, changing repo.pigsty.io to repo.pigsty.cc.

INFRA_PACKAGE

These RPM/DEB packages are installed only on INFRA nodes.

infra_packages

name: infra_packages, type: string[], level: G

String array type, where each line is a space-separated list of software packages, specifying packages to install on Infra nodes.

This parameter has no single cross-platform default. If not explicitly specified, Pigsty loads infra_packages_default from the platform-specific file under roles/node_id/vars, based on the operating-system version and CPU architecture.

For example, the current mapping for EL 9 x86_64 is:

infra_packages_default:
  - grafana,grafana-plugins,grafana-victorialogs-ds,grafana-victoriametrics-ds,victoria-metrics,victoria-logs,victoria-traces,vmutils,vlogscli,alertmanager
  - node-exporter,blackbox-exporter,nginx-exporter,pg-exporter,pev2,nginx,dnsmasq,ansible,etcd,python3-requests,redis,mcli,restic,certbot,python3-certbot-nginx

The current mapping for Debian 13 x86_64 is:

infra_packages_default:
  - grafana,grafana-plugins,grafana-victorialogs-ds,grafana-victoriametrics-ds,victoria-metrics,victoria-logs,victoria-traces,vmutils,vlogscli,alertmanager
  - node-exporter,blackbox-exporter,nginx-exporter,pg-exporter,pev2,nginx,dnsmasq,ansible,etcd,python3-requests,redis,mcli,restic,certbot,python3-certbot-nginx
Note

Pigsty v4.x replaces Prometheus and Loki with the VictoriaMetrics suite, so its package list differs significantly from v3.x.

NGINX

Pigsty proxies all web service access through Nginx: Home Page, Grafana, VictoriaMetrics, etc., as well as other optional tools like PGWeb, Jupyter Lab, Pgadmin, Bytebase, and static resources and reports like pev, schemaspy, and pgbadger.

Most importantly, Nginx also serves as the web server for the local software repository (Yum/Apt), used to store and distribute Pigsty packages.

nginx_enabled: true               # enable Nginx on this infra node?
nginx_clean: false                # clean existing Nginx config during init?
nginx_exporter_enabled: true      # enable nginx_exporter?
nginx_exporter_port: 9113         # nginx_exporter listen port
nginx_sslmode: enable             # SSL mode: disable,enable,enforce
nginx_cert_validity: 397d         # self-signed cert validity
nginx_home: /www                  # Nginx content directory (symlink)
nginx_data: /data/nginx           # Nginx actual data directory
nginx_users: {}                   # basic auth users dictionary
nginx_port: 80                    # HTTP port
nginx_ssl_port: 443               # HTTPS port
certbot_sign: false               # sign cert with certbot?
certbot_email: [email protected]     # certbot email
certbot_options: ''               # certbot extra options

nginx_enabled

name: nginx_enabled, type: bool, level: G/I

Enable Nginx on this Infra node? Default value: true.

Nginx is a core component of Pigsty infrastructure, responsible for:

  • Providing local software repository service
  • Reverse proxying Grafana, VictoriaMetrics, and other web services
  • Hosting static files and reports

nginx_clean

name: nginx_clean, type: bool, level: G/A

Clean existing Nginx configuration during initialization? Default value: false.

When set to true, all existing configuration files under /etc/nginx/conf.d/ will be deleted during Nginx initialization, ensuring a clean start.

If you’re deploying for the first time or want to completely rebuild Nginx configuration, you can set this parameter to true.

nginx_exporter_enabled

name: nginx_exporter_enabled, type: bool, level: G/I

Enable nginx_exporter on this infrastructure node? Default value: true.

If this option is disabled, the /nginx health check stub will also be disabled. Consider disabling this when your Nginx version doesn’t support this feature.

nginx_exporter_port

name: nginx_exporter_port, type: port, level: G

nginx_exporter listen port, default value is 9113.

nginx_exporter is used to collect Nginx operational metrics for VictoriaMetrics to scrape and monitor.

nginx_sslmode

name: nginx_sslmode, type: enum, level: G

Nginx SSL operating mode. Three options: disable, enable, enforce, default value is enable, meaning SSL is enabled but not enforced.

  • disable: Only listen on the port specified by nginx_port to serve HTTP requests.
  • enable: Also listen on the port specified by nginx_ssl_port to serve HTTPS requests.
  • enforce: All links will be rendered to use https:// by default
    • Also redirect port 80 to port 443 for non-default servers in infra_portal

nginx_cert_validity

name: nginx_cert_validity, type: duration, level: G

Nginx self-signed certificate validity, default value is 397d (approximately 13 months).

Modern browsers require website certificate validity to be at most 397 days, hence this default value. Setting a longer validity is not recommended, as browsers may refuse to trust such certificates.

nginx_home

name: nginx_home, type: path, level: G

Nginx server static content directory, default: /www

This is a symlink that actually points to the nginx_data directory. This directory contains static resources and software repository files.

It’s best not to modify this parameter arbitrarily. If modified, it should be consistent with the repo_home parameter.

nginx_data

name: nginx_data, type: path, level: G

Nginx actual data directory, default is /data/nginx.

This is the actual storage location for Nginx static files; nginx_home is a symlink pointing to this directory.

It’s recommended to place this directory on a data disk for easier management of large package files.

nginx_users

name: nginx_users, type: dict, level: G

Nginx Basic Authentication user dictionary, default is an empty dictionary {}.

Format is { username: password } key-value pairs, for example:

nginx_users:
  admin: pigsty
  viewer: readonly

These users can be used to protect certain Nginx endpoints that require authentication.

nginx_port

name: nginx_port, type: port, level: G

Nginx default listening port (serving HTTP), default is port 80. It’s best not to modify this parameter.

When your server’s port 80 is occupied, you can consider using another port, but you need to also modify repo_endpoint and keep node_repo_local_urls consistent with the port used here.

nginx_ssl_port

name: nginx_ssl_port, type: port, level: G

Nginx SSL default listening port, default is 443. It’s best not to modify this parameter.

certbot_sign

name: certbot_sign, type: bool, level: G/A

Use certbot to sign Nginx certificates during installation? Default value is false.

When set to true, Pigsty will use certbot to automatically apply for free SSL certificates from Let’s Encrypt during the execution of infra.yml and deploy.yml playbooks (in the nginx role).

For domains defined in infra_portal, if a certbot parameter is defined, Pigsty will use certbot to apply for a certificate for that domain. The certificate name will be the value of the certbot parameter. If multiple servers/domains specify the same certbot parameter, Pigsty will merge and apply for certificates for these domains, using the certbot parameter value as the certificate name.

Enabling this option requires:

  • The current node can be accessed through a public domain name, and DNS resolution is correctly pointed to the current node’s public IP
  • The current node can access the Let’s Encrypt API interface

This option is disabled by default. You can manually execute the make cert command after installation, which actually calls the rendered /etc/nginx/sign-cert script to update or apply for certificates using certbot.

certbot_email

name: certbot_email, type: string, level: G/A

Email address for receiving certificate expiration reminder emails, default value is [email protected].

When certbot_sign is set to true, it’s recommended to provide this parameter. Let’s Encrypt will send reminder emails to this address when certificates are about to expire.

certbot_options

name: certbot_options, type: string, level: G/A

Additional configuration parameters passed to certbot, default value is an empty string.

You can pass additional command-line options to certbot through this parameter, for example --dry-run, which makes certbot perform a preview and test without actually applying for certificates.

DNS

Pigsty enables DNSMASQ on Infra nodes by default to resolve auxiliary names such as i.pigsty, m.pigsty, and api.pigsty; sss.pigsty can optionally provide the Silo endpoint.

Resolution records are stored in the /etc/dnsmasq.d/pigsty/default file on Infra nodes. To use this DNS server, you must add nameserver <ip> to /etc/resolv.conf. The node_dns_servers parameter handles this.

dns_enabled: true                 # setup dnsmasq on this infra node?
dns_port: 53                      # DNS server listen port
dns_records:                      # dynamic DNS records
  - "${admin_ip} i.pigsty"
  - "${admin_ip} m.pigsty supa.pigsty api.pigsty adm.pigsty cli.pigsty ddl.pigsty"

dns_enabled

name: dns_enabled, type: bool, level: G/I

Enable DNSMASQ service on this Infra node? Default value: true.

If you don’t want to use the default DNS server (e.g., you already have an external DNS server, or your provider doesn’t allow you to use a DNS server), you can set this value to false to disable it, and use node_default_etc_hosts and node_etc_hosts static resolution records instead.

dns_port

name: dns_port, type: port, level: G

DNSMASQ default listening port, default is 53. It’s not recommended to modify the default DNS service port.

dns_records

name: dns_records, type: string[], level: G

Dynamic DNS records resolved by dnsmasq, generally used to resolve auxiliary domain names to the admin node. These records are written to the /etc/dnsmasq.d/pigsty/default file on infrastructure nodes.

v4.x default value:

dns_records:
  - "${admin_ip} i.pigsty"
  - "${admin_ip} m.pigsty supa.pigsty api.pigsty adm.pigsty cli.pigsty ddl.pigsty"

The ${admin_ip} placeholder is used here and will be replaced with the actual admin_ip value during deployment.

Common domain name purposes:

  • i.pigsty: Pigsty home page
  • m.pigsty: Commonly used for the Silo console (optional)
  • p.pigsty: Commonly used for the VictoriaMetrics Web UI when explicitly configured in infra_portal
  • api.pigsty: API service
  • adm.pigsty: Admin service
  • Others customized based on actual deployment needs

VICTORIA

Pigsty v4.x uses the VictoriaMetrics suite to replace Prometheus and Loki, providing a superior observability solution:

  • VictoriaMetrics: Replaces Prometheus as the time series database for storing monitoring metrics
  • VictoriaLogs: Replaces Loki as the log aggregation storage
  • VictoriaTraces: Distributed trace storage
  • VMAlert: Replaces Prometheus Alerting for alert rule evaluation
vmetrics_enabled: true            # enable VictoriaMetrics?
vmetrics_clean: false             # clean data during init?
vmetrics_port: 8428               # listen port
vmetrics_scrape_interval: 10s     # global scrape interval
vmetrics_scrape_timeout: 8s       # global scrape timeout
vmetrics_options: >-
  -retentionPeriod=15d
  -promscrape.fileSDCheckInterval=5s
vlogs_enabled: true               # enable VictoriaLogs?
vlogs_clean: false                # clean data during init?
vlogs_port: 9428                  # listen port
vlogs_options: >-
  -retentionPeriod=15d
  -retention.maxDiskSpaceUsageBytes=50GiB
  -insert.maxLineSizeBytes=1MB
  -search.maxQueryDuration=120s
vtraces_enabled: true             # enable VictoriaTraces?
vtraces_clean: false              # clean data during init?
vtraces_port: 10428               # listen port
vtraces_options: >-
  -retentionPeriod=15d
  -retention.maxDiskSpaceUsageBytes=50GiB
vmalert_enabled: true             # enable VMAlert?
vmalert_port: 8880                # listen port
vmalert_options: ''               # extra CLI options

vmetrics_enabled

name: vmetrics_enabled, type: bool, level: G/I

Enable VictoriaMetrics on this Infra node? Default value is true.

VictoriaMetrics is the core monitoring component in Pigsty v4.x, replacing Prometheus as the time series database, responsible for:

  • Scraping monitoring metrics from various exporters
  • Storing time series data
  • Providing PromQL-compatible query interface
  • Supporting Grafana data sources

vmetrics_clean

name: vmetrics_clean, type: bool, level: G/A

Clean existing VictoriaMetrics data during initialization? Default value is false.

When set to true, existing time series data will be deleted during initialization. Use this option carefully unless you’re sure you want to rebuild monitoring data.

vmetrics_port

name: vmetrics_port, type: port, level: G

VictoriaMetrics listen port, default value is 8428.

This port is used for:

  • HTTP API access
  • Web UI access
  • Prometheus-compatible remote write/read
  • Grafana data source connections

vmetrics_scrape_interval

name: vmetrics_scrape_interval, type: interval, level: G

VictoriaMetrics global metrics scrape interval, default value is 10s.

In production environments, 10-30 seconds is a suitable scrape interval. If you need finer monitoring data granularity, you can adjust this parameter, but it will increase storage and CPU overhead.

vmetrics_scrape_timeout

name: vmetrics_scrape_timeout, type: interval, level: G

VictoriaMetrics global scrape timeout, default is 8s.

Setting a scrape timeout can effectively prevent avalanches caused by monitoring system queries. The principle is that this parameter must be less than and close to vmetrics_scrape_interval to ensure each scrape duration doesn’t exceed the scrape interval.

vmetrics_options

name: vmetrics_options, type: arg, level: G

VictoriaMetrics extra command line options, default value:

vmetrics_options: >-
  -retentionPeriod=15d
  -promscrape.fileSDCheckInterval=5s

Common parameter descriptions:

  • -retentionPeriod=15d: Data retention period, default 15 days
  • -promscrape.fileSDCheckInterval=5s: File service discovery refresh interval

You can add other VictoriaMetrics-supported parameters as needed.

vlogs_enabled

name: vlogs_enabled, type: bool, level: G/I

Enable VictoriaLogs on this Infra node? Default value is true.

VictoriaLogs replaces Loki as the log aggregation storage, responsible for:

  • Receiving log data from Vector
  • Storing and indexing logs
  • Providing log query interface
  • Supporting Grafana VictoriaLogs data source

vlogs_clean

name: vlogs_clean, type: bool, level: G/A

Clean existing VictoriaLogs data during initialization? Default value is false.

vlogs_port

name: vlogs_port, type: port, level: G

VictoriaLogs listen port, default value is 9428.

vlogs_options

name: vlogs_options, type: arg, level: G

VictoriaLogs extra command line options, default value:

vlogs_options: >-
  -retentionPeriod=15d
  -retention.maxDiskSpaceUsageBytes=50GiB
  -insert.maxLineSizeBytes=1MB
  -search.maxQueryDuration=120s

Common parameter descriptions:

  • -retentionPeriod=15d: Log retention period, default 15 days
  • -retention.maxDiskSpaceUsageBytes=50GiB: Maximum disk usage
  • -insert.maxLineSizeBytes=1MB: Maximum single log line size
  • -search.maxQueryDuration=120s: Maximum query execution time

vtraces_enabled

name: vtraces_enabled, type: bool, level: G/I

Enable VictoriaTraces on this Infra node? Default value is true.

VictoriaTraces is used for distributed trace data storage and query, supporting Jaeger, Zipkin, and other trace protocols.

vtraces_clean

name: vtraces_clean, type: bool, level: G/A

Clean existing VictoriaTraces data during initialization? Default value is false.

vtraces_port

name: vtraces_port, type: port, level: G

VictoriaTraces listen port, default value is 10428.

vtraces_options

name: vtraces_options, type: arg, level: G

VictoriaTraces extra command line options, default value:

vtraces_options: >-
  -retentionPeriod=15d
  -retention.maxDiskSpaceUsageBytes=50GiB

vmalert_enabled

name: vmalert_enabled, type: bool, level: G/I

Enable VMAlert on this Infra node? Default value is true.

VMAlert is responsible for alert rule evaluation, replacing Prometheus Alerting functionality, working with Alertmanager.

vmalert_port

name: vmalert_port, type: port, level: G

VMAlert listen port, default value is 8880.

vmalert_options

name: vmalert_options, type: arg, level: G

VMAlert extra command line options, default value is an empty string.

PROMETHEUS

This section now primarily contains Blackbox Exporter and Alertmanager configuration.

Note

Pigsty v4.x uses VictoriaMetrics instead of Prometheus. The legacy prometheus_* and pushgateway_* parameters are no longer part of the current interface; use the vmetrics_* and vmalert_* parameters under VICTORIA for metric storage and rule evaluation.

blackbox_enabled: true            # enable blackbox_exporter?
blackbox_port: 9115               # blackbox_exporter listen port
blackbox_options: ''              # extra CLI options
alertmanager_enabled: true        # enable alertmanager?
alertmanager_port: 9059           # alertmanager listen port
alertmanager_options: ''          # extra CLI options
exporter_metrics_path: /metrics   # exporter metrics path

blackbox_enabled

name: blackbox_enabled, type: bool, level: G/I

Enable BlackboxExporter on this Infra node? Default value is true.

BlackboxExporter sends ICMP packets to node IP addresses, VIP addresses, and PostgreSQL VIP addresses to test network connectivity. It can also perform HTTP, TCP, DNS, and other probes.

blackbox_port

name: blackbox_port, type: port, level: G

Blackbox Exporter listen port, default value is 9115.

blackbox_options

name: blackbox_options, type: arg, level: G

BlackboxExporter extra command line options, default value: empty string.

alertmanager_enabled

name: alertmanager_enabled, type: bool, level: G/I

Enable AlertManager on this Infra node? Default value is true.

AlertManager is responsible for receiving alert notifications from VMAlert and performing alert grouping, inhibition, silencing, routing, and other processing.

alertmanager_port

name: alertmanager_port, type: port, level: G

AlertManager listen port, default value is 9059.

If you modify this port, ensure you update the alertmanager entry’s endpoint configuration in infra_portal accordingly (if defined).

alertmanager_options

name: alertmanager_options, type: arg, level: G

AlertManager extra command line options, default value: empty string.

exporter_metrics_path

name: exporter_metrics_path, type: path, level: G

HTTP endpoint path where monitoring exporters expose metrics, default: /metrics. Not recommended to modify this parameter.

This parameter defines the standard path for all exporters to expose monitoring metrics.

GRAFANA

Pigsty uses Grafana as the monitoring system frontend. It can also serve as a data analysis and visualization platform, or for low-code data application development and data application prototyping.

grafana_enabled: true             # enable Grafana?
grafana_port: 3000                # Grafana listen port
grafana_clean: false              # clean data during init?
grafana_admin_username: admin     # admin username
grafana_admin_password: pigsty    # admin password
grafana_auth_proxy: false         # enable auth proxy?
grafana_pgurl: ''                 # external PostgreSQL URL
grafana_view_password: DBUser.Viewer  # PG datasource password

grafana_enabled

name: grafana_enabled, type: bool, level: G/I

Enable Grafana on Infra node? Default value: true, meaning all infrastructure nodes will install and enable Grafana by default.

grafana_port

name: grafana_port, type: port, level: G

Grafana listen port, default value is 3000.

If you need to access Grafana directly (not through Nginx reverse proxy), you can use this port.

grafana_clean

name: grafana_clean, type: bool, level: G/A

Clean Grafana data files during initialization? Default: false.

This operation removes /var/lib/grafana/grafana.db, ensuring a fresh Grafana installation.

If you want to preserve existing Grafana configuration (such as dashboards, users, data sources, etc.), set this parameter to false.

grafana_admin_username

name: grafana_admin_username, type: username, level: G

Grafana admin username, default is admin.

grafana_admin_password

name: grafana_admin_password, type: password, level: G

Grafana admin password, default is pigsty.

IMPORTANT: Be sure to change this password parameter before deploying to production!

grafana_auth_proxy

name: grafana_auth_proxy, type: bool, level: G

Enable Grafana auth proxy? Default is false.

When enabled, Grafana will trust user identity information passed by the reverse proxy (Nginx), enabling single sign-on (SSO) functionality.

This is typically used for integration with external identity authentication systems.

grafana_pgurl

name: grafana_pgurl, type: url, level: G

External PostgreSQL database URL for Grafana persistence storage. Default is an empty string.

If specified, Grafana will use this PostgreSQL database instead of the default SQLite database to store its configuration data.

Format example: postgres://grafana:password@pg-meta:5432/grafana?sslmode=disable

This is useful for scenarios requiring Grafana high availability deployment or data persistence.

grafana_view_password

name: grafana_view_password, type: password, level: G

Read-only user password used by Grafana metadb PG data source, default is DBUser.Viewer.

This password is used for Grafana to connect to the PostgreSQL CMDB data source to query metadata in read-only mode.

9.3 - Playbook

How to use built-in Ansible playbooks to manage the INFRA module, with a quick reference for common commands.

Pigsty provides three playbooks related to the INFRA module:

  • deploy.yml: Deploy the NODE, INFRA, ETCD, MINIO, and PGSQL core modules on all nodes in one pass
  • infra.yml: Initialize Pigsty infrastructure on infra nodes
  • infra-rm.yml: Remove infrastructure components from infra nodes

deploy.yml

Deploy the NODE, INFRA, ETCD, MINIO, and PGSQL core modules on all nodes in one pass, resolving INFRA/NODE circular dependency issues.

This playbook interleaves subtasks from infra.yml and node.yml, completing deployment of the core components in the following order:

  1. id: Generate node and PostgreSQL identities
  2. ca: Create self-signed CA on localhost
  3. repo: Create local software repository on infra nodes
  4. node-init: Initialize nodes and HAProxy
  5. infra: Initialize Nginx, DNS, VictoriaMetrics, Grafana, etc.
  6. node-monitor: Initialize node-exporter, vector
  7. etcd: Initialize etcd (required for PostgreSQL HA)
  8. minio: Initialize Silo (optional)
  9. pgsql: Initialize PostgreSQL clusters and configure PostgreSQL monitoring

This playbook is equivalent to executing the following five playbooks sequentially:

./infra.yml -l infra    # Deploy infrastructure on infra group
./node.yml              # Initialize all nodes
./etcd.yml              # Initialize etcd cluster
./minio.yml             # Initialize MINIO (Silo) cluster (optional)
./pgsql.yml             # Initialize PostgreSQL clusters

deploy.yml does not currently deploy the Docker module. If Docker is required, set docker_enabled: true and run docker.yml separately.

infra.yml

Initialize the infrastructure module on Infra nodes defined in the infra group of your configuration file.

This playbook performs the following tasks:

  • Configures directories and environment variables on Infra nodes
  • Downloads and creates a local software repository to accelerate subsequent installations
  • Incorporates the current Infra node as a common node managed by Pigsty
  • Deploys infrastructure components (VictoriaMetrics/Logs/Traces, VMAlert, Grafana, Alertmanager, Blackbox Exporter, etc.)

Playbook notes:

  • This is an idempotent playbook - repeated execution will overwrite infrastructure components on Infra nodes
  • To preserve historical monitoring data, set vmetrics_clean, vlogs_clean, vtraces_clean to false beforehand
  • Unless grafana_clean is set to false, Grafana dashboards and configuration changes will be lost
  • When /www/pigsty/repo_complete exists, this playbook skips internet downloads; the file is the SHA-256 manifest and completion marker generated by SOW
  • Complete execution takes approximately 1-3 minutes, depending on machine configuration and network conditions

Available Tasks 

# ca: create self-signed CA on localhost files/pki
#   - ca_dir        : create CA directory
#   - ca_private    : generate ca private key: files/pki/ca/ca.key
#   - ca_cert       : signing ca cert: files/pki/ca/ca.crt
#
# id: generate node identity
#
# repo: bootstrap a local yum repo from internet or offline packages
#   - repo_dir      : create repo directory
#   - repo_check    : check repo exists
#   - repo_prepare  : use existing repo if exists
#   - repo_build    : build repo from upstream if not exists
#     - repo_upstream    : handle upstream repo files in /etc/yum.repos.d
#       - repo_remove    : remove existing repo file if repo_remove == true
#       - repo_add       : add upstream repo files to /etc/yum.repos.d
#     - repo_url_pkg     : download packages from internet defined by repo_url_packages
#     - repo_cache       : make upstream yum cache with yum makecache
#     - repo_boot_pkg    : install sow and dnf/yum download utilities
#     - repo_pkg         : download packages & dependencies from upstream repo
#     - repo_create      : atomically create RPM/APT metadata with sow create --pigsty
#     - repo_use         : add newly built repo into /etc/yum.repos.d
#   - repo_nginx    : launch a nginx for repo if no nginx is serving
#
# node/haproxy/monitor: setup infra node as a common node
#   - node_name, node_hosts, node_resolv, node_firewall, node_ca, node_repo, node_pkg
#   - node_feature, node_kernel, node_tune, node_sysctl, node_profile, node_ulimit
#   - node_data, node_admin, node_timezone, node_ntp, node_crontab, node_vip
#   - haproxy_install, haproxy_config, haproxy_launch, haproxy_reload
#   - haproxy_register, node_exporter, node_register, vector
#
# infra: setup infra components
#   - infra_user     : setup infra os user group
#   - infra_dir      : create infra data/config/runtime directories
#   - infra_env      : env_patroni, env_pg, env_pgadmin, env_etcd, env_pglog, env_var
#   - infra_pkg      : install infra packages
#   - infra_cert     : issue cert for infra components
#   - dns            : dns_config, dns_record, dns_launch
#   - nginx          : nginx_dir, nginx_config, nginx_cert, nginx_static, nginx_launch, nginx_certbot, nginx_reload, nginx_exporter
#   - victoria       : vmetrics/vlogs/vtraces clean, config & launch; vmalert_config, vmalert_launch
#   - alertmanager   : alertmanager_config, alertmanager_launch
#   - blackbox       : blackbox_config, blackbox_launch
#   - grafana        : grafana_clean, grafana_dir, grafana_config, grafana_launch, grafana_provision
#   - infra_register : add_metrics, add_logs, add_ds

infra-rm.yml

Remove Pigsty infrastructure from Infra nodes defined in the infra group of your configuration file.

Common subtasks include:

./infra-rm.yml               # Run all phases: deregister, stop, remove config/environment/data, and uninstall packages
./infra-rm.yml -t deregister # Only deregister monitoring targets, Grafana datasources, and Nginx log collection
./infra-rm.yml -t service    # Stop infrastructure services on INFRA
./infra-rm.yml -t config     # Remove INFRA configuration and systemd units
./infra-rm.yml -t env        # Remove admin environment files
./infra-rm.yml -t data       # Remove INFRA data
./infra-rm.yml -t package    # Uninstall INFRA packages
Full removal deletes data

infra-rm.yml has no deletion safeguard. Without tags, it runs every phase above. The data phase recursively removes infra_data (default: /data/infra), nginx_data (default: /data/nginx), nginx_home (default: /www), and /var/lib/grafana, including metrics, logs, traces, the software repository, and local Grafana data. Use the corresponding tag if you only want to stop services or deregister targets. Before a full run, back up everything that must survive and verify the exact infra target.

9.4 - Monitoring

How to perform self-monitoring of infrastructure in Pigsty?

This document describes monitoring dashboards and alert rules for the INFRA module in Pigsty.

Dashboards

Pigsty provides the following monitoring dashboards for the Infra module:

Dashboard Description
Pigsty Home Pigsty monitoring system homepage
INFRA Overview Pigsty infrastructure self-monitoring overview
Nginx Instance Nginx metrics and logs
Grafana Instance Grafana metrics and logs
VictoriaMetrics Instance VictoriaMetrics scraping/query status
VMAlert Instance Alert rule execution status
Alertmanager Instance Alert aggregation and notifications
VictoriaLogs Instance Log ingestion, querying, and indexing
Logs Instance View log information on a single node
VictoriaTraces Instance Trace storage and querying
Inventory CMDB CMDB visualization
ETCD Overview etcd cluster monitoring

Alert Rules

Pigsty provides the following two alert rules for the INFRA module:

Alert Rule Description
InfraDown Infrastructure component is down
AgentDown Monitoring agent is down

You can modify or add new infrastructure alert rules in files/victoria/rules/infra.yml.

Alert Rule Configuration 

################################################################
#                Infrastructure Alert Rules                    #
################################################################
- name: infra-alert
  rules:

    #==============================================================#
    #                       Infra Aliveness                        #
    #==============================================================#
    # infra components (victoria,grafana) down for 1m triggers a P1 alert
    - alert: InfraDown
      expr: infra_up < 1
      for: 1m
      labels: { level: 0, severity: CRIT, category: infra }
      annotations:
        summary: "CRIT InfraDown {{ $labels.type }}@{{ $labels.instance }}"
        description: |
          infra_up[type={{ $labels.type }}, instance={{ $labels.instance }}] = {{ $value  | printf "%.2f" }} < 1

    #==============================================================#
    #                       Agent Aliveness                        #
    #==============================================================#

    # agent aliveness are determined directly by exporter aliveness
    # including: node_exporter, pg_exporter, pgbouncer_exporter, haproxy_exporter
    - alert: AgentDown
      expr: agent_up < 1
      for: 1m
      labels: { level: 0, severity: CRIT, category: infra }
      annotations:
        summary: 'CRIT AgentDown {{ $labels.ins }}@{{ $labels.instance }}'
        description: |
          agent_up[ins={{ $labels.ins }}, instance={{ $labels.instance }}] = {{ $value  | printf "%.2f" }} < 1

9.5 - Metrics

Complete list of monitoring metrics provided by the Pigsty INFRA module

Note: Pigsty v4.0 has replaced Prometheus/Loki with VictoriaMetrics/Logs/Traces. The following metric list is still based on v3.x generation, for reference when troubleshooting older versions only. To get the latest metrics, query directly in https://p.pigsty (VMUI) or Grafana. Future versions will regenerate metric reference sheets consistent with the Victoria suite.

INFRA Metrics

The INFRA module has 964 available metrics.

Metric Name Type Labels Description
alertmanager_alerts gauge ins, instance, ip, job, cls, state How many alerts by state.
alertmanager_alerts_invalid_total counter version, ins, instance, ip, job, cls The total number of received alerts that were invalid.
alertmanager_alerts_received_total counter version, ins, instance, ip, status, job, cls The total number of received alerts.
alertmanager_build_info gauge revision, version, ins, instance, ip, tags, goarch, goversion, job, cls, branch, goos A metric with a constant ‘1’ value labeled by version, revision, branch, goversion from which alertmanager was built, and the goos and goarch for the build.
alertmanager_cluster_alive_messages_total counter ins, instance, ip, peer, job, cls Total number of received alive messages.
alertmanager_cluster_enabled gauge ins, instance, ip, job, cls Indicates whether the clustering is enabled or not.
alertmanager_cluster_failed_peers gauge ins, instance, ip, job, cls Number indicating the current number of failed peers in the cluster.
alertmanager_cluster_health_score gauge ins, instance, ip, job, cls Health score of the cluster. Lower values are better and zero means ’totally healthy’.
alertmanager_cluster_members gauge ins, instance, ip, job, cls Number indicating current number of members in cluster.
alertmanager_cluster_messages_pruned_total counter ins, instance, ip, job, cls Total number of cluster messages pruned.
alertmanager_cluster_messages_queued gauge ins, instance, ip, job, cls Number of cluster messages which are queued.
alertmanager_cluster_messages_received_size_total counter ins, instance, ip, msg_type, job, cls Total size of cluster messages received.
alertmanager_cluster_messages_received_total counter ins, instance, ip, msg_type, job, cls Total number of cluster messages received.
alertmanager_cluster_messages_sent_size_total counter ins, instance, ip, msg_type, job, cls Total size of cluster messages sent.
alertmanager_cluster_messages_sent_total counter ins, instance, ip, msg_type, job, cls Total number of cluster messages sent.
alertmanager_cluster_peer_info gauge ins, instance, ip, peer, job, cls A metric with a constant ‘1’ value labeled by peer name.
alertmanager_cluster_peers_joined_total counter ins, instance, ip, job, cls A counter of the number of peers that have joined.
alertmanager_cluster_peers_left_total counter ins, instance, ip, job, cls A counter of the number of peers that have left.
alertmanager_cluster_peers_update_total counter ins, instance, ip, job, cls A counter of the number of peers that have updated metadata.
alertmanager_cluster_reconnections_failed_total counter ins, instance, ip, job, cls A counter of the number of failed cluster peer reconnection attempts.
alertmanager_cluster_reconnections_total counter ins, instance, ip, job, cls A counter of the number of cluster peer reconnections.
alertmanager_cluster_refresh_join_failed_total counter ins, instance, ip, job, cls A counter of the number of failed cluster peer joined attempts via refresh.
alertmanager_cluster_refresh_join_total counter ins, instance, ip, job, cls A counter of the number of cluster peer joined via refresh.
alertmanager_config_hash gauge ins, instance, ip, job, cls Hash of the currently loaded alertmanager configuration.
alertmanager_config_last_reload_success_timestamp_seconds gauge ins, instance, ip, job, cls Timestamp of the last successful configuration reload.
alertmanager_config_last_reload_successful gauge ins, instance, ip, job, cls Whether the last configuration reload attempt was successful.
alertmanager_dispatcher_aggregation_groups gauge ins, instance, ip, job, cls Number of active aggregation groups
alertmanager_dispatcher_alert_processing_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
alertmanager_dispatcher_alert_processing_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
alertmanager_http_concurrency_limit_exceeded_total counter ins, instance, method, ip, job, cls Total number of times an HTTP request failed because the concurrency limit was reached.
alertmanager_http_request_duration_seconds_bucket Unknown ins, instance, method, ip, le, job, cls, handler N/A
alertmanager_http_request_duration_seconds_count Unknown ins, instance, method, ip, job, cls, handler N/A
alertmanager_http_request_duration_seconds_sum Unknown ins, instance, method, ip, job, cls, handler N/A
alertmanager_http_requests_in_flight gauge ins, instance, method, ip, job, cls Current number of HTTP requests being processed.
alertmanager_http_response_size_bytes_bucket Unknown ins, instance, method, ip, le, job, cls, handler N/A
alertmanager_http_response_size_bytes_count Unknown ins, instance, method, ip, job, cls, handler N/A
alertmanager_http_response_size_bytes_sum Unknown ins, instance, method, ip, job, cls, handler N/A
alertmanager_integrations gauge ins, instance, ip, job, cls Number of configured integrations.
alertmanager_marked_alerts gauge ins, instance, ip, job, cls, state How many alerts by state are currently marked in the Alertmanager regardless of their expiry.
alertmanager_nflog_gc_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
alertmanager_nflog_gc_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
alertmanager_nflog_gossip_messages_propagated_total counter ins, instance, ip, job, cls Number of received gossip messages that have been further gossiped.
alertmanager_nflog_maintenance_errors_total counter ins, instance, ip, job, cls How many maintenances were executed for the notification log that failed.
alertmanager_nflog_maintenance_total counter ins, instance, ip, job, cls How many maintenances were executed for the notification log.
alertmanager_nflog_queries_total counter ins, instance, ip, job, cls Number of notification log queries were received.
alertmanager_nflog_query_duration_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
alertmanager_nflog_query_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
alertmanager_nflog_query_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
alertmanager_nflog_query_errors_total counter ins, instance, ip, job, cls Number notification log received queries that failed.
alertmanager_nflog_snapshot_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
alertmanager_nflog_snapshot_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
alertmanager_nflog_snapshot_size_bytes gauge ins, instance, ip, job, cls Size of the last notification log snapshot in bytes.
alertmanager_notification_latency_seconds_bucket Unknown integration, ins, instance, ip, le, job, cls N/A
alertmanager_notification_latency_seconds_count Unknown integration, ins, instance, ip, job, cls N/A
alertmanager_notification_latency_seconds_sum Unknown integration, ins, instance, ip, job, cls N/A
alertmanager_notification_requests_failed_total counter integration, ins, instance, ip, job, cls The total number of failed notification requests.
alertmanager_notification_requests_total counter integration, ins, instance, ip, job, cls The total number of attempted notification requests.
alertmanager_notifications_failed_total counter integration, ins, instance, ip, reason, job, cls The total number of failed notifications.
alertmanager_notifications_total counter integration, ins, instance, ip, job, cls The total number of attempted notifications.
alertmanager_oversize_gossip_message_duration_seconds_bucket Unknown ins, instance, ip, le, key, job, cls N/A
alertmanager_oversize_gossip_message_duration_seconds_count Unknown ins, instance, ip, key, job, cls N/A
alertmanager_oversize_gossip_message_duration_seconds_sum Unknown ins, instance, ip, key, job, cls N/A
alertmanager_oversized_gossip_message_dropped_total counter ins, instance, ip, key, job, cls Number of oversized gossip messages that were dropped due to a full message queue.
alertmanager_oversized_gossip_message_failure_total counter ins, instance, ip, key, job, cls Number of oversized gossip message sends that failed.
alertmanager_oversized_gossip_message_sent_total counter ins, instance, ip, key, job, cls Number of oversized gossip message sent.
alertmanager_peer_position gauge ins, instance, ip, job, cls Position the Alertmanager instance believes it’s in. The position determines a peer’s behavior in the cluster.
alertmanager_receivers gauge ins, instance, ip, job, cls Number of configured receivers.
alertmanager_silences gauge ins, instance, ip, job, cls, state How many silences by state.
alertmanager_silences_gc_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
alertmanager_silences_gc_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
alertmanager_silences_gossip_messages_propagated_total counter ins, instance, ip, job, cls Number of received gossip messages that have been further gossiped.
alertmanager_silences_maintenance_errors_total counter ins, instance, ip, job, cls How many maintenances were executed for silences that failed.
alertmanager_silences_maintenance_total counter ins, instance, ip, job, cls How many maintenances were executed for silences.
alertmanager_silences_queries_total counter ins, instance, ip, job, cls How many silence queries were received.
alertmanager_silences_query_duration_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
alertmanager_silences_query_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
alertmanager_silences_query_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
alertmanager_silences_query_errors_total counter ins, instance, ip, job, cls How many silence received queries did not succeed.
alertmanager_silences_snapshot_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
alertmanager_silences_snapshot_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
alertmanager_silences_snapshot_size_bytes gauge ins, instance, ip, job, cls Size of the last silence snapshot in bytes.
blackbox_exporter_build_info gauge revision, version, ins, instance, ip, tags, goarch, goversion, job, cls, branch, goos A metric with a constant ‘1’ value labeled by version, revision, branch, goversion from which blackbox_exporter was built, and the goos and goarch for the build.
blackbox_exporter_config_last_reload_success_timestamp_seconds gauge ins, instance, ip, job, cls Timestamp of the last successful configuration reload.
blackbox_exporter_config_last_reload_successful gauge ins, instance, ip, job, cls Blackbox exporter config loaded successfully.
blackbox_module_unknown_total counter ins, instance, ip, job, cls Count of unknown modules requested by probes
cortex_distributor_ingester_clients gauge ins, instance, ip, job, cls The current number of ingester clients.
cortex_dns_failures_total Unknown ins, instance, ip, job, cls N/A
cortex_dns_lookups_total Unknown ins, instance, ip, job, cls N/A
cortex_frontend_query_range_duration_seconds_bucket Unknown ins, instance, method, ip, le, job, cls, status_code N/A
cortex_frontend_query_range_duration_seconds_count Unknown ins, instance, method, ip, job, cls, status_code N/A
cortex_frontend_query_range_duration_seconds_sum Unknown ins, instance, method, ip, job, cls, status_code N/A
cortex_ingester_flush_queue_length gauge ins, instance, ip, job, cls The total number of series pending in the flush queue.
cortex_kv_request_duration_seconds_bucket Unknown ins, instance, role, ip, le, kv_name, type, operation, job, cls, status_code N/A
cortex_kv_request_duration_seconds_count Unknown ins, instance, role, ip, kv_name, type, operation, job, cls, status_code N/A
cortex_kv_request_duration_seconds_sum Unknown ins, instance, role, ip, kv_name, type, operation, job, cls, status_code N/A
cortex_member_consul_heartbeats_total Unknown ins, instance, ip, job, cls N/A
cortex_prometheus_notifications_alertmanagers_discovered gauge ins, instance, ip, user, job, cls The number of alertmanagers discovered and active.
cortex_prometheus_notifications_dropped_total Unknown ins, instance, ip, user, job, cls N/A
cortex_prometheus_notifications_queue_capacity gauge ins, instance, ip, user, job, cls The capacity of the alert notifications queue.
cortex_prometheus_notifications_queue_length gauge ins, instance, ip, user, job, cls The number of alert notifications in the queue.
cortex_prometheus_rule_evaluation_duration_seconds summary ins, instance, ip, user, job, cls, quantile The duration for a rule to execute.
cortex_prometheus_rule_evaluation_duration_seconds_count Unknown ins, instance, ip, user, job, cls N/A
cortex_prometheus_rule_evaluation_duration_seconds_sum Unknown ins, instance, ip, user, job, cls N/A
cortex_prometheus_rule_group_duration_seconds summary ins, instance, ip, user, job, cls, quantile The duration of rule group evaluations.
cortex_prometheus_rule_group_duration_seconds_count Unknown ins, instance, ip, user, job, cls N/A
cortex_prometheus_rule_group_duration_seconds_sum Unknown ins, instance, ip, user, job, cls N/A
cortex_query_frontend_connected_schedulers gauge ins, instance, ip, job, cls Number of schedulers this frontend is connected to.
cortex_query_frontend_queries_in_progress gauge ins, instance, ip, job, cls Number of queries in progress handled by this frontend.
cortex_query_frontend_retries_bucket Unknown ins, instance, ip, le, job, cls N/A
cortex_query_frontend_retries_count Unknown ins, instance, ip, job, cls N/A
cortex_query_frontend_retries_sum Unknown ins, instance, ip, job, cls N/A
cortex_query_scheduler_connected_frontend_clients gauge ins, instance, ip, job, cls Number of query-frontend worker clients currently connected to the query-scheduler.
cortex_query_scheduler_connected_querier_clients gauge ins, instance, ip, job, cls Number of querier worker clients currently connected to the query-scheduler.
cortex_query_scheduler_inflight_requests summary ins, instance, ip, job, cls, quantile Number of inflight requests (either queued or processing) sampled at a regular interval. Quantile buckets keep track of inflight requests over the last 60s.
cortex_query_scheduler_inflight_requests_count Unknown ins, instance, ip, job, cls N/A
cortex_query_scheduler_inflight_requests_sum Unknown ins, instance, ip, job, cls N/A
cortex_query_scheduler_queue_duration_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
cortex_query_scheduler_queue_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
cortex_query_scheduler_queue_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
cortex_query_scheduler_queue_length Unknown ins, instance, ip, user, job, cls N/A
cortex_query_scheduler_running gauge ins, instance, ip, job, cls Value will be 1 if the scheduler is in the ReplicationSet and actively receiving/processing requests
cortex_ring_member_heartbeats_total Unknown ins, instance, ip, job, cls N/A
cortex_ring_member_tokens_owned gauge ins, instance, ip, job, cls The number of tokens owned in the ring.
cortex_ring_member_tokens_to_own gauge ins, instance, ip, job, cls The number of tokens to own in the ring.
cortex_ring_members gauge ins, instance, ip, job, cls, state Number of members in the ring
cortex_ring_oldest_member_timestamp gauge ins, instance, ip, job, cls, state Timestamp of the oldest member in the ring.
cortex_ring_tokens_total gauge ins, instance, ip, job, cls Number of tokens in the ring
cortex_ruler_clients gauge ins, instance, ip, job, cls The current number of ruler clients in the pool.
cortex_ruler_config_last_reload_successful gauge ins, instance, ip, user, job, cls Boolean set to 1 whenever the last configuration reload attempt was successful.
cortex_ruler_config_last_reload_successful_seconds gauge ins, instance, ip, user, job, cls Timestamp of the last successful configuration reload.
cortex_ruler_config_updates_total Unknown ins, instance, ip, user, job, cls N/A
cortex_ruler_managers_total gauge ins, instance, ip, job, cls Total number of managers registered and running in the ruler
cortex_ruler_ring_check_errors_total Unknown ins, instance, ip, job, cls N/A
cortex_ruler_sync_rules_total Unknown ins, instance, ip, reason, job, cls N/A
deprecated_flags_inuse_total Unknown ins, instance, ip, job, cls N/A
go_cgo_go_to_c_calls_calls_total Unknown ins, instance, ip, job, cls N/A
go_cpu_classes_gc_mark_assist_cpu_seconds_total Unknown ins, instance, ip, job, cls N/A
go_cpu_classes_gc_mark_dedicated_cpu_seconds_total Unknown ins, instance, ip, job, cls N/A
go_cpu_classes_gc_mark_idle_cpu_seconds_total Unknown ins, instance, ip, job, cls N/A
go_cpu_classes_gc_pause_cpu_seconds_total Unknown ins, instance, ip, job, cls N/A
go_cpu_classes_gc_total_cpu_seconds_total Unknown ins, instance, ip, job, cls N/A
go_cpu_classes_idle_cpu_seconds_total Unknown ins, instance, ip, job, cls N/A
go_cpu_classes_scavenge_assist_cpu_seconds_total Unknown ins, instance, ip, job, cls N/A
go_cpu_classes_scavenge_background_cpu_seconds_total Unknown ins, instance, ip, job, cls N/A
go_cpu_classes_scavenge_total_cpu_seconds_total Unknown ins, instance, ip, job, cls N/A
go_cpu_classes_total_cpu_seconds_total Unknown ins, instance, ip, job, cls N/A
go_cpu_classes_user_cpu_seconds_total Unknown ins, instance, ip, job, cls N/A
go_gc_cycles_automatic_gc_cycles_total Unknown ins, instance, ip, job, cls N/A
go_gc_cycles_forced_gc_cycles_total Unknown ins, instance, ip, job, cls N/A
go_gc_cycles_total_gc_cycles_total Unknown ins, instance, ip, job, cls N/A
go_gc_duration_seconds summary ins, instance, ip, job, cls, quantile A summary of the pause duration of garbage collection cycles.
go_gc_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
go_gc_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
go_gc_gogc_percent gauge ins, instance, ip, job, cls Heap size target percentage configured by the user, otherwise 100. This value is set by the GOGC environment variable, and the runtime/debug.SetGCPercent function.
go_gc_gomemlimit_bytes gauge ins, instance, ip, job, cls Go runtime memory limit configured by the user, otherwise math.MaxInt64. This value is set by the GOMEMLIMIT environment variable, and the runtime/debug.SetMemoryLimit function.
go_gc_heap_allocs_by_size_bytes_bucket Unknown ins, instance, ip, le, job, cls N/A
go_gc_heap_allocs_by_size_bytes_count Unknown ins, instance, ip, job, cls N/A
go_gc_heap_allocs_by_size_bytes_sum Unknown ins, instance, ip, job, cls N/A
go_gc_heap_allocs_bytes_total Unknown ins, instance, ip, job, cls N/A
go_gc_heap_allocs_objects_total Unknown ins, instance, ip, job, cls N/A
go_gc_heap_frees_by_size_bytes_bucket Unknown ins, instance, ip, le, job, cls N/A
go_gc_heap_frees_by_size_bytes_count Unknown ins, instance, ip, job, cls N/A
go_gc_heap_frees_by_size_bytes_sum Unknown ins, instance, ip, job, cls N/A
go_gc_heap_frees_bytes_total Unknown ins, instance, ip, job, cls N/A
go_gc_heap_frees_objects_total Unknown ins, instance, ip, job, cls N/A
go_gc_heap_goal_bytes gauge ins, instance, ip, job, cls Heap size target for the end of the GC cycle.
go_gc_heap_live_bytes gauge ins, instance, ip, job, cls Heap memory occupied by live objects that were marked by the previous GC.
go_gc_heap_objects_objects gauge ins, instance, ip, job, cls Number of objects, live or unswept, occupying heap memory.
go_gc_heap_tiny_allocs_objects_total Unknown ins, instance, ip, job, cls N/A
go_gc_limiter_last_enabled_gc_cycle gauge ins, instance, ip, job, cls GC cycle the last time the GC CPU limiter was enabled. This metric is useful for diagnosing the root cause of an out-of-memory error, because the limiter trades memory for CPU time when the GC’s CPU time gets too high. This is most likely to occur with use of SetMemoryLimit. The first GC cycle is cycle 1, so a value of 0 indicates that it was never enabled.
go_gc_pauses_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
go_gc_pauses_seconds_count Unknown ins, instance, ip, job, cls N/A
go_gc_pauses_seconds_sum Unknown ins, instance, ip, job, cls N/A
go_gc_scan_globals_bytes gauge ins, instance, ip, job, cls The total amount of global variable space that is scannable.
go_gc_scan_heap_bytes gauge ins, instance, ip, job, cls The total amount of heap space that is scannable.
go_gc_scan_stack_bytes gauge ins, instance, ip, job, cls The number of bytes of stack that were scanned last GC cycle.
go_gc_scan_total_bytes gauge ins, instance, ip, job, cls The total amount space that is scannable. Sum of all metrics in /gc/scan.
go_gc_stack_starting_size_bytes gauge ins, instance, ip, job, cls The stack size of new goroutines.
go_godebug_non_default_behavior_execerrdot_events_total Unknown ins, instance, ip, job, cls N/A
go_godebug_non_default_behavior_gocachehash_events_total Unknown ins, instance, ip, job, cls N/A
go_godebug_non_default_behavior_gocachetest_events_total Unknown ins, instance, ip, job, cls N/A
go_godebug_non_default_behavior_gocacheverify_events_total Unknown ins, instance, ip, job, cls N/A
go_godebug_non_default_behavior_http2client_events_total Unknown ins, instance, ip, job, cls N/A
go_godebug_non_default_behavior_http2server_events_total Unknown ins, instance, ip, job, cls N/A
go_godebug_non_default_behavior_installgoroot_events_total Unknown ins, instance, ip, job, cls N/A
go_godebug_non_default_behavior_jstmpllitinterp_events_total Unknown ins, instance, ip, job, cls N/A
go_godebug_non_default_behavior_multipartmaxheaders_events_total Unknown ins, instance, ip, job, cls N/A
go_godebug_non_default_behavior_multipartmaxparts_events_total Unknown ins, instance, ip, job, cls N/A
go_godebug_non_default_behavior_multipathtcp_events_total Unknown ins, instance, ip, job, cls N/A
go_godebug_non_default_behavior_panicnil_events_total Unknown ins, instance, ip, job, cls N/A
go_godebug_non_default_behavior_randautoseed_events_total Unknown ins, instance, ip, job, cls N/A
go_godebug_non_default_behavior_tarinsecurepath_events_total Unknown ins, instance, ip, job, cls N/A
go_godebug_non_default_behavior_tlsmaxrsasize_events_total Unknown ins, instance, ip, job, cls N/A
go_godebug_non_default_behavior_x509sha1_events_total Unknown ins, instance, ip, job, cls N/A
go_godebug_non_default_behavior_x509usefallbackroots_events_total Unknown ins, instance, ip, job, cls N/A
go_godebug_non_default_behavior_zipinsecurepath_events_total Unknown ins, instance, ip, job, cls N/A
go_goroutines gauge ins, instance, ip, job, cls Number of goroutines that currently exist.
go_info gauge version, ins, instance, ip, job, cls Information about the Go environment.
go_memory_classes_heap_free_bytes gauge ins, instance, ip, job, cls Memory that is completely free and eligible to be returned to the underlying system, but has not been. This metric is the runtime’s estimate of free address space that is backed by physical memory.
go_memory_classes_heap_objects_bytes gauge ins, instance, ip, job, cls Memory occupied by live objects and dead objects that have not yet been marked free by the garbage collector.
go_memory_classes_heap_released_bytes gauge ins, instance, ip, job, cls Memory that is completely free and has been returned to the underlying system. This metric is the runtime’s estimate of free address space that is still mapped into the process, but is not backed by physical memory.
go_memory_classes_heap_stacks_bytes gauge ins, instance, ip, job, cls Memory allocated from the heap that is reserved for stack space, whether or not it is currently in-use. Currently, this represents all stack memory for goroutines. It also includes all OS thread stacks in non-cgo programs. Note that stacks may be allocated differently in the future, and this may change.
go_memory_classes_heap_unused_bytes gauge ins, instance, ip, job, cls Memory that is reserved for heap objects but is not currently used to hold heap objects.
go_memory_classes_metadata_mcache_free_bytes gauge ins, instance, ip, job, cls Memory that is reserved for runtime mcache structures, but not in-use.
go_memory_classes_metadata_mcache_inuse_bytes gauge ins, instance, ip, job, cls Memory that is occupied by runtime mcache structures that are currently being used.
go_memory_classes_metadata_mspan_free_bytes gauge ins, instance, ip, job, cls Memory that is reserved for runtime mspan structures, but not in-use.
go_memory_classes_metadata_mspan_inuse_bytes gauge ins, instance, ip, job, cls Memory that is occupied by runtime mspan structures that are currently being used.
go_memory_classes_metadata_other_bytes gauge ins, instance, ip, job, cls Memory that is reserved for or used to hold runtime metadata.
go_memory_classes_os_stacks_bytes gauge ins, instance, ip, job, cls Stack memory allocated by the underlying operating system. In non-cgo programs this metric is currently zero. This may change in the future.In cgo programs this metric includes OS thread stacks allocated directly from the OS. Currently, this only accounts for one stack in c-shared and c-archive build modes, and other sources of stacks from the OS are not measured. This too may change in the future.
go_memory_classes_other_bytes gauge ins, instance, ip, job, cls Memory used by execution trace buffers, structures for debugging the runtime, finalizer and profiler specials, and more.
go_memory_classes_profiling_buckets_bytes gauge ins, instance, ip, job, cls Memory that is used by the stack trace hash map used for profiling.
go_memory_classes_total_bytes gauge ins, instance, ip, job, cls All memory mapped by the Go runtime into the current process as read-write. Note that this does not include memory mapped by code called via cgo or via the syscall package. Sum of all metrics in /memory/classes.
go_memstats_alloc_bytes counter ins, instance, ip, job, cls Total number of bytes allocated, even if freed.
go_memstats_alloc_bytes_total counter ins, instance, ip, job, cls Total number of bytes allocated, even if freed.
go_memstats_buck_hash_sys_bytes gauge ins, instance, ip, job, cls Number of bytes used by the profiling bucket hash table.
go_memstats_frees_total counter ins, instance, ip, job, cls Total number of frees.
go_memstats_gc_sys_bytes gauge ins, instance, ip, job, cls Number of bytes used for garbage collection system metadata.
go_memstats_heap_alloc_bytes gauge ins, instance, ip, job, cls Number of heap bytes allocated and still in use.
go_memstats_heap_idle_bytes gauge ins, instance, ip, job, cls Number of heap bytes waiting to be used.
go_memstats_heap_inuse_bytes gauge ins, instance, ip, job, cls Number of heap bytes that are in use.
go_memstats_heap_objects gauge ins, instance, ip, job, cls Number of allocated objects.
go_memstats_heap_released_bytes gauge ins, instance, ip, job, cls Number of heap bytes released to OS.
go_memstats_heap_sys_bytes gauge ins, instance, ip, job, cls Number of heap bytes obtained from system.
go_memstats_last_gc_time_seconds gauge ins, instance, ip, job, cls Number of seconds since 1970 of last garbage collection.
go_memstats_lookups_total counter ins, instance, ip, job, cls Total number of pointer lookups.
go_memstats_mallocs_total counter ins, instance, ip, job, cls Total number of mallocs.
go_memstats_mcache_inuse_bytes gauge ins, instance, ip, job, cls Number of bytes in use by mcache structures.
go_memstats_mcache_sys_bytes gauge ins, instance, ip, job, cls Number of bytes used for mcache structures obtained from system.
go_memstats_mspan_inuse_bytes gauge ins, instance, ip, job, cls Number of bytes in use by mspan structures.
go_memstats_mspan_sys_bytes gauge ins, instance, ip, job, cls Number of bytes used for mspan structures obtained from system.
go_memstats_next_gc_bytes gauge ins, instance, ip, job, cls Number of heap bytes when next garbage collection will take place.
go_memstats_other_sys_bytes gauge ins, instance, ip, job, cls Number of bytes used for other system allocations.
go_memstats_stack_inuse_bytes gauge ins, instance, ip, job, cls Number of bytes in use by the stack allocator.
go_memstats_stack_sys_bytes gauge ins, instance, ip, job, cls Number of bytes obtained from system for stack allocator.
go_memstats_sys_bytes gauge ins, instance, ip, job, cls Number of bytes obtained from system.
go_sched_gomaxprocs_threads gauge ins, instance, ip, job, cls The current runtime.GOMAXPROCS setting, or the number of operating system threads that can execute user-level Go code simultaneously.
go_sched_goroutines_goroutines gauge ins, instance, ip, job, cls Count of live goroutines.
go_sched_latencies_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
go_sched_latencies_seconds_count Unknown ins, instance, ip, job, cls N/A
go_sched_latencies_seconds_sum Unknown ins, instance, ip, job, cls N/A
go_sql_stats_connections_blocked_seconds unknown ins, instance, db_name, ip, job, cls The total time blocked waiting for a new connection.
go_sql_stats_connections_closed_max_idle unknown ins, instance, db_name, ip, job, cls The total number of connections closed due to SetMaxIdleConns.
go_sql_stats_connections_closed_max_idle_time unknown ins, instance, db_name, ip, job, cls The total number of connections closed due to SetConnMaxIdleTime.
go_sql_stats_connections_closed_max_lifetime unknown ins, instance, db_name, ip, job, cls The total number of connections closed due to SetConnMaxLifetime.
go_sql_stats_connections_idle gauge ins, instance, db_name, ip, job, cls The number of idle connections.
go_sql_stats_connections_in_use gauge ins, instance, db_name, ip, job, cls The number of connections currently in use.
go_sql_stats_connections_max_open gauge ins, instance, db_name, ip, job, cls Maximum number of open connections to the database.
go_sql_stats_connections_open gauge ins, instance, db_name, ip, job, cls The number of established connections both in use and idle.
go_sql_stats_connections_waited_for unknown ins, instance, db_name, ip, job, cls The total number of connections waited for.
go_sync_mutex_wait_total_seconds_total Unknown ins, instance, ip, job, cls N/A
go_threads gauge ins, instance, ip, job, cls Number of OS threads created.
grafana_access_evaluation_count unknown ins, instance, ip, job, cls number of evaluation calls
grafana_access_evaluation_duration_bucket Unknown ins, instance, ip, le, job, cls N/A
grafana_access_evaluation_duration_count Unknown ins, instance, ip, job, cls N/A
grafana_access_evaluation_duration_sum Unknown ins, instance, ip, job, cls N/A
grafana_access_permissions_duration_bucket Unknown ins, instance, ip, le, job, cls N/A
grafana_access_permissions_duration_count Unknown ins, instance, ip, job, cls N/A
grafana_access_permissions_duration_sum Unknown ins, instance, ip, job, cls N/A
grafana_aggregator_discovery_aggregation_count_total Unknown ins, instance, ip, job, cls N/A
grafana_alerting_active_alerts gauge ins, instance, ip, job, cls amount of active alerts
grafana_alerting_active_configurations gauge ins, instance, ip, job, cls The number of active Alertmanager configurations.
grafana_alerting_alertmanager_config_match gauge ins, instance, ip, job, cls The total number of match
grafana_alerting_alertmanager_config_match_re gauge ins, instance, ip, job, cls The total number of matchRE
grafana_alerting_alertmanager_config_matchers gauge ins, instance, ip, job, cls The total number of matchers
grafana_alerting_alertmanager_config_object_matchers gauge ins, instance, ip, job, cls The total number of object_matchers
grafana_alerting_discovered_configurations gauge ins, instance, ip, job, cls The number of organizations we’ve discovered that require an Alertmanager configuration.
grafana_alerting_dispatcher_aggregation_groups gauge ins, instance, ip, job, cls Number of active aggregation groups
grafana_alerting_dispatcher_alert_processing_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
grafana_alerting_dispatcher_alert_processing_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_alerting_execution_time_milliseconds summary ins, instance, ip, job, cls, quantile summary of alert execution duration
grafana_alerting_execution_time_milliseconds_count Unknown ins, instance, ip, job, cls N/A
grafana_alerting_execution_time_milliseconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_alerting_nflog_gc_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
grafana_alerting_nflog_gc_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_alerting_nflog_gossip_messages_propagated_total Unknown ins, instance, ip, job, cls N/A
grafana_alerting_nflog_queries_total Unknown ins, instance, ip, job, cls N/A
grafana_alerting_nflog_query_duration_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
grafana_alerting_nflog_query_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
grafana_alerting_nflog_query_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_alerting_nflog_query_errors_total Unknown ins, instance, ip, job, cls N/A
grafana_alerting_nflog_snapshot_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
grafana_alerting_nflog_snapshot_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_alerting_nflog_snapshot_size_bytes gauge ins, instance, ip, job, cls Size of the last notification log snapshot in bytes.
grafana_alerting_notification_latency_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
grafana_alerting_notification_latency_seconds_count Unknown ins, instance, ip, job, cls N/A
grafana_alerting_notification_latency_seconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_alerting_schedule_alert_rules gauge ins, instance, ip, job, cls The number of alert rules that could be considered for evaluation at the next tick.
grafana_alerting_schedule_alert_rules_hash gauge ins, instance, ip, job, cls A hash of the alert rules that could be considered for evaluation at the next tick.
grafana_alerting_schedule_periodic_duration_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
grafana_alerting_schedule_periodic_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
grafana_alerting_schedule_periodic_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_alerting_schedule_query_alert_rules_duration_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
grafana_alerting_schedule_query_alert_rules_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
grafana_alerting_schedule_query_alert_rules_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_alerting_scheduler_behind_seconds gauge ins, instance, ip, job, cls The total number of seconds the scheduler is behind.
grafana_alerting_silences_gc_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
grafana_alerting_silences_gc_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_alerting_silences_gossip_messages_propagated_total Unknown ins, instance, ip, job, cls N/A
grafana_alerting_silences_queries_total Unknown ins, instance, ip, job, cls N/A
grafana_alerting_silences_query_duration_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
grafana_alerting_silences_query_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
grafana_alerting_silences_query_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_alerting_silences_query_errors_total Unknown ins, instance, ip, job, cls N/A
grafana_alerting_silences_snapshot_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
grafana_alerting_silences_snapshot_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_alerting_silences_snapshot_size_bytes gauge ins, instance, ip, job, cls Size of the last silence snapshot in bytes.
grafana_alerting_state_calculation_duration_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
grafana_alerting_state_calculation_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
grafana_alerting_state_calculation_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_alerting_state_history_writes_bytes_total Unknown ins, instance, ip, job, cls N/A
grafana_alerting_ticker_interval_seconds gauge ins, instance, ip, job, cls Interval at which the ticker is meant to tick.
grafana_alerting_ticker_last_consumed_tick_timestamp_seconds gauge ins, instance, ip, job, cls Timestamp of the last consumed tick in seconds.
grafana_alerting_ticker_next_tick_timestamp_seconds gauge ins, instance, ip, job, cls Timestamp of the next tick in seconds before it is consumed.
grafana_api_admin_user_created_total Unknown ins, instance, ip, job, cls N/A
grafana_api_dashboard_get_milliseconds summary ins, instance, ip, job, cls, quantile summary for dashboard get duration
grafana_api_dashboard_get_milliseconds_count Unknown ins, instance, ip, job, cls N/A
grafana_api_dashboard_get_milliseconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_api_dashboard_save_milliseconds summary ins, instance, ip, job, cls, quantile summary for dashboard save duration
grafana_api_dashboard_save_milliseconds_count Unknown ins, instance, ip, job, cls N/A
grafana_api_dashboard_save_milliseconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_api_dashboard_search_milliseconds summary ins, instance, ip, job, cls, quantile summary for dashboard search duration
grafana_api_dashboard_search_milliseconds_count Unknown ins, instance, ip, job, cls N/A
grafana_api_dashboard_search_milliseconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_api_dashboard_snapshot_create_total Unknown ins, instance, ip, job, cls N/A
grafana_api_dashboard_snapshot_external_total Unknown ins, instance, ip, job, cls N/A
grafana_api_dashboard_snapshot_get_total Unknown ins, instance, ip, job, cls N/A
grafana_api_dataproxy_request_all_milliseconds summary ins, instance, ip, job, cls, quantile summary for dataproxy request duration
grafana_api_dataproxy_request_all_milliseconds_count Unknown ins, instance, ip, job, cls N/A
grafana_api_dataproxy_request_all_milliseconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_api_login_oauth_total Unknown ins, instance, ip, job, cls N/A
grafana_api_login_post_total Unknown ins, instance, ip, job, cls N/A
grafana_api_login_saml_total Unknown ins, instance, ip, job, cls N/A
grafana_api_models_dashboard_insert_total Unknown ins, instance, ip, job, cls N/A
grafana_api_org_create_total Unknown ins, instance, ip, job, cls N/A
grafana_api_response_status_total Unknown ins, instance, ip, job, cls, code N/A
grafana_api_user_signup_completed_total Unknown ins, instance, ip, job, cls N/A
grafana_api_user_signup_invite_total Unknown ins, instance, ip, job, cls N/A
grafana_api_user_signup_started_total Unknown ins, instance, ip, job, cls N/A
grafana_apiserver_audit_event_total Unknown ins, instance, ip, job, cls N/A
grafana_apiserver_audit_requests_rejected_total Unknown ins, instance, ip, job, cls N/A
grafana_apiserver_client_certificate_expiration_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
grafana_apiserver_client_certificate_expiration_seconds_count Unknown ins, instance, ip, job, cls N/A
grafana_apiserver_client_certificate_expiration_seconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_apiserver_envelope_encryption_dek_cache_fill_percent gauge ins, instance, ip, job, cls [ALPHA] Percent of the cache slots currently occupied by cached DEKs.
grafana_apiserver_flowcontrol_seat_fair_frac gauge ins, instance, ip, job, cls [ALPHA] Fair fraction of server’s concurrency to allocate to each priority level that can use it
grafana_apiserver_storage_data_key_generation_duration_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
grafana_apiserver_storage_data_key_generation_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
grafana_apiserver_storage_data_key_generation_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_apiserver_storage_data_key_generation_failures_total Unknown ins, instance, ip, job, cls N/A
grafana_apiserver_storage_envelope_transformation_cache_misses_total Unknown ins, instance, ip, job, cls N/A
grafana_apiserver_tls_handshake_errors_total Unknown ins, instance, ip, job, cls N/A
grafana_apiserver_webhooks_x509_insecure_sha1_total Unknown ins, instance, ip, job, cls N/A
grafana_apiserver_webhooks_x509_missing_san_total Unknown ins, instance, ip, job, cls N/A
grafana_authn_authn_failed_authentication_total Unknown ins, instance, ip, job, cls N/A
grafana_authn_authn_successful_authentication_total Unknown ins, instance, ip, client, job, cls N/A
grafana_authn_authn_successful_login_total Unknown ins, instance, ip, client, job, cls N/A
grafana_aws_cloudwatch_get_metric_data_total Unknown ins, instance, ip, job, cls N/A
grafana_aws_cloudwatch_get_metric_statistics_total Unknown ins, instance, ip, job, cls N/A
grafana_aws_cloudwatch_list_metrics_total Unknown ins, instance, ip, job, cls N/A
grafana_build_info gauge revision, version, ins, instance, edition, ip, goversion, job, cls, branch A metric with a constant ‘1’ value labeled by version, revision, branch, and goversion from which Grafana was built
grafana_build_timestamp gauge revision, version, ins, instance, edition, ip, goversion, job, cls, branch A metric exposing when the binary was built in epoch
grafana_cardinality_enforcement_unexpected_categorizations_total Unknown ins, instance, ip, job, cls N/A
grafana_database_conn_idle gauge ins, instance, ip, job, cls The number of idle connections
grafana_database_conn_in_use gauge ins, instance, ip, job, cls The number of connections currently in use
grafana_database_conn_max_idle_closed_seconds unknown ins, instance, ip, job, cls The total number of connections closed due to SetConnMaxIdleTime
grafana_database_conn_max_idle_closed_total Unknown ins, instance, ip, job, cls N/A
grafana_database_conn_max_lifetime_closed_total Unknown ins, instance, ip, job, cls N/A
grafana_database_conn_max_open gauge ins, instance, ip, job, cls Maximum number of open connections to the database
grafana_database_conn_open gauge ins, instance, ip, job, cls The number of established connections both in use and idle
grafana_database_conn_wait_count_total Unknown ins, instance, ip, job, cls N/A
grafana_database_conn_wait_duration_seconds unknown ins, instance, ip, job, cls The total time blocked waiting for a new connection
grafana_datasource_request_duration_seconds_bucket Unknown datasource, ins, instance, method, ip, le, datasource_type, job, cls, code N/A
grafana_datasource_request_duration_seconds_count Unknown datasource, ins, instance, method, ip, datasource_type, job, cls, code N/A
grafana_datasource_request_duration_seconds_sum Unknown datasource, ins, instance, method, ip, datasource_type, job, cls, code N/A
grafana_datasource_request_in_flight gauge datasource, ins, instance, ip, datasource_type, job, cls A gauge of outgoing data source requests currently being sent by Grafana
grafana_datasource_request_total Unknown datasource, ins, instance, method, ip, datasource_type, job, cls, code N/A
grafana_datasource_response_size_bytes_bucket Unknown datasource, ins, instance, ip, le, datasource_type, job, cls N/A
grafana_datasource_response_size_bytes_count Unknown datasource, ins, instance, ip, datasource_type, job, cls N/A
grafana_datasource_response_size_bytes_sum Unknown datasource, ins, instance, ip, datasource_type, job, cls N/A
grafana_db_datasource_query_by_id_total Unknown ins, instance, ip, job, cls N/A
grafana_disabled_metrics_total Unknown ins, instance, ip, job, cls N/A
grafana_emails_sent_failed unknown ins, instance, ip, job, cls Number of emails Grafana failed to send
grafana_emails_sent_total Unknown ins, instance, ip, job, cls N/A
grafana_encryption_cache_reads_total Unknown ins, instance, method, ip, hit, job, cls N/A
grafana_encryption_ops_total Unknown ins, instance, ip, success, operation, job, cls N/A
grafana_environment_info gauge version, ins, instance, ip, job, cls, commit A metric with a constant ‘1’ value labeled by environment information about the running instance.
grafana_feature_toggles_info gauge ins, instance, ip, job, cls info metric that exposes what feature toggles are enabled or not
grafana_frontend_boot_css_time_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
grafana_frontend_boot_css_time_seconds_count Unknown ins, instance, ip, job, cls N/A
grafana_frontend_boot_css_time_seconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_frontend_boot_first_contentful_paint_time_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
grafana_frontend_boot_first_contentful_paint_time_seconds_count Unknown ins, instance, ip, job, cls N/A
grafana_frontend_boot_first_contentful_paint_time_seconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_frontend_boot_first_paint_time_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
grafana_frontend_boot_first_paint_time_seconds_count Unknown ins, instance, ip, job, cls N/A
grafana_frontend_boot_first_paint_time_seconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_frontend_boot_js_done_time_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
grafana_frontend_boot_js_done_time_seconds_count Unknown ins, instance, ip, job, cls N/A
grafana_frontend_boot_js_done_time_seconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_frontend_boot_load_time_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
grafana_frontend_boot_load_time_seconds_count Unknown ins, instance, ip, job, cls N/A
grafana_frontend_boot_load_time_seconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_frontend_plugins_preload_ms_bucket Unknown ins, instance, ip, le, job, cls N/A
grafana_frontend_plugins_preload_ms_count Unknown ins, instance, ip, job, cls N/A
grafana_frontend_plugins_preload_ms_sum Unknown ins, instance, ip, job, cls N/A
grafana_hidden_metrics_total Unknown ins, instance, ip, job, cls N/A
grafana_http_request_duration_seconds_bucket Unknown ins, instance, method, ip, le, job, cls, status_code, handler N/A
grafana_http_request_duration_seconds_count Unknown ins, instance, method, ip, job, cls, status_code, handler N/A
grafana_http_request_duration_seconds_sum Unknown ins, instance, method, ip, job, cls, status_code, handler N/A
grafana_http_request_in_flight gauge ins, instance, ip, job, cls A gauge of requests currently being served by Grafana.
grafana_idforwarding_idforwarding_failed_token_signing_total Unknown ins, instance, ip, job, cls N/A
grafana_idforwarding_idforwarding_token_signing_duration_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
grafana_idforwarding_idforwarding_token_signing_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
grafana_idforwarding_idforwarding_token_signing_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_idforwarding_idforwarding_token_signing_from_cache_total Unknown ins, instance, ip, job, cls N/A
grafana_idforwarding_idforwarding_token_signing_total Unknown ins, instance, ip, job, cls N/A
grafana_instance_start_total Unknown ins, instance, ip, job, cls N/A
grafana_ldap_users_sync_execution_time summary ins, instance, ip, job, cls, quantile summary for LDAP users sync execution duration
grafana_ldap_users_sync_execution_time_count Unknown ins, instance, ip, job, cls N/A
grafana_ldap_users_sync_execution_time_sum Unknown ins, instance, ip, job, cls N/A
grafana_live_client_command_duration_seconds summary ins, instance, method, ip, job, cls, quantile Client command duration summary.
grafana_live_client_command_duration_seconds_count Unknown ins, instance, method, ip, job, cls N/A
grafana_live_client_command_duration_seconds_sum Unknown ins, instance, method, ip, job, cls N/A
grafana_live_client_num_reply_errors unknown ins, instance, method, ip, job, cls, code Number of errors in replies sent to clients.
grafana_live_client_num_server_disconnects unknown ins, instance, ip, job, cls, code Number of server initiated disconnects.
grafana_live_client_recover unknown ins, instance, ip, recovered, job, cls Count of recover operations.
grafana_live_node_action_count unknown action, ins, instance, ip, job, cls Number of node actions called.
grafana_live_node_build gauge version, ins, instance, ip, job, cls Node build info.
grafana_live_node_messages_received_count unknown ins, instance, ip, type, job, cls Number of messages received.
grafana_live_node_messages_sent_count unknown ins, instance, ip, type, job, cls Number of messages sent.
grafana_live_node_num_channels gauge ins, instance, ip, job, cls Number of channels with one or more subscribers.
grafana_live_node_num_clients gauge ins, instance, ip, job, cls Number of clients connected.
grafana_live_node_num_nodes gauge ins, instance, ip, job, cls Number of nodes in cluster.
grafana_live_node_num_subscriptions gauge ins, instance, ip, job, cls Number of subscriptions.
grafana_live_node_num_users gauge ins, instance, ip, job, cls Number of unique users connected.
grafana_live_transport_connect_count unknown ins, instance, ip, transport, job, cls Number of connections to specific transport.
grafana_live_transport_messages_sent unknown ins, instance, ip, transport, job, cls Number of messages sent over specific transport.
grafana_loki_plugin_parse_response_duration_seconds_bucket Unknown endpoint, ins, instance, ip, le, status, job, cls N/A
grafana_loki_plugin_parse_response_duration_seconds_count Unknown endpoint, ins, instance, ip, status, job, cls N/A
grafana_loki_plugin_parse_response_duration_seconds_sum Unknown endpoint, ins, instance, ip, status, job, cls N/A
grafana_page_response_status_total Unknown ins, instance, ip, job, cls, code N/A
grafana_plugin_build_info gauge version, signature_status, ins, instance, plugin_type, ip, plugin_id, job, cls A metric with a constant ‘1’ value labeled by pluginId, pluginType and version from which Grafana plugin was built
grafana_plugin_request_duration_milliseconds_bucket Unknown endpoint, ins, instance, target, ip, le, plugin_id, job, cls N/A
grafana_plugin_request_duration_milliseconds_count Unknown endpoint, ins, instance, target, ip, plugin_id, job, cls N/A
grafana_plugin_request_duration_milliseconds_sum Unknown endpoint, ins, instance, target, ip, plugin_id, job, cls N/A
grafana_plugin_request_duration_seconds_bucket Unknown endpoint, ins, instance, target, ip, le, status, plugin_id, source, job, cls N/A
grafana_plugin_request_duration_seconds_count Unknown endpoint, ins, instance, target, ip, status, plugin_id, source, job, cls N/A
grafana_plugin_request_duration_seconds_sum Unknown endpoint, ins, instance, target, ip, status, plugin_id, source, job, cls N/A
grafana_plugin_request_size_bytes_bucket Unknown endpoint, ins, instance, target, ip, le, plugin_id, source, job, cls N/A
grafana_plugin_request_size_bytes_count Unknown endpoint, ins, instance, target, ip, plugin_id, source, job, cls N/A
grafana_plugin_request_size_bytes_sum Unknown endpoint, ins, instance, target, ip, plugin_id, source, job, cls N/A
grafana_plugin_request_total Unknown endpoint, ins, instance, target, ip, status, plugin_id, job, cls N/A
grafana_process_cpu_seconds_total Unknown ins, instance, ip, job, cls N/A
grafana_process_max_fds gauge ins, instance, ip, job, cls Maximum number of open file descriptors.
grafana_process_open_fds gauge ins, instance, ip, job, cls Number of open file descriptors.
grafana_process_resident_memory_bytes gauge ins, instance, ip, job, cls Resident memory size in bytes.
grafana_process_start_time_seconds gauge ins, instance, ip, job, cls Start time of the process since unix epoch in seconds.
grafana_process_virtual_memory_bytes gauge ins, instance, ip, job, cls Virtual memory size in bytes.
grafana_process_virtual_memory_max_bytes gauge ins, instance, ip, job, cls Maximum amount of virtual memory available in bytes.
grafana_prometheus_plugin_backend_request_count unknown endpoint, ins, instance, ip, status, errorSource, job, cls The total amount of prometheus backend plugin requests
grafana_proxy_response_status_total Unknown ins, instance, ip, job, cls, code N/A
grafana_public_dashboard_request_count unknown ins, instance, ip, job, cls counter for public dashboards requests
grafana_registered_metrics_total Unknown ins, instance, ip, stability_level, deprecated_version, job, cls N/A
grafana_rendering_queue_size gauge ins, instance, ip, job, cls size of rendering queue
grafana_search_dashboard_search_failures_duration_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
grafana_search_dashboard_search_failures_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
grafana_search_dashboard_search_failures_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_search_dashboard_search_successes_duration_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
grafana_search_dashboard_search_successes_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
grafana_search_dashboard_search_successes_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
grafana_stat_active_users gauge ins, instance, ip, job, cls number of active users
grafana_stat_total_orgs gauge ins, instance, ip, job, cls total amount of orgs
grafana_stat_total_playlists gauge ins, instance, ip, job, cls total amount of playlists
grafana_stat_total_service_account_tokens gauge ins, instance, ip, job, cls total amount of service account tokens
grafana_stat_total_service_accounts gauge ins, instance, ip, job, cls total amount of service accounts
grafana_stat_total_service_accounts_role_none gauge ins, instance, ip, job, cls total amount of service accounts with no role
grafana_stat_total_teams gauge ins, instance, ip, job, cls total amount of teams
grafana_stat_total_users gauge ins, instance, ip, job, cls total amount of users
grafana_stat_totals_active_admins gauge ins, instance, ip, job, cls total amount of active admins
grafana_stat_totals_active_editors gauge ins, instance, ip, job, cls total amount of active editors
grafana_stat_totals_active_viewers gauge ins, instance, ip, job, cls total amount of active viewers
grafana_stat_totals_admins gauge ins, instance, ip, job, cls total amount of admins
grafana_stat_totals_alert_rules gauge ins, instance, ip, job, cls total amount of alert rules in the database
grafana_stat_totals_annotations gauge ins, instance, ip, job, cls total amount of annotations in the database
grafana_stat_totals_correlations gauge ins, instance, ip, job, cls total amount of correlations
grafana_stat_totals_dashboard gauge ins, instance, ip, job, cls total amount of dashboards
grafana_stat_totals_dashboard_versions gauge ins, instance, ip, job, cls total amount of dashboard versions in the database
grafana_stat_totals_data_keys gauge ins, instance, ip, job, cls, active total amount of data keys in the database
grafana_stat_totals_datasource gauge ins, instance, ip, plugin_id, job, cls total number of defined datasources, labeled by pluginId
grafana_stat_totals_editors gauge ins, instance, ip, job, cls total amount of editors
grafana_stat_totals_folder gauge ins, instance, ip, job, cls total amount of folders
grafana_stat_totals_library_panels gauge ins, instance, ip, job, cls total amount of library panels in the database
grafana_stat_totals_library_variables gauge ins, instance, ip, job, cls total amount of library variables in the database
grafana_stat_totals_public_dashboard gauge ins, instance, ip, job, cls total amount of public dashboards
grafana_stat_totals_rule_groups gauge ins, instance, ip, job, cls total amount of alert rule groups in the database
grafana_stat_totals_viewers gauge ins, instance, ip, job, cls total amount of viewers
infra_up Unknown ins, instance, ip, job, cls N/A
jaeger_tracer_baggage_restrictions_updates_total Unknown result, ins, instance, ip, job, cls N/A
jaeger_tracer_baggage_truncations_total Unknown ins, instance, ip, job, cls N/A
jaeger_tracer_baggage_updates_total Unknown result, ins, instance, ip, job, cls N/A
jaeger_tracer_finished_spans_total Unknown ins, instance, ip, sampled, job, cls N/A
jaeger_tracer_reporter_queue_length gauge ins, instance, ip, job, cls Current number of spans in the reporter queue
jaeger_tracer_reporter_spans_total Unknown result, ins, instance, ip, job, cls N/A
jaeger_tracer_sampler_queries_total Unknown result, ins, instance, ip, job, cls N/A
jaeger_tracer_sampler_updates_total Unknown result, ins, instance, ip, job, cls N/A
jaeger_tracer_span_context_decoding_errors_total Unknown ins, instance, ip, job, cls N/A
jaeger_tracer_started_spans_total Unknown ins, instance, ip, sampled, job, cls N/A
jaeger_tracer_throttled_debug_spans_total Unknown ins, instance, ip, job, cls N/A
jaeger_tracer_throttler_updates_total Unknown result, ins, instance, ip, job, cls N/A
jaeger_tracer_traces_total Unknown ins, instance, ip, sampled, job, cls, state N/A
kv_request_duration_seconds_bucket Unknown ins, instance, role, ip, le, kv_name, type, operation, job, cls, status_code N/A
kv_request_duration_seconds_count Unknown ins, instance, role, ip, kv_name, type, operation, job, cls, status_code N/A
kv_request_duration_seconds_sum Unknown ins, instance, role, ip, kv_name, type, operation, job, cls, status_code N/A
legacy_grafana_alerting_ticker_interval_seconds gauge ins, instance, ip, job, cls Interval at which the ticker is meant to tick.
legacy_grafana_alerting_ticker_last_consumed_tick_timestamp_seconds gauge ins, instance, ip, job, cls Timestamp of the last consumed tick in seconds.
legacy_grafana_alerting_ticker_next_tick_timestamp_seconds gauge ins, instance, ip, job, cls Timestamp of the next tick in seconds before it is consumed.
logql_query_duration_seconds_bucket Unknown ins, instance, query_type, ip, le, job, cls N/A
logql_query_duration_seconds_count Unknown ins, instance, query_type, ip, job, cls N/A
logql_query_duration_seconds_sum Unknown ins, instance, query_type, ip, job, cls N/A
loki_azure_blob_egress_bytes_total Unknown ins, instance, ip, job, cls N/A
loki_boltdb_shipper_apply_retention_last_successful_run_timestamp_seconds gauge ins, instance, ip, job, cls Unix timestamp of the last successful retention run
loki_boltdb_shipper_compact_tables_operation_duration_seconds gauge ins, instance, ip, job, cls Time (in seconds) spent in compacting all the tables
loki_boltdb_shipper_compact_tables_operation_last_successful_run_timestamp_seconds gauge ins, instance, ip, job, cls Unix timestamp of the last successful compaction run
loki_boltdb_shipper_compact_tables_operation_total Unknown ins, instance, ip, status, job, cls N/A
loki_boltdb_shipper_compactor_running gauge ins, instance, ip, job, cls Value will be 1 if compactor is currently running on this instance
loki_boltdb_shipper_open_existing_file_failures_total Unknown ins, instance, ip, component, job, cls N/A
loki_boltdb_shipper_query_time_table_download_duration_seconds unknown ins, instance, ip, component, job, cls, table Time (in seconds) spent in downloading of files per table at query time
loki_boltdb_shipper_request_duration_seconds_bucket Unknown ins, instance, ip, le, component, operation, job, cls, status_code N/A
loki_boltdb_shipper_request_duration_seconds_count Unknown ins, instance, ip, component, operation, job, cls, status_code N/A
loki_boltdb_shipper_request_duration_seconds_sum Unknown ins, instance, ip, component, operation, job, cls, status_code N/A
loki_boltdb_shipper_tables_download_operation_duration_seconds gauge ins, instance, ip, component, job, cls Time (in seconds) spent in downloading updated files for all the tables
loki_boltdb_shipper_tables_sync_operation_total Unknown ins, instance, ip, status, component, job, cls N/A
loki_boltdb_shipper_tables_upload_operation_total Unknown ins, instance, ip, status, component, job, cls N/A
loki_build_info gauge revision, version, ins, instance, ip, tags, goarch, goversion, job, cls, branch, goos A metric with a constant ‘1’ value labeled by version, revision, branch, goversion from which loki was built, and the goos and goarch for the build.
loki_bytes_per_line_bucket Unknown ins, instance, ip, le, job, cls N/A
loki_bytes_per_line_count Unknown ins, instance, ip, job, cls N/A
loki_bytes_per_line_sum Unknown ins, instance, ip, job, cls N/A
loki_cache_corrupt_chunks_total Unknown ins, instance, ip, job, cls N/A
loki_cache_fetched_keys unknown ins, instance, ip, job, cls Total count of keys requested from cache.
loki_cache_hits unknown ins, instance, ip, job, cls Total count of keys found in cache.
loki_cache_request_duration_seconds_bucket Unknown ins, instance, method, ip, le, job, cls, status_code N/A
loki_cache_request_duration_seconds_count Unknown ins, instance, method, ip, job, cls, status_code N/A
loki_cache_request_duration_seconds_sum Unknown ins, instance, method, ip, job, cls, status_code N/A
loki_cache_value_size_bytes_bucket Unknown ins, instance, method, ip, le, job, cls N/A
loki_cache_value_size_bytes_count Unknown ins, instance, method, ip, job, cls N/A
loki_cache_value_size_bytes_sum Unknown ins, instance, method, ip, job, cls N/A
loki_chunk_fetcher_cache_dequeued_total Unknown ins, instance, ip, job, cls N/A
loki_chunk_fetcher_cache_enqueued_total Unknown ins, instance, ip, job, cls N/A
loki_chunk_fetcher_cache_skipped_buffer_full_total Unknown ins, instance, ip, job, cls N/A
loki_chunk_fetcher_fetched_size_bytes_bucket Unknown ins, instance, ip, le, source, job, cls N/A
loki_chunk_fetcher_fetched_size_bytes_count Unknown ins, instance, ip, source, job, cls N/A
loki_chunk_fetcher_fetched_size_bytes_sum Unknown ins, instance, ip, source, job, cls N/A
loki_chunk_store_chunks_per_query_bucket Unknown ins, instance, ip, le, job, cls N/A
loki_chunk_store_chunks_per_query_count Unknown ins, instance, ip, job, cls N/A
loki_chunk_store_chunks_per_query_sum Unknown ins, instance, ip, job, cls N/A
loki_chunk_store_deduped_bytes_total Unknown ins, instance, ip, job, cls N/A
loki_chunk_store_deduped_chunks_total Unknown ins, instance, ip, job, cls N/A
loki_chunk_store_fetched_chunk_bytes_total Unknown ins, instance, ip, user, job, cls N/A
loki_chunk_store_fetched_chunks_total Unknown ins, instance, ip, user, job, cls N/A
loki_chunk_store_index_entries_per_chunk_bucket Unknown ins, instance, ip, le, job, cls N/A
loki_chunk_store_index_entries_per_chunk_count Unknown ins, instance, ip, job, cls N/A
loki_chunk_store_index_entries_per_chunk_sum Unknown ins, instance, ip, job, cls N/A
loki_chunk_store_index_lookups_per_query_bucket Unknown ins, instance, ip, le, job, cls N/A
loki_chunk_store_index_lookups_per_query_count Unknown ins, instance, ip, job, cls N/A
loki_chunk_store_index_lookups_per_query_sum Unknown ins, instance, ip, job, cls N/A
loki_chunk_store_series_post_intersection_per_query_bucket Unknown ins, instance, ip, le, job, cls N/A
loki_chunk_store_series_post_intersection_per_query_count Unknown ins, instance, ip, job, cls N/A
loki_chunk_store_series_post_intersection_per_query_sum Unknown ins, instance, ip, job, cls N/A
loki_chunk_store_series_pre_intersection_per_query_bucket Unknown ins, instance, ip, le, job, cls N/A
loki_chunk_store_series_pre_intersection_per_query_count Unknown ins, instance, ip, job, cls N/A
loki_chunk_store_series_pre_intersection_per_query_sum Unknown ins, instance, ip, job, cls N/A
loki_chunk_store_stored_chunk_bytes_total Unknown ins, instance, ip, user, job, cls N/A
loki_chunk_store_stored_chunks_total Unknown ins, instance, ip, user, job, cls N/A
loki_consul_request_duration_seconds_bucket Unknown ins, instance, ip, le, kv_name, operation, job, cls, status_code N/A
loki_consul_request_duration_seconds_count Unknown ins, instance, ip, kv_name, operation, job, cls, status_code N/A
loki_consul_request_duration_seconds_sum Unknown ins, instance, ip, kv_name, operation, job, cls, status_code N/A
loki_delete_request_lookups_failed_total Unknown ins, instance, ip, job, cls N/A
loki_delete_request_lookups_total Unknown ins, instance, ip, job, cls N/A
loki_discarded_bytes_total Unknown ins, instance, ip, reason, job, cls, tenant N/A
loki_discarded_samples_total Unknown ins, instance, ip, reason, job, cls, tenant N/A
loki_distributor_bytes_received_total Unknown ins, instance, retention_hours, ip, job, cls, tenant N/A
loki_distributor_ingester_appends_total Unknown ins, instance, ip, ingester, job, cls N/A
loki_distributor_lines_received_total Unknown ins, instance, ip, job, cls, tenant N/A
loki_distributor_replication_factor gauge ins, instance, ip, job, cls The configured replication factor.
loki_distributor_structured_metadata_bytes_received_total Unknown ins, instance, retention_hours, ip, job, cls, tenant N/A
loki_experimental_features_in_use_total Unknown ins, instance, ip, job, cls N/A
loki_index_chunk_refs_total Unknown ins, instance, ip, status, job, cls N/A
loki_index_request_duration_seconds_bucket Unknown ins, instance, ip, le, component, operation, job, cls, status_code N/A
loki_index_request_duration_seconds_count Unknown ins, instance, ip, component, operation, job, cls, status_code N/A
loki_index_request_duration_seconds_sum Unknown ins, instance, ip, component, operation, job, cls, status_code N/A
loki_inflight_requests gauge ins, instance, method, ip, route, job, cls Current number of inflight requests.
loki_ingester_autoforget_unhealthy_ingesters_total Unknown ins, instance, ip, job, cls N/A
loki_ingester_blocks_per_chunk_bucket Unknown ins, instance, ip, le, job, cls N/A
loki_ingester_blocks_per_chunk_count Unknown ins, instance, ip, job, cls N/A
loki_ingester_blocks_per_chunk_sum Unknown ins, instance, ip, job, cls N/A
loki_ingester_checkpoint_creations_failed_total Unknown ins, instance, ip, job, cls N/A
loki_ingester_checkpoint_creations_total Unknown ins, instance, ip, job, cls N/A
loki_ingester_checkpoint_deletions_failed_total Unknown ins, instance, ip, job, cls N/A
loki_ingester_checkpoint_deletions_total Unknown ins, instance, ip, job, cls N/A
loki_ingester_checkpoint_duration_seconds summary ins, instance, ip, job, cls, quantile Time taken to create a checkpoint.
loki_ingester_checkpoint_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
loki_ingester_checkpoint_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
loki_ingester_checkpoint_logged_bytes_total Unknown ins, instance, ip, job, cls N/A
loki_ingester_chunk_age_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
loki_ingester_chunk_age_seconds_count Unknown ins, instance, ip, job, cls N/A
loki_ingester_chunk_age_seconds_sum Unknown ins, instance, ip, job, cls N/A
loki_ingester_chunk_bounds_hours_bucket Unknown ins, instance, ip, le, job, cls N/A
loki_ingester_chunk_bounds_hours_count Unknown ins, instance, ip, job, cls N/A
loki_ingester_chunk_bounds_hours_sum Unknown ins, instance, ip, job, cls N/A
loki_ingester_chunk_compression_ratio_bucket Unknown ins, instance, ip, le, job, cls N/A
loki_ingester_chunk_compression_ratio_count Unknown ins, instance, ip, job, cls N/A
loki_ingester_chunk_compression_ratio_sum Unknown ins, instance, ip, job, cls N/A
loki_ingester_chunk_encode_time_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
loki_ingester_chunk_encode_time_seconds_count Unknown ins, instance, ip, job, cls N/A
loki_ingester_chunk_encode_time_seconds_sum Unknown ins, instance, ip, job, cls N/A
loki_ingester_chunk_entries_bucket Unknown ins, instance, ip, le, job, cls N/A
loki_ingester_chunk_entries_count Unknown ins, instance, ip, job, cls N/A
loki_ingester_chunk_entries_sum Unknown ins, instance, ip, job, cls N/A
loki_ingester_chunk_size_bytes_bucket Unknown ins, instance, ip, le, job, cls N/A
loki_ingester_chunk_size_bytes_count Unknown ins, instance, ip, job, cls N/A
loki_ingester_chunk_size_bytes_sum Unknown ins, instance, ip, job, cls N/A
loki_ingester_chunk_stored_bytes_total Unknown ins, instance, ip, job, cls, tenant N/A
loki_ingester_chunk_utilization_bucket Unknown ins, instance, ip, le, job, cls N/A
loki_ingester_chunk_utilization_count Unknown ins, instance, ip, job, cls N/A
loki_ingester_chunk_utilization_sum Unknown ins, instance, ip, job, cls N/A
loki_ingester_chunks_created_total Unknown ins, instance, ip, job, cls N/A
loki_ingester_chunks_flushed_total Unknown ins, instance, ip, reason, job, cls N/A
loki_ingester_chunks_stored_total Unknown ins, instance, ip, job, cls, tenant N/A
loki_ingester_client_request_duration_seconds_bucket Unknown ins, instance, ip, le, operation, job, cls, status_code N/A
loki_ingester_client_request_duration_seconds_count Unknown ins, instance, ip, operation, job, cls, status_code N/A
loki_ingester_client_request_duration_seconds_sum Unknown ins, instance, ip, operation, job, cls, status_code N/A
loki_ingester_limiter_enabled gauge ins, instance, ip, job, cls Whether the ingester’s limiter is enabled
loki_ingester_memory_chunks gauge ins, instance, ip, job, cls The total number of chunks in memory.
loki_ingester_memory_streams gauge ins, instance, ip, job, cls, tenant The total number of streams in memory per tenant.
loki_ingester_memory_streams_labels_bytes gauge ins, instance, ip, job, cls Total bytes of labels of the streams in memory.
loki_ingester_received_chunks unknown ins, instance, ip, job, cls The total number of chunks received by this ingester whilst joining.
loki_ingester_samples_per_chunk_bucket Unknown ins, instance, ip, le, job, cls N/A
loki_ingester_samples_per_chunk_count Unknown ins, instance, ip, job, cls N/A
loki_ingester_samples_per_chunk_sum Unknown ins, instance, ip, job, cls N/A
loki_ingester_sent_chunks unknown ins, instance, ip, job, cls The total number of chunks sent by this ingester whilst leaving.
loki_ingester_shutdown_marker gauge ins, instance, ip, job, cls 1 if prepare shutdown has been called, 0 otherwise
loki_ingester_streams_created_total Unknown ins, instance, ip, job, cls, tenant N/A
loki_ingester_streams_removed_total Unknown ins, instance, ip, job, cls, tenant N/A
loki_ingester_wal_bytes_in_use gauge ins, instance, ip, job, cls Total number of bytes in use by the WAL recovery process.
loki_ingester_wal_disk_full_failures_total Unknown ins, instance, ip, job, cls N/A
loki_ingester_wal_duplicate_entries_total Unknown ins, instance, ip, job, cls N/A
loki_ingester_wal_logged_bytes_total Unknown ins, instance, ip, job, cls N/A
loki_ingester_wal_records_logged_total Unknown ins, instance, ip, job, cls N/A
loki_ingester_wal_recovered_bytes_total Unknown ins, instance, ip, job, cls N/A
loki_ingester_wal_recovered_chunks_total Unknown ins, instance, ip, job, cls N/A
loki_ingester_wal_recovered_entries_total Unknown ins, instance, ip, job, cls N/A
loki_ingester_wal_recovered_streams_total Unknown ins, instance, ip, job, cls N/A
loki_ingester_wal_replay_active gauge ins, instance, ip, job, cls Whether the WAL is replaying
loki_ingester_wal_replay_duration_seconds gauge ins, instance, ip, job, cls Time taken to replay the checkpoint and the WAL.
loki_ingester_wal_replay_flushing gauge ins, instance, ip, job, cls Whether the wal replay is in a flushing phase due to backpressure
loki_internal_log_messages_total Unknown ins, instance, ip, level, job, cls N/A
loki_kv_request_duration_seconds_bucket Unknown ins, instance, role, ip, le, kv_name, type, operation, job, cls, status_code N/A
loki_kv_request_duration_seconds_count Unknown ins, instance, role, ip, kv_name, type, operation, job, cls, status_code N/A
loki_kv_request_duration_seconds_sum Unknown ins, instance, role, ip, kv_name, type, operation, job, cls, status_code N/A
loki_log_flushes_bucket Unknown ins, instance, ip, le, job, cls N/A
loki_log_flushes_count Unknown ins, instance, ip, job, cls N/A
loki_log_flushes_sum Unknown ins, instance, ip, job, cls N/A
loki_log_messages_total Unknown ins, instance, ip, level, job, cls N/A
loki_logql_querystats_bytes_processed_per_seconds_bucket Unknown ins, instance, range, ip, le, sharded, type, job, cls, status_code, latency_type N/A
loki_logql_querystats_bytes_processed_per_seconds_count Unknown ins, instance, range, ip, sharded, type, job, cls, status_code, latency_type N/A
loki_logql_querystats_bytes_processed_per_seconds_sum Unknown ins, instance, range, ip, sharded, type, job, cls, status_code, latency_type N/A
loki_logql_querystats_chunk_download_latency_seconds_bucket Unknown ins, instance, range, ip, le, type, job, cls, status_code N/A
loki_logql_querystats_chunk_download_latency_seconds_count Unknown ins, instance, range, ip, type, job, cls, status_code N/A
loki_logql_querystats_chunk_download_latency_seconds_sum Unknown ins, instance, range, ip, type, job, cls, status_code N/A
loki_logql_querystats_downloaded_chunk_total Unknown ins, instance, range, ip, type, job, cls, status_code N/A
loki_logql_querystats_duplicates_total Unknown ins, instance, ip, job, cls N/A
loki_logql_querystats_ingester_sent_lines_total Unknown ins, instance, ip, job, cls N/A
loki_logql_querystats_latency_seconds_bucket Unknown ins, instance, range, ip, le, type, job, cls, status_code N/A
loki_logql_querystats_latency_seconds_count Unknown ins, instance, range, ip, type, job, cls, status_code N/A
loki_logql_querystats_latency_seconds_sum Unknown ins, instance, range, ip, type, job, cls, status_code N/A
loki_panic_total Unknown ins, instance, ip, job, cls N/A
loki_querier_index_cache_corruptions_total Unknown ins, instance, ip, job, cls N/A
loki_querier_index_cache_encode_errors_total Unknown ins, instance, ip, job, cls N/A
loki_querier_index_cache_gets_total Unknown ins, instance, ip, job, cls N/A
loki_querier_index_cache_hits_total Unknown ins, instance, ip, job, cls N/A
loki_querier_index_cache_puts_total Unknown ins, instance, ip, job, cls N/A
loki_querier_query_frontend_clients gauge ins, instance, ip, job, cls The current number of clients connected to query-frontend.
loki_querier_query_frontend_request_duration_seconds_bucket Unknown ins, instance, ip, le, operation, job, cls, status_code N/A
loki_querier_query_frontend_request_duration_seconds_count Unknown ins, instance, ip, operation, job, cls, status_code N/A
loki_querier_query_frontend_request_duration_seconds_sum Unknown ins, instance, ip, operation, job, cls, status_code N/A
loki_querier_tail_active gauge ins, instance, ip, job, cls Number of active tailers
loki_querier_tail_active_streams gauge ins, instance, ip, job, cls Number of active streams being tailed
loki_querier_tail_bytes_total Unknown ins, instance, ip, job, cls N/A
loki_querier_worker_concurrency gauge ins, instance, ip, job, cls Number of concurrent querier workers
loki_querier_worker_inflight_queries gauge ins, instance, ip, job, cls Number of queries being processed by the querier workers
loki_query_frontend_log_result_cache_hit_total Unknown ins, instance, ip, job, cls N/A
loki_query_frontend_log_result_cache_miss_total Unknown ins, instance, ip, job, cls N/A
loki_query_frontend_partitions_bucket Unknown ins, instance, ip, le, job, cls N/A
loki_query_frontend_partitions_count Unknown ins, instance, ip, job, cls N/A
loki_query_frontend_partitions_sum Unknown ins, instance, ip, job, cls N/A
loki_query_frontend_shard_factor_bucket Unknown ins, instance, ip, le, mapper, job, cls N/A
loki_query_frontend_shard_factor_count Unknown ins, instance, ip, mapper, job, cls N/A
loki_query_frontend_shard_factor_sum Unknown ins, instance, ip, mapper, job, cls N/A
loki_query_scheduler_enqueue_count Unknown ins, instance, ip, level, user, job, cls N/A
loki_rate_store_expired_streams_total Unknown ins, instance, ip, job, cls N/A
loki_rate_store_max_stream_rate_bytes gauge ins, instance, ip, job, cls The maximum stream rate for any stream reported by ingesters during a sync operation. Sharded Streams are combined.
loki_rate_store_max_stream_shards gauge ins, instance, ip, job, cls The number of shards for a single stream reported by ingesters during a sync operation.
loki_rate_store_max_unique_stream_rate_bytes gauge ins, instance, ip, job, cls The maximum stream rate for any stream reported by ingesters during a sync operation. Sharded Streams are considered separate.
loki_rate_store_stream_rate_bytes_bucket Unknown ins, instance, ip, le, job, cls N/A
loki_rate_store_stream_rate_bytes_count Unknown ins, instance, ip, job, cls N/A
loki_rate_store_stream_rate_bytes_sum Unknown ins, instance, ip, job, cls N/A
loki_rate_store_stream_shards_bucket Unknown ins, instance, ip, le, job, cls N/A
loki_rate_store_stream_shards_count Unknown ins, instance, ip, job, cls N/A
loki_rate_store_stream_shards_sum Unknown ins, instance, ip, job, cls N/A
loki_rate_store_streams gauge ins, instance, ip, job, cls The number of unique streams reported by all ingesters. Sharded streams are combined
loki_request_duration_seconds_bucket Unknown ins, instance, method, ip, le, ws, route, job, cls, status_code N/A
loki_request_duration_seconds_count Unknown ins, instance, method, ip, ws, route, job, cls, status_code N/A
loki_request_duration_seconds_sum Unknown ins, instance, method, ip, ws, route, job, cls, status_code N/A
loki_request_message_bytes_bucket Unknown ins, instance, method, ip, le, route, job, cls N/A
loki_request_message_bytes_count Unknown ins, instance, method, ip, route, job, cls N/A
loki_request_message_bytes_sum Unknown ins, instance, method, ip, route, job, cls N/A
loki_response_message_bytes_bucket Unknown ins, instance, method, ip, le, route, job, cls N/A
loki_response_message_bytes_count Unknown ins, instance, method, ip, route, job, cls N/A
loki_response_message_bytes_sum Unknown ins, instance, method, ip, route, job, cls N/A
loki_results_cache_version_comparisons_total Unknown ins, instance, ip, job, cls N/A
loki_store_chunks_downloaded_total Unknown ins, instance, ip, status, job, cls N/A
loki_store_chunks_per_batch_bucket Unknown ins, instance, ip, le, status, job, cls N/A
loki_store_chunks_per_batch_count Unknown ins, instance, ip, status, job, cls N/A
loki_store_chunks_per_batch_sum Unknown ins, instance, ip, status, job, cls N/A
loki_store_series_total Unknown ins, instance, ip, status, job, cls N/A
loki_stream_sharding_count unknown ins, instance, ip, job, cls Total number of times the distributor has sharded streams
loki_tcp_connections gauge ins, instance, ip, protocol, job, cls Current number of accepted TCP connections.
loki_tcp_connections_limit gauge ins, instance, ip, protocol, job, cls The max number of TCP connections that can be accepted (0 means no limit).
net_conntrack_dialer_conn_attempted_total counter ins, instance, ip, dialer_name, job, cls Total number of connections attempted by the given dialer a given name.
net_conntrack_dialer_conn_closed_total counter ins, instance, ip, dialer_name, job, cls Total number of connections closed which originated from the dialer of a given name.
net_conntrack_dialer_conn_established_total counter ins, instance, ip, dialer_name, job, cls Total number of connections successfully established by the given dialer a given name.
net_conntrack_dialer_conn_failed_total counter ins, instance, ip, dialer_name, reason, job, cls Total number of connections failed to dial by the dialer a given name.
net_conntrack_listener_conn_accepted_total counter ins, instance, ip, listener_name, job, cls Total number of connections opened to the listener of a given name.
net_conntrack_listener_conn_closed_total counter ins, instance, ip, listener_name, job, cls Total number of connections closed that were made to the listener of a given name.
nginx_connections_accepted counter ins, instance, ip, job, cls Accepted client connections
nginx_connections_active gauge ins, instance, ip, job, cls Active client connections
nginx_connections_handled counter ins, instance, ip, job, cls Handled client connections
nginx_connections_reading gauge ins, instance, ip, job, cls Connections where NGINX is reading the request header
nginx_connections_waiting gauge ins, instance, ip, job, cls Idle client connections
nginx_connections_writing gauge ins, instance, ip, job, cls Connections where NGINX is writing the response back to the client
nginx_exporter_build_info gauge revision, version, ins, instance, ip, tags, goarch, goversion, job, cls, branch, goos A metric with a constant ‘1’ value labeled by version, revision, branch, goversion from which nginx_exporter was built, and the goos and goarch for the build.
nginx_http_requests_total counter ins, instance, ip, job, cls Total http requests
nginx_up gauge ins, instance, ip, job, cls Status of the last metric scrape
plugins_active_instances gauge ins, instance, ip, job, cls The number of active plugin instances
plugins_datasource_instances_total Unknown ins, instance, ip, job, cls N/A
process_cpu_seconds_total counter ins, instance, ip, job, cls Total user and system CPU time spent in seconds.
process_max_fds gauge ins, instance, ip, job, cls Maximum number of open file descriptors.
process_open_fds gauge ins, instance, ip, job, cls Number of open file descriptors.
process_resident_memory_bytes gauge ins, instance, ip, job, cls Resident memory size in bytes.
process_start_time_seconds gauge ins, instance, ip, job, cls Start time of the process since unix epoch in seconds.
process_virtual_memory_bytes gauge ins, instance, ip, job, cls Virtual memory size in bytes.
process_virtual_memory_max_bytes gauge ins, instance, ip, job, cls Maximum amount of virtual memory available in bytes.
prometheus_api_remote_read_queries gauge ins, instance, ip, job, cls The current number of remote read queries being executed or waiting.
prometheus_build_info gauge revision, version, ins, instance, ip, tags, goarch, goversion, job, cls, branch, goos A metric with a constant ‘1’ value labeled by version, revision, branch, goversion from which prometheus was built, and the goos and goarch for the build.
prometheus_config_last_reload_success_timestamp_seconds gauge ins, instance, ip, job, cls Timestamp of the last successful configuration reload.
prometheus_config_last_reload_successful gauge ins, instance, ip, job, cls Whether the last configuration reload attempt was successful.
prometheus_engine_queries gauge ins, instance, ip, job, cls The current number of queries being executed or waiting.
prometheus_engine_queries_concurrent_max gauge ins, instance, ip, job, cls The max number of concurrent queries.
prometheus_engine_query_duration_seconds summary ins, instance, ip, job, cls, quantile, slice Query timings
prometheus_engine_query_duration_seconds_count Unknown ins, instance, ip, job, cls, slice N/A
prometheus_engine_query_duration_seconds_sum Unknown ins, instance, ip, job, cls, slice N/A
prometheus_engine_query_log_enabled gauge ins, instance, ip, job, cls State of the query log.
prometheus_engine_query_log_failures_total counter ins, instance, ip, job, cls The number of query log failures.
prometheus_engine_query_samples_total counter ins, instance, ip, job, cls The total number of samples loaded by all queries.
prometheus_http_request_duration_seconds_bucket Unknown ins, instance, ip, le, job, cls, handler N/A
prometheus_http_request_duration_seconds_count Unknown ins, instance, ip, job, cls, handler N/A
prometheus_http_request_duration_seconds_sum Unknown ins, instance, ip, job, cls, handler N/A
prometheus_http_requests_total counter ins, instance, ip, job, cls, code, handler Counter of HTTP requests.
prometheus_http_response_size_bytes_bucket Unknown ins, instance, ip, le, job, cls, handler N/A
prometheus_http_response_size_bytes_count Unknown ins, instance, ip, job, cls, handler N/A
prometheus_http_response_size_bytes_sum Unknown ins, instance, ip, job, cls, handler N/A
prometheus_notifications_alertmanagers_discovered gauge ins, instance, ip, job, cls The number of alertmanagers discovered and active.
prometheus_notifications_dropped_total counter ins, instance, ip, job, cls Total number of alerts dropped due to errors when sending to Alertmanager.
prometheus_notifications_errors_total counter ins, instance, ip, alertmanager, job, cls Total number of errors sending alert notifications.
prometheus_notifications_latency_seconds summary ins, instance, ip, alertmanager, job, cls, quantile Latency quantiles for sending alert notifications.
prometheus_notifications_latency_seconds_count Unknown ins, instance, ip, alertmanager, job, cls N/A
prometheus_notifications_latency_seconds_sum Unknown ins, instance, ip, alertmanager, job, cls N/A
prometheus_notifications_queue_capacity gauge ins, instance, ip, job, cls The capacity of the alert notifications queue.
prometheus_notifications_queue_length gauge ins, instance, ip, job, cls The number of alert notifications in the queue.
prometheus_notifications_sent_total counter ins, instance, ip, alertmanager, job, cls Total number of alerts sent.
prometheus_ready gauge ins, instance, ip, job, cls Whether Prometheus startup was fully completed and the server is ready for normal operation.
prometheus_remote_storage_exemplars_in_total counter ins, instance, ip, job, cls Exemplars in to remote storage, compare to exemplars out for queue managers.
prometheus_remote_storage_highest_timestamp_in_seconds gauge ins, instance, ip, job, cls Highest timestamp that has come into the remote storage via the Appender interface, in seconds since epoch.
prometheus_remote_storage_histograms_in_total counter ins, instance, ip, job, cls HistogramSamples in to remote storage, compare to histograms out for queue managers.
prometheus_remote_storage_samples_in_total counter ins, instance, ip, job, cls Samples in to remote storage, compare to samples out for queue managers.
prometheus_remote_storage_string_interner_zero_reference_releases_total counter ins, instance, ip, job, cls The number of times release has been called for strings that are not interned.
prometheus_rule_evaluation_duration_seconds summary ins, instance, ip, job, cls, quantile The duration for a rule to execute.
prometheus_rule_evaluation_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
prometheus_rule_evaluation_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
prometheus_rule_evaluation_failures_total counter ins, instance, ip, job, cls, rule_group The total number of rule evaluation failures.
prometheus_rule_evaluations_total counter ins, instance, ip, job, cls, rule_group The total number of rule evaluations.
prometheus_rule_group_duration_seconds summary ins, instance, ip, job, cls, quantile The duration of rule group evaluations.
prometheus_rule_group_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
prometheus_rule_group_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
prometheus_rule_group_interval_seconds gauge ins, instance, ip, job, cls, rule_group The interval of a rule group.
prometheus_rule_group_iterations_missed_total counter ins, instance, ip, job, cls, rule_group The total number of rule group evaluations missed due to slow rule group evaluation.
prometheus_rule_group_iterations_total counter ins, instance, ip, job, cls, rule_group The total number of scheduled rule group evaluations, whether executed or missed.
prometheus_rule_group_last_duration_seconds gauge ins, instance, ip, job, cls, rule_group The duration of the last rule group evaluation.
prometheus_rule_group_last_evaluation_samples gauge ins, instance, ip, job, cls, rule_group The number of samples returned during the last rule group evaluation.
prometheus_rule_group_last_evaluation_timestamp_seconds gauge ins, instance, ip, job, cls, rule_group The timestamp of the last rule group evaluation in seconds.
prometheus_rule_group_rules gauge ins, instance, ip, job, cls, rule_group The number of rules.
prometheus_sd_azure_cache_hit_total counter ins, instance, ip, job, cls Number of cache hit during refresh.
prometheus_sd_azure_failures_total counter ins, instance, ip, job, cls Number of Azure service discovery refresh failures.
prometheus_sd_consul_rpc_duration_seconds summary endpoint, ins, instance, ip, job, cls, call, quantile The duration of a Consul RPC call in seconds.
prometheus_sd_consul_rpc_duration_seconds_count Unknown endpoint, ins, instance, ip, job, cls, call N/A
prometheus_sd_consul_rpc_duration_seconds_sum Unknown endpoint, ins, instance, ip, job, cls, call N/A
prometheus_sd_consul_rpc_failures_total counter ins, instance, ip, job, cls The number of Consul RPC call failures.
prometheus_sd_discovered_targets gauge ins, instance, ip, config, job, cls Current number of discovered targets.
prometheus_sd_dns_lookup_failures_total counter ins, instance, ip, job, cls The number of DNS-SD lookup failures.
prometheus_sd_dns_lookups_total counter ins, instance, ip, job, cls The number of DNS-SD lookups.
prometheus_sd_failed_configs gauge ins, instance, ip, job, cls Current number of service discovery configurations that failed to load.
prometheus_sd_file_mtime_seconds gauge ins, instance, ip, filename, job, cls Timestamp (mtime) of files read by FileSD. Timestamp is set at read time.
prometheus_sd_file_read_errors_total counter ins, instance, ip, job, cls The number of File-SD read errors.
prometheus_sd_file_scan_duration_seconds summary ins, instance, ip, job, cls, quantile The duration of the File-SD scan in seconds.
prometheus_sd_file_scan_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
prometheus_sd_file_scan_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
prometheus_sd_file_watcher_errors_total counter ins, instance, ip, job, cls The number of File-SD errors caused by filesystem watch failures.
prometheus_sd_http_failures_total counter ins, instance, ip, job, cls Number of HTTP service discovery refresh failures.
prometheus_sd_kubernetes_events_total counter event, ins, instance, role, ip, job, cls The number of Kubernetes events handled.
prometheus_sd_kuma_fetch_duration_seconds summary ins, instance, ip, job, cls, quantile The duration of a Kuma MADS fetch call.
prometheus_sd_kuma_fetch_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
prometheus_sd_kuma_fetch_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
prometheus_sd_kuma_fetch_failures_total counter ins, instance, ip, job, cls The number of Kuma MADS fetch call failures.
prometheus_sd_kuma_fetch_skipped_updates_total counter ins, instance, ip, job, cls The number of Kuma MADS fetch calls that result in no updates to the targets.
prometheus_sd_linode_failures_total counter ins, instance, ip, job, cls Number of Linode service discovery refresh failures.
prometheus_sd_nomad_failures_total counter ins, instance, ip, job, cls Number of nomad service discovery refresh failures.
prometheus_sd_received_updates_total counter ins, instance, ip, job, cls Total number of update events received from the SD providers.
prometheus_sd_updates_total counter ins, instance, ip, job, cls Total number of update events sent to the SD consumers.
prometheus_target_interval_length_seconds summary ins, instance, interval, ip, job, cls, quantile Actual intervals between scrapes.
prometheus_target_interval_length_seconds_count Unknown ins, instance, interval, ip, job, cls N/A
prometheus_target_interval_length_seconds_sum Unknown ins, instance, interval, ip, job, cls N/A
prometheus_target_metadata_cache_bytes gauge ins, instance, ip, scrape_job, job, cls The number of bytes that are currently used for storing metric metadata in the cache
prometheus_target_metadata_cache_entries gauge ins, instance, ip, scrape_job, job, cls Total number of metric metadata entries in the cache
prometheus_target_scrape_pool_exceeded_label_limits_total counter ins, instance, ip, job, cls Total number of times scrape pools hit the label limits, during sync or config reload.
prometheus_target_scrape_pool_exceeded_target_limit_total counter ins, instance, ip, job, cls Total number of times scrape pools hit the target limit, during sync or config reload.
prometheus_target_scrape_pool_reloads_failed_total counter ins, instance, ip, job, cls Total number of failed scrape pool reloads.
prometheus_target_scrape_pool_reloads_total counter ins, instance, ip, job, cls Total number of scrape pool reloads.
prometheus_target_scrape_pool_sync_total counter ins, instance, ip, scrape_job, job, cls Total number of syncs that were executed on a scrape pool.
prometheus_target_scrape_pool_target_limit gauge ins, instance, ip, scrape_job, job, cls Maximum number of targets allowed in this scrape pool.
prometheus_target_scrape_pool_targets gauge ins, instance, ip, scrape_job, job, cls Current number of targets in this scrape pool.
prometheus_target_scrape_pools_failed_total counter ins, instance, ip, job, cls Total number of scrape pool creations that failed.
prometheus_target_scrape_pools_total counter ins, instance, ip, job, cls Total number of scrape pool creation attempts.
prometheus_target_scrapes_cache_flush_forced_total counter ins, instance, ip, job, cls How many times a scrape cache was flushed due to getting big while scrapes are failing.
prometheus_target_scrapes_exceeded_body_size_limit_total counter ins, instance, ip, job, cls Total number of scrapes that hit the body size limit
prometheus_target_scrapes_exceeded_native_histogram_bucket_limit_total counter ins, instance, ip, job, cls Total number of scrapes that hit the native histogram bucket limit and were rejected.
prometheus_target_scrapes_exceeded_sample_limit_total counter ins, instance, ip, job, cls Total number of scrapes that hit the sample limit and were rejected.
prometheus_target_scrapes_exemplar_out_of_order_total counter ins, instance, ip, job, cls Total number of exemplar rejected due to not being out of the expected order.
prometheus_target_scrapes_sample_duplicate_timestamp_total counter ins, instance, ip, job, cls Total number of samples rejected due to duplicate timestamps but different values.
prometheus_target_scrapes_sample_out_of_bounds_total counter ins, instance, ip, job, cls Total number of samples rejected due to timestamp falling outside of the time bounds.
prometheus_target_scrapes_sample_out_of_order_total counter ins, instance, ip, job, cls Total number of samples rejected due to not being out of the expected order.
prometheus_target_sync_failed_total counter ins, instance, ip, scrape_job, job, cls Total number of target sync failures.
prometheus_target_sync_length_seconds summary ins, instance, ip, scrape_job, job, cls, quantile Actual interval to sync the scrape pool.
prometheus_target_sync_length_seconds_count Unknown ins, instance, ip, scrape_job, job, cls N/A
prometheus_target_sync_length_seconds_sum Unknown ins, instance, ip, scrape_job, job, cls N/A
prometheus_template_text_expansion_failures_total counter ins, instance, ip, job, cls The total number of template text expansion failures.
prometheus_template_text_expansions_total counter ins, instance, ip, job, cls The total number of template text expansions.
prometheus_treecache_watcher_goroutines gauge ins, instance, ip, job, cls The current number of watcher goroutines.
prometheus_treecache_zookeeper_failures_total counter ins, instance, ip, job, cls The total number of ZooKeeper failures.
prometheus_tsdb_blocks_loaded gauge ins, instance, ip, job, cls Number of currently loaded data blocks
prometheus_tsdb_checkpoint_creations_failed_total counter ins, instance, ip, job, cls Total number of checkpoint creations that failed.
prometheus_tsdb_checkpoint_creations_total counter ins, instance, ip, job, cls Total number of checkpoint creations attempted.
prometheus_tsdb_checkpoint_deletions_failed_total counter ins, instance, ip, job, cls Total number of checkpoint deletions that failed.
prometheus_tsdb_checkpoint_deletions_total counter ins, instance, ip, job, cls Total number of checkpoint deletions attempted.
prometheus_tsdb_clean_start gauge ins, instance, ip, job, cls -1: lockfile is disabled. 0: a lockfile from a previous execution was replaced. 1: lockfile creation was clean
prometheus_tsdb_compaction_chunk_range_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
prometheus_tsdb_compaction_chunk_range_seconds_count Unknown ins, instance, ip, job, cls N/A
prometheus_tsdb_compaction_chunk_range_seconds_sum Unknown ins, instance, ip, job, cls N/A
prometheus_tsdb_compaction_chunk_samples_bucket Unknown ins, instance, ip, le, job, cls N/A
prometheus_tsdb_compaction_chunk_samples_count Unknown ins, instance, ip, job, cls N/A
prometheus_tsdb_compaction_chunk_samples_sum Unknown ins, instance, ip, job, cls N/A
prometheus_tsdb_compaction_chunk_size_bytes_bucket Unknown ins, instance, ip, le, job, cls N/A
prometheus_tsdb_compaction_chunk_size_bytes_count Unknown ins, instance, ip, job, cls N/A
prometheus_tsdb_compaction_chunk_size_bytes_sum Unknown ins, instance, ip, job, cls N/A
prometheus_tsdb_compaction_duration_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
prometheus_tsdb_compaction_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
prometheus_tsdb_compaction_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
prometheus_tsdb_compaction_populating_block gauge ins, instance, ip, job, cls Set to 1 when a block is currently being written to the disk.
prometheus_tsdb_compactions_failed_total counter ins, instance, ip, job, cls Total number of compactions that failed for the partition.
prometheus_tsdb_compactions_skipped_total counter ins, instance, ip, job, cls Total number of skipped compactions due to disabled auto compaction.
prometheus_tsdb_compactions_total counter ins, instance, ip, job, cls Total number of compactions that were executed for the partition.
prometheus_tsdb_compactions_triggered_total counter ins, instance, ip, job, cls Total number of triggered compactions for the partition.
prometheus_tsdb_data_replay_duration_seconds gauge ins, instance, ip, job, cls Time taken to replay the data on disk.
prometheus_tsdb_exemplar_exemplars_appended_total counter ins, instance, ip, job, cls Total number of appended exemplars.
prometheus_tsdb_exemplar_exemplars_in_storage gauge ins, instance, ip, job, cls Number of exemplars currently in circular storage.
prometheus_tsdb_exemplar_last_exemplars_timestamp_seconds gauge ins, instance, ip, job, cls The timestamp of the oldest exemplar stored in circular storage. Useful to check for what timerange the current exemplar buffer limit allows. This usually means the last timestampfor all exemplars for a typical setup. This is not true though if one of the series timestamp is in future compared to rest series.
prometheus_tsdb_exemplar_max_exemplars gauge ins, instance, ip, job, cls Total number of exemplars the exemplar storage can store, resizeable.
prometheus_tsdb_exemplar_out_of_order_exemplars_total counter ins, instance, ip, job, cls Total number of out of order exemplar ingestion failed attempts.
prometheus_tsdb_exemplar_series_with_exemplars_in_storage gauge ins, instance, ip, job, cls Number of series with exemplars currently in circular storage.
prometheus_tsdb_head_active_appenders gauge ins, instance, ip, job, cls Number of currently active appender transactions
prometheus_tsdb_head_chunks gauge ins, instance, ip, job, cls Total number of chunks in the head block.
prometheus_tsdb_head_chunks_created_total counter ins, instance, ip, job, cls Total number of chunks created in the head
prometheus_tsdb_head_chunks_removed_total counter ins, instance, ip, job, cls Total number of chunks removed in the head
prometheus_tsdb_head_chunks_storage_size_bytes gauge ins, instance, ip, job, cls Size of the chunks_head directory.
prometheus_tsdb_head_gc_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
prometheus_tsdb_head_gc_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
prometheus_tsdb_head_max_time gauge ins, instance, ip, job, cls Maximum timestamp of the head block. The unit is decided by the library consumer.
prometheus_tsdb_head_max_time_seconds gauge ins, instance, ip, job, cls Maximum timestamp of the head block.
prometheus_tsdb_head_min_time gauge ins, instance, ip, job, cls Minimum time bound of the head block. The unit is decided by the library consumer.
prometheus_tsdb_head_min_time_seconds gauge ins, instance, ip, job, cls Minimum time bound of the head block.
prometheus_tsdb_head_out_of_order_samples_appended_total counter ins, instance, ip, job, cls Total number of appended out of order samples.
prometheus_tsdb_head_samples_appended_total counter ins, instance, ip, type, job, cls Total number of appended samples.
prometheus_tsdb_head_series gauge ins, instance, ip, job, cls Total number of series in the head block.
prometheus_tsdb_head_series_created_total counter ins, instance, ip, job, cls Total number of series created in the head
prometheus_tsdb_head_series_not_found_total counter ins, instance, ip, job, cls Total number of requests for series that were not found.
prometheus_tsdb_head_series_removed_total counter ins, instance, ip, job, cls Total number of series removed in the head
prometheus_tsdb_head_truncations_failed_total counter ins, instance, ip, job, cls Total number of head truncations that failed.
prometheus_tsdb_head_truncations_total counter ins, instance, ip, job, cls Total number of head truncations attempted.
prometheus_tsdb_isolation_high_watermark gauge ins, instance, ip, job, cls The highest TSDB append ID that has been given out.
prometheus_tsdb_isolation_low_watermark gauge ins, instance, ip, job, cls The lowest TSDB append ID that is still referenced.
prometheus_tsdb_lowest_timestamp gauge ins, instance, ip, job, cls Lowest timestamp value stored in the database. The unit is decided by the library consumer.
prometheus_tsdb_lowest_timestamp_seconds gauge ins, instance, ip, job, cls Lowest timestamp value stored in the database.
prometheus_tsdb_mmap_chunk_corruptions_total counter ins, instance, ip, job, cls Total number of memory-mapped chunk corruptions.
prometheus_tsdb_mmap_chunks_total counter ins, instance, ip, job, cls Total number of chunks that were memory-mapped.
prometheus_tsdb_out_of_bound_samples_total counter ins, instance, ip, type, job, cls Total number of out of bound samples ingestion failed attempts with out of order support disabled.
prometheus_tsdb_out_of_order_samples_total counter ins, instance, ip, type, job, cls Total number of out of order samples ingestion failed attempts due to out of order being disabled.
prometheus_tsdb_reloads_failures_total counter ins, instance, ip, job, cls Number of times the database failed to reloadBlocks block data from disk.
prometheus_tsdb_reloads_total counter ins, instance, ip, job, cls Number of times the database reloaded block data from disk.
prometheus_tsdb_retention_limit_bytes gauge ins, instance, ip, job, cls Max number of bytes to be retained in the tsdb blocks, configured 0 means disabled
prometheus_tsdb_retention_limit_seconds gauge ins, instance, ip, job, cls How long to retain samples in storage.
prometheus_tsdb_size_retentions_total counter ins, instance, ip, job, cls The number of times that blocks were deleted because the maximum number of bytes was exceeded.
prometheus_tsdb_snapshot_replay_error_total counter ins, instance, ip, job, cls Total number snapshot replays that failed.
prometheus_tsdb_storage_blocks_bytes gauge ins, instance, ip, job, cls The number of bytes that are currently used for local storage by all blocks.
prometheus_tsdb_symbol_table_size_bytes gauge ins, instance, ip, job, cls Size of symbol table in memory for loaded blocks
prometheus_tsdb_time_retentions_total counter ins, instance, ip, job, cls The number of times that blocks were deleted because the maximum time limit was exceeded.
prometheus_tsdb_tombstone_cleanup_seconds_bucket Unknown ins, instance, ip, le, job, cls N/A
prometheus_tsdb_tombstone_cleanup_seconds_count Unknown ins, instance, ip, job, cls N/A
prometheus_tsdb_tombstone_cleanup_seconds_sum Unknown ins, instance, ip, job, cls N/A
prometheus_tsdb_too_old_samples_total counter ins, instance, ip, type, job, cls Total number of out of order samples ingestion failed attempts with out of support enabled, but sample outside of time window.
prometheus_tsdb_vertical_compactions_total counter ins, instance, ip, job, cls Total number of compactions done on overlapping blocks.
prometheus_tsdb_wal_completed_pages_total counter ins, instance, ip, job, cls Total number of completed pages.
prometheus_tsdb_wal_corruptions_total counter ins, instance, ip, job, cls Total number of WAL corruptions.
prometheus_tsdb_wal_fsync_duration_seconds summary ins, instance, ip, job, cls, quantile Duration of write log fsync.
prometheus_tsdb_wal_fsync_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
prometheus_tsdb_wal_fsync_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
prometheus_tsdb_wal_page_flushes_total counter ins, instance, ip, job, cls Total number of page flushes.
prometheus_tsdb_wal_segment_current gauge ins, instance, ip, job, cls Write log segment index that TSDB is currently writing to.
prometheus_tsdb_wal_storage_size_bytes gauge ins, instance, ip, job, cls Size of the write log directory.
prometheus_tsdb_wal_truncate_duration_seconds_count Unknown ins, instance, ip, job, cls N/A
prometheus_tsdb_wal_truncate_duration_seconds_sum Unknown ins, instance, ip, job, cls N/A
prometheus_tsdb_wal_truncations_failed_total counter ins, instance, ip, job, cls Total number of write log truncations that failed.
prometheus_tsdb_wal_truncations_total counter ins, instance, ip, job, cls Total number of write log truncations attempted.
prometheus_tsdb_wal_writes_failed_total counter ins, instance, ip, job, cls Total number of write log writes that failed.
prometheus_web_federation_errors_total counter ins, instance, ip, job, cls Total number of errors that occurred while sending federation responses.
prometheus_web_federation_warnings_total counter ins, instance, ip, job, cls Total number of warnings that occurred while sending federation responses.
promhttp_metric_handler_requests_in_flight gauge ins, instance, ip, job, cls Current number of scrapes being served.
promhttp_metric_handler_requests_total counter ins, instance, ip, job, cls, code Total number of scrapes by HTTP status code.
pushgateway_build_info gauge revision, version, ins, instance, ip, tags, goarch, goversion, job, cls, branch, goos A metric with a constant ‘1’ value labeled by version, revision, branch, goversion from which pushgateway was built, and the goos and goarch for the build.
pushgateway_http_requests_total counter ins, instance, method, ip, job, cls, code, handler Total HTTP requests processed by the Pushgateway, excluding scrapes.
querier_cache_added_new_total Unknown ins, instance, ip, job, cache, cls N/A
querier_cache_added_total Unknown ins, instance, ip, job, cache, cls N/A
querier_cache_entries gauge ins, instance, ip, job, cache, cls The total number of entries
querier_cache_evicted_total Unknown ins, instance, ip, job, reason, cache, cls N/A
querier_cache_gets_total Unknown ins, instance, ip, job, cache, cls N/A
querier_cache_memory_bytes gauge ins, instance, ip, job, cache, cls The current cache size in bytes
querier_cache_misses_total Unknown ins, instance, ip, job, cache, cls N/A
querier_cache_stale_gets_total Unknown ins, instance, ip, job, cache, cls N/A
ring_member_heartbeats_total Unknown ins, instance, ip, job, cls N/A
ring_member_tokens_owned gauge ins, instance, ip, job, cls The number of tokens owned in the ring.
ring_member_tokens_to_own gauge ins, instance, ip, job, cls The number of tokens to own in the ring.
scrape_duration_seconds Unknown ins, instance, ip, job, cls N/A
scrape_samples_post_metric_relabeling Unknown ins, instance, ip, job, cls N/A
scrape_samples_scraped Unknown ins, instance, ip, job, cls N/A
scrape_series_added Unknown ins, instance, ip, job, cls N/A
up Unknown ins, instance, ip, job, cls N/A

PING Metrics

PING job has 54 metrics, provided by blackbox_exporter.

Metric Name Type Labels Description
agent_up Unknown ins, ip, job, instance, cls N/A
probe_dns_lookup_time_seconds gauge ins, ip, job, instance, cls Returns the time taken for probe dns lookup in seconds
probe_duration_seconds gauge ins, ip, job, instance, cls Returns how long the probe took to complete in seconds
probe_icmp_duration_seconds gauge ins, ip, job, phase, instance, cls Duration of icmp request by phase
probe_icmp_reply_hop_limit gauge ins, ip, job, instance, cls Replied packet hop limit (TTL for ipv4)
probe_ip_addr_hash gauge ins, ip, job, instance, cls Specifies the hash of IP address. It’s useful to detect if the IP address changes.
probe_ip_protocol gauge ins, ip, job, instance, cls Specifies whether probe ip protocol is IP4 or IP6
probe_success gauge ins, ip, job, instance, cls Displays whether or not the probe was a success
scrape_duration_seconds Unknown ins, ip, job, instance, cls N/A
scrape_samples_post_metric_relabeling Unknown ins, ip, job, instance, cls N/A
scrape_samples_scraped Unknown ins, ip, job, instance, cls N/A
scrape_series_added Unknown ins, ip, job, instance, cls N/A
up Unknown ins, ip, job, instance, cls N/A

PUSH Metrics

PushGateway provides 44 metrics.

Metric Name Type Labels Description
agent_up Unknown job, cls, instance, ins, ip N/A
go_gc_duration_seconds summary job, cls, instance, ins, quantile, ip A summary of the pause duration of garbage collection cycles.
go_gc_duration_seconds_count Unknown job, cls, instance, ins, ip N/A
go_gc_duration_seconds_sum Unknown job, cls, instance, ins, ip N/A
go_goroutines gauge job, cls, instance, ins, ip Number of goroutines that currently exist.
go_info gauge job, cls, instance, ins, ip, version Information about the Go environment.
go_memstats_alloc_bytes counter job, cls, instance, ins, ip Total number of bytes allocated, even if freed.
go_memstats_alloc_bytes_total counter job, cls, instance, ins, ip Total number of bytes allocated, even if freed.
go_memstats_buck_hash_sys_bytes gauge job, cls, instance, ins, ip Number of bytes used by the profiling bucket hash table.
go_memstats_frees_total counter job, cls, instance, ins, ip Total number of frees.
go_memstats_gc_sys_bytes gauge job, cls, instance, ins, ip Number of bytes used for garbage collection system metadata.
go_memstats_heap_alloc_bytes gauge job, cls, instance, ins, ip Number of heap bytes allocated and still in use.
go_memstats_heap_idle_bytes gauge job, cls, instance, ins, ip Number of heap bytes waiting to be used.
go_memstats_heap_inuse_bytes gauge job, cls, instance, ins, ip Number of heap bytes that are in use.
go_memstats_heap_objects gauge job, cls, instance, ins, ip Number of allocated objects.
go_memstats_heap_released_bytes gauge job, cls, instance, ins, ip Number of heap bytes released to OS.
go_memstats_heap_sys_bytes gauge job, cls, instance, ins, ip Number of heap bytes obtained from system.
go_memstats_last_gc_time_seconds gauge job, cls, instance, ins, ip Number of seconds since 1970 of last garbage collection.
go_memstats_lookups_total counter job, cls, instance, ins, ip Total number of pointer lookups.
go_memstats_mallocs_total counter job, cls, instance, ins, ip Total number of mallocs.
go_memstats_mcache_inuse_bytes gauge job, cls, instance, ins, ip Number of bytes in use by mcache structures.
go_memstats_mcache_sys_bytes gauge job, cls, instance, ins, ip Number of bytes used for mcache structures obtained from system.
go_memstats_mspan_inuse_bytes gauge job, cls, instance, ins, ip Number of bytes in use by mspan structures.
go_memstats_mspan_sys_bytes gauge job, cls, instance, ins, ip Number of bytes used for mspan structures obtained from system.
go_memstats_next_gc_bytes gauge job, cls, instance, ins, ip Number of heap bytes when next garbage collection will take place.
go_memstats_other_sys_bytes gauge job, cls, instance, ins, ip Number of bytes used for other system allocations.
go_memstats_stack_inuse_bytes gauge job, cls, instance, ins, ip Number of bytes in use by the stack allocator.
go_memstats_stack_sys_bytes gauge job, cls, instance, ins, ip Number of bytes obtained from system for stack allocator.
go_memstats_sys_bytes gauge job, cls, instance, ins, ip Number of bytes obtained from system.
go_threads gauge job, cls, instance, ins, ip Number of OS threads created.
process_cpu_seconds_total counter job, cls, instance, ins, ip Total user and system CPU time spent in seconds.
process_max_fds gauge job, cls, instance, ins, ip Maximum number of open file descriptors.
process_open_fds gauge job, cls, instance, ins, ip Number of open file descriptors.
process_resident_memory_bytes gauge job, cls, instance, ins, ip Resident memory size in bytes.
process_start_time_seconds gauge job, cls, instance, ins, ip Start time of the process since unix epoch in seconds.
process_virtual_memory_bytes gauge job, cls, instance, ins, ip Virtual memory size in bytes.
process_virtual_memory_max_bytes gauge job, cls, instance, ins, ip Maximum amount of virtual memory available in bytes.
pushgateway_build_info gauge job, goversion, cls, branch, instance, tags, revision, goarch, ins, ip, version, goos A metric with a constant ‘1’ value labeled by version, revision, branch, goversion from which pushgateway was built, and the goos and goarch for the build.
pushgateway_http_requests_total counter job, cls, method, code, handler, instance, ins, ip Total HTTP requests processed by the Pushgateway, excluding scrapes.
scrape_duration_seconds Unknown job, cls, instance, ins, ip N/A
scrape_samples_post_metric_relabeling Unknown job, cls, instance, ins, ip N/A
scrape_samples_scraped Unknown job, cls, instance, ins, ip N/A
scrape_series_added Unknown job, cls, instance, ins, ip N/A
up Unknown job, cls, instance, ins, ip N/A

9.6 - FAQ

Frequently asked questions about the Pigsty INFRA infrastructure module

What components are included in the INFRA module?

Strictly following the current source, the infra role directly manages:

  • Nginx: Exposes Grafana, VictoriaMetrics (VMUI), Alertmanager, and other WebUIs, and hosts local YUM/APT repositories.
  • DNSMasq: Provides DNS registration and resolution.
  • VictoriaMetrics suite: VictoriaMetrics, VMAlert, VictoriaLogs, and VictoriaTraces.
  • Alertmanager, Blackbox Exporter, and Grafana: Alert dispatch, blackbox probing, and visualization.

infra.yml also chains the CA, repository, NODE, HAProxy, and node-monitoring roles, so it configures supporting capabilities such as the self-signed CA, Chronyd, Node Exporter, and Vector on Infra nodes. ETCD, PostgreSQL, and Docker are separate modules and are not deployed by infra.yml; use etcd.yml, pgsql.yml, and docker.yml, respectively.

How to re-register monitoring targets to VictoriaMetrics?

VictoriaMetrics uses static service discovery through the /infra/targets/<job>/*.yml directory. If target files are accidentally deleted, use the following commands to re-register:

./infra.yml  -t infra_register   # Re-render Infra self-monitoring targets
./node.yml   -t node_register    # Re-render node / HAProxy / Vector targets
./etcd.yml   -t etcd_register    # Re-render Etcd targets
./minio.yml  -t minio_register   # Re-render Silo module targets
./pgsql.yml  -t pg_register      # Re-render PGSQL/Patroni targets
./redis.yml  -t redis_register   # Re-render Redis targets

Other modules (such as pg_monitor.yml and mysql.yml) also provide corresponding *_register tags that can be executed as needed.

How to re-register PostgreSQL datasources to Grafana?

PGSQL databases defined in pg_databases are registered as Grafana datasources by default (for use by PGCAT applications).

If you accidentally delete postgres datasources registered in Grafana, you can register them again using the following command:

# Register all pgsql databases (defined in pg_databases) as grafana datasources
./pgsql.yml -t register_grafana

How to re-register node HAProxy admin pages to Nginx?

If you accidentally delete the registered haproxy proxy settings in /etc/nginx/conf.d/haproxy, you can restore them using the following command:

./node.yml -t register_nginx     # Register all haproxy admin page proxy settings to nginx on infra nodes

How to restore DNS registration records in DNSMASQ?

PGSQL cluster/instance domains are registered by default to /etc/dnsmasq.d/pigsty/<name> on infra nodes. You can restore them using the following command:

./pgsql.yml -t pg_dns    # Register pg DNS names to dnsmasq on infra nodes

How to expose new upstream services via Nginx?

Although you can access services directly via IP:Port, we still recommend consolidating access entry points by using domain names and accessing various WebUI services through Nginx proxy. This helps consolidate access, reduce exposed ports, and facilitate access control and auditing.

If you want to expose new WebUI services through the Nginx portal, you can add service definitions to the infra_portal parameter. For example, here’s the Infra portal configuration used by Pigsty’s official demo, exposing several additional services:

infra_portal:
  home         : { domain: home.pigsty.cc }
  grafana      : { domain: demo.pigsty.io ,endpoint: "${admin_ip}:3000" ,websocket: true }
  prometheus   : { domain: p.pigsty.cc ,endpoint: "${admin_ip}:8428" }
  alertmanager : { domain: a.pigsty.cc ,endpoint: "${admin_ip}:9059" }
  blackbox     : { endpoint: "${admin_ip}:9115" }
  vmalert      : { endpoint: "${admin_ip}:8880" }
  # Additional web portals
  minio        : { domain: sss.pigsty  ,endpoint: "${admin_ip}:9001" ,scheme: https ,websocket: true }
  postgrest    : { domain: api.pigsty.cc  ,endpoint: "127.0.0.1:8884"   }
  pgadmin      : { domain: adm.pigsty.cc  ,endpoint: "127.0.0.1:8885"   }
  pgweb        : { domain: cli.pigsty.cc  ,endpoint: "127.0.0.1:8886"   }
  bytebase     : { domain: ddl.pigsty.cc  ,endpoint: "127.0.0.1:8887"   }
  gitea        : { domain: git.pigsty.cc  ,endpoint: "127.0.0.1:8889"   }
  wiki         : { domain: wiki.pigsty.cc ,endpoint: "127.0.0.1:9002"   }
  noco         : { domain: noco.pigsty.cc ,endpoint: "127.0.0.1:9003"   }
  supa         : { domain: supa.pigsty.cc ,endpoint: "127.0.0.1:8000", websocket: true }

After completing the Nginx upstream service definition, use the following configuration and commands to register new services to Nginx.

./infra.yml -t nginx_config           # Regenerate Nginx configuration files
./infra.yml -t nginx_launch           # Update and apply Nginx configuration

# You can also manually reload Nginx config with Ansible
ansible infra -b -a 'nginx -s reload'  # Reload Nginx config

If you want HTTPS access, you must delete files/pki/csr/pigsty.csr and files/pki/nginx/pigsty.{key,crt} to force regeneration of Nginx SSL/TLS certificates to include new upstream domains. If you want to use certificates issued by an authoritative CA instead of Pigsty self-signed CA certificates, you can place them in the /etc/nginx/conf.d/cert/ directory and modify the corresponding configuration: /etc/nginx/conf.d/<name>.conf.

How to manually add upstream repo files to nodes?

Pigsty has a built-in wrapper script bin/repo-add that calls the ansible playbook node.yml to add repo files to corresponding nodes.

bin/repo-add <selector> [modules]
bin/repo-add 10.10.10.10           # Add node repo for node 10.10.10.10
bin/repo-add infra   node,infra    # Add node and infra repos for infra group
bin/repo-add infra   node,local    # Add node repo and local pigsty repo for infra group
bin/repo-add pg-test node,pgsql    # Add node and pgsql repos for pg-test group

9.7 - Administration

Infrastructure components and INFRA cluster administration procedures.

This section covers daily administration and operations for Pigsty deployments.

9.7.1 - Nginx Management

Nginx management, web portal configuration, web server, upstream services

Pigsty installs Nginx on INFRA nodes as the entry point for all web services, listening on standard ports 80/443.

In Pigsty, you can configure Nginx to provide various services through inventory:

  • Expose web interfaces for monitoring components like Grafana, VictoriaMetrics (VMUI), Alertmanager, and VictoriaLogs
  • Serve static files (software repos, documentation sites, websites, etc.)
  • Proxy custom application services (internal apps, database management UIs, Docker application interfaces, etc.)
  • Automatically issue self-signed HTTPS certificates, or use Certbot to obtain free Let’s Encrypt certificates
  • Expose services through a single port using different subdomains for unified access

Basic Configuration

Customize Nginx behavior via infra_portal parameter:

infra_portal:
  home: { domain: i.pigsty }

infra_portal is a dictionary where each key defines a service and the value is the service configuration. Only services with a domain defined will generate corresponding Nginx config files.

  • home: Special default server for homepage and built-in monitoring component reverse proxies
  • Proxy services: Specify upstream service address via endpoint for reverse proxy
  • Static services: Specify local directory via path for static file serving

Server Parameters

Basic Parameters

Parameter Description
domain Optional proxy domain
endpoint Upstream service address (IP:PORT or socket)
path Local directory for static content
scheme Protocol type (http/https), default http
domains Additional domain list (aliases)

SSL/TLS Options

Parameter Description
certbot Enable Let’s Encrypt cert management, value is cert name
cert Custom certificate file path
key Custom private key file path
enforce_https Force HTTPS redirect (301)

Advanced Settings

Parameter Description
config Custom Nginx config snippet
index Enable directory listing (for static)
log Custom log file name
websocket Enable WebSocket support
auth Enable Basic Auth
realm Basic Auth prompt message

Configuration Examples

Reverse Proxy Services 

grafana: { domain: g.pigsty, endpoint: "${admin_ip}:3000", websocket: true }
pgadmin: { domain: adm.pigsty, endpoint: "127.0.0.1:8885" }

Static Files and Directory Listing 

repo: { domain: repo.pigsty.io, path: "/www/repo", index: true }

Custom SSL Certificate 

secure_app:
  domain: secure.pigsty.io
  endpoint: "${admin_ip}:8443"
  cert: "/etc/ssl/certs/custom.crt"
  key: "/etc/ssl/private/custom.key"

Using Let’s Encrypt Certificates 

grafana:
  domain: demo.pigsty.io
  endpoint: "${admin_ip}:3000"
  websocket: true
  certbot: pigsty.demo    # Cert name, multiple domains can share one cert

Force HTTPS Redirect 

web.io:
  domain: en.pigsty.io
  path: "/www/web.io"
  certbot: pigsty.doc
  enforce_https: true

Custom Config Snippet 

web.cc:
  domain: pigsty.io
  path: "/www/web.io"
  domains: [ en.pigsty.io ]
  certbot: pigsty.doc
  config: |
    # rewrite /en/ to /
        location /en/ {
            rewrite ^/en/(.*)$ /$1 permanent;
        }

Management Commands 

./infra.yml -t nginx           # Full Nginx reconfiguration
./infra.yml -t nginx_config    # Regenerate config files
./infra.yml -t nginx_launch    # Restart Nginx service
./infra.yml -t nginx_cert      # Regenerate SSL certificates
./infra.yml -t nginx_certbot   # Sign certificates with certbot
./infra.yml -t nginx_reload    # Reload Nginx configuration

Domain Resolution

Three ways to resolve domains to Pigsty servers:

  1. Public domains: Configure via DNS provider
  2. Internal DNS server: Configure internal DNS resolution
  3. Local hosts file: Modify /etc/hosts

For local development, add to /etc/hosts:

<your_public_ip_address> i.pigsty

Pigsty includes dnsmasq service, configurable via dns_records parameter for internal DNS resolution.

HTTPS Configuration

Configure HTTPS via nginx_sslmode parameter:

Mode Description
disable Listen HTTP only (nginx_port)
enable Also listen HTTPS (nginx_ssl_port), default self-signed cert
enforce Force redirect to HTTPS, all port 80 requests get 301 redirect

For self-signed certificates, several access options:

  • Trust the self-signed CA in browser (download at http://<ip>/ca.crt)
  • Use browser security bypass (type “thisisunsafe” in Chrome)
  • Configure proper CA-signed certs or Let’s Encrypt for production

Certbot Certificates

Pigsty supports using Certbot to request free Let’s Encrypt certificates.

Enable Certbot

  1. Add certbot parameter to services in infra_portal, specifying cert name
  2. Configure certbot_email with a valid email
  3. Set certbot_sign to true for auto-signing during deployment
certbot_sign: true
certbot_email: [email protected]

Manual Certificate Signing 

./infra.yml -t nginx_certbot   # Sign Let's Encrypt certificates

Or run the scripts directly on the server:

/etc/nginx/sign-cert           # Sign certificates
/etc/nginx/link-cert           # Link certificates to Nginx config directory

For more info, see Certbot: Request and Renew HTTPS Certificates

Default Homepage

Pigsty’s default home server provides these built-in routes:

Path Description
/ Homepage navigation
/ui/ Grafana monitoring dashboards
/vmetrics/ VictoriaMetrics VMUI
/vlogs/ VictoriaLogs log query
/vtraces/ VictoriaTraces tracing
/vmalert/ VMAlert alerting rules
/alertmgr/ AlertManager alert management
/blackbox/ Blackbox Exporter
/pev PostgreSQL Explain visualization
/haproxy/<cluster>/ HAProxy admin interface (if any)

These routes allow accessing all monitoring components through a single entry point, no need for multiple domain configurations.

Best Practices

  • Use domain names instead of IP:PORT for service access
  • Properly configure DNS resolution or hosts file
  • Enable WebSocket for real-time apps (e.g., Grafana, Jupyter)
  • Enable HTTPS for production
  • Use meaningful subdomains to organize services
  • Monitor Let’s Encrypt certificate expiration
  • Use config parameter for custom Nginx configurations

Full Example

Here’s the Nginx configuration used by Pigsty’s public demo site demo.pigsty.io:

infra_portal:
  home         : { domain: i.pigsty }
  io           : { domain: pigsty.io      ,path: "/www/pigsty.io"   ,cert: /etc/cert/pigsty.io.crt ,key: /etc/cert/pigsty.io.key }
  minio        : { domain: m.pigsty.io    ,endpoint: "${admin_ip}:9001" ,scheme: https ,websocket: true }
  postgrest    : { domain: api.pigsty.io  ,endpoint: "127.0.0.1:8884" }
  pgadmin      : { domain: adm.pigsty.io  ,endpoint: "127.0.0.1:8885" }
  pgweb        : { domain: cli.pigsty.io  ,endpoint: "127.0.0.1:8886" }
  bytebase     : { domain: ddl.pigsty.io  ,endpoint: "127.0.0.1:8887" }
  jupyter      : { domain: lab.pigsty.io  ,endpoint: "127.0.0.1:8888" ,websocket: true }
  gitea        : { domain: git.pigsty.io  ,endpoint: "127.0.0.1:8889" }
  wiki         : { domain: wiki.pigsty.io ,endpoint: "127.0.0.1:9002" }
  noco         : { domain: noco.pigsty.io ,endpoint: "127.0.0.1:9003" }
  supa         : { domain: supa.pigsty.io ,endpoint: "10.10.10.10:8000" ,websocket: true }
  dify         : { domain: dify.pigsty.io ,endpoint: "10.10.10.10:8001" ,websocket: true }
  odoo         : { domain: odoo.pigsty.io ,endpoint: "127.0.0.1:8069"   ,websocket: true }
  mm           : { domain: mm.pigsty.io   ,endpoint: "10.10.10.10:8065" ,websocket: true }

9.7.2 - Software Repository

Create and maintain Pigsty local RPM/APT repositories with SOW, including completion markers, ModuleMD, and forced rebuild semantics.

Pigsty’s REPO role downloads required packages and creates a local YUM/APT repository under /www/pigsty that Nginx can serve. The current package candidate is SOW 0.3.0; the source uses SOW to generate metadata for both repository types instead of calling createrepo_c, modifyrepo_c, or dpkg-scanpackages separately.

Quick Start

Add packages to repo_packages or repo_extra_packages, then run:

./infra.yml -t repo_build   # Download and build only when the repository is absent
./node.yml -t node_repo     # Refresh repository definitions and caches on each node

If /www/pigsty/repo_complete already exists, the default repo_build skips the build. To force a rebuild, override it explicitly:

./infra.yml -t repo_build -e repo_build=true

To rebuild metadata for packages already present without downloading new ones:

./infra.yml -t repo_create

SOW Prerequisites

Both repo_create and cache_create require sow on the target node. A fresh online build automatically adds infra to the effective repo_modules list and installs SOW from the Pigsty INFRA upstream repository.

Offline bundles and local repositories created before this change may not contain SOW. Before rebuilding from old media, refresh the bundle/local repository or install the current SOW 0.3.0 candidate from the Pigsty INFRA repository. Do not assume that older environments can still fall back to createrepo_c.

On a fresh installation, if /www does not exist, the role creates /data/nginx and makes /www point to it. Existing directories and symlinks are preserved rather than forcibly replaced.

Build Flow

Task Purpose
repo_check Check repo_complete to determine whether the local repository is complete
repo_prepare Configure and use an existing repository
repo_dir Create /www/pigsty and ACME directories
repo_upstream Back up and add upstream YUM or APT definitions
repo_url_pkg Download packages from direct URLs
repo_cache Run yum makecache or apt update
repo_boot_pkg Install sow and the RPM platform’s dnf-utils / yum-utils
repo_pkg Download packages and dependencies
repo_create Run SOW to clean and atomically publish repository metadata
repo_use Write the local Pigsty repository definition on the current host
repo_nginx Start a temporary Nginx when no service is already running

The command executed by repo_create is:

sow create --pigsty --timeout 10m -- /www/pigsty

--pigsty removes unneeded or conflict-prone packages and atomically publishes the result only after all metadata has been generated. A typical layout is:

/www/pigsty/
├── *.rpm / *.deb
├── repodata/            # RPM repository
├── Packages             # APT repository
├── Packages.gz
└── repo_complete        # SHA-256 manifest and completion marker

Do not treat repo_complete as an empty sentinel; it contains SHA-256 checksums. Its presence means SOW completed publication of local repository metadata. It does not prove that remote mirrors, signed repositories, or offline bundles have been synchronized.

DNF Module Streams

Pigsty no longer fabricates modules.yaml / ModuleMD metadata for the aggregated local repository. System upstream repositories retain native DNF module filtering. Only an upstream repository that truly needs to replace an EL module stream should set meta explicitly:

- name: example
  module: pgsql
  # ... releases, arch, baseurl ...
  meta: { module_hotfixes: 1 }

The aggregated Pigsty local repository itself is configured with module_hotfixes=1 so local PostgreSQL packages are not hidden by system module streams. This is distinct from generating fake ModuleMD metadata.

Package Aliases

The default repo_packages uses these alias groups:

[node-bootstrap, infra-package, infra-addons, node-package1,
 node-package2, node-package3, pgsql-utility, extra-modules]

node-bootstrap includes Ansible, Python dependencies, SOW, and SSH tools. infra-package includes Nginx, etcd, HAProxy, Victoria exporters, Redis/Valkey, Silo, mcli, SOW, and Pig. Exact names vary by OS mapping; always use roles/node_id/vars/<os>.<arch>.yml as the authority.

Common Commands 

./infra.yml -t repo                           # Check, prepare, or build, then start repository service
./infra.yml -t repo_check,repo_prepare        # Only check and use an existing repository
./infra.yml -t repo_upstream                  # Refresh upstream repository definitions
./infra.yml -t repo_pkg                       # Download configured packages and dependencies
./infra.yml -t repo_create                    # Rebuild metadata in the existing directory with SOW
./infra.yml -t repo_build -e repo_build=true  # Force the complete build stage
./infra.yml -t repo_nginx                     # Configure/start repository Nginx
./node.yml -t node_repo                       # Refresh managed-node repository caches
./cache.yml                                   # Rebuild metadata with SOW, then create an offline bundle

9.7.3 - Domain Management

Configure local or public domain names to access Pigsty services.

Use domain names instead of IP addresses to access Pigsty’s various web services.

Quick Start

Add the following static resolution records to /etc/hosts:

10.10.10.10 i.pigsty

Replace IP address with your actual Pigsty node’s IP.

Why Use Domain Names

  • Easier to remember than IP addresses
  • Flexible pointing to different IPs
  • Unified service management through Nginx
  • Support for HTTPS encryption
  • Prevent ISP hijacking in some regions
  • Allow access to internally bound services via proxy

DNS Mechanism

DNS Protocol: Resolves domain names to IP addresses. Multiple domains can point to same IP.

HTTP Protocol: Uses Host header to route requests to different sites on same port (80/443).

Default Domains

Pigsty predefines the following default domains:

Domain Service Port Purpose
i.pigsty Nginx 80/443 Default homepage, local repo, unified entry
m.pigsty Silo 9001 Object storage console

Grafana, VictoriaMetrics, and Alertmanager are accessed by default through the /ui/, /vmetrics/, and /alertmgr/ subpaths under i.pigsty. To use dedicated domains such as g.pigsty, p.pigsty, and a.pigsty, configure them explicitly in infra_portal and dns_records.

Resolution Methods

Local Static Resolution

Add entries to /etc/hosts on the client machine:

# Linux/macOS
sudo vim /etc/hosts

# Windows
notepad C:\Windows\System32\drivers\etc\hosts

Add content:

10.10.10.10 i.pigsty m.pigsty

Internal Dynamic Resolution

Pigsty includes dnsmasq as an internal DNS server. Configure managed nodes to use INFRA node as DNS server:

node_dns_servers: ['${admin_ip}']   # Use INFRA node as DNS server
node_dns_method: add                # Add to existing DNS server list

Configure domain records resolved by dnsmasq via dns_records:

dns_records:
  - "${admin_ip} i.pigsty"
  - "${admin_ip} m.pigsty sss.pigsty api.pigsty adm.pigsty cli.pigsty ddl.pigsty"

Public Domain Names

Purchase a domain and add DNS A record pointing to public IP:

  1. Purchase domain from registrar (e.g., example.com)
  2. Configure A record pointing to server public IP
  3. Use real domain in infra_portal

Built-in DNS Service

Pigsty runs dnsmasq on INFRA nodes as a DNS server.

Related Parameters

Parameter Default Description
dns_enabled true Enable DNS service
dns_port 53 DNS listen port
dns_records See below Default DNS records

Default DNS records:

dns_records:
  - "${admin_ip} i.pigsty"
  - "${admin_ip} m.pigsty sss.pigsty api.pigsty adm.pigsty cli.pigsty ddl.pigsty"

Dynamic DNS Registration

Pigsty automatically registers DNS records for PostgreSQL clusters and instances:

  • Instance-level DNS: <pg_instance> points to instance IP (e.g., pg-meta-1)
  • Cluster-level DNS: <pg_cluster> points to primary IP or VIP (e.g., pg-meta)

Cluster-level DNS target controlled by pg_dns_target:

Value Description
auto Auto-select: use VIP if available, else primary IP
primary Always point to primary IP
vip Always point to VIP (requires VIP enabled)
none Don’t register cluster DNS
<ip> Specify fixed IP address

Add suffix to cluster DNS via pg_dns_suffix.

Node DNS Configuration

Pigsty manages DNS configuration on managed nodes.

Static hosts Records

Configure static /etc/hosts records via node_etc_hosts:

node_etc_hosts:
  - "${admin_ip} i.pigsty"
  - "${admin_ip} sss.pigsty"      # Optional: Silo S3 endpoint domain
  - "10.10.10.20 db.example.com"

DNS Server Configuration

Parameter Default Description
node_dns_method add DNS config method
node_dns_servers ['${admin_ip}'] DNS server list
node_dns_options See below resolv.conf options

node_dns_method options:

Value Description
add Prepend to existing DNS server list
overwrite Completely overwrite DNS config
none Don’t modify DNS config

Default DNS options:

node_dns_options:
  - options single-request-reopen timeout:1

HTTPS Certificates

Pigsty uses self-signed certificates by default. Options include:

  • Ignore warnings, use HTTP
  • Trust self-signed CA certificate (download at http://<ip>/ca.crt)
  • Use real CA or get free public domain certs via Certbot

See CA and Certificates documentation for details.

Extended Domains

Pigsty reserves the following domains for various application services:

Domain Purpose
adm.pigsty PgAdmin interface
ddl.pigsty Bytebase DDL management
cli.pigsty PgWeb CLI interface
api.pigsty PostgREST API service
lab.pigsty Jupyter environment
git.pigsty Gitea Git service
wiki.pigsty Wiki.js docs
noco.pigsty NocoDB
supa.pigsty Supabase
dify.pigsty Dify AI
odoo.pigsty Odoo ERP
mm.pigsty Mattermost

Using these domains requires configuring corresponding services in infra_portal.

Management Commands 

./infra.yml -t dns            # Full DNS service configuration
./infra.yml -t dns_config     # Regenerate dnsmasq config
./infra.yml -t dns_record     # Update default DNS records
./infra.yml -t dns_launch     # Restart dnsmasq service

./node.yml -t node_hosts      # Configure node /etc/hosts
./node.yml -t node_resolv     # Configure node DNS resolver

./pgsql.yml -t pg_dns         # Register PostgreSQL DNS records
./pgsql.yml -t pg_dns_ins     # Register instance-level DNS only
./pgsql.yml -t pg_dns_cls     # Register cluster-level DNS only

9.7.4 - Module Management

INFRA module management SOP: define, create, destroy, scale out, scale in

This document covers daily management operations for the INFRA module, including installation, uninstallation, scaling, and component maintenance.

Install INFRA Module

Use the infra.yml playbook to install the INFRA module on the infra group:

./infra.yml     # Install INFRA module on infra group

Uninstall INFRA Module

Use the infra-rm.yml playbook to uninstall the INFRA module from the infra group:

./infra-rm.yml -l infra # Full removal: deregister, stop, remove config/environment/data, and uninstall packages

This playbook has no deletion safeguard. Full execution removes infra_data, nginx_data, nginx_home (default: /www), and /var/lib/grafana. If you only need to stop services or deregister targets, use -t service or -t deregister. Read the complete removal scope and back up required data before running it.

Scale Out INFRA Module

Assign infra_seq to new nodes and add them to the infra group in the inventory:

all:
  children:
    infra:
      hosts:
        10.10.10.10: { infra_seq: 1 }  # Existing node
        10.10.10.11: { infra_seq: 2 }  # New node

Use the -l limit option to execute the playbook on the new node only:

./infra.yml -l 10.10.10.11    # Install INFRA module on new node

Manage Local Repository

Local repository management tasks:

./infra.yml -t repo              # Create repo from internet or offline packages
./infra.yml -t repo_upstream     # Add upstream repositories
./infra.yml -t repo_pkg          # Download packages and dependencies
./infra.yml -t repo_create       # Create local yum/apt repository

Complete subtask list:

./infra.yml -t repo_dir          # Create local repository directory
./infra.yml -t repo_check        # Check if local repo exists
./infra.yml -t repo_prepare      # Use existing repo directly
./infra.yml -t repo_build        # Build repo from upstream
./infra.yml -t repo_upstream     # Add upstream repositories
./infra.yml -t repo_remove       # Delete existing repo files
./infra.yml -t repo_add          # Add repo to system directory
./infra.yml -t repo_url_pkg      # Download packages from internet
./infra.yml -t repo_cache        # Create metadata cache
./infra.yml -t repo_boot_pkg     # Install bootstrap packages
./infra.yml -t repo_pkg          # Download packages and dependencies
./infra.yml -t repo_create       # Create local repository
./infra.yml -t repo_use          # Add new repo to system
./infra.yml -t repo_nginx        # Start Nginx file server

Manage Nginx

Nginx management tasks:

./infra.yml -t nginx                       # Reset Nginx component
./infra.yml -t nginx_index                 # Re-render homepage
./infra.yml -t nginx_config,nginx_reload   # Re-render config and reload

Request HTTPS certificate:

./infra.yml -t nginx_certbot,nginx_reload -e certbot_sign=true

Manage Infrastructure Components

Management commands for various infrastructure components:

./infra.yml -t infra           # Configure infrastructure
./infra.yml -t infra_user      # Set up OS user
./infra.yml -t infra_dir       # Create infrastructure directories
./infra.yml -t infra_env       # Configure environment variables
./infra.yml -t infra_pkg       # Install packages
./infra.yml -t infra_cert      # Issue certificates
./infra.yml -t dns             # Configure DNSMasq
./infra.yml -t nginx           # Configure Nginx
./infra.yml -t victoria        # Configure VictoriaMetrics/Logs/Traces
./infra.yml -t alertmanager    # Configure AlertManager
./infra.yml -t blackbox        # Configure Blackbox Exporter
./infra.yml -t grafana         # Configure Grafana
./infra.yml -t infra_register  # Register to VictoriaMetrics/Grafana

Common maintenance commands:

./infra.yml -t nginx_index                        # Re-render homepage
./infra.yml -t nginx_config,nginx_reload          # Reconfigure and reload
./infra.yml -t vmetrics_config,vmetrics_launch    # Regenerate VictoriaMetrics config and restart
./infra.yml -t vlogs_config,vlogs_launch          # Update VictoriaLogs config
./infra.yml -t grafana_provision                  # Reload Grafana dashboards and data-source definitions

Manage Grafana Passwords

Grafana uses two password parameters: grafana_admin_password, whose public default is pigsty, and grafana_view_password, whose default is DBUser.Viewer.

Parameter Rendered configuration
grafana_admin_password /etc/grafana/grafana.ini and /infra/env/pigsty
grafana_view_password /etc/grafana/provisioning/datasources/pigsty.yml

Change the public defaults before production. After Grafana initializes, changing grafana_admin_password in inventory does not by itself reset the live Grafana account password; change it through Grafana (or its supported administration interface), keep the inventory consistent, and rerender the environment when needed:

./infra.yml -t env_var            # Re-render environment variables

grafana_view_password is the password used by the default PostgreSQL metadb data source as dbuser_view. If that database role password changes, update both the declared value and the Grafana data source; changing only one side breaks the dashboard connection.

9.7.5 - CA and Certificates

Manage Pigsty’s self-signed CA, service certificates, and public Certbot certificates.

Pigsty maintains a self-signed Certificate Authority (CA) on the admin node by default. It signs certificates for PostgreSQL, Patroni, etcd, Silo, Nginx, and other internal services. Public Nginx entries can use Certbot/Let’s Encrypt certificates configured through infra_portal.

Protect the CA private key

files/pki/ca/ca.key is the trust-root private key for the entire deployment. Never print, commit, upload, or transmit it over an unprotected channel. Back it up together with ca.crt, encrypted, with tightly restricted read access.

Self-Signed CA

The ca stage of infra.yml creates or reuses the CA locally on the admin node running Ansible, not on a remote Infra node. The default layout is:

files/pki/
├── ca/                       # CA key, certificate, and OpenSSL CA state
│   ├── ca.key
│   └── ca.crt
├── csr/                      # certificate signing requests
├── misc/                     # generic certificates issued by cert.yml
├── etcd/
├── infra/
├── kafka/
├── minio/                    # MINIO module (Silo) certificates
├── mongo/
├── mysql/
├── nginx/
└── pgsql/

The core defaults match the v4.5.0 roles:

Parameter Default Meaning
ca_create true Allow creation when ca.key is missing
ca_cn pigsty-ca Common Name of the CA certificate
cert_validity 7300d Default internal service/client validity (20 years)
nginx_cert_validity 397d Nginx self-signed HTTPS certificate validity

The role hard-codes the CA certificate lifetime to 36500d (about 100 years). These long-lived certificates are for a controlled internal trust domain; they are not publicly browser-trusted. Clients must explicitly trust ca.crt, while public endpoints should use a publicly trusted CA.

Initialize the local CA stage:

./infra.yml -t ca

Running ./infra.yml -t ca may create a missing key or certificate and therefore changes PKI state. Confirm the admin node, configuration, and existing CA backup first.

Use an External CA

To reuse an enterprise CA:

  1. Set ca_create: false in pigsty.yml.
  2. Place a matching files/pki/ca/ca.key and files/pki/ca/ca.crt pair on the admin node.
  3. Set permissions and verify that the public-key digests match.
chmod 700 files/pki/ca
chmod 600 files/pki/ca/ca.key
chmod 644 files/pki/ca/ca.crt

# The public-key digests must match; neither command prints private-key material
openssl pkey -in files/pki/ca/ca.key -pubout -outform PEM | openssl sha256
openssl x509 -in files/pki/ca/ca.crt -pubkey -noout | openssl sha256

ca_create: false prevents generation of a new private key only when ca.key is missing. If the key exists but ca.crt does not, the role still creates a new self-signed CA certificate from that key. Always restore the pair together instead of relying on certificate regeneration.

Before running the CA stage, verify the files that will be used, the existing CA backup, and the admin node.

Back Up and Restore the CA

Retain at least:

  • files/pki/ca/ca.key and ca.crt
  • CA state such as ca.srl, index.txt, and CRL files if issuance/revocation management uses them
  • backup time, the CA certificate SHA-256 fingerprint, and restoration instructions
# Inspect the public CA certificate only
openssl x509 -in files/pki/ca/ca.crt -noout -subject -issuer -dates -fingerprint -sha256

Encrypt the backup and keep it in controlled offline media or a secrets-management system; do not leave an unencrypted tar archive. Restore into an isolated temporary directory first, then verify file count, type, permissions, public-key match, and certificate fingerprint before replacement.

Losing ca.key does not immediately make existing certificates unverifiable: they remain verifiable while clients trust ca.crt and the certificates remain valid and unrevoked. You can no longer issue, renew, or revoke with the original CA, so recovery usually requires a new CA, reissuing every certificate, and rolling out a new trust chain.

Issue Certificates with cert.yml

cert.yml runs locally on the admin node and issues generic certificates with the Pigsty CA. Pass cn explicitly instead of relying on the generic script default:

./cert.yml -e cn=dbuser_dba

Default outputs:

files/pki/misc/<cn>.key   # 0600
files/pki/misc/<cn>.crt   # 0600
files/pki/csr/<cn>.csr
Parameter Default Meaning
cn pigsty Common Name; set explicitly in real use
san [DNS:localhost, IP:127.0.0.1] Subject Alternative Names
org pigsty Organization
unit pigsty Organizational Unit
expire 7300d Validity
key files/pki/misc/<cn>.key Private-key output path
crt files/pki/misc/<cn>.crt Certificate output path

Advanced examples:

# Pass DNS/IP SANs as a JSON list
./cert.yml -e cn=myservice \
  -e '{"san":["DNS:myservice.local","DNS:myservice","IP:10.10.10.50"]}'

# Issue a one-year certificate
./cert.yml -e cn=myservice \
  -e '{"san":["DNS:myservice.local","DNS:myservice","IP:10.10.10.50"]}' \
  -e expire=365d

# When customizing key/crt, provide both paths together
./cert.yml -e cn=custom \
  -e key=/secure/path/custom.key \
  -e crt=/secure/path/custom.crt

Verify the certificate without displaying or copying private-key content:

openssl x509 -in files/pki/misc/myservice.crt -noout -subject -issuer -dates -ext subjectAltName
openssl verify -CAfile files/pki/ca/ca.crt files/pki/misc/myservice.crt

For PostgreSQL client certificates, cn must match the database role expected by HBA/cert authentication. Install the certificate, key, and root certificate on the client with the private key at mode 0600. With sslmode=verify-full, the connection hostname must appear in the server certificate SAN.

Trust the CA Certificate

Distribute only public ca.crt, never ca.key. Verify its SHA-256 fingerprint through a separate trusted channel before installation.

Debian / Ubuntu 

sudo cp ca.crt /usr/local/share/ca-certificates/pigsty-ca.crt
sudo update-ca-certificates

RHEL / Rocky / AlmaLinux 

sudo cp ca.crt /etc/pki/ca-trust/source/anchors/pigsty-ca.crt
sudo update-ca-trust

macOS 

sudo security add-trusted-cert -d -r trustRoot \
  -k /Library/Keychains/System.keychain ca.crt

Windows (Administrator PowerShell) 

Import-Certificate -FilePath .\ca.crt -CertStoreLocation Cert:\LocalMachine\Root

Infra Nginx normally exposes the public CA certificate at http://<infra_ip>/ca.crt. Verify the fingerprint after download; HTTP transport alone does not prove certificate authenticity.

Nginx and Let’s Encrypt

Each infra_portal entry can name a certbot certificate. Pigsty’s /etc/nginx/sign-cert uses Certbot webroot mode, groups domain and domains entries that share a certificate name, and then /etc/nginx/link-cert links the result into Nginx.

Prerequisites:

  • Public DNS A/AAAA records resolve to the intended Infra node.
  • Port 80 is publicly reachable for HTTP-01, and Nginx serves the ACME webroot.
  • certbot_email is valid and the Certbot package is installed.
  • Portal domains, additional domains, and certificate names are exact.
certbot_email: [email protected]
infra_portal:
  home:
    domain: example.com
    domains: [www.example.com]
    certbot: example.com
  grafana:
    domain: grafana.example.com
    endpoint: "${admin_ip}:3000"
    websocket: true
    certbot: grafana.example.com

Update the Nginx configuration and issue certificates:

dig +short example.com
./infra.yml -l infra -t nginx_config,nginx_launch

./infra.yml -l infra -t nginx_certbot,nginx_reload -e certbot_sign=true
Verify issuance separately

In v4.5.0, the nginx_certbot task has ignore_errors: true. A playbook that continues or reports overall success does not prove certificate issuance. Inspect Certbot state, certificate files, Nginx configuration, and a real TLS handshake.

certbot certificates
test -r /etc/letsencrypt/live/example.com/fullchain.pem
nginx -t
openssl s_client -connect example.com:443 -servername example.com </dev/null

Renewal scheduling depends on the Certbot package for the operating system. Do not add a duplicate cron job before checking existing timers and cron configuration:

systemctl list-timers --all | grep -i certbot
certbot renew --dry-run

After Certbot replaces certificates on disk, Nginx still needs a reload to use them. Configure and verify a renewal deploy hook such as systemctl reload nginx, or an equivalent managed process. Treat automatic renewal as proven only after a real or staging renewal exercise.

Troubleshooting and Acceptance

Symptom Check
Browser rejects an internal cert Correct ca.crt, SAN hostname, and system time
verify-full fails Connection hostname, SAN, chain, and root certificate
Certbot HTTP-01 fails DNS, port 80, ACME webroot, proxy/CDN, and rate limits
Playbook succeeds but old cert remains Ignored nginx_certbot errors, link-cert, and Nginx reload
Permission denied Private key 0600 (deployed Nginx key is 0640 root:nginx)
Trust breaks after CA rotation Roll out client trust, then service certs, then reload services

Final acceptance should separately prove correct certificate content and SANs, successful chain validation, that the service loaded the new certificate, that target clients trust it, and that renewal exists and passes a dry run. A generated file or successful playbook alone proves none of those later layers.

9.7.6 - Grafana High Availability: Using PostgreSQL Backend

Use PostgreSQL instead of SQLite as Grafana’s remote storage backend for better performance and availability.

You can use PostgreSQL as Grafana’s backend database.

This is a great opportunity to understand Pigsty’s deployment system. By completing this tutorial, you’ll learn:

TL;DR 

vi pigsty.yml # Uncomment DB/User definitions: dbuser_grafana  grafana
bin/pgsql-user  pg-meta  dbuser_grafana
bin/pgsql-db    pg-meta  grafana

psql postgres://dbuser_grafana:DBUser.Grafana@meta:5436/grafana -c \
  'CREATE TABLE t(); DROP TABLE t;' # Verify connection string works

vi /etc/grafana/grafana.ini # Modify [database] type url
systemctl restart grafana-server

Create Database Cluster

We can define a new database grafana on pg-meta, or create a dedicated Grafana database cluster pg-grafana on new nodes.

Define Cluster

To create a new dedicated cluster pg-grafana on machines 10.10.10.11 and 10.10.10.12, use this config:

pg-grafana:
  hosts:
    10.10.10.11: {pg_seq: 1, pg_role: primary}
    10.10.10.12: {pg_seq: 2, pg_role: replica}
  vars:
    pg_cluster: pg-grafana
    pg_databases:
      - name: grafana
        owner: dbuser_grafana
        revokeconn: true
        comment: grafana primary database
    pg_users:
      - name: dbuser_grafana
        password: DBUser.Grafana
        pgbouncer: true
        roles: [dbrole_admin]
        comment: admin user for grafana database

Create Cluster

Use this command to create the pg-grafana cluster: pgsql.yml.

./pgsql.yml -l pg-grafana    # Initialize pg-grafana cluster

This command is the Ansible Playbook pgsql.yml for creating database clusters.

Users and databases defined in pg_users and pg_databases are automatically created during cluster initialization. With this config, after cluster creation (without DNS), you can access the database using these connection strings (any one works):

postgres://dbuser_grafana:[email protected]:5432/grafana # Direct primary connection
postgres://dbuser_grafana:[email protected]:5436/grafana # Direct default service
postgres://dbuser_grafana:[email protected]:5433/grafana # Primary read-write service

postgres://dbuser_grafana:[email protected]:5432/grafana # Direct primary connection
postgres://dbuser_grafana:[email protected]:5436/grafana # Direct default service
postgres://dbuser_grafana:[email protected]:5433/grafana # Primary read-write service

Since Pigsty is installed on a single meta node by default, the following steps will create Grafana’s user and database on the existing pg-meta cluster, not the pg-grafana cluster created here.

Create Grafana Business User

The usual convention for business object management: create user first, then database. Because if the database has an owner configured, it depends on the corresponding user.

Define User

To create user dbuser_grafana on the pg-meta cluster, first add this user definition to pg-meta’s cluster definition:

Location: all.children.pg-meta.vars.pg_users

- name: dbuser_grafana
  password: DBUser.Grafana
  comment: admin user for grafana database
  pgbouncer: true
  roles: [ dbrole_admin ]

If you define a different password here, replace the corresponding parameter in subsequent steps

Create User

Use this command to create the dbuser_grafana user (either works):

bin/pgsql-user pg-meta dbuser_grafana # Create `dbuser_grafana` user on pg-meta cluster

This actually calls the Ansible Playbook pgsql-user.yml to create the user:

./pgsql-user.yml -l pg-meta -e pg_user=dbuser_grafana  # Ansible

The dbrole_admin role has permission to execute DDL changes in the database, which is exactly what Grafana needs.

Create Grafana Business Database

Define Database

Creating a business database follows the same pattern as users. First add the new database grafana definition to pg-meta’s cluster definition.

Location: all.children.pg-meta.vars.pg_databases

- { name: grafana, owner: dbuser_grafana, revokeconn: true }

Create Database

Use this command to create the grafana database (either works):

bin/pgsql-db pg-meta grafana # Create `grafana` database on `pg-meta` cluster

This actually calls the Ansible Playbook pgsql-db.yml to create the database:

./pgsql-db.yml -l pg-meta -e pg_database=grafana # Actual Ansible playbook executed

Use Grafana Business Database

Verify Connection String Reachability

You can access the database using different services or access methods, for example:

postgres://dbuser_grafana:DBUser.Grafana@meta:5432/grafana # Direct connection
postgres://dbuser_grafana:DBUser.Grafana@meta:5436/grafana # Default service
postgres://dbuser_grafana:DBUser.Grafana@meta:5433/grafana # Primary service

Here, we’ll use the Default service that directly accesses the primary through load balancer.

First verify the connection string is reachable and has DDL execution permissions:

psql postgres://dbuser_grafana:DBUser.Grafana@meta:5436/grafana -c \
  'CREATE TABLE t(); DROP TABLE t;'

Directly Modify Grafana Config

To make Grafana use a Postgres datasource, edit /etc/grafana/grafana.ini and modify the config:

[database]
;type = sqlite3
;host = 127.0.0.1:3306
;name = grafana
;user = root
# If the password contains # or ; you have to wrap it with triple quotes. Ex """#password;"""
;password =
;url =

Change the default config to:

[database]
type = postgres
url =  postgres://dbuser_grafana:DBUser.Grafana@meta/grafana

Then restart Grafana:

systemctl restart grafana-server

When you see activity in the newly added grafana database from the monitoring system, Grafana is now using Postgres as its primary backend database. But there’s a new issue—the original Dashboards and Datasources in Grafana have disappeared! You need to re-import dashboards and Postgres datasources.

Manage Grafana Dashboards

As admin user, navigate to the files/grafana directory under the Pigsty directory and run grafana.py init to reload Pigsty dashboards.

cd ~/pigsty/files/grafana
./grafana.py init    # Initialize Grafana dashboards using Dashboards in current directory

Execution result:

vagrant@meta:~/pigsty/files/grafana
$ ./grafana.py init
Grafana API: admin:pigsty @ http://10.10.10.10:3000
init dashboard : home.json
init folder pgcat
init dashboard: pgcat / pgcat-table.json
init dashboard: pgcat / pgcat-bloat.json
init dashboard: pgcat / pgcat-query.json
init folder pgsql
init dashboard: pgsql / pgsql-replication.json
...

This script detects the current environment (defined in ~/pigsty during installation), gets Grafana access info, and replaces dashboard URL placeholder domains (*.pigsty) with actual domains used.

export GRAFANA_ENDPOINT=http://10.10.10.10:3000
export GRAFANA_USERNAME=admin
export GRAFANA_PASSWORD=pigsty

export NGINX_UPSTREAM_YUMREPO=yum.pigsty
export NGINX_UPSTREAM_CONSUL=c.pigsty
export NGINX_UPSTREAM_PROMETHEUS=p.pigsty
export NGINX_UPSTREAM_ALERTMANAGER=a.pigsty
export NGINX_UPSTREAM_GRAFANA=g.pigsty
export NGINX_UPSTREAM_HAPROXY=h.pigsty

As a side note, use grafana.py clean to clear target dashboards, and grafana.py load to load all dashboards from the current directory. When Pigsty dashboards change, use these two commands to upgrade all dashboards.

Manage Postgres Datasources

When creating a new PostgreSQL cluster with pgsql.yml or a new business database with pgsql-db.yml, Pigsty registers new PostgreSQL datasources in Grafana. You can directly access target database instances through Grafana using the default monitoring user. Most pgcat application features depend on this.

To register Postgres databases, use the register_grafana task in pgsql.yml:

./pgsql.yml -t register_grafana             # Re-register all Postgres datasources in current environment
./pgsql.yml -t register_grafana -l pg-test  # Re-register all databases in pg-test cluster

One-Step Grafana Upgrade

You can directly modify the Pigsty config file to change Grafana’s backend datasource, completing the database switch in one step. Edit the grafana_pgurl parameter in pigsty.yml:

grafana_pgurl: postgres://dbuser_grafana:DBUser.Grafana@meta:5436/grafana

Then re-run the grafana task from infra.yml to complete the Grafana upgrade:

./infra.yml -t grafana

10 - Module: NODE

Tune nodes into the desired state and monitor it, manage node, VIP, HAProxy, and exporters.

Tune nodes into the desired state and monitor it, manage node, VIP, HAProxy, and exporters.

10.1 - Configuration

Configure node identity, cluster, and identity borrowing from PostgreSQL

Pigsty uses IP address as the unique identifier for nodes. This IP should be the internal IP address on which the database instance listens and provides external services.

node-test:
  hosts:
    10.10.10.11: { nodename: node-test-1 }
    10.10.10.12: { nodename: node-test-2 }
    10.10.10.13: { nodename: node-test-3 }
  vars:
    node_cluster: node-test

This IP address must be the address on which the database instance listens and provides external services, but should not be a public IP address. That said, you don’t necessarily have to connect to the database via this IP. For example, managing target nodes indirectly through SSH tunnels or jump hosts is also feasible. However, when identifying database nodes, the primary IPv4 address remains the node’s core identifier. This is critical, and you should ensure this during configuration.

The IP address is the inventory_hostname in the inventory, represented as the key in the <cluster>.hosts object. In addition, each node has two optional identity parameters:

Name Type Level Necessity Comment
inventory_hostname ip - Required Node IP
nodename string I Optional Node Name
node_cluster string C Optional Node cluster name

The parameters nodename and node_cluster are optional. If not provided, the node’s existing hostname and the fixed value nodes will be used as defaults. In Pigsty’s monitoring system, these two will be used as the node’s cluster identifier (cls) and instance identifier (ins).

For PGSQL nodes, because Pigsty defaults to a 1:1 exclusive deployment of PG to node, you can use the node_id_from_pg parameter to borrow the PostgreSQL instance’s identity parameters (pg_cluster and pg_seq) for the node’s ins and cls labels. This allows database and node monitoring metrics to share the same labels for cross-analysis.

#nodename:                # [instance] # node instance identity, uses existing hostname if missing, optional
node_cluster: nodes       # [cluster] # node cluster identity, uses 'nodes' if missing, optional
nodename_overwrite: true          # overwrite node's hostname with nodename?
nodename_exchange: false          # exchange nodename among play hosts?
node_id_from_pg: true             # borrow postgres identity as node identity if applicable?

You can also configure rich functionality for host clusters. For example, use HAProxy on the node cluster for load balancing and service exposure, or bind an L2 VIP to the cluster.

10.2 - Parameters

NODE module provides 11 sections with 74 parameters

The NODE module tunes target nodes into the desired state and integrates them into the Pigsty monitoring system.

Parameter Section Description
NODE_ID Node identity parameters
NODE_DNS Node DNS resolution
NODE_PACKAGE Upstream repo & package install
NODE_TUNE Node tuning & kernel features
NODE_SEC Node security configurations
NODE_ADMIN Admin user & SSH keys
NODE_TIME Timezone, NTP, crontab
NODE_VIP Optional L2 VIP for cluster
HAPROXY HAProxy load balancer
NODE_EXPORTER Node monitoring exporter
VECTOR Vector log collector

Parameters Overview

NODE_ID section defines node identity parameters, including node name, cluster name, and whether to borrow identity from PostgreSQL.

Parameter Type Level Description
nodename string I node instance identity, use hostname if missing
node_cluster string C node cluster identity, use ’nodes’ if missing
nodename_overwrite bool C overwrite node’s hostname with nodename?
nodename_exchange bool C exchange nodename among play hosts?
node_id_from_pg bool C use postgres identity as node identity if applicable?

NODE_DNS section configures node DNS resolution, including static hosts records and dynamic DNS servers.

Parameter Type Level Description
node_write_etc_hosts bool G/C/I modify /etc/hosts on target node?
node_default_etc_hosts string[] G static dns records in /etc/hosts
node_etc_hosts string[] C extra static dns records in /etc/hosts
node_dns_method enum C how to handle dns servers: add,none,overwrite
node_dns_servers string[] C dynamic nameserver in /etc/resolv.conf
node_dns_options string[] C dns resolv options in /etc/resolv.conf

NODE_PACKAGE section configures node software repositories, package installation, and uv Python virtual environment.

Parameter Type Level Description
node_repo_modules enum C which repo modules to enable on node? local default
node_repo_remove bool C remove existing repo on node when configuring?
node_packages string[] C packages to be installed on current nodes
node_default_packages string[] G default packages to be installed on all nodes
node_uv_env path C uv venv path, /data/venv by default, empty to skip
node_pip_packages string C pip packages to install in uv venv

NODE_TUNE section configures node kernel parameters, feature toggles, and tuning templates.

Parameter Type Level Description
node_disable_numa bool C disable node numa, reboot required
node_disable_swap bool C disable node swap, use with caution
node_static_network bool C preserve dns resolver settings after reboot
node_disk_prefetch bool C setup disk prefetch on HDD to increase performance
node_kernel_modules string[] C kernel modules to be enabled on this node
node_hugepage_count int C number of 2MB hugepage, take precedence over ratio
node_hugepage_ratio float C node mem hugepage ratio, 0 disable it by default
node_overcommit_ratio float C node mem overcommit ratio (50-100), 0 disable it
node_tune enum C node tuned profile: none,oltp,olap,crit,tiny
node_tuned_profile_dir path C tuned profile directory selected by platform mapping
node_sysctl_params dict C extra sysctl parameters in k:v format

NODE_SEC section configures node security options, including SELinux and firewall.

Parameter Type Level Description
node_selinux_mode enum C SELinux mode: disabled, permissive, enforcing
node_firewall_mode enum C firewall mode: zone (default, enabled), off (disable), none (self-managed)
node_firewall_intranet cidr[] C intranet CIDR list for firewall rules
node_firewall_public_port port[] C public exposed port list, default [22, 80, 443]

NODE_ADMIN section configures admin user, data directory, and shell aliases.

Parameter Type Level Description
node_data path C node main data directory, /data by default
node_admin_enabled bool C create a admin user on target node?
node_admin_uid int C uid and gid for node admin user
node_admin_username username C name of node admin user, dba by default
node_admin_sudo enum C admin sudo privilege: limited, nopass, all, none
node_admin_ssh_exchange bool C exchange admin ssh key among node cluster
node_admin_pk_current bool C add current user’s ssh pk to admin authorized_keys
node_admin_pk_list string[] C ssh public keys to be added to admin user
node_aliases dict C shell aliases in K:V dict format

NODE_TIME section configures timezone, NTP time sync, and crontab.

Parameter Type Level Description
node_timezone string C setup node timezone, empty string to skip
node_ntp_enabled bool C enable chronyd time sync service?
node_ntp_servers string[] C ntp servers in /etc/chrony.conf
node_crontab_overwrite bool C overwrite or append to /etc/crontab?
node_crontab string[] C crontab entries in /etc/crontab

NODE_VIP section configures L2 VIP for node cluster, implemented by keepalived.

Parameter Type Level Description
vip_enabled bool C enable L2 vip on this node cluster?
vip_address ip C node vip address in ipv4 format, required if enabled
vip_vrid int C integer 1-254, should be unique in same VLAN
vip_role enum I optional, master/backup, backup by default
vip_preempt bool C/I optional, true/false, enable vip preemption
vip_interface string C/I node vip network interface, auto by default
vip_dns_suffix string C node vip dns name suffix, empty string by default
vip_auth_pass password C vrrp authentication password, auto-generated if empty
vip_exporter_port port C keepalived exporter listen port, 9650 by default

HAPROXY section configures HAProxy load balancer and service exposure.

Parameter Type Level Description
haproxy_enabled bool C enable haproxy on this node?
haproxy_clean bool G/C/A cleanup all existing haproxy config?
haproxy_reload bool A reload haproxy after config?
haproxy_auth_enabled bool G enable authentication for admin page
haproxy_admin_username username G haproxy admin username, admin default
haproxy_admin_password password G haproxy admin password, pigsty default
haproxy_exporter_port port C haproxy exporter port, 9101 by default
haproxy_client_timeout interval C client connection timeout, 24h default
haproxy_server_timeout interval C server connection timeout, 24h default
haproxy_services service[] C list of haproxy services to expose

NODE_EXPORTER section configures node monitoring exporter.

Parameter Type Level Description
node_exporter_enabled bool C setup node_exporter on this node?
node_exporter_port port C node exporter listen port, 9100 default
node_exporter_options arg C extra server options for node_exporter

VECTOR section configures Vector log collector.

Parameter Type Level Description
vector_enabled bool C enable vector log collector?
vector_clean bool G/A purge vector data dir during init?
vector_data path C vector data directory, /data/vector default
vector_port port C vector metrics listen port, 9598 default
vector_read_from enum C read log from beginning or end
vector_log_endpoint string[] C log endpoint, default send to infra group

NODE_ID

Each node has identity parameters that are configured through the parameters in <cluster>.hosts and <cluster>.vars.

Pigsty uses IP address as the unique identifier for database nodes. This IP address must be the one that the database instance listens on and provides services, but should not be a public IP address. However, users don’t have to connect to the database via this IP address. For example, managing target nodes indirectly through SSH tunnels or jump servers is feasible. When identifying database nodes, the primary IPv4 address remains the core identifier. This is very important, and users should ensure this when configuring. The IP address is the inventory_hostname in the inventory, which is the key of the <cluster>.hosts object.

node-test:
  hosts:
    10.10.10.11: { nodename: node-test-1 }
    10.10.10.12: { nodename: node-test-2 }
    10.10.10.13: { nodename: node-test-3 }
  vars:
    node_cluster: node-test

In addition, nodes have two important identity parameters in the Pigsty monitoring system: nodename and node_cluster, which are used as the instance identity (ins) and cluster identity (cls) in the monitoring system.

node_load1{cls="pg-meta", ins="pg-meta-1", ip="10.10.10.10", job="nodes"}
node_load1{cls="pg-test", ins="pg-test-1", ip="10.10.10.11", job="nodes"}
node_load1{cls="pg-test", ins="pg-test-2", ip="10.10.10.12", job="nodes"}
node_load1{cls="pg-test", ins="pg-test-3", ip="10.10.10.13", job="nodes"}

When executing the default PostgreSQL deployment, since Pigsty uses exclusive 1:1 deployment by default, you can borrow the database instance’s identity parameters (pg_cluster) to the node’s ins and cls labels through the node_id_from_pg parameter.

Name Type Level Required Description
inventory_hostname ip - Required Node IP Address
nodename string I Optional Node Name
node_cluster string C Optional Node Cluster Name 
#nodename:                # [instance] # node instance identity, use hostname if missing, optional
node_cluster: nodes       # [cluster] # node cluster identity, use 'nodes' if missing, optional
nodename_overwrite: true          # overwrite node's hostname with nodename?
nodename_exchange: false          # exchange nodename among play hosts?
node_id_from_pg: true             # use postgres identity as node identity if applicable?

nodename

name: nodename, type: string, level: I

Node instance identity parameter. If not explicitly set, the existing hostname will be used as the node name. This parameter is optional since it has a reasonable default value.

If node_id_from_pg is enabled (default), and nodename is not explicitly specified, nodename will try to use ${pg_cluster}-${pg_seq} as the instance identity. If the PGSQL module is not defined on this cluster, it will fall back to the default, which is the node’s HOSTNAME.

node_cluster

name: node_cluster, type: string, level: C

This option allows explicitly specifying a cluster name for the node, which is only meaningful when defined at the node cluster level. Using the default empty value will use the fixed value nodes as the node cluster identity.

If node_id_from_pg is enabled (default), and node_cluster is not explicitly specified, node_cluster will try to use ${pg_cluster} as the cluster identity. If the PGSQL module is not defined on this cluster, it will fall back to the default nodes.

nodename_overwrite

name: nodename_overwrite, type: bool, level: C

Overwrite node’s hostname with nodename? Default is true. In this case, if you set a non-empty nodename, it will be used as the current host’s HOSTNAME.

When nodename is empty, if node_id_from_pg is true (default), Pigsty will try to borrow the identity parameters of the PostgreSQL instance defined 1:1 on the node as the node name, i.e., {{ pg_cluster }}-{{ pg_seq }}. If the PGSQL module is not installed on this node, it will fall back to not doing anything.

Therefore, if you leave nodename empty and don’t enable node_id_from_pg, Pigsty will not make any changes to the existing hostname.

nodename_exchange

name: nodename_exchange, type: bool, level: C

Exchange nodename among play hosts? Default is false.

When enabled, nodes executing the node.yml playbook in the same batch will exchange node names with each other, writing them to /etc/hosts.

node_id_from_pg

name: node_id_from_pg, type: bool, level: C

Borrow identity parameters from the PostgreSQL instance/cluster deployed 1:1 on the node? Default is true.

PostgreSQL instances and nodes in Pigsty use 1:1 deployment by default, so you can “borrow” identity parameters from the database instance. This parameter is enabled by default, meaning that if a PostgreSQL cluster has no special configuration, the host node cluster and instance identity parameters will default to matching the database identity parameters. This provides extra convenience for problem analysis and monitoring data processing.

NODE_DNS

Pigsty configures static DNS records and dynamic DNS servers for nodes.

If your node provider has already configured DNS servers for you, you can set node_dns_method to none to skip DNS setup.

node_write_etc_hosts: true        # modify `/etc/hosts` on target node?
node_default_etc_hosts:           # static dns records in `/etc/hosts`
  - "${admin_ip} i.pigsty"
node_etc_hosts: []                # extra static dns records in `/etc/hosts`
node_dns_method: add              # how to handle dns servers: add,none,overwrite
node_dns_servers: ['${admin_ip}'] # dynamic nameserver in `/etc/resolv.conf`
node_dns_options:                 # dns resolv options in `/etc/resolv.conf`
  - options single-request-reopen timeout:1

node_write_etc_hosts

name: node_write_etc_hosts, type: bool, level: G|C|I

Modify /etc/hosts on the target node? Default is true. Container environments often prohibit modifying this file; set this parameter to false to skip the change.

node_default_etc_hosts

name: node_default_etc_hosts, type: string[], level: G

Static DNS records to be written to all nodes’ /etc/hosts. Default value:

["${admin_ip} i.pigsty"]

node_default_etc_hosts is an array. Each element is a DNS record with format <ip> <name>. You can specify multiple domain names separated by spaces.

This parameter is used to configure global static DNS records. If you want to configure specific static DNS records for individual clusters and instances, use the node_etc_hosts parameter.

node_etc_hosts

name: node_etc_hosts, type: string[], level: C

Extra static DNS records to write to node’s /etc/hosts. Default is [] (empty array).

Same format as node_default_etc_hosts, but suitable for configuration at the cluster/instance level.

node_dns_method

name: node_dns_method, type: enum, level: C

How to configure DNS servers? Three options: add, none, overwrite. Default is add.

  • add: Append the records in node_dns_servers to /etc/resolv.conf and keep existing DNS servers. (default)
  • overwrite: Overwrite /etc/resolv.conf with the records in node_dns_servers
  • none: Skip DNS server configuration. If your environment already has DNS servers configured, you can skip DNS configuration directly.

node_dns_servers

name: node_dns_servers, type: string[], level: C

Configure the dynamic DNS server list in /etc/resolv.conf. Default is ["${admin_ip}"], using the admin node as the primary DNS server.

node_dns_options

name: node_dns_options, type: string[], level: C

DNS resolution options in /etc/resolv.conf. Default value:

- "options single-request-reopen timeout:1"

If node_dns_method is configured as add or overwrite, the records in this configuration will be written to /etc/resolv.conf first. Refer to Linux documentation for /etc/resolv.conf format details.

NODE_PACKAGE

Pigsty configures software repositories and installs packages on managed nodes.

node_repo_modules: local          # upstream repo to be added on node, local by default.
node_repo_remove: true            # remove existing repo on node?
node_packages: [openssh-server]   # packages to be installed current nodes with latest version
#node_default_packages:           # default packages to be installed on all nodes

node_repo_modules

name: node_repo_modules, type: string, level: C/A

List of software repository modules to be added on the node, same format as repo_modules. Default is local, using the local software repository specified in repo_upstream.

When Pigsty manages nodes, it filters entries in repo_upstream based on this parameter value. Only entries whose module field matches this parameter value will be added to the node’s software sources.

node_repo_remove

name: node_repo_remove, type: bool, level: C/A

Remove existing software repository definitions on the node? Default is true.

When enabled, Pigsty will remove existing configuration files in /etc/yum.repos.d on the node and back them up to /etc/yum.repos.d/backup. On Debian/Ubuntu systems, it backs up /etc/apt/sources.list(.d) to /etc/apt/backup.

node_packages

name: node_packages, type: string[], level: C

List of software packages to install and upgrade on the current node. Default is [openssh-server], which upgrades sshd to the latest version during installation (to avoid security vulnerabilities).

Each array element is a string of comma-separated package names. Same format as node_default_packages. This parameter is usually used to specify additional packages to install at the node/cluster level.

Packages specified in this parameter will be upgraded to the latest available version. If you need to keep existing node software versions unchanged (just ensure they exist), use the node_default_packages parameter.

node_default_packages

name: node_default_packages, type: string[], level: G

Default packages to install on every node. This parameter has no single cross-platform default. If it is not set explicitly, the node_id role loads node_packages_default from the corresponding <os>.<arch>.yml file under roles/node_id/vars, according to operating-system version and CPU architecture.

This is a string array in which each line is a comma-separated package list. Mappings differ across distributions, releases, and architectures; no single EL or Debian list should be treated as the universal default for that family.

Packages specified in this variable only require existence, not latest. If you need to install the latest version, use the node_packages parameter.

For example, the current mapping for EL 9 x86_64 is:

- bash,python3,sudo,acl,ca-certificates,openssl,curl,wget,lz4,zstd,unzip,bzip2,gzip,tar,tzdata,chrony,openssh-server,util-linux,rsync,psmisc,logrotate
- pv,jq,git,make,patch,lsof,less,ncdu,htop,iotop,socat,net-tools,telnet,ipvsadm,tuned,numactl,nvme-cli,sysstat,keepalived,etcd,haproxy,vector,pig,uv
- zlib,readline,xz,glibc-langpack-en,cronie,openssh-clients,node-exporter,bind-utils,iproute,iputils,nmap-ncat,procps-ng,vim-minimal,yum,audit,grubby,chkconfig

The current mapping for Debian 13 x86_64 is:

- bash,python3,sudo,acl,ca-certificates,openssl,curl,wget,lz4,zstd,unzip,bzip2,gzip,tar,tzdata,chrony,openssh-server,util-linux,rsync,psmisc,logrotate
- pv,jq,git,make,patch,lsof,less,ncdu,htop,iotop,socat,net-tools,telnet,ipvsadm,tuned,numactl,nvme-cli,sysstat,keepalived,etcd,haproxy,vector,pig,uv
- zlib1g,libreadline-dev,xz-utils,locales,cron,openssh-client,node-exporter,bind9-dnsutils,iproute2,iputils-ping,netcat-openbsd,procps,vim-tiny

This parameter uses the same format as node_packages, but is normally used as a global override of the platform mapping for packages required on every node.

node_uv_env

name: node_uv_env, type: path, level: C

uv virtual environment path, default is /data/venv. Set to empty string '' to skip uv venv configuration.

When non-empty, Pigsty creates a Python virtual environment on the node using uv venv and installs pip packages specified in node_pip_packages.

In the China region (region: china), /etc/uv/uv.toml is automatically configured to use the Tencent Cloud PyPI mirror at https://mirrors.cloud.tencent.com/pypi/simple/.

node_pip_packages

name: node_pip_packages, type: string, level: C

Pip packages to install in uv virtual environment, default is empty string ''.

Use space-separated package names, e.g.: 'ansible pgcli requests pandas'.

Only takes effect when node_uv_env is non-empty.

NODE_TUNE

Host node features, kernel modules, and tuning templates.

node_disable_numa: false          # disable node numa, reboot required
node_disable_swap: false          # disable node swap, use with caution
node_static_network: true         # preserve dns resolver settings after reboot
node_disk_prefetch: false         # setup disk prefetch on HDD to increase performance
node_kernel_modules: [ softdog, ip_vs, ip_vs_rr, ip_vs_wrr, ip_vs_sh ]
node_hugepage_count: 0            # number of 2MB hugepage, take precedence over ratio
node_hugepage_ratio: 0            # node mem hugepage ratio, 0 disable it by default
node_overcommit_ratio: 0          # node mem overcommit ratio, 0 disable it by default
node_tune: oltp                   # node tuned profile: none,oltp,olap,crit,tiny
node_tuned_profile_dir: /etc/tuned # node tuned profile directory
node_sysctl_params:               # sysctl parameters in k:v format in addition to tuned
  fs.nr_open: 8388608

node_disable_numa

name: node_disable_numa, type: bool, level: C

Disable NUMA? Default is false (NUMA not disabled).

Note that disabling NUMA requires a machine reboot to take effect! If you don’t know how to set CPU affinity, it’s recommended to disable NUMA when using databases in production environments.

node_disable_swap

name: node_disable_swap, type: bool, level: C

Disable SWAP? Default is false (SWAP not disabled).

Disabling SWAP is generally not recommended. The exception is if you have enough memory for exclusive PostgreSQL deployment, you can disable SWAP to improve performance.

Exception: SWAP should be disabled when your node is used for Kubernetes deployments.

node_static_network

name: node_static_network, type: bool, level: C

Use static DNS servers? Default is true (enabled).

Enabling static networking means your DNS Resolv configuration won’t be overwritten by machine reboots or NIC changes. Recommended to enable, or have network engineers handle the configuration.

node_disk_prefetch

name: node_disk_prefetch, type: bool, level: C

Enable disk prefetch? Default is false (not enabled).

Can optimize performance for HDD-deployed instances. Recommended to enable when using mechanical hard drives.

node_kernel_modules

name: node_kernel_modules, type: string[], level: C

Which kernel modules to enable? Default enables the following kernel modules:

node_kernel_modules: [ softdog, ip_vs, ip_vs_rr, ip_vs_wrr, ip_vs_sh ]

An array of kernel module names declaring the kernel modules that need to be installed on the node.

node_hugepage_count

name: node_hugepage_count, type: int, level: C

Number of 2MB hugepages to allocate on the node. Default is 0. Related parameter is node_hugepage_ratio.

If both node_hugepage_count and node_hugepage_ratio are 0 (default), hugepages will be completely disabled. This parameter has higher priority than node_hugepage_ratio because it’s more precise.

If a non-zero value is set, it will be written to /etc/sysctl.d/hugepage.conf to take effect. Negative values won’t work, and numbers higher than 90% of node memory will be capped at 90% of node memory.

If not zero, it should be slightly larger than the corresponding pg_shared_buffer_ratio value so PostgreSQL can use hugepages.

node_hugepage_ratio

name: node_hugepage_ratio, type: float, level: C

Ratio of node memory for hugepages. Default is 0. Valid range: 0 ~ 0.40.

This memory ratio will be allocated as hugepages and reserved for PostgreSQL. node_hugepage_count is the higher priority and more precise version of this parameter.

Default: 0, which sets vm.nr_hugepages=0 and completely disables hugepages.

This parameter should equal or be slightly larger than pg_shared_buffer_ratio if not zero.

For example, if you allocate 25% of memory for Postgres shared buffers by default, you can set this value to 0.27 ~ 0.30, and use /pg/bin/pg-tune-hugepage after initialization to precisely reclaim wasted hugepages.

node_overcommit_ratio

name: node_overcommit_ratio, type: int, level: C

Node memory overcommit ratio. Default is 0. This is an integer from 0 to 100+.

Default: 0, which sets vm.overcommit_memory=0. Otherwise, vm.overcommit_memory=2 will be used with this value as vm.overcommit_ratio.

Recommended to set vm.overcommit_ratio on dedicated pgsql nodes to avoid memory overcommit.

node_tune

name: node_tune, type: enum, level: C

Preset tuning profiles for machines, provided through tuned. Four preset modes:

  • tiny: Micro virtual machine
  • oltp: Regular OLTP template, optimizes latency (default)
  • olap: Regular OLAP template, optimizes throughput
  • crit: Core financial business template, optimizes dirty page count

Typically, the database tuning template pg_conf should match the machine tuning template.

node_tuned_profile_dir

name: node_tuned_profile_dir, type: path, level: C

Directory where Pigsty writes the tiny, oltp, olap, and crit tuned profiles. The role default is /etc/tuned, then platform variables adapt it to the distribution layout: EL 10, Debian 13, and Ubuntu 26 use /etc/tuned/profiles; EL 8/9, Debian 12, and Ubuntu 22/24 use /etc/tuned.

Normally leave this unchanged. Override it only when the target system’s tuned profile directory differs from Pigsty’s known platform mapping.

node_sysctl_params

name: node_sysctl_params, type: dict, level: C

Sysctl kernel parameters in K:V format (written and applied immediately by Ansible sysctl module) as a supplement to the tuned profile.

Default:

node_sysctl_params:
  fs.nr_open: 8388608

This default ensures the kernel per-process FD ceiling is not lower than LimitNOFILE=8388608 used by several Pigsty systemd units, avoiding setrlimit failures on some distro/systemd combinations.

This is a KV dictionary parameter where Key is the kernel sysctl parameter name and Value is the parameter value. You can also consider defining extra sysctl parameters directly in the tuned templates in roles/node/templates.

NODE_SEC

Node security related parameters, including SELinux and firewall configuration.

node_selinux_mode: permissive             # selinux mode: disabled, permissive, enforcing
node_firewall_mode: zone                  # firewall mode: zone (default, enabled), off (disable), none (skip & self-managed)
node_firewall_intranet:           # which intranet cidr considered as internal network
  - 10.0.0.0/8
  - 192.168.0.0/16
  - 172.16.0.0/12
node_firewall_public_port:        # expose these ports to public network in zone mode
  - 22                            # enable ssh access
  - 80                            # enable http access
  - 443                           # enable https access

node_selinux_mode

name: node_selinux_mode, type: enum, level: C

SELinux running mode. Default is permissive.

Options:

  • disabled: Completely disable SELinux (equivalent to old version’s node_disable_selinux: true)
  • permissive: Permissive mode, logs violations but doesn’t block (recommended, default)
  • enforcing: Enforcing mode, strictly enforces SELinux policies

If you don’t have professional OS/security experts, it’s recommended to use permissive or disabled mode.

Note that SELinux is only enabled by default on EL-based systems. If you want to enable SELinux on Debian/Ubuntu systems, you need to install and enable SELinux configuration yourself. Also, SELinux mode changes may require a system reboot to fully take effect.

node_firewall_mode

name: node_firewall_mode, type: enum, level: C

Firewall running mode. Default is zone (firewall enabled and zone-managed). Since v4.1, the default changed from none to zone.

Options:

  • zone: Enable firewall and configure rules: trust intranet, only open specified ports to public (default)
  • off: Turn off and disable firewall (equivalent to old version’s node_disable_firewall: true)
  • none: Do not manage firewall state/rules; fully self-managed by user

Uses firewalld service on EL systems, ufw service on Debian/Ubuntu systems. To align behavior across distros, Pigsty now defaults to zone: firewall enabled by default, intranet trusted, and public access limited to node_firewall_public_port.

If you need full manual firewall control (for example, relying only on cloud security groups or enterprise firewall policies), set node_firewall_mode to none. Use off only when you explicitly want to disable the system firewall.

Production environments with public network exposure should use zone mode with node_firewall_intranet and node_firewall_public_port for fine-grained access control. The zone mode will enable the firewall if not already running.

node_firewall_intranet

name: node_firewall_intranet, type: cidr[], level: C

Intranet CIDR address list. Introduced in v4.0. Default value:

node_firewall_intranet:
  - 10.0.0.0/8
  - 172.16.0.0/12
  - 192.168.0.0/16

This parameter defines IP address ranges considered as “internal network”. Traffic from these networks will be allowed to access all service ports without separate open rules.

Hosts within these CIDR ranges are treated as trusted intranet hosts with more permissive firewall rules. The same ranges are treated as “intranet” by PG/PGB HBA rules. Because the default firewall mode is zone, this list is active by default.

node_firewall_public_port

name: node_firewall_public_port, type: port[], level: C

Public exposed port list. Default is [22, 80, 443].

This parameter defines ports exposed to public network (non-intranet CIDR). Default exposed ports include:

  • 22: SSH service port
  • 80: HTTP service port
  • 443: HTTPS service port

You can adjust this list according to actual needs. For example, if you need to expose PostgreSQL to public network, explicitly add 5432:

node_firewall_public_port: [22, 80, 443, 5432]

PostgreSQL default security policy in Pigsty only allows administrators to access the database port from public networks. If you want other users to access the database from public networks, make sure to correctly configure corresponding access permissions in PG/PGB HBA rules.

If you want to expose other service ports to public networks, you can add them to this list. Always keep the minimum-exposure principle and open only ports you really need.

Note that this parameter only takes effect when node_firewall_mode is set to zone; it is not applied in none or off mode.

NODE_ADMIN

This section is about administrators on host nodes - who can log in and how.

node_data: /data                  # node main data directory, `/data` by default
node_admin_enabled: true          # create a admin user on target node?
node_admin_uid: 88                # uid and gid for node admin user
node_admin_username: dba          # name of node admin user, `dba` by default
node_admin_sudo: nopass           # admin user's sudo privilege: limited, nopass, all, none
node_admin_ssh_exchange: true     # exchange admin ssh key among node cluster
node_admin_pk_current: true       # add current user's ssh pk to admin authorized_keys
node_admin_pk_list: []            # ssh public keys to be added to admin user
node_aliases: {}                  # alias name -> IP address dict for `/etc/hosts`

node_data

name: node_data, type: path, level: C

Node’s main data directory. Default is /data.

If this directory doesn’t exist, it will be created. This directory should be owned by root with 777 permissions.

node_admin_enabled

name: node_admin_enabled, type: bool, level: C

Create a dedicated admin user on this node? Default is true.

Pigsty creates an admin user on each node by default (with password-free sudo and ssh). The default admin is named dba (uid=88), which can access other nodes in the environment from the admin node via password-free SSH and execute password-free sudo.

node_admin_uid

name: node_admin_uid, type: int, level: C

Admin user UID. Default is 88.

Please ensure the UID is the same across all nodes whenever possible to avoid unnecessary permission issues.

If the default UID 88 is already taken, you can choose another UID. Be careful about UID namespace conflicts when manually assigning.

node_admin_username

name: node_admin_username, type: username, level: C

Admin username. Default is dba.

node_admin_sudo

name: node_admin_sudo, type: enum, level: C

Admin user’s sudo privilege level. Default is nopass (password-free sudo).

Options:

  • none: No sudo privileges
  • limited: Limited sudo privileges (only allowed to execute specific commands)
  • nopass: Password-free sudo privileges (default, allows all commands without password)
  • all: Full sudo privileges (requires password)

Pigsty uses nopass mode by default, allowing admin users to execute any sudo command without password, which is very convenient for automated operations.

In production environments with high security requirements, you may need to adjust this parameter to limited or all to restrict admin privileges.

node_admin_ssh_exchange

name: node_admin_ssh_exchange, type: bool, level: C

Exchange node admin SSH keys between node clusters. Default is true.

When enabled, Pigsty will exchange SSH public keys between members during playbook execution, allowing admin node_admin_username to access each other from different nodes.

node_admin_pk_current

name: node_admin_pk_current, type: bool, level: C

Add current node & user’s public key to admin account? Default is true.

When enabled, the SSH public key (~/.ssh/id_rsa.pub) of the admin user executing this playbook on the current node will be copied to the target node admin user’s authorized_keys.

When deploying in production environments, please pay attention to this parameter, as it will install the default public key of the user currently executing the command to the admin user on all machines.

node_admin_pk_list

name: node_admin_pk_list, type: string[], level: C

List of public keys for admins who can log in. Default is [] (empty array).

Each array element is a string containing the public key to be written to the admin user’s ~/.ssh/authorized_keys. Users with the corresponding private key can log in as admin.

When deploying in production environments, please pay attention to this parameter and only add trusted keys to this list.

node_aliases

name: node_aliases, type: dict, level: C

Shell aliases to be written to host’s /etc/profile.d/node.alias.sh. Default is {} (empty dict).

This parameter allows you to configure convenient shell aliases for the host’s shell environment. The K:V dict defined here will be written to the target node’s profile.d file in the format alias k=v.

For example, the following declares an alias named dp for quickly executing docker compose pull:

node_aliases:
  dp: 'docker compose pull'

NODE_TIME

Configuration related to host time/timezone/NTP/scheduled tasks.

Time synchronization is very important for database services. Please ensure the system chronyd time service is running properly.

node_timezone: ''                 # setup node timezone, empty string to skip
node_ntp_enabled: true            # enable chronyd time sync service?
node_ntp_servers:                 # ntp servers in `/etc/chrony.conf`
  - pool pool.ntp.org iburst
node_crontab_overwrite: true      # overwrite or append to `/etc/crontab`?
node_crontab: [ ]                 # crontab entries in `/etc/crontab`

node_timezone

name: node_timezone, type: string, level: C

Set node timezone. Empty string means skip. Default is empty string, which won’t modify the default timezone (usually UTC).

When using in China region, it’s recommended to set to Asia/Hong_Kong / Asia/Shanghai.

node_ntp_enabled

name: node_ntp_enabled, type: bool, level: C

Enable chronyd time sync service? Default is true.

Pigsty will override the node’s /etc/chrony.conf with the NTP server list specified in node_ntp_servers.

If your node already has NTP servers configured, you can set this parameter to false to skip time sync configuration.

node_ntp_servers

name: node_ntp_servers, type: string[], level: C

NTP server list used in /etc/chrony.conf. Default: ["pool pool.ntp.org iburst"]

This parameter is an array where each element is a string representing one line of NTP server configuration. Only takes effect when node_ntp_enabled is enabled.

Pigsty uses the global NTP server pool.ntp.org by default. You can modify this parameter according to your network environment, e.g., cn.pool.ntp.org iburst, or internal time services.

You can also use the ${admin_ip} placeholder in the configuration to use the time server on the admin node.

node_ntp_servers: [ 'pool ${admin_ip} iburst' ]

node_crontab_overwrite

name: node_crontab_overwrite, type: bool, level: C

When handling scheduled tasks in node_crontab, append or overwrite? Default is true (overwrite).

If you want to append scheduled tasks on the node, set this parameter to false, and Pigsty will append rather than overwrite all scheduled tasks on the node’s crontab.

node_crontab

name: node_crontab, type: string[], level: C

Scheduled tasks defined in node’s /etc/crontab. Default is [] (empty array).

Each array element is a string representing one scheduled task line. Use standard cron format for definition.

For example, the following configuration will execute a system task as root at 3am every day:

node_crontab:
  - '00 03 * * * root /usr/bin/some-system-task'

Note: For PostgreSQL backup tasks and other postgres user cron jobs, use the pg_crontab parameter instead of node_crontab. Because node_crontab is written to /etc/crontab during NODE initialization, the postgres user may not exist yet, which will cause cron to report bad username and ignore the entire crontab file.

When node_crontab_overwrite is true (default), the default /etc/crontab will be restored when removing the node.

NODE_VIP

You can bind an optional L2 VIP to a node cluster. This feature is disabled by default. L2 VIP only makes sense for a group of node clusters. The VIP will switch between nodes in the cluster according to configured priorities, ensuring high availability of node services.

Note that L2 VIP can only be used within the same L2 network segment, which may impose additional restrictions on your network topology. If you don’t want this restriction, you can consider using DNS LB or HAProxy for similar functionality.

When enabling this feature, you need to explicitly assign available vip_address and vip_vrid for this L2 VIP. Users should ensure both are unique within the same network segment.

Note that NODE VIP is different from PG VIP. PG VIP is a VIP serving PostgreSQL instances, managed by vip-manager and bound to the PG cluster primary. NODE VIP is managed by Keepalived and bound to node clusters. It can be in master-backup mode or load-balanced mode, and both can coexist.

vip_enabled: false                # enable vip on this node cluster?
# vip_address:         [IDENTITY] # node vip address in ipv4 format, required if vip is enabled
# vip_vrid:            [IDENTITY] # required, integer, 1-254, should be unique among same VLAN
vip_role: backup                  # optional, `master/backup`, backup by default, use as init role
vip_preempt: false                # optional, `true/false`, false by default, enable vip preemption
vip_interface: auto               # node vip network interface to listen, `auto` by default
vip_dns_suffix: ''                # node vip dns name suffix, empty string by default
vip_auth_pass: ''                 # vrrp auth password, empty to use `<cls>-<vrid>` as default
vip_exporter_port: 9650           # keepalived exporter listen port, 9650 by default

vip_enabled

name: vip_enabled, type: bool, level: C

Enable an L2 VIP managed by Keepalived on this node cluster? Default is false.

vip_address

name: vip_address, type: ip, level: C

Node VIP address in IPv4 format (without CIDR suffix). This is a required parameter when vip_enabled is enabled.

This parameter has no default value, meaning you must explicitly assign a unique VIP address for the node cluster.

vip_vrid

name: vip_vrid, type: int, level: C

VRID is a positive integer from 1 to 254 used to identify a VIP in the network. This is a required parameter when vip_enabled is enabled.

This parameter has no default value, meaning you must explicitly assign a unique ID within the network segment for the node cluster.

vip_role

name: vip_role, type: enum, level: I

Node VIP role. Options are master or backup. Default is backup.

This parameter value will be set as keepalived’s initial state.

vip_preempt

name: vip_preempt, type: bool, level: C/I

Enable VIP preemption? Optional parameter. Default is false (no preemption).

Preemption means when a backup node has higher priority than the currently alive and working master node, should it preempt the VIP?

vip_interface

name: vip_interface, type: string, level: C/I

Network interface for the node VIP. The default is auto; Pigsty detects the interface associated with the node IP in the inventory.

For non-standard routing, policy routing, or other unusual network environments where auto-detection is unsuitable, explicitly override the interface name at the instance or node level.

You should use the same interface name as the node’s primary IP address (the IP address you put in the inventory).

If your nodes have different interface names, you can override it at the instance/node level.

vip_dns_suffix

name: vip_dns_suffix, type: string, level: C/I

DNS name for node cluster L2 VIP. Default is empty string, meaning the cluster name itself is used as the DNS name.

vip_auth_pass

name: vip_auth_pass, type: password, level: C

VRRP authentication password for keepalived. Default is empty string.

When empty, Pigsty will auto-generate a password using the pattern <cluster_name>-<vrid>. For production environments with security requirements, set an explicit strong password.

vip_exporter_port

name: vip_exporter_port, type: port, level: C/I

Keepalived exporter listen port. Default is 9650.

HAPROXY

HAProxy is installed and enabled on all nodes by default, exposing services in a manner similar to Kubernetes NodePort.

The PGSQL module uses HAProxy for services.

haproxy_enabled: true             # enable haproxy on this node?
haproxy_clean: false              # cleanup all existing haproxy config?
haproxy_reload: true              # reload haproxy after config?
haproxy_auth_enabled: true        # enable authentication for haproxy admin page
haproxy_admin_username: admin     # haproxy admin username, `admin` by default
haproxy_admin_password: pigsty    # haproxy admin password, `pigsty` by default
haproxy_exporter_port: 9101       # haproxy admin/exporter port, 9101 by default
haproxy_client_timeout: 24h       # client connection timeout, 24h by default
haproxy_server_timeout: 24h       # server connection timeout, 24h by default
haproxy_services: []              # list of haproxy services to be exposed on node

haproxy_enabled

name: haproxy_enabled, type: bool, level: C

Enable haproxy on this node? Default is true.

haproxy_clean

name: haproxy_clean, type: bool, level: G/C/A

Cleanup all existing haproxy config? Default is false.

haproxy_reload

name: haproxy_reload, type: bool, level: A

Reload haproxy after config? Default is true, will reload haproxy after config changes.

If you want to check before applying, you can disable this option with command arguments, check, then apply.

haproxy_auth_enabled

name: haproxy_auth_enabled, type: bool, level: G

Enable authentication for haproxy admin page. Default is true, which requires HTTP basic auth for the admin page.

Not recommended to disable authentication, as your traffic control page will be exposed, which is risky.

haproxy_admin_username

name: haproxy_admin_username, type: username, level: G

HAProxy admin username. Default is admin.

haproxy_admin_password

name: haproxy_admin_password, type: password, level: G

HAProxy admin password. Default is pigsty.

PLEASE CHANGE THIS PASSWORD IN YOUR PRODUCTION ENVIRONMENT!

haproxy_exporter_port

name: haproxy_exporter_port, type: port, level: C

HAProxy traffic management/metrics exposed port. Default is 9101.

haproxy_client_timeout

name: haproxy_client_timeout, type: interval, level: C

Client connection timeout. Default is 24h.

Setting a timeout can avoid long-lived connections that are difficult to clean up. If you really need long connections, you can set it to a longer time.

haproxy_server_timeout

name: haproxy_server_timeout, type: interval, level: C

Server connection timeout. Default is 24h.

Setting a timeout can avoid long-lived connections that are difficult to clean up. If you really need long connections, you can set it to a longer time.

haproxy_services

name: haproxy_services, type: service[], level: C

List of services to expose via HAProxy on this node. Default is [] (empty array).

Each array element is a service definition. Here’s an example service definition:

haproxy_services:                   # list of haproxy service

  # expose pg-test read only replicas
  - name: pg-test-ro                # [REQUIRED] service name, unique
    port: 5440                      # [REQUIRED] service port, unique
    ip: "*"                         # [OPTIONAL] service listen addr, "*" by default
    protocol: tcp                   # [OPTIONAL] service protocol, 'tcp' by default
    balance: leastconn              # [OPTIONAL] load balance algorithm, roundrobin by default (or leastconn)
    maxconn: 20000                  # [OPTIONAL] max allowed front-end connection, 20000 by default
    default: 'inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100'
    options:
      - option httpchk
      - option http-keep-alive
      - http-check send meth OPTIONS uri /read-only
      - http-check expect status 200
    servers:
      - { name: pg-test-1 ,ip: 10.10.10.11 , port: 5432 , options: check port 8008 , backup: true }
      - { name: pg-test-2 ,ip: 10.10.10.12 , port: 5432 , options: check port 8008 }
      - { name: pg-test-3 ,ip: 10.10.10.13 , port: 5432 , options: check port 8008 }

Each service definition is rendered as /etc/haproxy/conf.d/<service.name>.cfg and takes effect after HAProxy reload. The main configuration remains /etc/haproxy/haproxy.cfg.

Pigsty writes the HAProxy unit to /etc/systemd/system/haproxy.service. The optional environment file is /etc/default/haproxy, where only EXTRAOPTS is recognized. Do not repeat -f there because it conflicts with the main configuration and configuration directory fixed in the unit. If you override EXTRAOPTS, keep the default -S /run/haproxy-master.sock so seamless reload continues to work; restart the service after changing it.

NODE_EXPORTER 

node_exporter_enabled: true       # setup node_exporter on this node?
node_exporter_port: 9100          # node exporter listen port, 9100 by default
node_exporter_options: '--no-collector.softnet --no-collector.nvme --collector.tcpstat --collector.processes'

node_exporter_enabled

name: node_exporter_enabled, type: bool, level: C

Enable node metrics collector on current node? Default is true.

node_exporter_port

name: node_exporter_port, type: port, level: C

Port used to expose node metrics. Default is 9100.

node_exporter_options

name: node_exporter_options, type: arg, level: C

Command line arguments for node metrics collector. Default value:

--no-collector.softnet --no-collector.nvme --collector.tcpstat --collector.processes

This option enables/disables some metrics collectors. Please adjust according to your needs.

VECTOR

Vector is the log collection component used by Pigsty since v4. It collects logs from various modules and sends them to VictoriaLogs service on infrastructure nodes.

  • INFRA: Infrastructure component logs, collected only on Infra nodes.

    • nginx-access: /var/log/nginx/access.log
    • nginx-error: /var/log/nginx/error.log
    • grafana: /var/log/grafana/grafana.log

  • NODES: Host-related logs, collection enabled on all nodes.

    • syslog: /var/log/messages (/var/log/syslog on Debian)
    • dmesg: /var/log/dmesg
    • cron: /var/log/cron

  • PGSQL: PostgreSQL-related logs, collection enabled only when node has PGSQL module configured.

    • postgres: /pg/log/postgres/*
    • patroni: /pg/log/patroni/patroni.log (job=patroni)
    • pgbouncer: /pg/log/pgbouncer/pgbouncer.log
    • pgbackrest: /pg/log/pgbackrest/*.log

  • REDIS: Redis-related logs, collection enabled only when node has REDIS module configured.

    • redis: /var/log/redis/*.log

Log directories are automatically adjusted according to these parameter configurations: pg_log_dir, patroni_log_dir, pgbouncer_log_dir, pgbackrest_log_dir

vector_enabled: true              # enable vector log collector?
vector_clean: false               # purge vector data dir during init?
vector_data: /data/vector         # vector data directory, /data/vector by default
vector_port: 9598                 # vector metrics port, 9598 by default
vector_read_from: beginning       # read log from beginning or end
vector_log_endpoint: [ infra ]    # log endpoint, default send to infra group

vector_enabled

name: vector_enabled, type: bool, level: C

Enable Vector log collection service? Default is true.

Vector is the log collection agent used by Pigsty since v4, replacing Promtail from previous versions. It collects node and service logs and sends them to VictoriaLogs.

vector_clean

name: vector_clean, type: bool, level: G/A

Clean existing data directory when installing Vector? Default is false.

By default, it won’t clean. When you choose to clean, Pigsty will remove the existing data directory vector_data when deploying Vector. This means Vector will re-collect all logs on the current node and send them to VictoriaLogs.

vector_data

name: vector_data, type: path, level: C

Vector data directory path. Default is /data/vector.

Vector stores log read offsets and buffered data in this directory.

vector_port

name: vector_port, type: port, level: C

Vector metrics listen port. Default is 9598.

This port is used to expose Vector’s own monitoring metrics, which can be scraped by VictoriaMetrics.

vector_read_from

name: vector_read_from, type: enum, level: C

Vector log reading start position. Default is beginning.

Options are beginning (start from beginning) or end (start from end). beginning reads the entire content of existing log files, end only reads newly generated logs.

vector_log_endpoint

name: vector_log_endpoint, type: string[], level: C

Log destination endpoint list. Default is [ infra ].

Specifies which node group’s VictoriaLogs service to send logs to. Default sends to nodes in the infra group.

10.3 - Playbook

How to use built-in Ansible playbooks to manage NODE clusters, with a quick reference for common commands.

Pigsty provides two playbooks related to the NODE module:

  • node.yml: Add nodes to Pigsty and configure them to the desired state
  • node-rm.yml: Remove managed nodes from Pigsty

Two wrapper scripts are also provided: bin/node-add and bin/node-rm, for quickly invoking these playbooks.

node.yml

The node.yml playbook for adding nodes to Pigsty contains the following subtasks:

node-id       : generate node identity
node_name     : setup hostname
node_hosts    : setup /etc/hosts records
node_resolv   : setup DNS resolver /etc/resolv.conf
node_firewall : setup firewall & selinux
node_ca       : add & trust CA certificate
node_repo     : add upstream software repository
node_pkg      : install rpm/deb packages
node_uv       : setup uv Python virtual environment
node_feature  : setup numa, grub, static network
node_kernel   : enable kernel modules
node_tune     : setup tuned profile
node_sysctl   : setup additional sysctl parameters
node_profile  : write /etc/profile.d/node.sh
node_alias    : write /etc/profile.d/node.alias.sh
node_ulimit   : setup resource limits
node_data     : setup data directory
node_admin    : setup admin user and ssh key
node_timezone : setup timezone
node_ntp      : setup NTP server/client
node_crontab  : add/overwrite crontab tasks
node_vip      : setup optional L2 VIP for node cluster
haproxy       : setup haproxy on node to expose services
monitor       : setup node monitoring: node_exporter & vector

node-rm.yml

The node-rm.yml playbook for removing nodes from Pigsty contains the following subtasks:

register       : remove registration from prometheus & nginx
  - prometheus : remove registered prometheus monitoring target
  - nginx      : remove nginx proxy record for haproxy admin
vip            : remove keepalived & L2 VIP (if VIP enabled)
haproxy        : remove haproxy load balancer
node_exporter  : remove node monitoring: Node Exporter
vip_exporter   : remove keepalived_exporter (if VIP enabled)
vector         : remove log collection agent vector
node_crontab   : restore default /etc/crontab (if node_crontab_overwrite=true)
profile        : remove /etc/profile.d/node.sh

node-rm.yml removes Pigsty management and stops NODE-related services; it is not an operating-system teardown or full uninstall:

  • It deregisters Node, Docker, Ping, VIP, and other monitoring targets, plus the HAProxy administration portal.
  • It stops and disables HAProxy, Node Exporter, Vector, and Keepalived/Exporter when enabled.
  • It removes HAProxy configuration and NODE Vector configuration, and deletes vector_data (default: /data/vector).
  • It does not uninstall packages, remove the admin user, delete node_data, stop Docker, or delete Docker data.

The current removal role deletes vector_data directly and does not use the installation role’s vector_clean switch. Confirm that no Vector buffer data must be retained before running it.

Quick Reference 

# Basic node management
./node.yml -l <cls|ip|group>          # Add node to Pigsty
./node-rm.yml -l <cls|ip|group> --check # Preflight; execute after exact-target confirmation

# Node management shortcuts
bin/node-add node-test                 # Initialize node cluster 'node-test'
bin/node-add 10.10.10.10               # Initialize node '10.10.10.10'
./node-rm.yml -l node-test --check     # Preflight before bin/node-rm node-test
./node-rm.yml -l 10.10.10.10 --check   # Preflight before bin/node-rm 10.10.10.10

# Node main initialization
./node.yml -t node                     # Complete node main init (excludes haproxy, monitor)
./node.yml -t haproxy                  # Setup haproxy on node
./node.yml -t monitor                  # Setup node monitoring: node_exporter & vector

# VIP management
./node.yml -t node_vip                 # Setup optional L2 VIP for node cluster
./node.yml -t vip_config,vip_reload    # Refresh node L2 VIP configuration

# HAProxy management
./node.yml -t haproxy_config,haproxy_reload   # Refresh service definitions on node

# Registration management
./node.yml -t register_prometheus      # Re-register node to Prometheus
./node.yml -t register_nginx           # Re-register node haproxy admin to Nginx

# Specific tasks
./node.yml -t node-id                  # Generate node identity
./node.yml -t node_name                # Setup hostname
./node.yml -t node_hosts               # Setup node /etc/hosts records
./node.yml -t node_resolv              # Setup node DNS resolver /etc/resolv.conf
./node.yml -t node_firewall            # Setup firewall & selinux
./node.yml -t node_ca                  # Setup node CA certificate
./node.yml -t node_repo                # Setup node upstream software repository
./node.yml -t node_pkg                 # Install yum packages on node
./node.yml -t node_uv                  # Setup uv Python virtual environment
./node.yml -t node_feature             # Setup numa, grub, static network
./node.yml -t node_kernel              # Enable kernel modules
./node.yml -t node_tune                # Setup tuned profile
./node.yml -t node_sysctl              # Setup additional sysctl parameters
./node.yml -t node_profile             # Setup node environment: /etc/profile.d/node.sh
./node.yml -t node_alias               # Setup command aliases: /etc/profile.d/node.alias.sh
./node.yml -t node_ulimit              # Setup node resource limits
./node.yml -t node_data                # Setup node primary data directory
./node.yml -t node_admin               # Setup admin user and ssh key
./node.yml -t node_timezone            # Setup node timezone
./node.yml -t node_ntp                 # Setup node NTP server/client
./node.yml -t node_crontab             # Add/overwrite crontab tasks

10.4 - Administration

Node cluster management SOP - create, destroy, expand, shrink, and handle node/disk failures

Here are common administration operations for the NODE module:

For more questions, see FAQ: NODE

Add Node

To add a node to Pigsty, you need passwordless ssh/sudo access to that node.

You can also add an entire cluster at once, or use wildcards to match nodes in the inventory to add to Pigsty.

# ./node.yml -l <cls|ip|group>        # actual playbook to add nodes to Pigsty
# bin/node-add <selector|ip...>       # add node to Pigsty
bin/node-add node-test                # init node cluster 'node-test'
bin/node-add 10.10.10.10              # init node '10.10.10.10'

Example: Add three nodes of PG cluster pg-test to Pigsty management

demo/node-add.cast

Remove Node

To remove a node from Pigsty, you can use the following commands:

First confirm that every business module on the node has been removed through its own workflow and check whether any vector_data buffer must be retained. After confirming the exact target, call the wrapper:

# ./node-rm.yml -l <cls|ip|group>    # actual playbook that removes a node from Pigsty
# bin/node-rm <cls|ip|selector> ...  # remove node from Pigsty
bin/node-rm node-test                # remove node cluster 'node-test'
bin/node-rm 10.10.10.10              # remove node '10.10.10.10'

You can also remove an entire cluster at once, or use wildcards to match nodes in the inventory to remove from Pigsty.

Here, “remove node” means removing NODE management. The playbook deregisters monitoring, logging, and the HAProxy portal; stops Node Exporter, Vector, HAProxy, and optional VIP services; and deletes vector_data (default: /data/vector). It does not uninstall packages, remove the admin user or node_data, stop Docker, or delete Docker data. See node-rm.yml for the exact boundary.

demo/node-rm.cast

Create Admin

If the current user doesn’t have passwordless ssh/sudo access to the node, you can use another admin user to bootstrap it:

node.yml -t node_admin -k -K -e ansible_user=<another admin>   # enter ssh/sudo password for another admin to complete this task

Bind VIP

You can bind an optional L2 VIP on a node cluster using the vip_enabled parameter.

proxy:
  hosts:
    10.10.10.29: { nodename: proxy-1 }   # you can explicitly specify initial VIP role: MASTER / BACKUP
    10.10.10.30: { nodename: proxy-2 }   # , vip_role: master }
  vars:
    node_cluster: proxy
    vip_enabled: true
    vip_vrid: 128
    vip_address: 10.10.10.99
    vip_interface: eth1
./node.yml -l proxy -t node_vip     # enable VIP for the first time
./node.yml -l proxy -t vip_refresh  # refresh VIP config (e.g., designate master)

Add Node Monitoring

If you want to add or reconfigure monitoring on existing nodes, use the following commands:

./node.yml -t node_exporter,node_register  # configure monitoring and register
./node.yml -t vector                        # configure log collection

Other Tasks 

# Play
./node.yml -t node                            # complete node initialization (excludes haproxy, monitoring)
./node.yml -t haproxy                         # setup haproxy on node
./node.yml -t monitor                         # configure node monitoring: node_exporter & vector
./node.yml -t node_vip                        # install, configure, enable L2 VIP for clusters without VIP
./node.yml -t vip_config,vip_reload           # refresh node L2 VIP configuration
./node.yml -t haproxy_config,haproxy_reload   # refresh service definitions on node
./node.yml -t register_prometheus             # re-register node with Prometheus
./node.yml -t register_nginx                  # re-register node haproxy admin page with Nginx

# Task
./node.yml -t node-id        # generate node identity
./node.yml -t node_name      # setup hostname
./node.yml -t node_hosts     # configure node /etc/hosts records
./node.yml -t node_resolv    # configure node DNS resolver /etc/resolv.conf
./node.yml -t node_firewall  # configure firewall & selinux
./node.yml -t node_ca        # configure node CA certificate
./node.yml -t node_repo      # configure node upstream software repository
./node.yml -t node_pkg       # install yum packages on node
./node.yml -t node_feature   # configure numa, grub, static network, etc.
./node.yml -t node_kernel    # configure OS kernel modules
./node.yml -t node_tune      # configure tuned profile
./node.yml -t node_sysctl    # set additional sysctl parameters
./node.yml -t node_profile   # configure node environment variables: /etc/profile.d/node.sh
./node.yml -t node_alias     # configure command aliases: /etc/profile.d/node.alias.sh
./node.yml -t node_ulimit    # configure node resource limits
./node.yml -t node_data      # configure node primary data directory
./node.yml -t node_admin     # configure admin user and ssh keys
./node.yml -t node_timezone  # configure node timezone
./node.yml -t node_ntp       # configure node NTP server/client
./node.yml -t node_crontab   # add/overwrite crontab entries
./node.yml -t node_vip       # setup optional L2 VIP for node cluster

HAProxy Password

haproxy_admin_password (default pigsty) is used for HAProxy admin UI authentication, rendered to /etc/haproxy/haproxy.cfg.

After changing the password, use the following to reload config (hot reload, no connection interruption):

./node.yml -l <target> -t haproxy_config,haproxy_reload

Firewall Management

Pigsty uses node_firewall_mode to control firewall behavior. Uses firewalld on RHEL/Rocky and ufw on Debian/Ubuntu.

Since v4.1, this defaults to zone: Pigsty enables the system firewall consistently across distros with an “intranet trusted, public minimized” policy. In zone mode, intranet traffic is unrestricted, but external access is limited to specific ports. Set node_firewall_mode: none only when you want to fully self-manage firewall state and rules. This is especially important when deploying on cloud servers exposed to the internet.

We recommend opening only necessary ports: 22 (SSH), 80/443 (HTTP/HTTPS) are essential. Be cautious about exposing port 5432 (PostgreSQL).

Apply Firewall Rules

zone is already the default. If you previously set none/off, set it back to zone and apply:

node_firewall_mode: zone              # enable firewall with zone rules
node_firewall_intranet:               # trust these CIDRs (full access)
  - 10.0.0.0/8
  - 192.168.0.0/16
  - 172.16.0.0/12
node_firewall_public_port:            # open these ports to public
  - 22                                # SSH
  - 80                                # HTTP
  - 443                               # HTTPS

Then execute: ./node.yml -l <target> -t node_firewall

Open More Ports

To open additional ports, add them to node_firewall_public_port and re-run:

node_firewall_public_port: [22, 80, 443, 5432, 6379]  # add PostgreSQL and Redis ports
./node.yml -l <target> -t node_firewall

Configure Intranet CIDRs

CIDRs in node_firewall_intranet are added to the trusted zone with full access:

node_firewall_intranet:
  - 10.0.0.0/8           # Class A private
  - 192.168.0.0/16       # Class C private
  - 172.16.0.0/12        # Class B private
  - 100.64.0.0/10        # Carrier-grade NAT (if needed)

Remove Rules (Manual)

Important: Pigsty’s firewall management is add-only. Removing entries from config and re-running will NOT delete existing rules. You must remove them manually.

EL (firewalld)
# Remove port from public zone
sudo firewall-cmd --zone=public --remove-port=5432/tcp
sudo firewall-cmd --runtime-to-permanent

# Remove CIDR from trusted zone
sudo firewall-cmd --zone=trusted --remove-source=10.0.0.0/8
sudo firewall-cmd --runtime-to-permanent

# View current rules
sudo firewall-cmd --zone=public --list-ports
sudo firewall-cmd --zone=trusted --list-sources

# Reset to initial state (remove all custom rules)
sudo firewall-cmd --complete-reload
Debian (ufw)
# Delete port rule
sudo ufw delete allow 5432/tcp

# Delete CIDR rule
sudo ufw delete allow from 10.0.0.0/8

# View current rules (numbered)
sudo ufw status numbered

# Delete by rule number
sudo ufw delete <rule_number>

# Reset to initial state (remove all rules, keep ufw enabled)
sudo ufw reset

Disable Firewall

To completely disable the firewall, set node_firewall_mode to off:

node_firewall_mode: off    # completely disable firewall
./node.yml -l <target> -t node_firewall

Or disable manually:

EL (firewalld)
sudo systemctl disable --now firewalld
Debian (ufw)
sudo ufw disable

10.5 - Monitoring

Monitor NODE in Pigsty with dashboards and alerting rules

Pigsty currently provides 10 monitoring dashboards in the NODE dashboard directory, along with comprehensive alerting rules.

Dashboards

The NODE dashboard directory currently contains 10 dashboards. The JuiceFS and Claude Code dashboards show data only after those components are deployed and emit metrics.

NODE Overview

Displays an overall overview of all host nodes in the current environment.

node-overview.jpg

NODE Cluster

Shows detailed monitoring data for a specific host cluster.

node-cluster.jpg

Node Instance

Presents detailed monitoring information for a single host node.

node-instance.jpg

NODE Alert

Centrally displays alert information for all hosts in the environment.

node-alert.jpg

NODE VIP

Monitors detailed status of L2 virtual IPs.

node-vip.jpg

Node Haproxy

Tracks the operational status of HAProxy load balancers.

node-haproxy.jpg

Node Disk

Focuses on per-disk I/O latency, throughput, queue depth, and other storage metrics.

node-disk.webp

Node Vector

Shows Vector collection and forwarding status, plus log-pipeline health.

node-vector.webp

Node JuiceFS

Shows JuiceFS client cache, object storage, metadata operations, and read/write performance.

Open the Node JuiceFS Dashboard

Claude Code

Shows sessions, tokens, costs, and logs reported by Claude Code through OpenTelemetry.

Open the Claude Code Dashboard

Alert Rules

Pigsty implements the following alerting rules for NODE:

Availability Alerts

Rule Level Description
NodeDown CRIT Node is offline
HaproxyDown CRIT HAProxy service is offline
VectorDown WARN Log collecting agent offline (Vector)
DockerDown WARN Container engine offline
KeepalivedDown WARN Keepalived daemon offline

CPU Alerts

Rule Level Description
NodeCpuHigh WARN CPU usage exceeds 70%

Scheduling Alerts

Rule Level Description
NodeLoadHigh WARN Normalized load exceeds 100%

Memory Alerts

Rule Level Description
NodeOutOfMem WARN Available memory less than 10%
NodeMemSwapped WARN Swap usage exceeds 1%

Filesystem Alerts

Rule Level Description
NodeFsSpaceFull WARN Disk usage exceeds 90%
NodeFsFilesFull WARN Inode usage exceeds 90%
NodeFdFull WARN File descriptor usage exceeds 90%

Disk Alerts

Rule Level Description
NodeDiskSlow WARN Read/write latency exceeds 32ms

Network Protocol Alerts

Rule Level Description
NodeTcpErrHigh WARN TCP error rate exceeds 1/min
NodeTcpRetransHigh WARN TCP retransmission rate exceeds 1%

Time Synchronization Alerts

Rule Level Description
NodeTimeDrift WARN System time not synchronized

10.6 - Metrics

Complete list of monitoring metrics provided by Pigsty NODE module

This page is a snapshot of 727 monitoring metric categories for the NODE module. The actual runtime metric set varies with package version, enabled collectors, and target state.

Metric Name Type Labels Description
ALERTS Unknown alertname, ip, level, severity, ins, job, alertstate, category, instance, cls N/A
ALERTS_FOR_STATE Unknown alertname, ip, level, severity, ins, job, category, instance, cls N/A
deprecated_flags_inuse_total Unknown instance, ins, job, ip, cls N/A
go_gc_duration_seconds summary quantile, instance, ins, job, ip, cls A summary of the pause duration of garbage collection cycles.
go_gc_duration_seconds_count Unknown instance, ins, job, ip, cls N/A
go_gc_duration_seconds_sum Unknown instance, ins, job, ip, cls N/A
go_goroutines gauge instance, ins, job, ip, cls Number of goroutines that currently exist.
go_info gauge version, instance, ins, job, ip, cls Information about the Go environment.
go_memstats_alloc_bytes gauge instance, ins, job, ip, cls Number of bytes allocated and still in use.
go_memstats_alloc_bytes_total counter instance, ins, job, ip, cls Total number of bytes allocated, even if freed.
go_memstats_buck_hash_sys_bytes gauge instance, ins, job, ip, cls Number of bytes used by the profiling bucket hash table.
go_memstats_frees_total counter instance, ins, job, ip, cls Total number of frees.
go_memstats_gc_sys_bytes gauge instance, ins, job, ip, cls Number of bytes used for garbage collection system metadata.
go_memstats_heap_alloc_bytes gauge instance, ins, job, ip, cls Number of heap bytes allocated and still in use.
go_memstats_heap_idle_bytes gauge instance, ins, job, ip, cls Number of heap bytes waiting to be used.
go_memstats_heap_inuse_bytes gauge instance, ins, job, ip, cls Number of heap bytes that are in use.
go_memstats_heap_objects gauge instance, ins, job, ip, cls Number of allocated objects.
go_memstats_heap_released_bytes gauge instance, ins, job, ip, cls Number of heap bytes released to OS.
go_memstats_heap_sys_bytes gauge instance, ins, job, ip, cls Number of heap bytes obtained from system.
go_memstats_last_gc_time_seconds gauge instance, ins, job, ip, cls Number of seconds since 1970 of last garbage collection.
go_memstats_lookups_total counter instance, ins, job, ip, cls Total number of pointer lookups.
go_memstats_mallocs_total counter instance, ins, job, ip, cls Total number of mallocs.
go_memstats_mcache_inuse_bytes gauge instance, ins, job, ip, cls Number of bytes in use by mcache structures.
go_memstats_mcache_sys_bytes gauge instance, ins, job, ip, cls Number of bytes used for mcache structures obtained from system.
go_memstats_mspan_inuse_bytes gauge instance, ins, job, ip, cls Number of bytes in use by mspan structures.
go_memstats_mspan_sys_bytes gauge instance, ins, job, ip, cls Number of bytes used for mspan structures obtained from system.
go_memstats_next_gc_bytes gauge instance, ins, job, ip, cls Number of heap bytes when next garbage collection will take place.
go_memstats_other_sys_bytes gauge instance, ins, job, ip, cls Number of bytes used for other system allocations.
go_memstats_stack_inuse_bytes gauge instance, ins, job, ip, cls Number of bytes in use by the stack allocator.
go_memstats_stack_sys_bytes gauge instance, ins, job, ip, cls Number of bytes obtained from system for stack allocator.
go_memstats_sys_bytes gauge instance, ins, job, ip, cls Number of bytes obtained from system.
go_threads gauge instance, ins, job, ip, cls Number of OS threads created.
haproxy:cls:usage Unknown job, cls N/A
haproxy:ins:uptime Unknown instance, ins, job, ip, cls N/A
haproxy:ins:usage Unknown instance, ins, job, ip, cls N/A
haproxy_backend_active_servers gauge proxy, instance, ins, job, ip, cls Total number of active UP servers with a non-zero weight
haproxy_backend_agg_check_status gauge state, proxy, instance, ins, job, ip, cls Backend’s aggregated gauge of servers’ state check status
haproxy_backend_agg_server_check_status gauge state, proxy, instance, ins, job, ip, cls [DEPRECATED] Backend’s aggregated gauge of servers’ status
haproxy_backend_agg_server_status gauge state, proxy, instance, ins, job, ip, cls Backend’s aggregated gauge of servers’ status
haproxy_backend_backup_servers gauge proxy, instance, ins, job, ip, cls Total number of backup UP servers with a non-zero weight
haproxy_backend_bytes_in_total counter proxy, instance, ins, job, ip, cls Total number of request bytes since process started
haproxy_backend_bytes_out_total counter proxy, instance, ins, job, ip, cls Total number of response bytes since process started
haproxy_backend_check_last_change_seconds gauge proxy, instance, ins, job, ip, cls How long ago the last server state changed, in seconds
haproxy_backend_check_up_down_total counter proxy, instance, ins, job, ip, cls Total number of failed checks causing UP to DOWN server transitions, per server/backend, since the worker process started
haproxy_backend_client_aborts_total counter proxy, instance, ins, job, ip, cls Total number of requests or connections aborted by the client since the worker process started
haproxy_backend_connect_time_average_seconds gauge proxy, instance, ins, job, ip, cls Avg. connect time for last 1024 successful connections.
haproxy_backend_connection_attempts_total counter proxy, instance, ins, job, ip, cls Total number of outgoing connection attempts on this backend/server since the worker process started
haproxy_backend_connection_errors_total counter proxy, instance, ins, job, ip, cls Total number of failed connections to server since the worker process started
haproxy_backend_connection_reuses_total counter proxy, instance, ins, job, ip, cls Total number of reused connection on this backend/server since the worker process started
haproxy_backend_current_queue gauge proxy, instance, ins, job, ip, cls Number of current queued connections
haproxy_backend_current_sessions gauge proxy, instance, ins, job, ip, cls Number of current sessions on the frontend, backend or server
haproxy_backend_downtime_seconds_total counter proxy, instance, ins, job, ip, cls Total time spent in DOWN state, for server or backend
haproxy_backend_failed_header_rewriting_total counter proxy, instance, ins, job, ip, cls Total number of failed HTTP header rewrites since the worker process started
haproxy_backend_http_cache_hits_total counter proxy, instance, ins, job, ip, cls Total number of HTTP requests not found in the cache on this frontend/backend since the worker process started
haproxy_backend_http_cache_lookups_total counter proxy, instance, ins, job, ip, cls Total number of HTTP requests looked up in the cache on this frontend/backend since the worker process started
haproxy_backend_http_comp_bytes_bypassed_total counter proxy, instance, ins, job, ip, cls Total number of bytes that bypassed HTTP compression for this object since the worker process started (CPU/memory/bandwidth limitation)
haproxy_backend_http_comp_bytes_in_total counter proxy, instance, ins, job, ip, cls Total number of bytes submitted to the HTTP compressor for this object since the worker process started
haproxy_backend_http_comp_bytes_out_total counter proxy, instance, ins, job, ip, cls Total number of bytes emitted by the HTTP compressor for this object since the worker process started
haproxy_backend_http_comp_responses_total counter proxy, instance, ins, job, ip, cls Total number of HTTP responses that were compressed for this object since the worker process started
haproxy_backend_http_requests_total counter proxy, instance, ins, job, ip, cls Total number of HTTP requests processed by this object since the worker process started
haproxy_backend_http_responses_total counter ip, proxy, ins, code, job, instance, cls Total number of HTTP responses with status 100-199 returned by this object since the worker process started
haproxy_backend_internal_errors_total counter proxy, instance, ins, job, ip, cls Total number of internal errors since process started
haproxy_backend_last_session_seconds gauge proxy, instance, ins, job, ip, cls How long ago some traffic was seen on this object on this worker process, in seconds
haproxy_backend_limit_sessions gauge proxy, instance, ins, job, ip, cls Frontend/listener/server’s maxconn, backend’s fullconn
haproxy_backend_loadbalanced_total counter proxy, instance, ins, job, ip, cls Total number of requests routed by load balancing since the worker process started (ignores queue pop and stickiness)
haproxy_backend_max_connect_time_seconds gauge proxy, instance, ins, job, ip, cls Maximum observed time spent waiting for a connection to complete
haproxy_backend_max_queue gauge proxy, instance, ins, job, ip, cls Highest value of queued connections encountered since process started
haproxy_backend_max_queue_time_seconds gauge proxy, instance, ins, job, ip, cls Maximum observed time spent in the queue
haproxy_backend_max_response_time_seconds gauge proxy, instance, ins, job, ip, cls Maximum observed time spent waiting for a server response
haproxy_backend_max_session_rate gauge proxy, instance, ins, job, ip, cls Highest value of sessions per second observed since the worker process started
haproxy_backend_max_sessions gauge proxy, instance, ins, job, ip, cls Highest value of current sessions encountered since process started
haproxy_backend_max_total_time_seconds gauge proxy, instance, ins, job, ip, cls Maximum observed total request+response time (request+queue+connect+response+processing)
haproxy_backend_queue_time_average_seconds gauge proxy, instance, ins, job, ip, cls Avg. queue time for last 1024 successful connections.
haproxy_backend_redispatch_warnings_total counter proxy, instance, ins, job, ip, cls Total number of server redispatches due to connection failures since the worker process started
haproxy_backend_requests_denied_total counter proxy, instance, ins, job, ip, cls Total number of denied requests since process started
haproxy_backend_response_errors_total counter proxy, instance, ins, job, ip, cls Total number of invalid responses since the worker process started
haproxy_backend_response_time_average_seconds gauge proxy, instance, ins, job, ip, cls Avg. response time for last 1024 successful connections.
haproxy_backend_responses_denied_total counter proxy, instance, ins, job, ip, cls Total number of denied responses since process started
haproxy_backend_retry_warnings_total counter proxy, instance, ins, job, ip, cls Total number of server connection retries since the worker process started
haproxy_backend_server_aborts_total counter proxy, instance, ins, job, ip, cls Total number of requests or connections aborted by the server since the worker process started
haproxy_backend_sessions_total counter proxy, instance, ins, job, ip, cls Total number of sessions since process started
haproxy_backend_status gauge state, proxy, instance, ins, job, ip, cls Current status of the service, per state label value.
haproxy_backend_total_time_average_seconds gauge proxy, instance, ins, job, ip, cls Avg. total time for last 1024 successful connections.
haproxy_backend_uweight gauge proxy, instance, ins, job, ip, cls Server’s user weight, or sum of active servers’ user weights for a backend
haproxy_backend_weight gauge proxy, instance, ins, job, ip, cls Server’s effective weight, or sum of active servers’ effective weights for a backend
haproxy_frontend_bytes_in_total counter proxy, instance, ins, job, ip, cls Total number of request bytes since process started
haproxy_frontend_bytes_out_total counter proxy, instance, ins, job, ip, cls Total number of response bytes since process started
haproxy_frontend_connections_rate_max gauge proxy, instance, ins, job, ip, cls Highest value of connections per second observed since the worker process started
haproxy_frontend_connections_total counter proxy, instance, ins, job, ip, cls Total number of new connections accepted on this frontend since the worker process started
haproxy_frontend_current_sessions gauge proxy, instance, ins, job, ip, cls Number of current sessions on the frontend, backend or server
haproxy_frontend_denied_connections_total counter proxy, instance, ins, job, ip, cls Total number of incoming connections blocked on a listener/frontend by a tcp-request connection rule since the worker process started
haproxy_frontend_denied_sessions_total counter proxy, instance, ins, job, ip, cls Total number of incoming sessions blocked on a listener/frontend by a tcp-request connection rule since the worker process started
haproxy_frontend_failed_header_rewriting_total counter proxy, instance, ins, job, ip, cls Total number of failed HTTP header rewrites since the worker process started
haproxy_frontend_http_cache_hits_total counter proxy, instance, ins, job, ip, cls Total number of HTTP requests not found in the cache on this frontend/backend since the worker process started
haproxy_frontend_http_cache_lookups_total counter proxy, instance, ins, job, ip, cls Total number of HTTP requests looked up in the cache on this frontend/backend since the worker process started
haproxy_frontend_http_comp_bytes_bypassed_total counter proxy, instance, ins, job, ip, cls Total number of bytes that bypassed HTTP compression for this object since the worker process started (CPU/memory/bandwidth limitation)
haproxy_frontend_http_comp_bytes_in_total counter proxy, instance, ins, job, ip, cls Total number of bytes submitted to the HTTP compressor for this object since the worker process started
haproxy_frontend_http_comp_bytes_out_total counter proxy, instance, ins, job, ip, cls Total number of bytes emitted by the HTTP compressor for this object since the worker process started
haproxy_frontend_http_comp_responses_total counter proxy, instance, ins, job, ip, cls Total number of HTTP responses that were compressed for this object since the worker process started
haproxy_frontend_http_requests_rate_max gauge proxy, instance, ins, job, ip, cls Highest value of http requests observed since the worker process started
haproxy_frontend_http_requests_total counter proxy, instance, ins, job, ip, cls Total number of HTTP requests processed by this object since the worker process started
haproxy_frontend_http_responses_total counter ip, proxy, ins, code, job, instance, cls Total number of HTTP responses with status 100-199 returned by this object since the worker process started
haproxy_frontend_intercepted_requests_total counter proxy, instance, ins, job, ip, cls Total number of HTTP requests intercepted on the frontend (redirects/stats/services) since the worker process started
haproxy_frontend_internal_errors_total counter proxy, instance, ins, job, ip, cls Total number of internal errors since process started
haproxy_frontend_limit_session_rate gauge proxy, instance, ins, job, ip, cls Limit on the number of sessions accepted in a second (frontend only, ‘rate-limit sessions’ setting)
haproxy_frontend_limit_sessions gauge proxy, instance, ins, job, ip, cls Frontend/listener/server’s maxconn, backend’s fullconn
haproxy_frontend_max_session_rate gauge proxy, instance, ins, job, ip, cls Highest value of sessions per second observed since the worker process started
haproxy_frontend_max_sessions gauge proxy, instance, ins, job, ip, cls Highest value of current sessions encountered since process started
haproxy_frontend_request_errors_total counter proxy, instance, ins, job, ip, cls Total number of invalid requests since process started
haproxy_frontend_requests_denied_total counter proxy, instance, ins, job, ip, cls Total number of denied requests since process started
haproxy_frontend_responses_denied_total counter proxy, instance, ins, job, ip, cls Total number of denied responses since process started
haproxy_frontend_sessions_total counter proxy, instance, ins, job, ip, cls Total number of sessions since process started
haproxy_frontend_status gauge state, proxy, instance, ins, job, ip, cls Current status of the service, per state label value.
haproxy_process_active_peers gauge instance, ins, job, ip, cls Current number of verified active peers connections on the current worker process
haproxy_process_build_info gauge version, instance, ins, job, ip, cls Build info
haproxy_process_busy_polling_enabled gauge instance, ins, job, ip, cls 1 if busy-polling is currently in use on the worker process, otherwise zero (config.busy-polling)
haproxy_process_bytes_out_rate gauge instance, ins, job, ip, cls Number of bytes emitted by current worker process over the last second
haproxy_process_bytes_out_total counter instance, ins, job, ip, cls Total number of bytes emitted by current worker process since started
haproxy_process_connected_peers gauge instance, ins, job, ip, cls Current number of peers having passed the connection step on the current worker process
haproxy_process_connections_total counter instance, ins, job, ip, cls Total number of connections on this worker process since started
haproxy_process_current_backend_ssl_key_rate gauge instance, ins, job, ip, cls Number of SSL keys created on backends in this worker process over the last second
haproxy_process_current_connection_rate gauge instance, ins, job, ip, cls Number of front connections created on this worker process over the last second
haproxy_process_current_connections gauge instance, ins, job, ip, cls Current number of connections on this worker process
haproxy_process_current_frontend_ssl_key_rate gauge instance, ins, job, ip, cls Number of SSL keys created on frontends in this worker process over the last second
haproxy_process_current_run_queue gauge instance, ins, job, ip, cls Total number of active tasks+tasklets in the current worker process
haproxy_process_current_session_rate gauge instance, ins, job, ip, cls Number of sessions created on this worker process over the last second
haproxy_process_current_ssl_connections gauge instance, ins, job, ip, cls Current number of SSL endpoints on this worker process (front+back)
haproxy_process_current_ssl_rate gauge instance, ins, job, ip, cls Number of SSL connections created on this worker process over the last second
haproxy_process_current_tasks gauge instance, ins, job, ip, cls Total number of tasks in the current worker process (active + sleeping)
haproxy_process_current_zlib_memory gauge instance, ins, job, ip, cls Amount of memory currently used by HTTP compression on the current worker process (in bytes)
haproxy_process_dropped_logs_total counter instance, ins, job, ip, cls Total number of dropped logs for current worker process since started
haproxy_process_failed_resolutions counter instance, ins, job, ip, cls Total number of failed DNS resolutions in current worker process since started
haproxy_process_frontend_ssl_reuse gauge instance, ins, job, ip, cls Percent of frontend SSL connections which did not require a new key
haproxy_process_hard_max_connections gauge instance, ins, job, ip, cls Hard limit on the number of per-process connections (imposed by Memmax_MB or Ulimit-n)
haproxy_process_http_comp_bytes_in_total counter instance, ins, job, ip, cls Number of bytes submitted to the HTTP compressor in this worker process over the last second
haproxy_process_http_comp_bytes_out_total counter instance, ins, job, ip, cls Number of bytes emitted by the HTTP compressor in this worker process over the last second
haproxy_process_idle_time_percent gauge instance, ins, job, ip, cls Percentage of last second spent waiting in the current worker thread
haproxy_process_jobs gauge instance, ins, job, ip, cls Current number of active jobs on the current worker process (frontend connections, master connections, listeners)
haproxy_process_limit_connection_rate gauge instance, ins, job, ip, cls Hard limit for ConnRate (global.maxconnrate)
haproxy_process_limit_http_comp gauge instance, ins, job, ip, cls Limit of CompressBpsOut beyond which HTTP compression is automatically disabled
haproxy_process_limit_session_rate gauge instance, ins, job, ip, cls Hard limit for SessRate (global.maxsessrate)
haproxy_process_limit_ssl_rate gauge instance, ins, job, ip, cls Hard limit for SslRate (global.maxsslrate)
haproxy_process_listeners gauge instance, ins, job, ip, cls Current number of active listeners on the current worker process
haproxy_process_max_backend_ssl_key_rate gauge instance, ins, job, ip, cls Highest SslBackendKeyRate reached on this worker process since started (in SSL keys per second)
haproxy_process_max_connection_rate gauge instance, ins, job, ip, cls Highest ConnRate reached on this worker process since started (in connections per second)
haproxy_process_max_connections gauge instance, ins, job, ip, cls Hard limit on the number of per-process connections (configured or imposed by Ulimit-n)
haproxy_process_max_fds gauge instance, ins, job, ip, cls Hard limit on the number of per-process file descriptors
haproxy_process_max_frontend_ssl_key_rate gauge instance, ins, job, ip, cls Highest SslFrontendKeyRate reached on this worker process since started (in SSL keys per second)
haproxy_process_max_memory_bytes gauge instance, ins, job, ip, cls Worker process’s hard limit on memory usage in byes (-m on command line)
haproxy_process_max_pipes gauge instance, ins, job, ip, cls Hard limit on the number of pipes for splicing, 0=unlimited
haproxy_process_max_session_rate gauge instance, ins, job, ip, cls Highest SessRate reached on this worker process since started (in sessions per second)
haproxy_process_max_sockets gauge instance, ins, job, ip, cls Hard limit on the number of per-process sockets
haproxy_process_max_ssl_connections gauge instance, ins, job, ip, cls Hard limit on the number of per-process SSL endpoints (front+back), 0=unlimited
haproxy_process_max_ssl_rate gauge instance, ins, job, ip, cls Highest SslRate reached on this worker process since started (in connections per second)
haproxy_process_max_zlib_memory gauge instance, ins, job, ip, cls Limit on the amount of memory used by HTTP compression above which it is automatically disabled (in bytes, see global.maxzlibmem)
haproxy_process_nbproc gauge instance, ins, job, ip, cls Number of started worker processes (historical, always 1)
haproxy_process_nbthread gauge instance, ins, job, ip, cls Number of started threads (global.nbthread)
haproxy_process_pipes_free_total counter instance, ins, job, ip, cls Current number of allocated and available pipes in this worker process
haproxy_process_pipes_used_total counter instance, ins, job, ip, cls Current number of pipes in use in this worker process
haproxy_process_pool_allocated_bytes gauge instance, ins, job, ip, cls Amount of memory allocated in pools (in bytes)
haproxy_process_pool_failures_total counter instance, ins, job, ip, cls Number of failed pool allocations since this worker was started
haproxy_process_pool_used_bytes gauge instance, ins, job, ip, cls Amount of pool memory currently used (in bytes)
haproxy_process_recv_logs_total counter instance, ins, job, ip, cls Total number of log messages received by log-forwarding listeners on this worker process since started
haproxy_process_relative_process_id gauge instance, ins, job, ip, cls Relative worker process number (1)
haproxy_process_requests_total counter instance, ins, job, ip, cls Total number of requests on this worker process since started
haproxy_process_spliced_bytes_out_total counter instance, ins, job, ip, cls Total number of bytes emitted by current worker process through a kernel pipe since started
haproxy_process_ssl_cache_lookups_total counter instance, ins, job, ip, cls Total number of SSL session ID lookups in the SSL session cache on this worker since started
haproxy_process_ssl_cache_misses_total counter instance, ins, job, ip, cls Total number of SSL session ID lookups that didn’t find a session in the SSL session cache on this worker since started
haproxy_process_ssl_connections_total counter instance, ins, job, ip, cls Total number of SSL endpoints on this worker process since started (front+back)
haproxy_process_start_time_seconds gauge instance, ins, job, ip, cls Start time in seconds
haproxy_process_stopping gauge instance, ins, job, ip, cls 1 if the worker process is currently stopping, otherwise zero
haproxy_process_unstoppable_jobs gauge instance, ins, job, ip, cls Current number of unstoppable jobs on the current worker process (master connections)
haproxy_process_uptime_seconds gauge instance, ins, job, ip, cls How long ago this worker process was started (seconds)
haproxy_server_bytes_in_total counter proxy, instance, ins, job, server, ip, cls Total number of request bytes since process started
haproxy_server_bytes_out_total counter proxy, instance, ins, job, server, ip, cls Total number of response bytes since process started
haproxy_server_check_code gauge proxy, instance, ins, job, server, ip, cls layer5-7 code, if available of the last health check.
haproxy_server_check_duration_seconds gauge proxy, instance, ins, job, server, ip, cls Total duration of the latest server health check, in seconds.
haproxy_server_check_failures_total counter proxy, instance, ins, job, server, ip, cls Total number of failed individual health checks per server/backend, since the worker process started
haproxy_server_check_last_change_seconds gauge proxy, instance, ins, job, server, ip, cls How long ago the last server state changed, in seconds
haproxy_server_check_status gauge state, proxy, instance, ins, job, server, ip, cls Status of last health check, per state label value.
haproxy_server_check_up_down_total counter proxy, instance, ins, job, server, ip, cls Total number of failed checks causing UP to DOWN server transitions, per server/backend, since the worker process started
haproxy_server_client_aborts_total counter proxy, instance, ins, job, server, ip, cls Total number of requests or connections aborted by the client since the worker process started
haproxy_server_connect_time_average_seconds gauge proxy, instance, ins, job, server, ip, cls Avg. connect time for last 1024 successful connections.
haproxy_server_connection_attempts_total counter proxy, instance, ins, job, server, ip, cls Total number of outgoing connection attempts on this backend/server since the worker process started
haproxy_server_connection_errors_total counter proxy, instance, ins, job, server, ip, cls Total number of failed connections to server since the worker process started
haproxy_server_connection_reuses_total counter proxy, instance, ins, job, server, ip, cls Total number of reused connection on this backend/server since the worker process started
haproxy_server_current_queue gauge proxy, instance, ins, job, server, ip, cls Number of current queued connections
haproxy_server_current_sessions gauge proxy, instance, ins, job, server, ip, cls Number of current sessions on the frontend, backend or server
haproxy_server_current_throttle gauge proxy, instance, ins, job, server, ip, cls Throttling ratio applied to a server’s maxconn and weight during the slowstart period (0 to 100%)
haproxy_server_downtime_seconds_total counter proxy, instance, ins, job, server, ip, cls Total time spent in DOWN state, for server or backend
haproxy_server_failed_header_rewriting_total counter proxy, instance, ins, job, server, ip, cls Total number of failed HTTP header rewrites since the worker process started
haproxy_server_idle_connections_current gauge proxy, instance, ins, job, server, ip, cls Current number of idle connections available for reuse on this server
haproxy_server_idle_connections_limit gauge proxy, instance, ins, job, server, ip, cls Limit on the number of available idle connections on this server (server ‘pool_max_conn’ directive)
haproxy_server_internal_errors_total counter proxy, instance, ins, job, server, ip, cls Total number of internal errors since process started
haproxy_server_last_session_seconds gauge proxy, instance, ins, job, server, ip, cls How long ago some traffic was seen on this object on this worker process, in seconds
haproxy_server_limit_sessions gauge proxy, instance, ins, job, server, ip, cls Frontend/listener/server’s maxconn, backend’s fullconn
haproxy_server_loadbalanced_total counter proxy, instance, ins, job, server, ip, cls Total number of requests routed by load balancing since the worker process started (ignores queue pop and stickiness)
haproxy_server_max_connect_time_seconds gauge proxy, instance, ins, job, server, ip, cls Maximum observed time spent waiting for a connection to complete
haproxy_server_max_queue gauge proxy, instance, ins, job, server, ip, cls Highest value of queued connections encountered since process started
haproxy_server_max_queue_time_seconds gauge proxy, instance, ins, job, server, ip, cls Maximum observed time spent in the queue
haproxy_server_max_response_time_seconds gauge proxy, instance, ins, job, server, ip, cls Maximum observed time spent waiting for a server response
haproxy_server_max_session_rate gauge proxy, instance, ins, job, server, ip, cls Highest value of sessions per second observed since the worker process started
haproxy_server_max_sessions gauge proxy, instance, ins, job, server, ip, cls Highest value of current sessions encountered since process started
haproxy_server_max_total_time_seconds gauge proxy, instance, ins, job, server, ip, cls Maximum observed total request+response time (request+queue+connect+response+processing)
haproxy_server_need_connections_current gauge proxy, instance, ins, job, server, ip, cls Estimated needed number of connections
haproxy_server_queue_limit gauge proxy, instance, ins, job, server, ip, cls Limit on the number of connections in queue, for servers only (maxqueue argument)
haproxy_server_queue_time_average_seconds gauge proxy, instance, ins, job, server, ip, cls Avg. queue time for last 1024 successful connections.
haproxy_server_redispatch_warnings_total counter proxy, instance, ins, job, server, ip, cls Total number of server redispatches due to connection failures since the worker process started
haproxy_server_response_errors_total counter proxy, instance, ins, job, server, ip, cls Total number of invalid responses since the worker process started
haproxy_server_response_time_average_seconds gauge proxy, instance, ins, job, server, ip, cls Avg. response time for last 1024 successful connections.
haproxy_server_responses_denied_total counter proxy, instance, ins, job, server, ip, cls Total number of denied responses since process started
haproxy_server_retry_warnings_total counter proxy, instance, ins, job, server, ip, cls Total number of server connection retries since the worker process started
haproxy_server_safe_idle_connections_current gauge proxy, instance, ins, job, server, ip, cls Current number of safe idle connections
haproxy_server_server_aborts_total counter proxy, instance, ins, job, server, ip, cls Total number of requests or connections aborted by the server since the worker process started
haproxy_server_sessions_total counter proxy, instance, ins, job, server, ip, cls Total number of sessions since process started
haproxy_server_status gauge state, proxy, instance, ins, job, server, ip, cls Current status of the service, per state label value.
haproxy_server_total_time_average_seconds gauge proxy, instance, ins, job, server, ip, cls Avg. total time for last 1024 successful connections.
haproxy_server_unsafe_idle_connections_current gauge proxy, instance, ins, job, server, ip, cls Current number of unsafe idle connections
haproxy_server_used_connections_current gauge proxy, instance, ins, job, server, ip, cls Current number of connections in use
haproxy_server_uweight gauge proxy, instance, ins, job, server, ip, cls Server’s user weight, or sum of active servers’ user weights for a backend
haproxy_server_weight gauge proxy, instance, ins, job, server, ip, cls Server’s effective weight, or sum of active servers’ effective weights for a backend
haproxy_up Unknown instance, ins, job, ip, cls N/A
inflight_requests gauge instance, ins, job, route, ip, cls, method Current number of inflight requests.
jaeger_tracer_baggage_restrictions_updates_total Unknown instance, ins, job, result, ip, cls N/A
jaeger_tracer_baggage_truncations_total Unknown instance, ins, job, ip, cls N/A
jaeger_tracer_baggage_updates_total Unknown instance, ins, job, result, ip, cls N/A
jaeger_tracer_finished_spans_total Unknown instance, ins, job, sampled, ip, cls N/A
jaeger_tracer_reporter_queue_length gauge instance, ins, job, ip, cls Current number of spans in the reporter queue
jaeger_tracer_reporter_spans_total Unknown instance, ins, job, result, ip, cls N/A
jaeger_tracer_sampler_queries_total Unknown instance, ins, job, result, ip, cls N/A
jaeger_tracer_sampler_updates_total Unknown instance, ins, job, result, ip, cls N/A
jaeger_tracer_span_context_decoding_errors_total Unknown instance, ins, job, ip, cls N/A
jaeger_tracer_started_spans_total Unknown instance, ins, job, sampled, ip, cls N/A
jaeger_tracer_throttled_debug_spans_total Unknown instance, ins, job, ip, cls N/A
jaeger_tracer_throttler_updates_total Unknown instance, ins, job, result, ip, cls N/A
jaeger_tracer_traces_total Unknown state, instance, ins, job, sampled, ip, cls N/A
loki_experimental_features_in_use_total Unknown instance, ins, job, ip, cls N/A
loki_internal_log_messages_total Unknown level, instance, ins, job, ip, cls N/A
loki_log_flushes_bucket Unknown instance, ins, job, le, ip, cls N/A
loki_log_flushes_count Unknown instance, ins, job, ip, cls N/A
loki_log_flushes_sum Unknown instance, ins, job, ip, cls N/A
loki_log_messages_total Unknown level, instance, ins, job, ip, cls N/A
loki_logql_querystats_duplicates_total Unknown instance, ins, job, ip, cls N/A
loki_logql_querystats_ingester_sent_lines_total Unknown instance, ins, job, ip, cls N/A
loki_querier_index_cache_corruptions_total Unknown instance, ins, job, ip, cls N/A
loki_querier_index_cache_encode_errors_total Unknown instance, ins, job, ip, cls N/A
loki_querier_index_cache_gets_total Unknown instance, ins, job, ip, cls N/A
loki_querier_index_cache_hits_total Unknown instance, ins, job, ip, cls N/A
loki_querier_index_cache_puts_total Unknown instance, ins, job, ip, cls N/A
net_conntrack_dialer_conn_attempted_total counter ip, ins, job, instance, cls, dialer_name Total number of connections attempted by the given dialer a given name.
net_conntrack_dialer_conn_closed_total counter ip, ins, job, instance, cls, dialer_name Total number of connections closed which originated from the dialer of a given name.
net_conntrack_dialer_conn_established_total counter ip, ins, job, instance, cls, dialer_name Total number of connections successfully established by the given dialer a given name.
net_conntrack_dialer_conn_failed_total counter ip, ins, job, reason, instance, cls, dialer_name Total number of connections failed to dial by the dialer a given name.
node:cls:avail_bytes Unknown job, cls N/A
node:cls:cpu_count Unknown job, cls N/A
node:cls:cpu_usage Unknown job, cls N/A
node:cls:cpu_usage_15m Unknown job, cls N/A
node:cls:cpu_usage_1m Unknown job, cls N/A
node:cls:cpu_usage_5m Unknown job, cls N/A
node:cls:disk_io_bytes_rate1m Unknown job, cls N/A
node:cls:disk_iops_1m Unknown job, cls N/A
node:cls:disk_mreads_rate1m Unknown job, cls N/A
node:cls:disk_mreads_ratio1m Unknown job, cls N/A
node:cls:disk_mwrites_rate1m Unknown job, cls N/A
node:cls:disk_mwrites_ratio1m Unknown job, cls N/A
node:cls:disk_read_bytes_rate1m Unknown job, cls N/A
node:cls:disk_reads_rate1m Unknown job, cls N/A
node:cls:disk_write_bytes_rate1m Unknown job, cls N/A
node:cls:disk_writes_rate1m Unknown job, cls N/A
node:cls:free_bytes Unknown job, cls N/A
node:cls:mem_usage Unknown job, cls N/A
node:cls:network_io_bytes_rate1m Unknown job, cls N/A
node:cls:network_rx_bytes_rate1m Unknown job, cls N/A
node:cls:network_rx_pps1m Unknown job, cls N/A
node:cls:network_tx_bytes_rate1m Unknown job, cls N/A
node:cls:network_tx_pps1m Unknown job, cls N/A
node:cls:size_bytes Unknown job, cls N/A
node:cls:space_usage Unknown job, cls N/A
node:cls:space_usage_max Unknown job, cls N/A
node:cls:stdload1 Unknown job, cls N/A
node:cls:stdload15 Unknown job, cls N/A
node:cls:stdload5 Unknown job, cls N/A
node:cls:time_drift_max Unknown job, cls N/A
node:cpu:idle_time_irate1m Unknown ip, ins, job, cpu, instance, cls N/A
node:cpu:sched_timeslices_rate1m Unknown ip, ins, job, cpu, instance, cls N/A
node:cpu:sched_wait_rate1m Unknown ip, ins, job, cpu, instance, cls N/A
node:cpu:time_irate1m Unknown ip, mode, ins, job, cpu, instance, cls N/A
node:cpu:total_time_irate1m Unknown ip, ins, job, cpu, instance, cls N/A
node:cpu:usage Unknown ip, ins, job, cpu, instance, cls N/A
node:cpu:usage_avg15m Unknown ip, ins, job, cpu, instance, cls N/A
node:cpu:usage_avg1m Unknown ip, ins, job, cpu, instance, cls N/A
node:cpu:usage_avg5m Unknown ip, ins, job, cpu, instance, cls N/A
node:dev:disk_avg_queue_size Unknown ip, device, ins, job, instance, cls N/A
node:dev:disk_io_batch_1m Unknown ip, device, ins, job, instance, cls N/A
node:dev:disk_io_bytes_rate1m Unknown ip, device, ins, job, instance, cls N/A
node:dev:disk_io_rt_1m Unknown ip, device, ins, job, instance, cls N/A
node:dev:disk_io_time_rate1m Unknown ip, device, ins, job, instance, cls N/A
node:dev:disk_iops_1m Unknown ip, device, ins, job, instance, cls N/A
node:dev:disk_mreads_rate1m Unknown ip, device, ins, job, instance, cls N/A
node:dev:disk_mreads_ratio1m Unknown ip, device, ins, job, instance, cls N/A
node:dev:disk_mwrites_rate1m Unknown ip, device, ins, job, instance, cls N/A
node:dev:disk_mwrites_ratio1m Unknown ip, device, ins, job, instance, cls N/A
node:dev:disk_read_batch_1m Unknown ip, device, ins, job, instance, cls N/A
node:dev:disk_read_bytes_rate1m Unknown ip, device, ins, job, instance, cls N/A
node:dev:disk_read_rt_1m Unknown ip, device, ins, job, instance, cls N/A
node:dev:disk_read_time_rate1m Unknown ip, device, ins, job, instance, cls N/A
node:dev:disk_reads_rate1m Unknown ip, device, ins, job, instance, cls N/A
node:dev:disk_util_1m Unknown ip, device, ins, job, instance, cls N/A
node:dev:disk_write_batch_1m Unknown ip, device, ins, job, instance, cls N/A
node:dev:disk_write_bytes_rate1m Unknown ip, device, ins, job, instance, cls N/A
node:dev:disk_write_rt_1m Unknown ip, device, ins, job, instance, cls N/A
node:dev:disk_write_time_rate1m Unknown ip, device, ins, job, instance, cls N/A
node:dev:disk_writes_rate1m Unknown ip, device, ins, job, instance, cls N/A
node:dev:network_io_bytes_rate1m Unknown ip, device, ins, job, instance, cls N/A
node:dev:network_rx_bytes_rate1m Unknown ip, device, ins, job, instance, cls N/A
node:dev:network_rx_pps1m Unknown ip, device, ins, job, instance, cls N/A
node:dev:network_tx_bytes_rate1m Unknown ip, device, ins, job, instance, cls N/A
node:dev:network_tx_pps1m Unknown ip, device, ins, job, instance, cls N/A
node:env:avail_bytes Unknown job N/A
node:env:cpu_count Unknown job N/A
node:env:cpu_usage Unknown job N/A
node:env:cpu_usage_15m Unknown job N/A
node:env:cpu_usage_1m Unknown job N/A
node:env:cpu_usage_5m Unknown job N/A
node:env:device_space_usage_max Unknown device, mountpoint, job, fstype N/A
node:env:free_bytes Unknown job N/A
node:env:mem_avail Unknown job N/A
node:env:mem_total Unknown job N/A
node:env:mem_usage Unknown job N/A
node:env:size_bytes Unknown job N/A
node:env:space_usage Unknown job N/A
node:env:stdload1 Unknown job N/A
node:env:stdload15 Unknown job N/A
node:env:stdload5 Unknown job N/A
node:fs:avail_bytes Unknown ip, device, mountpoint, ins, cls, job, instance, fstype N/A
node:fs:free_bytes Unknown ip, device, mountpoint, ins, cls, job, instance, fstype N/A
node:fs:inode_free Unknown ip, device, mountpoint, ins, cls, job, instance, fstype N/A
node:fs:inode_total Unknown ip, device, mountpoint, ins, cls, job, instance, fstype N/A
node:fs:inode_usage Unknown ip, device, mountpoint, ins, cls, job, instance, fstype N/A
node:fs:inode_used Unknown ip, device, mountpoint, ins, cls, job, instance, fstype N/A
node:fs:size_bytes Unknown ip, device, mountpoint, ins, cls, job, instance, fstype N/A
node:fs:space_deriv1h Unknown ip, device, mountpoint, ins, cls, job, instance, fstype N/A
node:fs:space_exhaust Unknown ip, device, mountpoint, ins, cls, job, instance, fstype N/A
node:fs:space_predict_1d Unknown ip, device, mountpoint, ins, cls, job, instance, fstype N/A
node:fs:space_usage Unknown ip, device, mountpoint, ins, cls, job, instance, fstype N/A
node:ins Unknown id, ip, ins, job, nodename, instance, cls N/A
node:ins:avail_bytes Unknown instance, ins, job, ip, cls N/A
node:ins:cpu_count Unknown instance, ins, job, ip, cls N/A
node:ins:cpu_usage Unknown instance, ins, job, ip, cls N/A
node:ins:cpu_usage_15m Unknown instance, ins, job, ip, cls N/A
node:ins:cpu_usage_1m Unknown instance, ins, job, ip, cls N/A
node:ins:cpu_usage_5m Unknown instance, ins, job, ip, cls N/A
node:ins:ctx_switch_rate1m Unknown instance, ins, job, ip, cls N/A
node:ins:disk_io_bytes_rate1m Unknown instance, ins, job, ip, cls N/A
node:ins:disk_iops_1m Unknown instance, ins, job, ip, cls N/A
node:ins:disk_mreads_rate1m Unknown instance, ins, job, ip, cls N/A
node:ins:disk_mreads_ratio1m Unknown instance, ins, job, ip, cls N/A
node:ins:disk_mwrites_rate1m Unknown instance, ins, job, ip, cls N/A
node:ins:disk_mwrites_ratio1m Unknown instance, ins, job, ip, cls N/A
node:ins:disk_read_bytes_rate1m Unknown instance, ins, job, ip, cls N/A
node:ins:disk_reads_rate1m Unknown instance, ins, job, ip, cls N/A
node:ins:disk_write_bytes_rate1m Unknown instance, ins, job, ip, cls N/A
node:ins:disk_writes_rate1m Unknown instance, ins, job, ip, cls N/A
node:ins:fd_alloc_rate1m Unknown instance, ins, job, ip, cls N/A
node:ins:fd_usage Unknown instance, ins, job, ip, cls N/A
node:ins:forks_rate1m Unknown instance, ins, job, ip, cls N/A
node:ins:free_bytes Unknown instance, ins, job, ip, cls N/A
node:ins:inode_usage Unknown instance, ins, job, ip, cls N/A
node:ins:interrupt_rate1m Unknown instance, ins, job, ip, cls N/A
node:ins:mem_avail Unknown instance, ins, job, ip, cls N/A
node:ins:mem_commit_ratio Unknown instance, ins, job, ip, cls N/A
node:ins:mem_kernel Unknown instance, ins, job, ip, cls N/A
node:ins:mem_rss Unknown instance, ins, job, ip, cls N/A
node:ins:mem_usage Unknown instance, ins, job, ip, cls N/A
node:ins:network_io_bytes_rate1m Unknown instance, ins, job, ip, cls N/A
node:ins:network_rx_bytes_rate1m Unknown instance, ins, job, ip, cls N/A
node:ins:network_rx_pps1m Unknown instance, ins, job, ip, cls N/A
node:ins:network_tx_bytes_rate1m Unknown instance, ins, job, ip, cls N/A
node:ins:network_tx_pps1m Unknown instance, ins, job, ip, cls N/A
node:ins:pagefault_rate1m Unknown instance, ins, job, ip, cls N/A
node:ins:pagein_rate1m Unknown instance, ins, job, ip, cls N/A
node:ins:pageout_rate1m Unknown instance, ins, job, ip, cls N/A
node:ins:pgmajfault_rate1m Unknown instance, ins, job, ip, cls N/A
node:ins:sched_wait_rate1m Unknown instance, ins, job, ip, cls N/A
node:ins:size_bytes Unknown instance, ins, job, ip, cls N/A
node:ins:space_usage_max Unknown instance, ins, job, ip, cls N/A
node:ins:stdload1 Unknown instance, ins, job, ip, cls N/A
node:ins:stdload15 Unknown instance, ins, job, ip, cls N/A
node:ins:stdload5 Unknown instance, ins, job, ip, cls N/A
node:ins:swap_usage Unknown instance, ins, job, ip, cls N/A
node:ins:swapin_rate1m Unknown instance, ins, job, ip, cls N/A
node:ins:swapout_rate1m Unknown instance, ins, job, ip, cls N/A
node:ins:tcp_active_opens_rate1m Unknown instance, ins, job, ip, cls N/A
node:ins:tcp_dropped_rate1m Unknown instance, ins, job, ip, cls N/A
node:ins:tcp_error Unknown instance, ins, job, ip, cls N/A
node:ins:tcp_error_rate1m Unknown instance, ins, job, ip, cls N/A
node:ins:tcp_insegs_rate1m Unknown instance, ins, job, ip, cls N/A
node:ins:tcp_outsegs_rate1m Unknown instance, ins, job, ip, cls N/A
node:ins:tcp_overflow_rate1m Unknown instance, ins, job, ip, cls N/A
node:ins:tcp_passive_opens_rate1m Unknown instance, ins, job, ip, cls N/A
node:ins:tcp_retrans_ratio1m Unknown instance, ins, job, ip, cls N/A
node:ins:tcp_retranssegs_rate1m Unknown instance, ins, job, ip, cls N/A
node:ins:tcp_segs_rate1m Unknown instance, ins, job, ip, cls N/A
node:ins:time_drift Unknown instance, ins, job, ip, cls N/A
node:ins:udp_in_rate1m Unknown instance, ins, job, ip, cls N/A
node:ins:udp_out_rate1m Unknown instance, ins, job, ip, cls N/A
node:ins:uptime Unknown instance, ins, job, ip, cls N/A
node_arp_entries gauge ip, device, ins, job, instance, cls ARP entries by device
node_boot_time_seconds gauge instance, ins, job, ip, cls Node boot time, in unixtime.
node_context_switches_total counter instance, ins, job, ip, cls Total number of context switches.
node_cooling_device_cur_state gauge instance, ins, job, type, ip, cls Current throttle state of the cooling device
node_cooling_device_max_state gauge instance, ins, job, type, ip, cls Maximum throttle state of the cooling device
node_cpu_guest_seconds_total counter ip, mode, ins, job, cpu, instance, cls Seconds the CPUs spent in guests (VMs) for each mode.
node_cpu_seconds_total counter ip, mode, ins, job, cpu, instance, cls Seconds the CPUs spent in each mode.
node_disk_discard_time_seconds_total counter ip, device, ins, job, instance, cls This is the total number of seconds spent by all discards.
node_disk_discarded_sectors_total counter ip, device, ins, job, instance, cls The total number of sectors discarded successfully.
node_disk_discards_completed_total counter ip, device, ins, job, instance, cls The total number of discards completed successfully.
node_disk_discards_merged_total counter ip, device, ins, job, instance, cls The total number of discards merged.
node_disk_filesystem_info gauge ip, usage, version, device, uuid, ins, type, job, instance, cls Info about disk filesystem.
node_disk_info gauge minor, ip, major, revision, device, model, serial, path, ins, job, instance, cls Info of /sys/block/<block_device>.
node_disk_io_now gauge ip, device, ins, job, instance, cls The number of I/Os currently in progress.
node_disk_io_time_seconds_total counter ip, device, ins, job, instance, cls Total seconds spent doing I/Os.
node_disk_io_time_weighted_seconds_total counter ip, device, ins, job, instance, cls The weighted # of seconds spent doing I/Os.
node_disk_read_bytes_total counter ip, device, ins, job, instance, cls The total number of bytes read successfully.
node_disk_read_time_seconds_total counter ip, device, ins, job, instance, cls The total number of seconds spent by all reads.
node_disk_reads_completed_total counter ip, device, ins, job, instance, cls The total number of reads completed successfully.
node_disk_reads_merged_total counter ip, device, ins, job, instance, cls The total number of reads merged.
node_disk_write_time_seconds_total counter ip, device, ins, job, instance, cls This is the total number of seconds spent by all writes.
node_disk_writes_completed_total counter ip, device, ins, job, instance, cls The total number of writes completed successfully.
node_disk_writes_merged_total counter ip, device, ins, job, instance, cls The number of writes merged.
node_disk_written_bytes_total counter ip, device, ins, job, instance, cls The total number of bytes written successfully.
node_dmi_info gauge bios_vendor, ip, product_family, product_version, product_uuid, system_vendor, bios_version, ins, bios_date, cls, job, product_name, instance, chassis_version, chassis_vendor, product_serial A metric with a constant ‘1’ value labeled by bios_date, bios_release, bios_vendor, bios_version, board_asset_tag, board_name, board_serial, board_vendor, board_version, chassis_asset_tag, chassis_serial, chassis_vendor, chassis_version, product_family, product_name, product_serial, product_sku, product_uuid, product_version, system_vendor if provided by DMI.
node_entropy_available_bits gauge instance, ins, job, ip, cls Bits of available entropy.
node_entropy_pool_size_bits gauge instance, ins, job, ip, cls Bits of entropy pool.
node_exporter_build_info gauge ip, version, revision, goversion, branch, ins, goarch, job, tags, instance, cls, goos A metric with a constant ‘1’ value labeled by version, revision, branch, goversion from which node_exporter was built, and the goos and goarch for the build.
node_filefd_allocated gauge instance, ins, job, ip, cls File descriptor statistics: allocated.
node_filefd_maximum gauge instance, ins, job, ip, cls File descriptor statistics: maximum.
node_filesystem_avail_bytes gauge ip, device, mountpoint, ins, cls, job, instance, fstype Filesystem space available to non-root users in bytes.
node_filesystem_device_error gauge ip, device, mountpoint, ins, cls, job, instance, fstype Whether an error occurred while getting statistics for the given device.
node_filesystem_files gauge ip, device, mountpoint, ins, cls, job, instance, fstype Filesystem total file nodes.
node_filesystem_files_free gauge ip, device, mountpoint, ins, cls, job, instance, fstype Filesystem total free file nodes.
node_filesystem_free_bytes gauge ip, device, mountpoint, ins, cls, job, instance, fstype Filesystem free space in bytes.
node_filesystem_readonly gauge ip, device, mountpoint, ins, cls, job, instance, fstype Filesystem read-only status.
node_filesystem_size_bytes gauge ip, device, mountpoint, ins, cls, job, instance, fstype Filesystem size in bytes.
node_forks_total counter instance, ins, job, ip, cls Total number of forks.
node_hwmon_chip_names gauge chip_name, ip, ins, chip, job, instance, cls Annotation metric for human-readable chip names
node_hwmon_energy_joule_total counter sensor, ip, ins, chip, job, instance, cls Hardware monitor for joules used so far (input)
node_hwmon_sensor_label gauge sensor, ip, ins, chip, job, label, instance, cls Label for given chip and sensor
node_intr_total counter instance, ins, job, ip, cls Total number of interrupts serviced.
node_ipvs_connections_total counter instance, ins, job, ip, cls The total number of connections made.
node_ipvs_incoming_bytes_total counter instance, ins, job, ip, cls The total amount of incoming data.
node_ipvs_incoming_packets_total counter instance, ins, job, ip, cls The total number of incoming packets.
node_ipvs_outgoing_bytes_total counter instance, ins, job, ip, cls The total amount of outgoing data.
node_ipvs_outgoing_packets_total counter instance, ins, job, ip, cls The total number of outgoing packets.
node_load1 gauge instance, ins, job, ip, cls 1m load average.
node_load15 gauge instance, ins, job, ip, cls 15m load average.
node_load5 gauge instance, ins, job, ip, cls 5m load average.
node_memory_Active_anon_bytes gauge instance, ins, job, ip, cls Memory information field Active_anon_bytes.
node_memory_Active_bytes gauge instance, ins, job, ip, cls Memory information field Active_bytes.
node_memory_Active_file_bytes gauge instance, ins, job, ip, cls Memory information field Active_file_bytes.
node_memory_AnonHugePages_bytes gauge instance, ins, job, ip, cls Memory information field AnonHugePages_bytes.
node_memory_AnonPages_bytes gauge instance, ins, job, ip, cls Memory information field AnonPages_bytes.
node_memory_Bounce_bytes gauge instance, ins, job, ip, cls Memory information field Bounce_bytes.
node_memory_Buffers_bytes gauge instance, ins, job, ip, cls Memory information field Buffers_bytes.
node_memory_Cached_bytes gauge instance, ins, job, ip, cls Memory information field Cached_bytes.
node_memory_CommitLimit_bytes gauge instance, ins, job, ip, cls Memory information field CommitLimit_bytes.
node_memory_Committed_AS_bytes gauge instance, ins, job, ip, cls Memory information field Committed_AS_bytes.
node_memory_DirectMap1G_bytes gauge instance, ins, job, ip, cls Memory information field DirectMap1G_bytes.
node_memory_DirectMap2M_bytes gauge instance, ins, job, ip, cls Memory information field DirectMap2M_bytes.
node_memory_DirectMap4k_bytes gauge instance, ins, job, ip, cls Memory information field DirectMap4k_bytes.
node_memory_Dirty_bytes gauge instance, ins, job, ip, cls Memory information field Dirty_bytes.
node_memory_FileHugePages_bytes gauge instance, ins, job, ip, cls Memory information field FileHugePages_bytes.
node_memory_FilePmdMapped_bytes gauge instance, ins, job, ip, cls Memory information field FilePmdMapped_bytes.
node_memory_HardwareCorrupted_bytes gauge instance, ins, job, ip, cls Memory information field HardwareCorrupted_bytes.
node_memory_HugePages_Free gauge instance, ins, job, ip, cls Memory information field HugePages_Free.
node_memory_HugePages_Rsvd gauge instance, ins, job, ip, cls Memory information field HugePages_Rsvd.
node_memory_HugePages_Surp gauge instance, ins, job, ip, cls Memory information field HugePages_Surp.
node_memory_HugePages_Total gauge instance, ins, job, ip, cls Memory information field HugePages_Total.
node_memory_Hugepagesize_bytes gauge instance, ins, job, ip, cls Memory information field Hugepagesize_bytes.
node_memory_Hugetlb_bytes gauge instance, ins, job, ip, cls Memory information field Hugetlb_bytes.
node_memory_Inactive_anon_bytes gauge instance, ins, job, ip, cls Memory information field Inactive_anon_bytes.
node_memory_Inactive_bytes gauge instance, ins, job, ip, cls Memory information field Inactive_bytes.
node_memory_Inactive_file_bytes gauge instance, ins, job, ip, cls Memory information field Inactive_file_bytes.
node_memory_KReclaimable_bytes gauge instance, ins, job, ip, cls Memory information field KReclaimable_bytes.
node_memory_KernelStack_bytes gauge instance, ins, job, ip, cls Memory information field KernelStack_bytes.
node_memory_Mapped_bytes gauge instance, ins, job, ip, cls Memory information field Mapped_bytes.
node_memory_MemAvailable_bytes gauge instance, ins, job, ip, cls Memory information field MemAvailable_bytes.
node_memory_MemFree_bytes gauge instance, ins, job, ip, cls Memory information field MemFree_bytes.
node_memory_MemTotal_bytes gauge instance, ins, job, ip, cls Memory information field MemTotal_bytes.
node_memory_Mlocked_bytes gauge instance, ins, job, ip, cls Memory information field Mlocked_bytes.
node_memory_NFS_Unstable_bytes gauge instance, ins, job, ip, cls Memory information field NFS_Unstable_bytes.
node_memory_PageTables_bytes gauge instance, ins, job, ip, cls Memory information field PageTables_bytes.
node_memory_Percpu_bytes gauge instance, ins, job, ip, cls Memory information field Percpu_bytes.
node_memory_SReclaimable_bytes gauge instance, ins, job, ip, cls Memory information field SReclaimable_bytes.
node_memory_SUnreclaim_bytes gauge instance, ins, job, ip, cls Memory information field SUnreclaim_bytes.
node_memory_ShmemHugePages_bytes gauge instance, ins, job, ip, cls Memory information field ShmemHugePages_bytes.
node_memory_ShmemPmdMapped_bytes gauge instance, ins, job, ip, cls Memory information field ShmemPmdMapped_bytes.
node_memory_Shmem_bytes gauge instance, ins, job, ip, cls Memory information field Shmem_bytes.
node_memory_Slab_bytes gauge instance, ins, job, ip, cls Memory information field Slab_bytes.
node_memory_SwapCached_bytes gauge instance, ins, job, ip, cls Memory information field SwapCached_bytes.
node_memory_SwapFree_bytes gauge instance, ins, job, ip, cls Memory information field SwapFree_bytes.
node_memory_SwapTotal_bytes gauge instance, ins, job, ip, cls Memory information field SwapTotal_bytes.
node_memory_Unevictable_bytes gauge instance, ins, job, ip, cls Memory information field Unevictable_bytes.
node_memory_VmallocChunk_bytes gauge instance, ins, job, ip, cls Memory information field VmallocChunk_bytes.
node_memory_VmallocTotal_bytes gauge instance, ins, job, ip, cls Memory information field VmallocTotal_bytes.
node_memory_VmallocUsed_bytes gauge instance, ins, job, ip, cls Memory information field VmallocUsed_bytes.
node_memory_WritebackTmp_bytes gauge instance, ins, job, ip, cls Memory information field WritebackTmp_bytes.
node_memory_Writeback_bytes gauge instance, ins, job, ip, cls Memory information field Writeback_bytes.
node_netstat_Icmp6_InErrors unknown instance, ins, job, ip, cls Statistic Icmp6InErrors.
node_netstat_Icmp6_InMsgs unknown instance, ins, job, ip, cls Statistic Icmp6InMsgs.
node_netstat_Icmp6_OutMsgs unknown instance, ins, job, ip, cls Statistic Icmp6OutMsgs.
node_netstat_Icmp_InErrors unknown instance, ins, job, ip, cls Statistic IcmpInErrors.
node_netstat_Icmp_InMsgs unknown instance, ins, job, ip, cls Statistic IcmpInMsgs.
node_netstat_Icmp_OutMsgs unknown instance, ins, job, ip, cls Statistic IcmpOutMsgs.
node_netstat_Ip6_InOctets unknown instance, ins, job, ip, cls Statistic Ip6InOctets.
node_netstat_Ip6_OutOctets unknown instance, ins, job, ip, cls Statistic Ip6OutOctets.
node_netstat_IpExt_InOctets unknown instance, ins, job, ip, cls Statistic IpExtInOctets.
node_netstat_IpExt_OutOctets unknown instance, ins, job, ip, cls Statistic IpExtOutOctets.
node_netstat_Ip_Forwarding unknown instance, ins, job, ip, cls Statistic IpForwarding.
node_netstat_TcpExt_ListenDrops unknown instance, ins, job, ip, cls Statistic TcpExtListenDrops.
node_netstat_TcpExt_ListenOverflows unknown instance, ins, job, ip, cls Statistic TcpExtListenOverflows.
node_netstat_TcpExt_SyncookiesFailed unknown instance, ins, job, ip, cls Statistic TcpExtSyncookiesFailed.
node_netstat_TcpExt_SyncookiesRecv unknown instance, ins, job, ip, cls Statistic TcpExtSyncookiesRecv.
node_netstat_TcpExt_SyncookiesSent unknown instance, ins, job, ip, cls Statistic TcpExtSyncookiesSent.
node_netstat_TcpExt_TCPSynRetrans unknown instance, ins, job, ip, cls Statistic TcpExtTCPSynRetrans.
node_netstat_TcpExt_TCPTimeouts unknown instance, ins, job, ip, cls Statistic TcpExtTCPTimeouts.
node_netstat_Tcp_ActiveOpens unknown instance, ins, job, ip, cls Statistic TcpActiveOpens.
node_netstat_Tcp_CurrEstab unknown instance, ins, job, ip, cls Statistic TcpCurrEstab.
node_netstat_Tcp_InErrs unknown instance, ins, job, ip, cls Statistic TcpInErrs.
node_netstat_Tcp_InSegs unknown instance, ins, job, ip, cls Statistic TcpInSegs.
node_netstat_Tcp_OutRsts unknown instance, ins, job, ip, cls Statistic TcpOutRsts.
node_netstat_Tcp_OutSegs unknown instance, ins, job, ip, cls Statistic TcpOutSegs.
node_netstat_Tcp_PassiveOpens unknown instance, ins, job, ip, cls Statistic TcpPassiveOpens.
node_netstat_Tcp_RetransSegs unknown instance, ins, job, ip, cls Statistic TcpRetransSegs.
node_netstat_Udp6_InDatagrams unknown instance, ins, job, ip, cls Statistic Udp6InDatagrams.
node_netstat_Udp6_InErrors unknown instance, ins, job, ip, cls Statistic Udp6InErrors.
node_netstat_Udp6_NoPorts unknown instance, ins, job, ip, cls Statistic Udp6NoPorts.
node_netstat_Udp6_OutDatagrams unknown instance, ins, job, ip, cls Statistic Udp6OutDatagrams.
node_netstat_Udp6_RcvbufErrors unknown instance, ins, job, ip, cls Statistic Udp6RcvbufErrors.
node_netstat_Udp6_SndbufErrors unknown instance, ins, job, ip, cls Statistic Udp6SndbufErrors.
node_netstat_UdpLite6_InErrors unknown instance, ins, job, ip, cls Statistic UdpLite6InErrors.
node_netstat_UdpLite_InErrors unknown instance, ins, job, ip, cls Statistic UdpLiteInErrors.
node_netstat_Udp_InDatagrams unknown instance, ins, job, ip, cls Statistic UdpInDatagrams.
node_netstat_Udp_InErrors unknown instance, ins, job, ip, cls Statistic UdpInErrors.
node_netstat_Udp_NoPorts unknown instance, ins, job, ip, cls Statistic UdpNoPorts.
node_netstat_Udp_OutDatagrams unknown instance, ins, job, ip, cls Statistic UdpOutDatagrams.
node_netstat_Udp_RcvbufErrors unknown instance, ins, job, ip, cls Statistic UdpRcvbufErrors.
node_netstat_Udp_SndbufErrors unknown instance, ins, job, ip, cls Statistic UdpSndbufErrors.
node_network_address_assign_type gauge ip, device, ins, job, instance, cls Network device property: address_assign_type
node_network_carrier gauge ip, device, ins, job, instance, cls Network device property: carrier
node_network_carrier_changes_total counter ip, device, ins, job, instance, cls Network device property: carrier_changes_total
node_network_carrier_down_changes_total counter ip, device, ins, job, instance, cls Network device property: carrier_down_changes_total
node_network_carrier_up_changes_total counter ip, device, ins, job, instance, cls Network device property: carrier_up_changes_total
node_network_device_id gauge ip, device, ins, job, instance, cls Network device property: device_id
node_network_dormant gauge ip, device, ins, job, instance, cls Network device property: dormant
node_network_flags gauge ip, device, ins, job, instance, cls Network device property: flags
node_network_iface_id gauge ip, device, ins, job, instance, cls Network device property: iface_id
node_network_iface_link gauge ip, device, ins, job, instance, cls Network device property: iface_link
node_network_iface_link_mode gauge ip, device, ins, job, instance, cls Network device property: iface_link_mode
node_network_info gauge broadcast, ip, device, operstate, ins, job, adminstate, duplex, address, instance, cls Non-numeric data from /sys/class/net/, value is always 1.
node_network_mtu_bytes gauge ip, device, ins, job, instance, cls Network device property: mtu_bytes
node_network_name_assign_type gauge ip, device, ins, job, instance, cls Network device property: name_assign_type
node_network_net_dev_group gauge ip, device, ins, job, instance, cls Network device property: net_dev_group
node_network_protocol_type gauge ip, device, ins, job, instance, cls Network device property: protocol_type
node_network_receive_bytes_total counter ip, device, ins, job, instance, cls Network device statistic receive_bytes.
node_network_receive_compressed_total counter ip, device, ins, job, instance, cls Network device statistic receive_compressed.
node_network_receive_drop_total counter ip, device, ins, job, instance, cls Network device statistic receive_drop.
node_network_receive_errs_total counter ip, device, ins, job, instance, cls Network device statistic receive_errs.
node_network_receive_fifo_total counter ip, device, ins, job, instance, cls Network device statistic receive_fifo.
node_network_receive_frame_total counter ip, device, ins, job, instance, cls Network device statistic receive_frame.
node_network_receive_multicast_total counter ip, device, ins, job, instance, cls Network device statistic receive_multicast.
node_network_receive_nohandler_total counter ip, device, ins, job, instance, cls Network device statistic receive_nohandler.
node_network_receive_packets_total counter ip, device, ins, job, instance, cls Network device statistic receive_packets.
node_network_speed_bytes gauge ip, device, ins, job, instance, cls Network device property: speed_bytes
node_network_transmit_bytes_total counter ip, device, ins, job, instance, cls Network device statistic transmit_bytes.
node_network_transmit_carrier_total counter ip, device, ins, job, instance, cls Network device statistic transmit_carrier.
node_network_transmit_colls_total counter ip, device, ins, job, instance, cls Network device statistic transmit_colls.
node_network_transmit_compressed_total counter ip, device, ins, job, instance, cls Network device statistic transmit_compressed.
node_network_transmit_drop_total counter ip, device, ins, job, instance, cls Network device statistic transmit_drop.
node_network_transmit_errs_total counter ip, device, ins, job, instance, cls Network device statistic transmit_errs.
node_network_transmit_fifo_total counter ip, device, ins, job, instance, cls Network device statistic transmit_fifo.
node_network_transmit_packets_total counter ip, device, ins, job, instance, cls Network device statistic transmit_packets.
node_network_transmit_queue_length gauge ip, device, ins, job, instance, cls Network device property: transmit_queue_length
node_network_up gauge ip, device, ins, job, instance, cls Value is 1 if operstate is ‘up’, 0 otherwise.
node_nf_conntrack_entries gauge instance, ins, job, ip, cls Number of currently allocated flow entries for connection tracking.
node_nf_conntrack_entries_limit gauge instance, ins, job, ip, cls Maximum size of connection tracking table.
node_nf_conntrack_stat_drop gauge instance, ins, job, ip, cls Number of packets dropped due to conntrack failure.
node_nf_conntrack_stat_early_drop gauge instance, ins, job, ip, cls Number of dropped conntrack entries to make room for new ones, if maximum table size was reached.
node_nf_conntrack_stat_found gauge instance, ins, job, ip, cls Number of searched entries which were successful.
node_nf_conntrack_stat_ignore gauge instance, ins, job, ip, cls Number of packets seen which are already connected to a conntrack entry.
node_nf_conntrack_stat_insert gauge instance, ins, job, ip, cls Number of entries inserted into the list.
node_nf_conntrack_stat_insert_failed gauge instance, ins, job, ip, cls Number of entries for which list insertion was attempted but failed.
node_nf_conntrack_stat_invalid gauge instance, ins, job, ip, cls Number of packets seen which can not be tracked.
node_nf_conntrack_stat_search_restart gauge instance, ins, job, ip, cls Number of conntrack table lookups which had to be restarted due to hashtable resizes.
node_os_info gauge id, ip, version, version_id, ins, instance, job, pretty_name, id_like, cls A metric with a constant ‘1’ value labeled by build_id, id, id_like, image_id, image_version, name, pretty_name, variant, variant_id, version, version_codename, version_id.
node_os_version gauge id, ip, ins, instance, job, id_like, cls Metric containing the major.minor part of the OS version.
node_processes_max_processes gauge instance, ins, job, ip, cls Number of max PIDs limit
node_processes_max_threads gauge instance, ins, job, ip, cls Limit of threads in the system
node_processes_pids gauge instance, ins, job, ip, cls Number of PIDs
node_processes_state gauge state, instance, ins, job, ip, cls Number of processes in each state.
node_processes_threads gauge instance, ins, job, ip, cls Allocated threads in system
node_processes_threads_state gauge instance, ins, job, thread_state, ip, cls Number of threads in each state.
node_procs_blocked gauge instance, ins, job, ip, cls Number of processes blocked waiting for I/O to complete.
node_procs_running gauge instance, ins, job, ip, cls Number of processes in runnable state.
node_schedstat_running_seconds_total counter ip, ins, job, cpu, instance, cls Number of seconds CPU spent running a process.
node_schedstat_timeslices_total counter ip, ins, job, cpu, instance, cls Number of timeslices executed by CPU.
node_schedstat_waiting_seconds_total counter ip, ins, job, cpu, instance, cls Number of seconds spent by processing waiting for this CPU.
node_scrape_collector_duration_seconds gauge ip, collector, ins, job, instance, cls node_exporter: Duration of a collector scrape.
node_scrape_collector_success gauge ip, collector, ins, job, instance, cls node_exporter: Whether a collector succeeded.
node_selinux_enabled gauge instance, ins, job, ip, cls SELinux is enabled, 1 is true, 0 is false
node_sockstat_FRAG6_inuse gauge instance, ins, job, ip, cls Number of FRAG6 sockets in state inuse.
node_sockstat_FRAG6_memory gauge instance, ins, job, ip, cls Number of FRAG6 sockets in state memory.
node_sockstat_FRAG_inuse gauge instance, ins, job, ip, cls Number of FRAG sockets in state inuse.
node_sockstat_FRAG_memory gauge instance, ins, job, ip, cls Number of FRAG sockets in state memory.
node_sockstat_RAW6_inuse gauge instance, ins, job, ip, cls Number of RAW6 sockets in state inuse.
node_sockstat_RAW_inuse gauge instance, ins, job, ip, cls Number of RAW sockets in state inuse.
node_sockstat_TCP6_inuse gauge instance, ins, job, ip, cls Number of TCP6 sockets in state inuse.
node_sockstat_TCP_alloc gauge instance, ins, job, ip, cls Number of TCP sockets in state alloc.
node_sockstat_TCP_inuse gauge instance, ins, job, ip, cls Number of TCP sockets in state inuse.
node_sockstat_TCP_mem gauge instance, ins, job, ip, cls Number of TCP sockets in state mem.
node_sockstat_TCP_mem_bytes gauge instance, ins, job, ip, cls Number of TCP sockets in state mem_bytes.
node_sockstat_TCP_orphan gauge instance, ins, job, ip, cls Number of TCP sockets in state orphan.
node_sockstat_TCP_tw gauge instance, ins, job, ip, cls Number of TCP sockets in state tw.
node_sockstat_UDP6_inuse gauge instance, ins, job, ip, cls Number of UDP6 sockets in state inuse.
node_sockstat_UDPLITE6_inuse gauge instance, ins, job, ip, cls Number of UDPLITE6 sockets in state inuse.
node_sockstat_UDPLITE_inuse gauge instance, ins, job, ip, cls Number of UDPLITE sockets in state inuse.
node_sockstat_UDP_inuse gauge instance, ins, job, ip, cls Number of UDP sockets in state inuse.
node_sockstat_UDP_mem gauge instance, ins, job, ip, cls Number of UDP sockets in state mem.
node_sockstat_UDP_mem_bytes gauge instance, ins, job, ip, cls Number of UDP sockets in state mem_bytes.
node_sockstat_sockets_used gauge instance, ins, job, ip, cls Number of IPv4 sockets in use.
node_tcp_connection_states gauge state, instance, ins, job, ip, cls Number of connection states.
node_textfile_scrape_error gauge instance, ins, job, ip, cls 1 if there was an error opening or reading a file, 0 otherwise
node_time_clocksource_available_info gauge ip, device, ins, clocksource, job, instance, cls Available clocksources read from ‘/sys/devices/system/clocksource’.
node_time_clocksource_current_info gauge ip, device, ins, clocksource, job, instance, cls Current clocksource read from ‘/sys/devices/system/clocksource’.
node_time_seconds gauge instance, ins, job, ip, cls System time in seconds since epoch (1970).
node_time_zone_offset_seconds gauge instance, ins, job, time_zone, ip, cls System time zone offset in seconds.
node_timex_estimated_error_seconds gauge instance, ins, job, ip, cls Estimated error in seconds.
node_timex_frequency_adjustment_ratio gauge instance, ins, job, ip, cls Local clock frequency adjustment.
node_timex_loop_time_constant gauge instance, ins, job, ip, cls Phase-locked loop time constant.
node_timex_maxerror_seconds gauge instance, ins, job, ip, cls Maximum error in seconds.
node_timex_offset_seconds gauge instance, ins, job, ip, cls Time offset in between local system and reference clock.
node_timex_pps_calibration_total counter instance, ins, job, ip, cls Pulse per second count of calibration intervals.
node_timex_pps_error_total counter instance, ins, job, ip, cls Pulse per second count of calibration errors.
node_timex_pps_frequency_hertz gauge instance, ins, job, ip, cls Pulse per second frequency.
node_timex_pps_jitter_seconds gauge instance, ins, job, ip, cls Pulse per second jitter.
node_timex_pps_jitter_total counter instance, ins, job, ip, cls Pulse per second count of jitter limit exceeded events.
node_timex_pps_shift_seconds gauge instance, ins, job, ip, cls Pulse per second interval duration.
node_timex_pps_stability_exceeded_total counter instance, ins, job, ip, cls Pulse per second count of stability limit exceeded events.
node_timex_pps_stability_hertz gauge instance, ins, job, ip, cls Pulse per second stability, average of recent frequency changes.
node_timex_status gauge instance, ins, job, ip, cls Value of the status array bits.
node_timex_sync_status gauge instance, ins, job, ip, cls Is clock synchronized to a reliable server (1 = yes, 0 = no).
node_timex_tai_offset_seconds gauge instance, ins, job, ip, cls International Atomic Time (TAI) offset.
node_timex_tick_seconds gauge instance, ins, job, ip, cls Seconds between clock ticks.
node_udp_queues gauge ip, queue, ins, job, exported_ip, instance, cls Number of allocated memory in the kernel for UDP datagrams in bytes.
node_uname_info gauge ip, sysname, version, domainname, release, ins, job, nodename, instance, cls, machine Labeled system information as provided by the uname system call.
node_up Unknown instance, ins, job, ip, cls N/A
node_vmstat_oom_kill unknown instance, ins, job, ip, cls /proc/vmstat information field oom_kill.
node_vmstat_pgfault unknown instance, ins, job, ip, cls /proc/vmstat information field pgfault.
node_vmstat_pgmajfault unknown instance, ins, job, ip, cls /proc/vmstat information field pgmajfault.
node_vmstat_pgpgin unknown instance, ins, job, ip, cls /proc/vmstat information field pgpgin.
node_vmstat_pgpgout unknown instance, ins, job, ip, cls /proc/vmstat information field pgpgout.
node_vmstat_pswpin unknown instance, ins, job, ip, cls /proc/vmstat information field pswpin.
node_vmstat_pswpout unknown instance, ins, job, ip, cls /proc/vmstat information field pswpout.
process_cpu_seconds_total counter instance, ins, job, ip, cls Total user and system CPU time spent in seconds.
process_max_fds gauge instance, ins, job, ip, cls Maximum number of open file descriptors.
process_open_fds gauge instance, ins, job, ip, cls Number of open file descriptors.
process_resident_memory_bytes gauge instance, ins, job, ip, cls Resident memory size in bytes.
process_start_time_seconds gauge instance, ins, job, ip, cls Start time of the process since unix epoch in seconds.
process_virtual_memory_bytes gauge instance, ins, job, ip, cls Virtual memory size in bytes.
process_virtual_memory_max_bytes gauge instance, ins, job, ip, cls Maximum amount of virtual memory available in bytes.
prometheus_remote_storage_exemplars_in_total counter instance, ins, job, ip, cls Exemplars in to remote storage, compare to exemplars out for queue managers.
prometheus_remote_storage_histograms_in_total counter instance, ins, job, ip, cls HistogramSamples in to remote storage, compare to histograms out for queue managers.
prometheus_remote_storage_samples_in_total counter instance, ins, job, ip, cls Samples in to remote storage, compare to samples out for queue managers.
prometheus_remote_storage_string_interner_zero_reference_releases_total counter instance, ins, job, ip, cls The number of times release has been called for strings that are not interned.
prometheus_sd_azure_failures_total counter instance, ins, job, ip, cls Number of Azure service discovery refresh failures.
prometheus_sd_consul_rpc_duration_seconds summary ip, call, quantile, ins, job, instance, cls, endpoint The duration of a Consul RPC call in seconds.
prometheus_sd_consul_rpc_duration_seconds_count Unknown ip, call, ins, job, instance, cls, endpoint N/A
prometheus_sd_consul_rpc_duration_seconds_sum Unknown ip, call, ins, job, instance, cls, endpoint N/A
prometheus_sd_consul_rpc_failures_total counter instance, ins, job, ip, cls The number of Consul RPC call failures.
prometheus_sd_consulagent_rpc_duration_seconds summary ip, call, quantile, ins, job, instance, cls, endpoint The duration of a Consul Agent RPC call in seconds.
prometheus_sd_consulagent_rpc_duration_seconds_count Unknown ip, call, ins, job, instance, cls, endpoint N/A
prometheus_sd_consulagent_rpc_duration_seconds_sum Unknown ip, call, ins, job, instance, cls, endpoint N/A
prometheus_sd_consulagent_rpc_failures_total Unknown instance, ins, job, ip, cls N/A
prometheus_sd_dns_lookup_failures_total counter instance, ins, job, ip, cls The number of DNS-SD lookup failures.
prometheus_sd_dns_lookups_total counter instance, ins, job, ip, cls The number of DNS-SD lookups.
prometheus_sd_file_read_errors_total counter instance, ins, job, ip, cls The number of File-SD read errors.
prometheus_sd_file_scan_duration_seconds summary quantile, instance, ins, job, ip, cls The duration of the File-SD scan in seconds.
prometheus_sd_file_scan_duration_seconds_count Unknown instance, ins, job, ip, cls N/A
prometheus_sd_file_scan_duration_seconds_sum Unknown instance, ins, job, ip, cls N/A
prometheus_sd_file_watcher_errors_total counter instance, ins, job, ip, cls The number of File-SD errors caused by filesystem watch failures.
prometheus_sd_kubernetes_events_total counter ip, event, ins, job, role, instance, cls The number of Kubernetes events handled.
prometheus_target_scrape_pool_exceeded_label_limits_total counter instance, ins, job, ip, cls Total number of times scrape pools hit the label limits, during sync or config reload.
prometheus_target_scrape_pool_exceeded_target_limit_total counter instance, ins, job, ip, cls Total number of times scrape pools hit the target limit, during sync or config reload.
prometheus_target_scrape_pool_reloads_failed_total counter instance, ins, job, ip, cls Total number of failed scrape pool reloads.
prometheus_target_scrape_pool_reloads_total counter instance, ins, job, ip, cls Total number of scrape pool reloads.
prometheus_target_scrape_pools_failed_total counter instance, ins, job, ip, cls Total number of scrape pool creations that failed.
prometheus_target_scrape_pools_total counter instance, ins, job, ip, cls Total number of scrape pool creation attempts.
prometheus_target_scrapes_cache_flush_forced_total counter instance, ins, job, ip, cls How many times a scrape cache was flushed due to getting big while scrapes are failing.
prometheus_target_scrapes_exceeded_body_size_limit_total counter instance, ins, job, ip, cls Total number of scrapes that hit the body size limit
prometheus_target_scrapes_exceeded_sample_limit_total counter instance, ins, job, ip, cls Total number of scrapes that hit the sample limit and were rejected.
prometheus_target_scrapes_exemplar_out_of_order_total counter instance, ins, job, ip, cls Total number of exemplar rejected due to not being out of the expected order.
prometheus_target_scrapes_sample_duplicate_timestamp_total counter instance, ins, job, ip, cls Total number of samples rejected due to duplicate timestamps but different values.
prometheus_target_scrapes_sample_out_of_bounds_total counter instance, ins, job, ip, cls Total number of samples rejected due to timestamp falling outside of the time bounds.
prometheus_target_scrapes_sample_out_of_order_total counter instance, ins, job, ip, cls Total number of samples rejected due to not being out of the expected order.
prometheus_template_text_expansion_failures_total counter instance, ins, job, ip, cls The total number of template text expansion failures.
prometheus_template_text_expansions_total counter instance, ins, job, ip, cls The total number of template text expansions.
prometheus_treecache_watcher_goroutines gauge instance, ins, job, ip, cls The current number of watcher goroutines.
prometheus_treecache_zookeeper_failures_total counter instance, ins, job, ip, cls The total number of ZooKeeper failures.
promhttp_metric_handler_errors_total counter ip, cause, ins, job, instance, cls Total number of internal errors encountered by the promhttp metric handler.
promhttp_metric_handler_requests_in_flight gauge instance, ins, job, ip, cls Current number of scrapes being served.
promhttp_metric_handler_requests_total counter ip, ins, code, job, instance, cls Total number of scrapes by HTTP status code.
promtail_batch_retries_total Unknown host, ip, ins, job, instance, cls N/A
promtail_build_info gauge ip, version, revision, goversion, branch, ins, goarch, job, tags, instance, cls, goos A metric with a constant ‘1’ value labeled by version, revision, branch, goversion from which promtail was built, and the goos and goarch for the build.
promtail_config_reload_fail_total Unknown instance, ins, job, ip, cls N/A
promtail_config_reload_success_total Unknown instance, ins, job, ip, cls N/A
promtail_dropped_bytes_total Unknown host, ip, ins, job, reason, instance, cls N/A
promtail_dropped_entries_total Unknown host, ip, ins, job, reason, instance, cls N/A
promtail_encoded_bytes_total Unknown host, ip, ins, job, instance, cls N/A
promtail_file_bytes_total gauge path, instance, ins, job, ip, cls Number of bytes total.
promtail_files_active_total gauge instance, ins, job, ip, cls Number of active files.
promtail_mutated_bytes_total Unknown host, ip, ins, job, reason, instance, cls N/A
promtail_mutated_entries_total Unknown host, ip, ins, job, reason, instance, cls N/A
promtail_read_bytes_total gauge path, instance, ins, job, ip, cls Number of bytes read.
promtail_read_lines_total Unknown path, instance, ins, job, ip, cls N/A
promtail_request_duration_seconds_bucket Unknown host, ip, ins, job, status_code, le, instance, cls N/A
promtail_request_duration_seconds_count Unknown host, ip, ins, job, status_code, instance, cls N/A
promtail_request_duration_seconds_sum Unknown host, ip, ins, job, status_code, instance, cls N/A
promtail_sent_bytes_total Unknown host, ip, ins, job, instance, cls N/A
promtail_sent_entries_total Unknown host, ip, ins, job, instance, cls N/A
promtail_targets_active_total gauge instance, ins, job, ip, cls Number of active total.
promtail_up Unknown instance, ins, job, ip, cls N/A
request_duration_seconds_bucket Unknown instance, ins, job, status_code, route, ws, le, ip, cls, method N/A
request_duration_seconds_count Unknown instance, ins, job, status_code, route, ws, ip, cls, method N/A
request_duration_seconds_sum Unknown instance, ins, job, status_code, route, ws, ip, cls, method N/A
request_message_bytes_bucket Unknown instance, ins, job, route, le, ip, cls, method N/A
request_message_bytes_count Unknown instance, ins, job, route, ip, cls, method N/A
request_message_bytes_sum Unknown instance, ins, job, route, ip, cls, method N/A
response_message_bytes_bucket Unknown instance, ins, job, route, le, ip, cls, method N/A
response_message_bytes_count Unknown instance, ins, job, route, ip, cls, method N/A
response_message_bytes_sum Unknown instance, ins, job, route, ip, cls, method N/A
scrape_duration_seconds Unknown instance, ins, job, ip, cls N/A
scrape_samples_post_metric_relabeling Unknown instance, ins, job, ip, cls N/A
scrape_samples_scraped Unknown instance, ins, job, ip, cls N/A
scrape_series_added Unknown instance, ins, job, ip, cls N/A
tcp_connections gauge instance, ins, job, protocol, ip, cls Current number of accepted TCP connections.
tcp_connections_limit gauge instance, ins, job, protocol, ip, cls The max number of TCP connections that can be accepted (0 means no limit).
up Unknown instance, ins, job, ip, cls N/A

10.7 - FAQ

Frequently asked questions about Pigsty NODE module

How to configure NTP service?

NTP is critical for various production services. If NTP is not configured, you can use public NTP services or the Chronyd on the admin node as the time standard.

If your nodes already have NTP configured, you can preserve the existing configuration without making any changes by setting node_ntp_enabled to false.

Otherwise, if you have Internet access, you can use public NTP services such as pool.ntp.org.

If you don’t have Internet access, you can use the following approach to ensure all nodes in the environment are synchronized with the admin node, or use another internal NTP time service.

node_ntp_servers:                 # NTP servers in /etc/chrony.conf
  - pool cn.pool.ntp.org iburst
  - pool ${admin_ip} iburst       # assume non-admin nodes do not have internet access, at least sync with admin node

How to force sync time on nodes?

Use chronyc to sync time. You must configure the NTP service first.

ansible all -b -a 'chronyc -a makestep'     # sync time

You can replace all with any group or host IP address to limit the execution scope.

Remote nodes are not accessible via SSH?

If the target machine is hidden behind an SSH jump host, or some customizations prevent direct access using ssh ip, you can use Ansible connection parameters to specify various SSH connection options, such as:

pg-test:
  vars: { pg_cluster: pg-test }
  hosts:
    10.10.10.11: {pg_seq: 1, pg_role: primary, ansible_host: node-1 }
    10.10.10.12: {pg_seq: 2, pg_role: replica, ansible_port: 22223, ansible_user: admin }
    10.10.10.13: {pg_seq: 3, pg_role: offline, ansible_port: 22224 }

Password required for remote node SSH and SUDO?

When performing deployments and changes, the admin user used must have ssh and sudo privileges for all nodes. Passwordless login is not required.

You can pass ssh and sudo passwords via the -k|-K parameters when executing playbooks, or even use another user to run playbooks via -eansible_host=<another_user>.

However, Pigsty strongly recommends configuring SSH passwordless login with passwordless sudo for the admin user.

How to create a dedicated admin user with an existing admin user?

Use the following command to create a new standard admin user defined by node_admin_username using an existing admin user on that node.

./node.yml -k -K -e ansible_user=<another_admin> -t node_admin

How to expose services using HAProxy on nodes?

You can use haproxy_services in the configuration to expose services, and use node.yml -t haproxy_config,haproxy_reload to update the configuration.

Here’s an example of exposing Silo: Silo Service Access

Why are all my /etc/yum.repos.d/* files gone?

Pigsty builds a local software repository on infra nodes that includes all dependencies. All regular nodes will reference and use the local software repository on Infra nodes according to the default configuration of node_repo_modules as local.

This design avoids Internet access and enhances installation stability and reliability. All original repo definition files are moved to the /etc/yum.repos.d/backup directory; you can copy them back as needed.

If you want to preserve the original repo definition files during regular node installation, set node_repo_remove to false.

If you want to preserve the original repo definition files during Infra node local repo construction, set repo_remove to false.

Why did my command line prompt change? How to restore it?

The shell command line prompt used by Pigsty is specified by the environment variable PS1, defined in the /etc/profile.d/node.sh file.

If you don’t like it and want to modify or restore it, you can remove this file and log in again.

Why did my hostname change?

Pigsty will modify your node hostname in two situations:

  • nodename value is explicitly defined (default is empty)
  • The PGSQL module is declared on the node and the node_id_from_pg parameter is enabled (default is true)

If you don’t want the hostname to be modified, you can set nodename_overwrite to false at the global/cluster/instance level (default is true).

For details, see the NODE_ID section.

What compatibility issues exist with Tencent OpenCloudOS?

The softdog kernel module is not available on OpenCloudOS and needs to be removed from node_kernel_modules. Add the following configuration item to the global variables in the config file to override:

node_kernel_modules: [ ip_vs, ip_vs_rr, ip_vs_wrr, ip_vs_sh ]

What common issues exist on Debian systems?

When using Pigsty on Debian/Ubuntu systems, you may encounter the following issues:

Missing locale

If the system reports locale-related errors, you can fix them with the following command:

localedef -i en_US -f UTF-8 en_US.UTF-8

Missing rsync tool

Pigsty relies on rsync for file synchronization. If the system doesn’t have it installed, you can install it with:

apt-get install rsync

11 - Module: ETCD

Pigsty deploys etcd as DCS for reliable distributed config storage, supporting PostgreSQL HA.

ETCD is a distributed, reliable key-value store for critical system config data.

Pigsty uses etcd as DCS (Distributed Config Store), critical for PostgreSQL HA and automatic failover.

The ETCD module depends on NODE module and is required by PGSQL module. Install NODE module to manage nodes before installing ETCD.

Deploy ETCD cluster before any PGSQL cluster—patroni and vip-manager for PG HA rely on etcd for HA and L2 VIP binding to primary.

flowchart LR
    subgraph PGSQL [PGSQL]
        patroni[Patroni]
        vip[VIP Manager]
    end

    subgraph ETCD [ETCD]
        etcd[DCS Service]
    end

    subgraph NODE [NODE]
        node[Software Repo]
    end

    PGSQL -->|depends| ETCD -->|depends| NODE

    style PGSQL fill:#3E668F,stroke:#2d4a66,color:#fff
    style ETCD fill:#5B9CD5,stroke:#4178a8,color:#fff
    style NODE fill:#FCDB72,stroke:#d4b85e,color:#333

    style patroni fill:#2d4a66,stroke:#1e3347,color:#fff
    style vip fill:#2d4a66,stroke:#1e3347,color:#fff
    style etcd fill:#4178a8,stroke:#2d5a7a,color:#fff
    style node fill:#d4b85e,stroke:#b89a4a,color:#333

One etcd cluster per Pigsty deployment serves multiple PG clusters.

Pigsty enables RBAC by default. Each PG cluster uses independent credentials for multi-tenant isolation. Admins use etcd root user with full permissions over all PG clusters.

11.1 - Configuration

Choose etcd cluster size based on requirements, provide reliable access.

Before deployment, define etcd cluster in config inventory. Typical choices:

  • One Node: No HA, suitable for dev, test, demo, or standalone deployments using external S3 backup for PITR
  • Three Nodes: Basic HA, tolerates 1 node failure, suitable for small-medium prod
  • Five Nodes: Better HA, tolerates 2 node failures, suitable for large prod

An even-member Etcd cluster is technically valid, but it does not tolerate more failures than an odd-member cluster with one fewer member, while increasing deployment and quorum cost. Production therefore usually uses one, three, or five members; clusters larger than five are uncommon.

Cluster Size Quorum Fault Tolerance Use Case
1 node 1 0 Dev, test, demo
3 nodes 2 1 Small-medium prod
5 nodes 3 2 Large prod
7 nodes 4 3 Special HA requirements

One Node

Define singleton etcd instance in Pigsty—single line of config:

etcd: { hosts: { 10.10.10.10: { etcd_seq: 1 } }, vars: { etcd_cluster: etcd } }

All single-node config templates include this line. Placeholder IP 10.10.10.10 replaced with current admin node’s IP.

Only required params: etcd_seq and etcd_cluster—uniquely identify each etcd instance.

Three Nodes

Most common config: 3-node etcd cluster tolerates 1 node failure, suitable for small-medium prod.

Example: Pigsty’s 3-node templates trio and safe use 3-node etcd:

etcd:
  hosts:
    10.10.10.10: { etcd_seq: 1 }  # etcd_seq (instance number) required
    10.10.10.11: { etcd_seq: 2 }  # positive integers, sequential from 0 or 1
    10.10.10.12: { etcd_seq: 3 }  # immutable for life, never recycled
  vars: # cluster-level params
    etcd_cluster: etcd    # default cluster name: 'etcd', don't change unless deploying multiple etcd clusters
    etcd_safeguard: false # enable safeguard? Enable after prod init to prevent accidental deletion
    etcd_clean: true      # force remove existing during init? Enable for testing for true idempotency

Five Nodes

5-node cluster tolerates 2 node failures, suitable for large prod.

Example: Pigsty’s production simulation template ha/simu uses a 5-node etcd cluster:

etcd:
  hosts:
    10.10.10.21 : { etcd_seq: 1 }
    10.10.10.22 : { etcd_seq: 2 }
    10.10.10.23 : { etcd_seq: 3 }
    10.10.10.24 : { etcd_seq: 4 }
    10.10.10.25 : { etcd_seq: 5 }
  vars: { etcd_cluster: etcd    }

Services Using etcd

Services using etcd in Pigsty:

Service Purpose Config File
Patroni PG HA, stores cluster state and config /etc/patroni/patroni.yml
VIP-Manager Binds L2 VIP on PG clusters /etc/default/vip-manager.yml

When etcd cluster membership changes permanently, reload related service configs to ensure correct access.

Update Patroni’s etcd endpoint ref:

./pgsql.yml -t pg_conf                            # regenerate patroni config
ansible all -f 1 -b -a 'systemctl reload patroni' # reload patroni config

Update VIP-Manager’s etcd endpoint ref (only for PGSQL L2 VIP):

./pgsql.yml -t pg_vip_config                           # regenerate vip-manager config
ansible all -f 1 -b -a 'systemctl restart vip-manager' # restart vip-manager

RBAC Authentication Config

Since v4.0, Pigsty has enabled etcd RBAC auth by default. Related params:

Parameter Description Default
etcd_root_password etcd root password Etcd.Root
pg_etcd_password Patroni’s password for etcd Empty (uses cluster name)

Prod recommendations:

all:
  vars:
    etcd_root_password: 'YourSecureEtcdPassword'  # change default

etcd:
  hosts:
    10.10.10.10: { etcd_seq: 1 }
    10.10.10.11: { etcd_seq: 2 }
    10.10.10.12: { etcd_seq: 3 }
  vars:
    etcd_cluster: etcd
    etcd_safeguard: true    # enable safeguard for production

Filesystem Layout

Module creates these directories/files on target hosts:

Path Purpose Permissions
/etc/etcd/ Config dir 0750, etcd:etcd
/etc/etcd/etcd.conf Main config file 0644, etcd:etcd
/etc/etcd/etcd.pass Root password file 0640, root:etcd
/etc/etcd/ca.crt CA cert 0644, etcd:etcd
/etc/etcd/server.crt Server cert 0644, etcd:etcd
/etc/etcd/server.key Server private key 0600, etcd:etcd
/var/lib/etcd/ Backup data dir 0770, etcd:etcd
/data/etcd/ Main data dir (configurable) 0700, etcd:etcd
/etc/profile.d/etcdctl.sh Client env vars 0644, root:root
/etc/systemd/system/etcd.service Systemd service 0644, root:root


11.2 - Parameters

ETCD module provides 13 configuration parameters for fine-grained control over cluster behavior.

The ETCD module has 13 parameters, divided into two sections:

  • ETCD: 10 parameters for etcd cluster deployment and configuration
  • ETCD_REMOVE: 3 parameters for controlling etcd cluster removal
Architecture Change: Pigsty v3.6+

Since Pigsty v3.6, the etcd.yml playbook no longer includes removal functionality—removal parameters have been migrated to a standalone etcd_remove role. Starting from v4.0, RBAC authentication is enabled by default, with a new etcd_root_password parameter.

Parameter Overview

The ETCD parameter group is used for etcd cluster deployment and configuration, including instance identification, cluster name, data directory, ports, and authentication password.

Parameter Type Level Description
etcd_seq int I etcd instance identifier, REQUIRED
etcd_cluster string C etcd cluster name, fixed to etcd by default
etcd_learner bool I/A initialize etcd instance as learner?
etcd_data path C etcd data directory, /data/etcd by default
etcd_port port C etcd client port, 2379 by default
etcd_peer_port port C etcd peer port, 2380 by default
etcd_init enum C etcd initial cluster state, new or existing
etcd_election_timeout int C etcd election timeout, 1000ms by default
etcd_heartbeat_interval int C etcd heartbeat interval, 100ms by default
etcd_root_password password G etcd root user password for RBAC authentication

The ETCD_REMOVE parameter group controls etcd cluster removal behavior, including safeguard protection, data cleanup, and package uninstallation.

Parameter Type Level Description
etcd_safeguard bool G/C/A unconditionally refuse removal when true
etcd_rm_data bool G/C/A remove etcd data during removal? default is true
etcd_rm_pkg bool G/C/A uninstall etcd packages during removal? default is false

ETCD

This section contains parameters for the etcd role, which are used by the etcd.yml playbook.

Parameters are defined in roles/etcd/defaults/main.yml

#etcd_seq: 1                      # etcd instance identifier, explicitly required
etcd_cluster: etcd                # etcd cluster & group name, etcd by default
etcd_learner: false               # run etcd instance as learner? default is false
etcd_data: /data/etcd             # etcd data directory, /data/etcd by default
etcd_port: 2379                   # etcd client port, 2379 by default
etcd_peer_port: 2380              # etcd peer port, 2380 by default
etcd_init: new                    # etcd initial cluster state, new or existing
etcd_election_timeout: 1000       # etcd election timeout, 1000ms by default
etcd_heartbeat_interval: 100      # etcd heartbeat interval, 100ms by default
etcd_root_password: Etcd.Root     # etcd root user password for RBAC authentication (please change!)

etcd_seq

Parameter: etcd_seq, Type: int, Level: I

etcd instance identifier. This is a required parameter—you must assign a unique identifier to each etcd instance.

Here is an example of a 3-node etcd cluster with identifiers 1 through 3:

etcd: # dcs service for postgres/patroni ha consensus
  hosts:  # 1 node for testing, 3 or 5 for production
    10.10.10.10: { etcd_seq: 1 }  # etcd_seq required
    10.10.10.11: { etcd_seq: 2 }  # assign from 1 ~ n
    10.10.10.12: { etcd_seq: 3 }  # use odd numbers
  vars: # cluster level parameter override roles/etcd
    etcd_cluster: etcd  # mark etcd cluster name etcd
    etcd_safeguard: false # safeguard against purging

etcd_cluster

Parameter: etcd_cluster, Type: string, Level: C

etcd cluster & group name, default value is the hard-coded etcd.

You can modify this parameter when you want to deploy an additional etcd cluster for backup purposes.

etcd_learner

Parameter: etcd_learner, Type: bool, Level: I/A

Initialize etcd instance as learner? Default value is false.

When set to true, the etcd instance will be initialized as a learner, meaning it cannot participate in voting elections within the etcd cluster.

Use Cases:

  • Cluster Expansion: When adding new members to an existing cluster, using learner mode prevents affecting cluster quorum before data synchronization completes
  • Safe Migration: In rolling upgrade or migration scenarios, join as a learner first, then promote after confirming data synchronization

Workflow:

  1. Set etcd_learner: true to initialize the new member as a learner
  2. Wait for data synchronization to complete (check with etcdctl endpoint status)
  3. Use etcdctl member promote <member_id> to promote it to a full member
Note

Learner instances do not count toward cluster quorum. For example, in a 3-node cluster with 1 learner, the actual voting members are 2, which cannot tolerate any node failure.

etcd_data

Parameter: etcd_data, Type: path, Level: C

etcd data directory, default is /data/etcd.

etcd_port

Parameter: etcd_port, Type: port, Level: C

etcd client port, default is 2379.

etcd_peer_port

Parameter: etcd_peer_port, Type: port, Level: C

etcd peer port, default is 2380.

etcd_init

Parameter: etcd_init, Type: enum, Level: C

etcd initial cluster state, can be new or existing, default value: new.

Option Values:

Value Description Use Case
new Create a new etcd cluster Initial deployment, cluster rebuild
existing Join an existing etcd cluster Cluster expansion, adding new members

Important Notes:

Must use existing when expanding

When adding new members to an existing etcd cluster, you must set etcd_init=existing. Otherwise, the new instance will attempt to create an independent new cluster, causing split-brain or initialization failure.

Usage Examples:

# Create new cluster (default behavior)
./etcd.yml

# Add new member to existing cluster
./etcd.yml -l <new_ip> -e etcd_init=existing

# Or use the convenience script (automatically sets etcd_init=existing)
bin/etcd-add <new_ip>

etcd_election_timeout

Parameter: etcd_election_timeout, Type: int, Level: C

etcd election timeout, default is 1000 (milliseconds), i.e., 1 second.

etcd_heartbeat_interval

Parameter: etcd_heartbeat_interval, Type: int, Level: C

etcd heartbeat interval, default is 100 (milliseconds).

etcd_root_password

Parameter: etcd_root_password, Type: password, Level: G

etcd root user password for RBAC authentication, default value is Etcd.Root.

Since v4.0, Pigsty has enabled etcd RBAC (Role-Based Access Control) authentication by default. During cluster initialization, the etcd_auth task automatically creates the root user and enables authentication.

Password Storage Location:

  • Password is stored in /etc/etcd/etcd.pass file
  • File permissions are 0640 (owned by root, readable by etcd group)
  • The etcdctl environment script /etc/profile.d/etcdctl.sh automatically reads this file

Integration with Other Components:

  • Patroni uses the pg_etcd_password parameter to configure the password for connecting to etcd
  • If pg_etcd_password is empty, Patroni will use the cluster name as password (not recommended)
  • VIP-Manager also requires the same authentication credentials to connect to etcd

Security Recommendations:

Production Security

In production environments, it is strongly recommended to change the default password Etcd.Root. Set it in global or cluster configuration:

etcd_root_password: 'YourSecurePassword'

Using configure -g will automatically generate and replace etcd_root_password

ETCD_REMOVE

This section contains parameters for the etcd_remove role, which are action flags used by the etcd-rm.yml playbook.

Parameters are defined in roles/etcd_remove/defaults/main.yml

etcd_safeguard: false             # unconditionally refuse removal when true
etcd_rm_data: true                # remove etcd data and config files during removal?
etcd_rm_pkg: false                # uninstall etcd packages during removal?

etcd_safeguard

Parameter: etcd_safeguard, Type: bool, Level: G/C/A

Removal safeguard, default false. When set to true, etcd-rm.yml aborts before deregistration, leaving the cluster, stopping the service, or deleting data. It is a static boolean switch and does not probe whether the instance is running. Override it explicitly with -e etcd_safeguard=false.

Recommended Settings:

Environment Recommended Description
Dev/Test false Convenient for rapid rebuilding and testing
Production true Prevents service interruption from accidental operations

In emergencies, you can override the configuration with command-line parameters:

./etcd-rm.yml -l etcd -e etcd_safeguard=false # Override the guard and remove the target Etcd cluster

etcd_rm_data

Parameter: etcd_rm_data, Type: bool, Level: G/C/A

Remove etcd data and configuration files during removal? Default value is true.

When enabled, the etcd-rm.yml playbook will delete the following contents when removing a cluster or member:

  • /etc/etcd/ - Configuration directory (including certificates and password files)
  • /var/lib/etcd/ - Alternate data directory
  • {{ etcd_data }} - Primary data directory (default /data/etcd)
  • /etc/systemd/system/etcd.service - Systemd service unit file
  • /etc/profile.d/etcdctl.sh - Client environment script
  • /etc/vector/etcd.yaml - Vector log collection config

Use Cases:

Scenario Recommended Description
Complete removal true (default) Full cleanup, free disk space
Stop service only false Preserve data for troubleshooting or recovery 
# Stop service only, preserve data
./etcd-rm.yml -l etcd -e etcd_rm_data=false

etcd_rm_pkg

Parameter: etcd_rm_pkg, Type: bool, Level: G/C/A

Uninstall etcd packages during removal? Default value is false.

When enabled, the etcd-rm.yml playbook will uninstall etcd packages when removing a cluster or member.

Use Cases:

Scenario Recommended Description
Normal removal false (default) Keep packages for quick redeployment
Complete cleanup true Full uninstall, save disk space 
# Uninstall packages during removal
./etcd-rm.yml -l etcd -e etcd_rm_pkg=true
Tip

Usually there’s no need to uninstall etcd packages. Keeping the packages speeds up subsequent redeployments since no re-download or installation is required.

11.3 - Administration

etcd cluster management SOP: create, destroy, scale, config, and RBAC.

Common etcd admin SOPs:

For more, refer to FAQ: ETCD.

Create Cluster

Define etcd cluster in config inventory:

etcd:
  hosts:
    10.10.10.10: { etcd_seq: 1 }
    10.10.10.11: { etcd_seq: 2 }
    10.10.10.12: { etcd_seq: 3 }
  vars: { etcd_cluster: etcd }

Run etcd.yml playbook:

./etcd.yml  # initialize etcd cluster
Architecture Change: Pigsty v3.6+

Since v3.6, etcd.yml focuses on cluster install and member addition—no longer includes removal. Use dedicated etcd-rm.yml for all removals.

For prod etcd clusters, enable safeguard etcd_safeguard to prevent accidental deletion.

Destroy Cluster

Use the dedicated etcd-rm.yml playbook to destroy an Etcd cluster. The default etcd_rm_data: true deletes local data and configuration. First confirm that no PostgreSQL cluster still uses it as DCS, verify a recent backup, and check the exact target name.

./etcd-rm.yml -l etcd                            # Destroy the entire Etcd cluster after confirmation
./etcd-rm.yml -l etcd -e etcd_safeguard=false   # Override only when the inventory enables the safeguard

Or use utility script:

bin/etcd-rm                           # remove entire etcd cluster

The removal playbook respects etcd_safeguard. If true, it aborts before leaving the cluster, deregistering, stopping the service, or deleting files. Its default is false, so the absence of an explicit override is not itself a confirmation.

Warning

Before removing etcd cluster, ensure no PG clusters use it as DCS. PG HA will break otherwise.

CLI Environment

Uses etcd v3 API by default (v2 removed in v3.6+). Pigsty auto-configures env script /etc/profile.d/etcdctl.sh on etcd nodes, loaded on login.

Example client env config:

alias e="etcdctl"
alias em="etcdctl member"
export ETCDCTL_ENDPOINTS=https://10.10.10.10:2379
export ETCDCTL_CACERT=/etc/etcd/ca.crt
export ETCDCTL_CERT=/etc/etcd/server.crt
export ETCDCTL_KEY=/etc/etcd/server.key

Since v4.0, Pigsty has enabled RBAC auth for etcd by default, so user auth is still required:

export ETCDCTL_USER="root:$(cat /etc/etcd/etcd.pass)"

After configuring client env, run etcd CRUD ops:

e put a 10 ; e get a; e del a   # basic KV ops
e member list                    # list cluster members
e endpoint health                # check endpoint health
e endpoint status                # view endpoint status

RBAC Authentication

Since v4.0, Pigsty has enabled etcd RBAC auth by default. During cluster init, etcd_auth task auto-creates root user and enables auth.

Root user password set by etcd_root_password, default: Etcd.Root. Stored in /etc/etcd/etcd.pass with 0640 perms (root-owned, etcd-group readable).

Strongly recommended to change default password in prod:

etcd:
  hosts:
    10.10.10.10: { etcd_seq: 1 }
    10.10.10.11: { etcd_seq: 2 }
    10.10.10.12: { etcd_seq: 3 }
  vars:
    etcd_cluster: etcd
    etcd_root_password: 'YourSecurePassword'  # change default

Client auth methods:

# Method 1: env vars (recommended, auto-configured in /etc/profile.d/etcdctl.sh)
export ETCDCTL_USER="root:$(cat /etc/etcd/etcd.pass)"

# Method 2: command line
etcdctl --user root:YourSecurePassword member list

Patroni and etcd auth:

Patroni uses pg_etcd_password to configure etcd connection password. If empty, Patroni uses cluster name as password (not recommended). Configure separate etcd password per PG cluster in prod.

Reload Config

If etcd cluster membership changes (add/remove members), refresh etcd service endpoint references. These etcd refs in Pigsty need updates:

Config Location Config File Update Method
etcd member config /etc/etcd/etcd.conf ./etcd.yml -t etcd_conf
etcdctl env vars /etc/profile.d/etcdctl.sh ./etcd.yml -t etcd_config
Patroni DCS config /etc/patroni/patroni.yml ./pgsql.yml -t pg_conf
VIP-Manager config /etc/default/vip-manager.yml ./pgsql.yml -t pg_vip_config

Refresh etcd member config:

./etcd.yml -t etcd_conf                           # refresh /etc/etcd/etcd.conf
ansible etcd -f 1 -b -a 'systemctl restart etcd'  # optional: restart etcd instances

Refresh etcdctl client env:

./etcd.yml -t etcd_config                         # refresh /etc/profile.d/etcdctl.sh

Update Patroni DCS endpoint config:

./pgsql.yml -t pg_conf                            # regenerate patroni config
ansible all -f 1 -b -a 'systemctl reload patroni' # reload patroni config

Update VIP-Manager endpoint config (only for PGSQL L2 VIP):

./pgsql.yml -t pg_vip_config                           # regenerate vip-manager config
ansible all -f 1 -b -a 'systemctl restart vip-manager' # restart vip-manager
Tip

Using bin/etcd-add / bin/etcd-rm utility scripts? Scripts prompt config refresh commands after completion.

Add Member

ETCD Reference: Add a member

Recommended: Utility Script

Use bin/etcd-add script to add new members to existing etcd cluster:

# First add new member definition to config inventory, then:
bin/etcd-add <ip>              # add single new member
bin/etcd-add <ip1> <ip2> ...   # add multiple new members

The script attempts these operations in order:

  • Validates IP address validity
  • Executes etcd.yml playbook (auto-sets etcd_init=existing)
  • Provides safety warnings and countdown
  • Prompts config refresh commands after completion

Manual: Step-by-Step

Add new member to existing etcd cluster:

  1. Update config inventory: Add new instance to etcd group
  2. Notify cluster: Run etcdctl member add (optional, playbook auto-does this)
  3. Initialize new member: Run playbook with etcd_init=existing parameter
  4. Promote member: Promote learner to full member (optional, required when using etcd_learner=true)
  5. Reload config: Update etcd endpoint references for all clients
# After config inventory update, initialize new member
./etcd.yml -l <new_ins_ip> -e etcd_init=existing

# If using learner mode, manually promote
etcdctl member promote <new_ins_server_id>
Important

When adding new members, must use etcd_init=existing parameter. New instance will create new cluster instead of joining existing one otherwise.

Detailed: Add member to etcd cluster

Detailed steps. Start from single-instance etcd cluster:

etcd:
  hosts:
    10.10.10.10: { etcd_seq: 1 } # <--- only existing instance in cluster
    10.10.10.11: { etcd_seq: 2 } # <--- add this new member to inventory
  vars: { etcd_cluster: etcd }

Add new member using utility script (recommended):

$ bin/etcd-add 10.10.10.11

Or manual. First use etcdctl member add to announce new learner instance etcd-2 to existing etcd cluster:

$ etcdctl member add etcd-2 --learner=true --peer-urls=https://10.10.10.11:2380
Member 33631ba6ced84cf8 added to cluster 6646fbcf5debc68f

ETCD_NAME="etcd-2"
ETCD_INITIAL_CLUSTER="etcd-2=https://10.10.10.11:2380,etcd-1=https://10.10.10.10:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.10.10.11:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"

Check member list with etcdctl member list (or em list), see unstarted new member:

33631ba6ced84cf8, unstarted, , https://10.10.10.11:2380, , true       # unstarted new member here
429ee12c7fbab5c1, started, etcd-1, https://10.10.10.10:2380, https://10.10.10.10:2379, false

Next, use etcd.yml playbook to initialize new etcd instance etcd-2. After completion, new member has started:

$ ./etcd.yml -l 10.10.10.11 -e etcd_init=existing    # must add existing parameter
...
33631ba6ced84cf8, started, etcd-2, https://10.10.10.11:2380, https://10.10.10.11:2379, true
429ee12c7fbab5c1, started, etcd-1, https://10.10.10.10:2380, https://10.10.10.10:2379, false

After new member initialized and running stably, promote from learner to follower:

$ etcdctl member promote 33631ba6ced84cf8   # promote learner to follower
Member 33631ba6ced84cf8 promoted in cluster 6646fbcf5debc68f

$ em list                # check again, new member promoted to full member
33631ba6ced84cf8, started, etcd-2, https://10.10.10.11:2380, https://10.10.10.11:2379, false
429ee12c7fbab5c1, started, etcd-1, https://10.10.10.10:2380, https://10.10.10.10:2379, false

New member added. Don’t forget to reload config so all clients know new member.

Repeat steps to add more members. Prod environments need at least 3 members.

Remove Member

Recommended: Utility Script

Use bin/etcd-rm script to remove members from etcd cluster:

bin/etcd-rm <ip>              # remove specified member
bin/etcd-rm <ip1> <ip2> ...   # remove multiple members
bin/etcd-rm                   # remove entire etcd cluster

Script auto-performs:

  • Gracefully removes members from cluster
  • Stops and disables etcd service
  • Cleans up data and config files
  • Deregisters from monitoring system

The underlying removal role tolerates some leave and cleanup errors. After the script finishes, inspect etcdctl member list, endpoint health, remaining quorum, and the actual service and data-directory state on the target.

Manual: Step-by-Step

Remove member instance from etcd cluster:

  1. Keep the member in the inventory: The removal playbook needs its etcd_seq, cluster members, and connection endpoints
  2. Clean up the instance: Run etcd-rm.yml against the target; it attempts member remove, stops the service, and cleans up according to the removal flags
  3. Update the inventory: Comment out or delete the member only after the playbook succeeds
  4. Reload references: Follow Reload Config for the remaining etcd members and the Patroni/VIP-Manager endpoints
# <ip> must still be in the etcd inventory group
./etcd-rm.yml -l <ip>                  # Leave the cluster and clean up automatically
# After success, remove the member from pigsty.yml and refresh the remaining members and clients

Do not delete the target from the inventory before running the removal playbook. The hosts: etcd scope in etcd-rm.yml would no longer select it, and the playbook could not derive the instance identity or cluster endpoints from inventory. There is also no need to repeat etcdctl member remove before or after the playbook.

Detailed: Remove member from etcd cluster

Example: 3-node etcd cluster, remove instance 3.

Method 1: Utility script (recommended)

$ bin/etcd-rm 10.10.10.12

The script attempts to remove the member, stop the service, and clean up data. Afterwards, still inspect the member list, quorum, and target files as described above.

Method 2: Manual

First keep the member to be removed in the inventory, then run the removal playbook:

$ ./etcd-rm.yml -l 10.10.10.12

The playbook attempts these operations in order:

  1. Get member list, find corresponding member ID
  2. Execute etcdctl member remove to kick from cluster
  3. Stop etcd service
  4. Clean up data and config files

The playbook queries the member ID and runs member remove automatically. Do this manually only when troubleshooting:

$ etcdctl member list
429ee12c7fbab5c1, started, etcd-1, https://10.10.10.10:2380, https://10.10.10.10:2379, false
33631ba6ced84cf8, started, etcd-2, https://10.10.10.11:2380, https://10.10.10.11:2379, false
93fcf23b220473fb, started, etcd-3, https://10.10.10.12:2380, https://10.10.10.12:2379, false  # <--- remove this

$ etcdctl member remove 93fcf23b220473fb # kick from cluster
Member 93fcf23b220473fb removed from cluster 6646fbcf5debc68f

After a manual member removal, run ./etcd-rm.yml -l 10.10.10.12 while the target remains in inventory to stop, deregister, and clean it up. Its leave step skips a member that has already been removed.

Only after confirming that the member has left the live cluster, the remaining members retain quorum, and the target service and files match expectations should you delete 10.10.10.12 from the inventory. Then follow Reload Config to refresh the remaining Etcd members and all client references.

Repeat to remove more members. Combined with Add Member, perform rolling upgrades and migrations of etcd cluster.

Utility Scripts

v3.6+ provides utility scripts to simplify etcd cluster scaling:

bin/etcd-add

Add new members to existing etcd cluster:

bin/etcd-add <ip>              # add single new member
bin/etcd-add <ip1> <ip2> ...   # add multiple new members

Script features:

  • Validates IP addresses in config inventory
  • Auto-sets etcd_init=existing parameter
  • Executes etcd.yml playbook to complete member addition
  • Prompts config refresh commands after completion

bin/etcd-rm

Remove members or entire cluster from etcd:

bin/etcd-rm <ip>              # remove specified member
bin/etcd-rm <ip1> <ip2> ...   # remove multiple members
bin/etcd-rm                   # remove entire etcd cluster

Script features:

  • Provides safety warnings and confirmation countdown
  • Auto-executes etcd-rm.yml playbook
  • Gracefully removes members from cluster
  • Cleans up data and config files

11.4 - Playbook

Manage etcd clusters with Ansible playbooks and quick command reference.

The ETCD module provides two core playbooks: etcd.yml for installing and configuring etcd clusters, and etcd-rm.yml for removing etcd clusters or members.

Architecture Change: Pigsty v3.6+

Since Pigsty v3.6, the etcd.yml playbook focuses on cluster installation and member addition. All removal operations have been moved to the dedicated etcd-rm.yml playbook using the etcd_remove role.

etcd.yml

Playbook source: etcd.yml

This playbook installs and configures an etcd cluster on the hardcoded etcd group, then launches the etcd service.

The following subtasks are available in etcd.yml:

  • etcd_assert : Validate etcd identity parameters (etcd_seq must be defined as a non-negative integer)
  • etcd_install : Install etcd packages
  • etcd_dir : Create etcd data and configuration directories
  • etcd_config : Generate etcd configuration
    • etcd_conf : Generate etcd main config file /etc/etcd/etcd.conf
    • etcd_cert : Generate etcd TLS certificates (CA, server cert, private key)
  • etcd_member : Add new member to existing cluster (only runs when etcd_init=existing)
  • etcd_launch : Launch etcd service
  • etcd_auth : Enable RBAC authentication (create root user and enable auth)
  • etcd_register : Register etcd to VictoriaMetrics monitoring

etcd-rm.yml

Playbook source: etcd-rm.yml

A dedicated playbook for removing etcd clusters or individual members. The following subtasks are available in etcd-rm.yml:

  • etcd_safeguard : Check safeguard and abort if enabled
  • etcd_pause : Pause for 3 seconds, allowing user to abort with Ctrl-C
  • etcd_deregister : Remove etcd registration from VictoriaMetrics monitoring targets
  • etcd_leave : Try graceful leaving etcd cluster before purge
  • etcd_svc : Stop and disable etcd service with systemd
  • etcd_data : Remove etcd data (disable with etcd_rm_data=false)
  • etcd_pkg : Uninstall etcd packages (enable with etcd_rm_pkg=true)

The removal playbook uses the etcd_remove role with the following configurable parameters:

  • etcd_safeguard: Prevents accidental removal when set to true
  • etcd_rm_data: Controls whether ETCD data is deleted (default: true)
  • etcd_rm_pkg: Controls whether ETCD packages are uninstalled (default: false)
Dangerous Operation

etcd_safeguard defaults to false, while etcd_rm_data defaults to true. A full etcd-rm.yml run therefore attempts to remove the target from the cluster, deregister and stop it, then delete local Etcd data, configuration, unit files, and the client environment file. The playbook ignores some leave and cleanup errors and does not prove that the remaining members retain quorum. Always use an exact -l, and verify a recent backup, the member list, and remaining quorum.

Demo

asciicast

Cheatsheet

Etcd Installation & Configuration:

./etcd.yml                                      # Initialize etcd cluster
./etcd.yml -t etcd_launch                       # Restart entire etcd cluster
./etcd.yml -t etcd_conf                         # Refresh /etc/etcd/etcd.conf with latest state
./etcd.yml -t etcd_cert                         # Regenerate etcd TLS certificates
./etcd.yml -l 10.10.10.12 -e etcd_init=existing # Scale out: add new member to existing cluster

Etcd Removal & Cleanup:

./etcd-rm.yml -l 10.10.10.12                       # Leave, deregister, stop, and delete local data by default
./etcd-rm.yml -l 10.10.10.12 -e etcd_rm_data=false # Leave, deregister, and stop, preserving local data/config
./etcd-rm.yml -l 10.10.10.12 -e etcd_rm_pkg=true   # Also uninstall Etcd packages
./etcd-rm.yml -l etcd                              # Destroy the entire cluster and its local data

Convenience Scripts:

bin/etcd-add <ip>                               # Add new member to existing cluster (recommended)
bin/etcd-rm <ip>                                # Remove specific member from cluster (recommended)
bin/etcd-rm                                     # Remove entire etcd cluster

Safeguard

To prevent accidental deletion, Pigsty’s ETCD module provides a safeguard mechanism controlled by the etcd_safeguard parameter, which defaults to false (safeguard disabled).

For production etcd clusters that have been initialized, it’s recommended to enable the safeguard to prevent accidental deletion of existing etcd instances:

etcd:
  hosts:
    10.10.10.10: { etcd_seq: 1 }
    10.10.10.11: { etcd_seq: 2 }
    10.10.10.12: { etcd_seq: 3 }
  vars:
    etcd_cluster: etcd
    etcd_safeguard: true  # Enable safeguard protection

When etcd_safeguard is set to true, etcd-rm.yml aborts before any deregistration, cluster-leave, service-stop, or deletion action. It is a boolean guard and does not probe whether the instance is alive. Override it with a command-line parameter:

./etcd-rm.yml -l <exact-target> -e etcd_safeguard=false  # Override only after verifying target and backups

Regardless of the safeguard value, inspect etcdctl member list, endpoint health, and remaining quorum after a real run; a successful task status is not runtime acceptance.

11.5 - Monitoring

etcd monitoring dashboards, metrics, and alert rules.

Dashboards

ETCD module provides one monitoring dashboard: Etcd Overview.

ETCD Overview Dashboard

ETCD Overview: Overview of ETCD cluster

Dashboard provides key ETCD status info. Notable: ETCD Aliveness—shows overall etcd cluster service status.

Red bands = instance downtime; blue-gray below = cluster unavailable.

etcd-overview.jpg

Alert Rules

Pigsty provides five preset alert rules for etcd, defined in files/victoria/rules/etcd.yml:

  • EtcdServerDown: etcd node down, CRIT alert
  • EtcdNoLeader: etcd cluster no leader, CRIT alert
  • EtcdQuotaFull: etcd quota > 90%, WARN alert
  • EtcdNetworkPeerRTSlow: etcd network latency slow, INFO alert
  • EtcdWalFsyncSlow: etcd disk fsync slow, INFO alert

The following excerpt mirrors the current rule source. Pigsty currently ships only the etcd-overview dashboard; the /ui/d/etcd-instance targets in the comments for the two latency alerts do not exist. Use /ui/d/etcd-overview to inspect cluster state. This is a known source-comment discrepancy and does not affect the alert expressions.

#==============================================================#
#                         Aliveness                            #
#==============================================================#
# etcd server instance down
- alert: EtcdServerDown
  expr: etcd_up < 1
  for: 1m
  labels: { level: 0, severity: CRIT, category: etcd }
  annotations:
    summary: "CRIT EtcdServerDown {{ $labels.ins }}@{{ $labels.instance }}"
    description: |
      etcd_up[ins={{ $labels.ins }}, instance={{ $labels.instance }}] = {{ $value }} < 1
      https://demo.pigsty.io/d/etcd-overview

#==============================================================#
#                         Error                                #
#==============================================================#
# Etcd no Leader triggers P0 alert immediately
# if dcs_failsafe mode not enabled, may cause global outage
- alert: EtcdNoLeader
  expr: min(etcd_server_has_leader) by (cls) < 1
  for: 15s
  labels: { level: 0, severity: CRIT, category: etcd }
  annotations:
    summary: "CRIT EtcdNoLeader: {{ $labels.cls }} {{ $value }}"
    description: |
      etcd_server_has_leader[cls={{ $labels.cls }}] = {{ $value }} < 1
      https://demo.pigsty.io/d/etcd-overview?from=now-5m&to=now&var-cls={{$labels.cls}}

#==============================================================#
#                        Saturation                            #
#==============================================================#
- alert: EtcdQuotaFull
  expr: etcd:cls:quota_usage > 0.90
  for: 1m
  labels: { level: 1, severity: WARN, category: etcd }
  annotations:
    summary: "WARN EtcdQuotaFull: {{ $labels.cls }}"
    description: |
      etcd:cls:quota_usage[cls={{ $labels.cls }}] = {{ $value | printf "%.3f" }} > 90%
      https://demo.pigsty.io/d/etcd-overview

#==============================================================#
#                         Latency                              #
#==============================================================#
# etcd network peer rt p95 > 200ms for 1m
- alert: EtcdNetworkPeerRTSlow
  expr: etcd:ins:network_peer_rt_p95_5m > 0.200
  for: 1m
  labels: { level: 2, severity: INFO, category: etcd }
  annotations:
    summary: "INFO EtcdNetworkPeerRTSlow: {{ $labels.cls }} {{ $labels.ins }}"
    description: |
      etcd:ins:network_peer_rt_p95_5m[cls={{ $labels.cls }}, ins={{ $labels.ins }}] = {{ $value }} > 200ms
      https://demo.pigsty.io/d/etcd-instance?from=now-10m&to=now&var-cls={{ $labels.cls }}
# Etcd wal fsync rt p95 > 50ms
- alert: EtcdWalFsyncSlow
  expr: etcd:ins:wal_fsync_rt_p95_5m > 0.050
  for: 1m
  labels: { level: 2, severity: INFO, category: etcd }
  annotations:
    summary: "INFO EtcdWalFsyncSlow: {{ $labels.cls }} {{ $labels.ins }}"
    description: |
      etcd:ins:wal_fsync_rt_p95_5m[cls={{ $labels.cls }}, ins={{ $labels.ins }}] = {{ $value }} > 50ms
      https://demo.pigsty.io/d/etcd-instance?from=now-10m&to=now&var-cls={{ $labels.cls }}

11.6 - Metrics

Complete monitoring metrics list provided by Pigsty ETCD module

This snapshot records 177 monitoring metric families for the ETCD module. The metrics present at runtime vary with package version, enabled collectors, and target state.

Metric Name Type Labels Description
etcd:ins:backend_commit_rt_p99_5m Unknown cls, ins, instance, job, ip N/A
etcd:ins:disk_fsync_rt_p99_5m Unknown cls, ins, instance, job, ip N/A
etcd:ins:network_peer_rt_p99_1m Unknown cls, To, ins, instance, job, ip N/A
etcd_cluster_version gauge cls, cluster_version, ins, instance, job, ip Running version. 1 = ‘cluster_version’ label with current version
etcd_debugging_auth_revision gauge cls, ins, instance, job, ip Current auth store revision.
etcd_debugging_disk_backend_commit_rebalance_duration_seconds_bucket Unknown cls, ins, instance, job, le, ip N/A
etcd_debugging_disk_backend_commit_rebalance_duration_seconds_count Unknown cls, ins, instance, job, ip N/A
etcd_debugging_disk_backend_commit_rebalance_duration_seconds_sum Unknown cls, ins, instance, job, ip N/A
etcd_debugging_disk_backend_commit_spill_duration_seconds_bucket Unknown cls, ins, instance, job, le, ip N/A
etcd_debugging_disk_backend_commit_spill_duration_seconds_count Unknown cls, ins, instance, job, ip N/A
etcd_debugging_disk_backend_commit_spill_duration_seconds_sum Unknown cls, ins, instance, job, ip N/A
etcd_debugging_disk_backend_commit_write_duration_seconds_bucket Unknown cls, ins, instance, job, le, ip N/A
etcd_debugging_disk_backend_commit_write_duration_seconds_count Unknown cls, ins, instance, job, ip N/A
etcd_debugging_disk_backend_commit_write_duration_seconds_sum Unknown cls, ins, instance, job, ip N/A
etcd_debugging_lease_granted_total counter cls, ins, instance, job, ip Total granted leases.
etcd_debugging_lease_renewed_total counter cls, ins, instance, job, ip Renewed leases seen by leader.
etcd_debugging_lease_revoked_total counter cls, ins, instance, job, ip Revoked leases.
etcd_debugging_lease_ttl_total_bucket Unknown cls, ins, instance, job, le, ip N/A
etcd_debugging_lease_ttl_total_count Unknown cls, ins, instance, job, ip N/A
etcd_debugging_lease_ttl_total_sum Unknown cls, ins, instance, job, ip N/A
etcd_debugging_mvcc_compact_revision gauge cls, ins, instance, job, ip Last compaction revision in store.
etcd_debugging_mvcc_current_revision gauge cls, ins, instance, job, ip Current store revision.
etcd_debugging_mvcc_db_compaction_keys_total counter cls, ins, instance, job, ip DB keys compacted.
etcd_debugging_mvcc_db_compaction_last gauge cls, ins, instance, job, ip Last db compaction unix time. Resets to 0 on start.
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket Unknown cls, ins, instance, job, le, ip N/A
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_count Unknown cls, ins, instance, job, ip N/A
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_sum Unknown cls, ins, instance, job, ip N/A
etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_bucket Unknown cls, ins, instance, job, le, ip N/A
etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_count Unknown cls, ins, instance, job, ip N/A
etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_sum Unknown cls, ins, instance, job, ip N/A
etcd_debugging_mvcc_events_total counter cls, ins, instance, job, ip Events sent by this member.
etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds_bucket Unknown cls, ins, instance, job, le, ip N/A
etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds_count Unknown cls, ins, instance, job, ip N/A
etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds_sum Unknown cls, ins, instance, job, ip N/A
etcd_debugging_mvcc_keys_total gauge cls, ins, instance, job, ip Total keys.
etcd_debugging_mvcc_pending_events_total gauge cls, ins, instance, job, ip Pending events to send.
etcd_debugging_mvcc_range_total counter cls, ins, instance, job, ip Ranges seen by this member.
etcd_debugging_mvcc_slow_watcher_total gauge cls, ins, instance, job, ip Unsynced slow watchers.
etcd_debugging_mvcc_total_put_size_in_bytes gauge cls, ins, instance, job, ip Total put kv size seen by this member.
etcd_debugging_mvcc_watch_stream_total gauge cls, ins, instance, job, ip Watch streams.
etcd_debugging_mvcc_watcher_total gauge cls, ins, instance, job, ip Watchers.
etcd_debugging_server_lease_expired_total counter cls, ins, instance, job, ip Expired leases.
etcd_debugging_snap_save_marshalling_duration_seconds_bucket Unknown cls, ins, instance, job, le, ip N/A
etcd_debugging_snap_save_marshalling_duration_seconds_count Unknown cls, ins, instance, job, ip N/A
etcd_debugging_snap_save_marshalling_duration_seconds_sum Unknown cls, ins, instance, job, ip N/A
etcd_debugging_snap_save_total_duration_seconds_bucket Unknown cls, ins, instance, job, le, ip N/A
etcd_debugging_snap_save_total_duration_seconds_count Unknown cls, ins, instance, job, ip N/A
etcd_debugging_snap_save_total_duration_seconds_sum Unknown cls, ins, instance, job, ip N/A
etcd_debugging_store_expires_total counter cls, ins, instance, job, ip Expired keys.
etcd_debugging_store_reads_total counter cls, action, ins, instance, job, ip Reads (get/getRecursive) to this member.
etcd_debugging_store_watch_requests_total counter cls, ins, instance, job, ip Incoming watch requests (new/reestablished).
etcd_debugging_store_watchers gauge cls, ins, instance, job, ip Active watchers.
etcd_debugging_store_writes_total counter cls, action, ins, instance, job, ip Writes (set/compareAndDelete) to this member.
etcd_disk_backend_commit_duration_seconds_bucket Unknown cls, ins, instance, job, le, ip N/A
etcd_disk_backend_commit_duration_seconds_count Unknown cls, ins, instance, job, ip N/A
etcd_disk_backend_commit_duration_seconds_sum Unknown cls, ins, instance, job, ip N/A
etcd_disk_backend_defrag_duration_seconds_bucket Unknown cls, ins, instance, job, le, ip N/A
etcd_disk_backend_defrag_duration_seconds_count Unknown cls, ins, instance, job, ip N/A
etcd_disk_backend_defrag_duration_seconds_sum Unknown cls, ins, instance, job, ip N/A
etcd_disk_backend_snapshot_duration_seconds_bucket Unknown cls, ins, instance, job, le, ip N/A
etcd_disk_backend_snapshot_duration_seconds_count Unknown cls, ins, instance, job, ip N/A
etcd_disk_backend_snapshot_duration_seconds_sum Unknown cls, ins, instance, job, ip N/A
etcd_disk_defrag_inflight gauge cls, ins, instance, job, ip Defrag active. 1 = active, 0 = not.
etcd_disk_wal_fsync_duration_seconds_bucket Unknown cls, ins, instance, job, le, ip N/A
etcd_disk_wal_fsync_duration_seconds_count Unknown cls, ins, instance, job, ip N/A
etcd_disk_wal_fsync_duration_seconds_sum Unknown cls, ins, instance, job, ip N/A
etcd_disk_wal_write_bytes_total gauge cls, ins, instance, job, ip WAL bytes written.
etcd_grpc_proxy_cache_hits_total gauge cls, ins, instance, job, ip Cache hits.
etcd_grpc_proxy_cache_keys_total gauge cls, ins, instance, job, ip Keys/ranges cached.
etcd_grpc_proxy_cache_misses_total gauge cls, ins, instance, job, ip Cache misses.
etcd_grpc_proxy_events_coalescing_total counter cls, ins, instance, job, ip Events coalescing.
etcd_grpc_proxy_watchers_coalescing_total gauge cls, ins, instance, job, ip Current watchers coalescing.
etcd_mvcc_db_open_read_transactions gauge cls, ins, instance, job, ip Open read transactions.
etcd_mvcc_db_total_size_in_bytes gauge cls, ins, instance, job, ip DB physical bytes allocated.
etcd_mvcc_db_total_size_in_use_in_bytes gauge cls, ins, instance, job, ip DB logical bytes in use.
etcd_mvcc_delete_total counter cls, ins, instance, job, ip Deletes seen by this member.
etcd_mvcc_hash_duration_seconds_bucket Unknown cls, ins, instance, job, le, ip N/A
etcd_mvcc_hash_duration_seconds_count Unknown cls, ins, instance, job, ip N/A
etcd_mvcc_hash_duration_seconds_sum Unknown cls, ins, instance, job, ip N/A
etcd_mvcc_hash_rev_duration_seconds_bucket Unknown cls, ins, instance, job, le, ip N/A
etcd_mvcc_hash_rev_duration_seconds_count Unknown cls, ins, instance, job, ip N/A
etcd_mvcc_hash_rev_duration_seconds_sum Unknown cls, ins, instance, job, ip N/A
etcd_mvcc_put_total counter cls, ins, instance, job, ip Puts seen by this member.
etcd_mvcc_range_total counter cls, ins, instance, job, ip Ranges seen by this member.
etcd_mvcc_txn_total counter cls, ins, instance, job, ip Txns seen by this member.
etcd_network_active_peers gauge cls, ins, Local, instance, job, ip, Remote Active peer connections.
etcd_network_client_grpc_received_bytes_total counter cls, ins, instance, job, ip gRPC client bytes received.
etcd_network_client_grpc_sent_bytes_total counter cls, ins, instance, job, ip gRPC client bytes sent.
etcd_network_peer_received_bytes_total counter cls, ins, instance, job, ip, From Peer bytes received.
etcd_network_peer_round_trip_time_seconds_bucket Unknown cls, To, ins, instance, job, le, ip N/A
etcd_network_peer_round_trip_time_seconds_count Unknown cls, To, ins, instance, job, ip N/A
etcd_network_peer_round_trip_time_seconds_sum Unknown cls, To, ins, instance, job, ip N/A
etcd_network_peer_sent_bytes_total counter cls, To, ins, instance, job, ip Peer bytes sent.
etcd_server_apply_duration_seconds_bucket Unknown cls, version, ins, instance, job, le, success, ip, op N/A
etcd_server_apply_duration_seconds_count Unknown cls, version, ins, instance, job, success, ip, op N/A
etcd_server_apply_duration_seconds_sum Unknown cls, version, ins, instance, job, success, ip, op N/A
etcd_server_client_requests_total counter client_api_version, cls, ins, instance, type, job, ip Client requests per version.
etcd_server_go_version gauge cls, ins, instance, job, server_go_version, ip Go version running. 1 = ‘server_go_version’ label with current version.
etcd_server_has_leader gauge cls, ins, instance, job, ip Leader exists. 1 = exists, 0 = not.
etcd_server_health_failures counter cls, ins, instance, job, ip Failed health checks.
etcd_server_health_success counter cls, ins, instance, job, ip Successful health checks.
etcd_server_heartbeat_send_failures_total counter cls, ins, instance, job, ip Leader heartbeat send failures (likely overloaded from slow disk).
etcd_server_id gauge cls, ins, instance, job, server_id, ip Server/member ID (hex). 1 = ‘server_id’ label with current ID.
etcd_server_is_leader gauge cls, ins, instance, job, ip Member is leader. 1 if is, 0 otherwise.
etcd_server_is_learner gauge cls, ins, instance, job, ip Member is learner. 1 if is, 0 otherwise.
etcd_server_leader_changes_seen_total counter cls, ins, instance, job, ip Leader changes seen.
etcd_server_learner_promote_successes counter cls, ins, instance, job, ip Successful learner promotions while this member is leader.
etcd_server_proposals_applied_total gauge cls, ins, instance, job, ip Consensus proposals applied.
etcd_server_proposals_committed_total gauge cls, ins, instance, job, ip Consensus proposals committed.
etcd_server_proposals_failed_total counter cls, ins, instance, job, ip Failed proposals seen.
etcd_server_proposals_pending gauge cls, ins, instance, job, ip Pending proposals to commit.
etcd_server_quota_backend_bytes gauge cls, ins, instance, job, ip Backend storage quota bytes.
etcd_server_read_indexes_failed_total counter cls, ins, instance, job, ip Failed read indexes seen.
etcd_server_slow_apply_total counter cls, ins, instance, job, ip Slow apply requests (likely overloaded from slow disk).
etcd_server_slow_read_indexes_total counter cls, ins, instance, job, ip Pending read indexes not in sync with leader or timed out read index requests.
etcd_server_snapshot_apply_in_progress_total gauge cls, ins, instance, job, ip 1 if server applying incoming snapshot. 0 if none.
etcd_server_version gauge cls, server_version, ins, instance, job, ip Version running. 1 = ‘server_version’ label with current version.
etcd_snap_db_fsync_duration_seconds_bucket Unknown cls, ins, instance, job, le, ip N/A
etcd_snap_db_fsync_duration_seconds_count Unknown cls, ins, instance, job, ip N/A
etcd_snap_db_fsync_duration_seconds_sum Unknown cls, ins, instance, job, ip N/A
etcd_snap_db_save_total_duration_seconds_bucket Unknown cls, ins, instance, job, le, ip N/A
etcd_snap_db_save_total_duration_seconds_count Unknown cls, ins, instance, job, ip N/A
etcd_snap_db_save_total_duration_seconds_sum Unknown cls, ins, instance, job, ip N/A
etcd_snap_fsync_duration_seconds_bucket Unknown cls, ins, instance, job, le, ip N/A
etcd_snap_fsync_duration_seconds_count Unknown cls, ins, instance, job, ip N/A
etcd_snap_fsync_duration_seconds_sum Unknown cls, ins, instance, job, ip N/A
etcd_up Unknown cls, ins, instance, job, ip N/A
go_gc_duration_seconds summary cls, ins, instance, job, quantile, ip GC pause duration summary.
go_gc_duration_seconds_count Unknown cls, ins, instance, job, ip N/A
go_gc_duration_seconds_sum Unknown cls, ins, instance, job, ip N/A
go_goroutines gauge cls, ins, instance, job, ip Goroutines.
go_info gauge cls, version, ins, instance, job, ip Go environment info.
go_memstats_alloc_bytes gauge cls, ins, instance, job, ip Bytes allocated and in use.
go_memstats_alloc_bytes_total counter cls, ins, instance, job, ip Bytes allocated, even if freed.
go_memstats_buck_hash_sys_bytes gauge cls, ins, instance, job, ip Bytes used by profiling bucket hash table.
go_memstats_frees_total counter cls, ins, instance, job, ip Frees.
go_memstats_gc_cpu_fraction gauge cls, ins, instance, job, ip GC CPU fraction since program started.
go_memstats_gc_sys_bytes gauge cls, ins, instance, job, ip Bytes used for GC system metadata.
go_memstats_heap_alloc_bytes gauge cls, ins, instance, job, ip Heap bytes allocated and in use.
go_memstats_heap_idle_bytes gauge cls, ins, instance, job, ip Heap bytes waiting to be used.
go_memstats_heap_inuse_bytes gauge cls, ins, instance, job, ip Heap bytes in use.
go_memstats_heap_objects gauge cls, ins, instance, job, ip Allocated objects.
go_memstats_heap_released_bytes gauge cls, ins, instance, job, ip Heap bytes released to OS.
go_memstats_heap_sys_bytes gauge cls, ins, instance, job, ip Heap bytes obtained from system.
go_memstats_last_gc_time_seconds gauge cls, ins, instance, job, ip Seconds since 1970 of last GC.
go_memstats_lookups_total counter cls, ins, instance, job, ip Pointer lookups.
go_memstats_mallocs_total counter cls, ins, instance, job, ip Mallocs.
go_memstats_mcache_inuse_bytes gauge cls, ins, instance, job, ip Bytes in use by mcache structures.
go_memstats_mcache_sys_bytes gauge cls, ins, instance, job, ip Bytes used for mcache structures from system.
go_memstats_mspan_inuse_bytes gauge cls, ins, instance, job, ip Bytes in use by mspan structures.
go_memstats_mspan_sys_bytes gauge cls, ins, instance, job, ip Bytes used for mspan structures from system.
go_memstats_next_gc_bytes gauge cls, ins, instance, job, ip Heap bytes when next GC will take place.
go_memstats_other_sys_bytes gauge cls, ins, instance, job, ip Bytes used for other system allocations.
go_memstats_stack_inuse_bytes gauge cls, ins, instance, job, ip Bytes in use by stack allocator.
go_memstats_stack_sys_bytes gauge cls, ins, instance, job, ip Bytes obtained from system for stack allocator.
go_memstats_sys_bytes gauge cls, ins, instance, job, ip Bytes obtained from system.
go_threads gauge cls, ins, instance, job, ip OS threads created.
grpc_server_handled_total counter cls, ins, instance, job, grpc_code, grpc_method, grpc_type, ip, grpc_service RPCs completed on server.
grpc_server_msg_received_total counter cls, ins, instance, job, grpc_type, grpc_method, ip, grpc_service RPC stream messages received on server.
grpc_server_msg_sent_total counter cls, ins, instance, job, grpc_type, grpc_method, ip, grpc_service gRPC stream messages sent on server.
grpc_server_started_total counter cls, ins, instance, job, grpc_type, grpc_method, ip, grpc_service RPCs started on server.
os_fd_limit gauge cls, ins, instance, job, ip FD limit.
os_fd_used gauge cls, ins, instance, job, ip Used FDs.
process_cpu_seconds_total counter cls, ins, instance, job, ip User + system CPU seconds.
process_max_fds gauge cls, ins, instance, job, ip Max FDs.
process_open_fds gauge cls, ins, instance, job, ip Open FDs.
process_resident_memory_bytes gauge cls, ins, instance, job, ip Resident memory bytes.
process_start_time_seconds gauge cls, ins, instance, job, ip Start time (unix epoch seconds).
process_virtual_memory_bytes gauge cls, ins, instance, job, ip Virtual memory bytes.
process_virtual_memory_max_bytes gauge cls, ins, instance, job, ip Max virtual memory bytes.
promhttp_metric_handler_requests_in_flight gauge cls, ins, instance, job, ip Current scrapes.
promhttp_metric_handler_requests_total counter cls, ins, instance, job, ip, code Scrapes by HTTP status code.
scrape_duration_seconds Unknown cls, ins, instance, job, ip N/A
scrape_samples_post_metric_relabeling Unknown cls, ins, instance, job, ip N/A
scrape_samples_scraped Unknown cls, ins, instance, job, ip N/A
scrape_series_added Unknown cls, ins, instance, job, ip N/A
up Unknown cls, ins, instance, job, ip N/A

11.7 - FAQ

Frequently asked questions about Pigsty etcd module

What is etcd’s role in Pigsty?

etcd is a distributed, reliable key-value store for critical system data. Pigsty uses etcd as DCS (Distributed Config Store) service for Patroni, storing PG HA status.

Patroni uses etcd for: cluster failure detection, auto failover, primary-replica switchover, and cluster config management.

etcd is critical for PostgreSQL HA, and its own availability depends on a reachable majority. Production deployments normally spread three or five voting members across independent failure domains.

What’s the appropriate etcd cluster size?

If more than half (including exactly half) of etcd instances unavailable, etcd cluster enters unavailable state—refuses service.

Example: 3-node cluster allows max 1 node failure while 2 others continue; 5-node cluster tolerates 2 node failures.

Note: Learner instances don’t count toward members—3-node cluster with 1 learner = 2 actual members, zero fault tolerance.

In prod, use odd number of instances. For prod, recommend 3-node or 5-node for reliability.

Impact of etcd unavailability?

If etcd cluster unavailable, affects PG control plane but not data plane—existing PG clusters continue running, but Patroni management ops fail.

During etcd failure: PG HA can’t auto failover, can’t use patronictl for PG management (config changes, manual failover, etc.).

Ansible playbooks unaffected by etcd failure: create DB, create user, refresh HBA/Service config. During etcd failure, operate PG clusters directly.

Note: Behavior applies to Patroni >=3.0 (Pigsty >=2.0). With older Patroni (<3.0, Pigsty 1.x), etcd/consul failure causes severe global impact:

All PG clusters demote: primaries → replicas, reject writes, etcd failure amplifies to global PG failure. Patroni 3.0 introduced DCS Failsafe—significantly improved.

What data does etcd store?

By default, Pigsty uses etcd as Patroni’s DCS, where it stores coordination data such as leader leases, member state, and dynamic configuration. Pigsty itself does not store application data there.

Patroni creates and manages this DCS data. During controlled maintenance, it can normally reconstruct coordination state from a healthy PostgreSQL cluster, but that does not make etcd stateless or make direct DCS deletion risk-free.

Rebuilding etcd interrupts automatic failover and patronictl management and clears the current DCS state. Before doing so, inspect the Patroni topology, current primary, remaining quorum, and recent backups, and execute a documented recovery procedure during a maintenance window.

If using etcd for other purposes (K8s metadata, custom storage), backup etcd data yourself and restore after cluster recovery.

Recover from etcd failure?

By default, Pigsty uses etcd only as Patroni DCS. Restarting services and rebuilding the whole cluster have very different risk: a restart preserves DCS data, while a rebuild clears coordination state and leaves PostgreSQL HA without DCS quorum until recovery. Diagnose and recover existing members first; rebuild only after verifying the topology, backups, and recovery path.

Restart etcd cluster:

./etcd.yml -t etcd_launch

If a full reset/rebuild is genuinely required, do it in a maintenance window and then verify etcdctl endpoint health, etcdctl member list, and patronictl list:

./etcd-rm.yml -l etcd          # Clean the cluster after verifying target and backups
./etcd.yml -l etcd             # Redeploy from inventory

For custom etcd data: backup and restore after recovery.

Etcd maintenance considerations?

Simple answer: don’t fill up etcd.

Pigsty enables etcd auto-compaction by default, with the current backend quota set to 8 GiB. This is usually sufficient, but actual usage should still be monitored.

etcd’s data model = each write generates new version.

Frequent writes (even few keys) = growing etcd DB size. At capacity limit, etcd rejects writes → PG HA breaks.

Pigsty’s default etcd config includes optimizations:

auto-compaction-mode: periodic      # periodic auto compaction
auto-compaction-retention: "24h"    # retain 24 hours history
quota-backend-bytes: 8589934592     # 8 GiB quota

More details: etcd official maintenance guide.

Note

Before Pigsty v2.6? Manually enable etcd auto GC.

Enable etcd auto garbage collection?

Earlier Pigsty (v2.0 - v2.5)? Enable etcd auto-compaction in prod to avoid quota-based unavailability.

Edit the etcd configuration template: roles/etcd/templates/etcd.conf:

auto-compaction-mode: periodic
auto-compaction-retention: "24h"
quota-backend-bytes: 17179869184

Then set related PG clusters to maintenance mode and redeploy etcd with ./etcd.yml.

This increases default quota from 2 GiB → 16 GiB, retains last 24h writes—avoids infinite growth.

Where is PG HA data stored in etcd?

By default, Patroni uses pg_namespace prefix (default: /pg) for all metadata keys, followed by PG cluster name.

Example: PG cluster pg-meta stores metadata under /pg/pg-meta.

etcdctl get /pg/pg-meta --prefix

Sample data:

/pg/pg-meta/config
{"ttl":30,"loop_wait":10,"retry_timeout":10,"primary_start_timeout":10,"maximum_lag_on_failover":1048576,"maximum_lag_on_syncnode":-1,"primary_stop_timeout":30,"synchronous_mode":false,"synchronous_mode_strict":false,"failsafe_mode":true,"pg_version":16,"pg_cluster":"pg-meta","pg_shard":"pg-meta","pg_group":0,"postgresql":{"use_slots":true,"use_pg_rewind":true,"remove_data_directory_on_rewind_failure":true,"parameters":{"max_connections":100,"superuser_reserved_connections":10,"max_locks_per_transaction":200,"max_prepared_transactions":0,"track_commit_timestamp":"on","wal_level":"logical","wal_log_hints":"on","max_worker_processes":16,"max_wal_senders":50,"max_replication_slots":50,"password_encryption":"scram-sha-256","ssl":"on","ssl_cert_file":"/pg/cert/server.crt","ssl_key_file":"/pg/cert/server.key","ssl_ca_file":"/pg/cert/ca.crt","shared_buffers":"7969MB","maintenance_work_mem":"1993MB","work_mem":"79MB","max_parallel_workers":8,"max_parallel_maintenance_workers":2,"max_parallel_workers_per_gather":0,"hash_mem_multiplier":8.0,"huge_pages":"try","temp_file_limit":"7GB","vacuum_cost_delay":"20ms","vacuum_cost_limit":2000,"bgwriter_delay":"10ms","bgwriter_lru_maxpages":800,"bgwriter_lru_multiplier":5.0,"min_wal_size":"7GB","max_wal_size":"28GB","max_slot_wal_keep_size":"42GB","wal_buffers":"16MB","wal_writer_delay":"20ms","wal_writer_flush_after":"1MB","commit_delay":20,"commit_siblings":10,"checkpoint_timeout":"15min","checkpoint_completion_target":0.8,"archive_mode":"on","archive_timeout":300,"archive_command":"pgbackrest --stanza=pg-meta archive-push %p","max_standby_archive_delay":"10min","max_standby_streaming_delay":"3min","wal_receiver_status_interval":"1s","hot_standby_feedback":"on","wal_receiver_timeout":"60s","max_logical_replication_workers":8,"max_sync_workers_per_subscription":6,"random_page_cost":1.1,"effective_io_concurrency":1000,"effective_cache_size":"23907MB","default_statistics_target":200,"log_destination":"csvlog","logging_collector":"on","l...
ode=prefer"}}
/pg/pg-meta/failsafe
{"pg-meta-2":"http://10.10.10.11:8008/patroni","pg-meta-1":"http://10.10.10.10:8008/patroni"}
/pg/pg-meta/initialize
7418384210787662172
/pg/pg-meta/leader
pg-meta-1
/pg/pg-meta/members/pg-meta-1
{"conn_url":"postgres://10.10.10.10:5432/postgres","api_url":"http://10.10.10.10:8008/patroni","state":"running","role":"primary","version":"4.0.1","tags":{"clonefrom":true,"version":"16","spec":"8C.32G.125G","conf":"tiny.yml"},"xlog_location":184549376,"timeline":1}
/pg/pg-meta/members/pg-meta-2
{"conn_url":"postgres://10.10.10.11:5432/postgres","api_url":"http://10.10.10.11:8008/patroni","state":"running","role":"replica","version":"4.0.1","tags":{"clonefrom":true,"version":"16","spec":"8C.32G.125G","conf":"tiny.yml"},"xlog_location":184549376,"replication_state":"streaming","timeline":1}
/pg/pg-meta/status
{"optime":184549376,"slots":{"pg_meta_2":184549376,"pg_meta_1":184549376},"retain_slots":["pg_meta_1","pg_meta_2"]}

Use external existing etcd cluster?

Config inventory hardcodes etcd group—members used as DCS servers for PGSQL. Initialize with etcd.yml or assume external cluster exists.

To use external etcd: define as usual. Skip etcd.yml execution since cluster exists—no deployment needed.

Requirement: external etcd cluster certificate must use same CA as Pigsty—otherwise clients can’t use Pigsty’s self-signed certs.

Add new member to existing etcd cluster?

For detailed process, refer to Add member to etcd cluster

Recommended: Utility script

# First add new member to config inventory, then:
bin/etcd-add <ip>      # add single new member
bin/etcd-add <ip1>     # add multiple new members

Manual method:

etcdctl member add <etcd-?> --learner=true --peer-urls=https://<new_ins_ip>:2380 # announce new member
./etcd.yml -l <new_ins_ip> -e etcd_init=existing                                 # initialize new member
etcdctl member promote <new_ins_server_id>                                       # promote to full member

Recommend: add one new member at a time.

Remove member from existing etcd cluster?

For detailed process, refer to Remove member from etcd cluster

Recommended: Utility script

bin/etcd-rm <ip>              # remove specified member
bin/etcd-rm                   # remove entire etcd cluster

Manual method:

./etcd-rm.yml -l <ins_ip>                    # Leave, stop, and delete local data by default

etcd-rm.yml already includes the etcdctl member remove step; do not repeat it before or after the normal workflow. Use a manual member remove only for troubleshooting. You can still run the removal playbook once while the target remains in inventory to stop, deregister, and clean it locally, then verify the remaining quorum.

Configure etcd RBAC authentication?

Since v4.0, Pigsty has enabled etcd RBAC auth by default. Root password set by etcd_root_password, default: Etcd.Root.

Prod recommendation: change default password

all:
  vars:
    etcd_root_password: 'YourSecurePassword'

Client auth:

# On etcd nodes, env vars auto-configured
source /etc/profile.d/etcdctl.sh
etcdctl member list

# Manual auth config
export ETCDCTL_USER="root:YourSecurePassword"
export ETCDCTL_CACERT=/etc/etcd/ca.crt
export ETCDCTL_CERT=/etc/etcd/server.crt
export ETCDCTL_KEY=/etc/etcd/server.key

More: RBAC Authentication.

12 - Module: MINIO

Deploy Silo S3-compatible object storage with Pigsty’s MINIO compatibility module and use it as a PostgreSQL backup repository.

MINIO is Pigsty’s compatibility module name for S3-compatible object storage. The current role deploys Silo, and minio_type accepts only silo.

Silo preserves the MinIO S3/Admin APIs, MINIO_* environment variables, disk format, and mcli client interface, and can serve as a PostgreSQL pgBackRest backup repository. The module name, parameter prefix, and monitoring job retain the MINIO / minio_* namespace for compatibility with existing inventories and operational entry points.

Important

minio and rustfs are no longer valid minio_type values and fail during identity validation. Before upgrading a MinIO cluster managed by an older release, complete a backup, validate MinIO-to-Silo compatibility, and rehearse rollback. Do not treat package replacement as an already-accepted data migration. External MinIO, RustFS, or other S3 services can still serve as pgBackRest repositories, but the current MINIO role does not manage them.

MINIO is an optional module. When using it as a pgBackRest S3 repository, deploy it before the PGSQL module. TLS certificates and the host baseline come from the NODE / CA capabilities.

Quick Start

The following inventory explicitly defines a single-node Silo cluster. Both minio_cluster and minio_seq are required identity parameters; production inventories should explicitly set minio_type: silo.

minio:
  hosts:
    10.10.10.10: { minio_seq: 1 }
  vars:
    minio_cluster: minio
    minio_type: silo
./minio.yml -l minio    # Deploy Silo on the minio group

The inventory group name may differ from minio_cluster. Roles calculate actual membership from each host’s minio_cluster identity. Do not define minio_cluster in all.vars, or every host will be treated as an object-storage member.

After deployment, use these entry points:

  • S3 API: https://sss.pigsty:9000 (configure DNS or /etc/hosts explicitly for the domain)
  • Administration UI: https://<node-ip>:9001
  • Command line: mcli ls sss/ (a preconfigured alias is written on the admin node and cluster members)

The default administrator credentials are minioadmin / S3User.MinIO. They are for demos only and must be changed before production deployment.

Deployment Modes

Silo uses the following Pigsty inventory deployment modes:

Mode Description Use Cases
Single-Node Single-Disk (SNSD) Single node, one data directory Development, testing, demos
Single-Node Multi-Disk (SNMD) Single node, multiple disks Resource-constrained small deployments
Multi-Node Single-Disk (MNSD) Multiple nodes, one data drive per node Compact HA deployments
Multi-Node Multi-Disk (MNMD) Multiple nodes with multiple disks per node Recommended for production

minio_data is always a directory path. Distributed and multi-drive deployments require these paths to reside on non-root, persistent filesystems. For example, /data/minio may be a subdirectory of a separately mounted /data filesystem, but not merely a directory on the root filesystem.

The multi-pool expansion semantics of minio_volumes come from the MinIO-compatible interface retained by Silo. Validate operations and rollback against the actual Silo version before production scaling.

Core Capabilities

  • Compatible interface: Silo retains minio_* parameters, the S3 port, TLS, and the mcli provisioning flow
  • HA topologies: Supports single-node, multi-node single-drive, and multi-node multi-drive deployments, with multiple independent clusters in one inventory
  • Backup repository: Can serve as a remote pgBackRest S3 repository
  • Security baseline: Enables HTTPS by default and uses the Pigsty CA to issue a certificate for every instance
  • Observability: Scrapes Silo metrics through /minio/metrics/v3 and provides Grafana dashboards and alerts
  • Operational compatibility: Module name, target directories, monitoring labels, and client aliases retain the MINIO namespace

12.1 - Usage

Quickly use Silo deployed by the MINIO module through mcli, rclone, and pgBackRest.

After you configure and deploy Silo with the playbook, use this page to access it through the compatible S3 and mcli interfaces.

Deploy Cluster

First, define a single-node, single-disk object-storage cluster in the config inventory and explicitly pin its engine:

minio: { hosts: { 10.10.10.10: { minio_seq: 1 } }, vars: { minio_cluster: minio, minio_type: silo } }

Then, run the minio.yml playbook provided by Pigsty against the defined group (here minio):

./minio.yml -l minio

Note that deploy.yml automatically creates predefined Silo clusters, so you do not need to run the minio.yml playbook again manually.

For a production multi-node deployment, read Pigsty’s configuration documentation and verify the operational constraints of the Silo version you deploy.

Access Cluster

Production environments should access object storage through a domain name and HTTPS, which is also the default configuration. If you explicitly set minio_https to false, HTTP is available instead. In either case, ensure that the object-storage service domain (default sss.pigsty) points to the service node or load balancer.

  1. You can add static resolution records in node_etc_hosts, or manually modify the /etc/hosts file
  2. You can add a record on the internal DNS server if you already have an existing DNS service
  3. If you have enabled the DNS server on Infra nodes, you can add records in dns_records

For production, we recommend the first method—static DNS records—so object storage does not depend on dynamic DNS.

Point the S3 service domain to the IP address and service port of a Silo node or load balancer. Pigsty uses sss.pigsty as the default S3 service domain and serves it on port 9000. The role does not automatically create a global DNS record for minio_domain; configure resolution explicitly as described above.

Some examples deploy HAProxy on the Silo cluster to expose the service; those templates use port 9002 as the unified service port.

Adding Alias

To access the Silo cluster with the mcli client, first configure a server alias:

mcli alias ls  # list minio alias (default is sss)
mcli alias set sss https://sss.pigsty:9000 minioadmin S3User.MinIO            # root user
mcli alias set sss https://sss.pigsty:9002 minioadmin S3User.MinIO            # root user, using load balancer port 9002

mcli alias set pgbackrest https://sss.pigsty:9000 pgbackrest S3User.Backup    # use backup user

After a full minio.yml run with minio_provision enabled, the role configures the default alias for the Ansible execution user on every Infra node and every actual object-storage member discovered by minio_cluster. A host in both sets is configured only once.

For the full mcli command reference, see the upstream MinIO Client documentation.

Note: Use Your Actual Password

The password S3User.MinIO in the above examples is the Pigsty default. If you modified minio_secret_key during deployment, please use your actual configured password.

User Management

You can manage Silo application users with mcli. Default provisioning already creates pgbackrest, s3user_meta, and s3user_data; the example below creates one additional user and attaches the generated policy for the default data bucket:

mcli admin user list sss
set +o history
mcli admin user add sss appuser 'Replace.With.Strong.Password'
mcli admin policy attach sss data --user=appuser
set -o history

Bucket Management

You can perform CRUD operations on buckets in Silo:

mcli ls sss/                         # list all buckets on alias 'sss'
mcli mb --ignore-existing sss/hello  # create a bucket named 'hello'
mcli rb --force sss/hello            # force delete the 'hello' bucket

Object Management

You can also perform CRUD operations on objects within buckets. For details, please refer to the official documentation: Object Management

mcli cp /www/pigsty/* sss/data/      # upload local repo content to the default data bucket
mcli cp sss/data/plugins.tgz /tmp/   # download file from Silo
mcli ls sss/data                     # list all files in the data bucket
mcli rm sss/data/plugins.tgz         # delete a specific file in the data bucket
mcli cat sss/data/repo_complete      # view file content in the data bucket

Using rclone

The Pigsty repository provides rclone, a convenient multi-cloud object-storage client that can access Silo.

yum install rclone;  # EL-compatible systems
apt install rclone;  # Debian/Ubuntu systems

mkdir -p ~/.config/rclone/;
tee ~/.config/rclone/rclone.conf > /dev/null <<EOF
[sss]
type = s3
access_key_id = minioadmin
secret_access_key = S3User.MinIO
endpoint = https://sss.pigsty:9000
EOF

rclone ls sss:/
Note: HTTPS and Certificate Trust

If Silo uses HTTPS (the default), ensure that the client trusts Pigsty’s CA certificate (/etc/pki/ca.crt), or add no_check_certificate = true to the rclone configuration to skip certificate verification (not recommended for production).

Configure Backup Repository

In Pigsty, the MINIO module’s primary use case is as an S3 backup repository for pgBackRest. When you set pgbackrest_method to minio, the PGSQL module uses the S3-compatible repository preset with that name. Silo deployed by the MINIO module works directly with this preset.

pgbackrest_method: local          # pgbackrest repo method: local,minio,[user-defined...]
pgbackrest_repo:                  # pgbackrest repo: https://pgbackrest.org/configuration.html#section-repository
  local:                          # default pgbackrest repo with local posix fs
    path: /pg/backup              # local backup directory, `/pg/backup` by default
    retention_full_type: count    # retention full backups by count
    retention_full: 2             # keep 2, at most 3 full backup when using local fs repo
  minio:                          # optional minio repo for pgbackrest
    type: s3                      # Silo uses the S3-compatible repository type
    s3_endpoint: sss.pigsty       # Silo endpoint domain, `sss.pigsty` by default
    s3_region: us-east-1          # compatibility region, `us-east-1` by default
    s3_bucket: pgsql              # backup bucket, `pgsql` by default
    s3_key: pgbackrest            # pgBackRest access key
    s3_key_secret: S3User.Backup  # pgBackRest secret key
    s3_uri_style: path            # use path-style rather than host-style URIs
    path: /pgbackrest             # backup path, `/pgbackrest` by default
    storage_port: 9000            # Silo service port, 9000 by default
    storage_ca_file: /pg/cert/ca.crt  # CA path, `/pg/cert/ca.crt` by default
    bundle: y                     # bundle small files into a single file
    cipher_type: aes-256-cbc      # enable AES encryption for remote backup repo
    cipher_pass: pgBackRest       # AES encryption password, default is 'pgBackRest'
    retention_full_type: time     # retain full backups by time on the remote repository
    retention_full: 14            # keep full backup for last 14 days

If you use a multi-node Silo cluster behind a load balancer, adjust s3_endpoint and storage_port accordingly.



12.2 - Configuration

Deploy Silo with the MINIO module and configure reliable S3 access in single-node, multi-drive, or multi-node modes.

Before deploying the MINIO module, define a Silo object-storage cluster in the config inventory. The current role requires minio_type: silo and supports these inventory deployment modes:

SNSD is suitable for development and testing, three-node MNSD for resource-constrained compact HA, and MNMD for production environments with higher capacity, throughput, and drive-redundancy requirements. SNMD handles drive failures within one server but cannot tolerate losing the server.

Silo can also use multi-pool deployment for expansion, or you can deploy multiple clusters.

With a multi-node cluster, any member can serve the S3 API, so the best practice is to place load balancing and high-availability service access in front of it.

Backend Selection 

minio_type: silo   # the only valid value

minio_type is retained as a selector for future expansion, but the current deployment and removal roles accept only silo. It maps to the silo package, silo.service, /etc/default/silo, and ~/.minio/certs/. To support in-place migration, silo.service first reads legacy /etc/default/minio and then the higher-priority /etc/default/silo; it also conflicts with the old minio.service. New deployments should maintain only the Silo configuration file.

Legacy inventories with minio_type: minio or minio_type: rustfs fail identity validation. Before upgrading an existing MinIO deployment, validate MinIO-to-Silo data compatibility, backups, and rollback. References below to MinIO topology terms and upstream links describe interfaces retained by Silo; they do not mean that the current role still installs the minio package.

Core Parameters

Pigsty uses minio_volumes to describe members and disks and renders it as Silo’s MINIO_VOLUMES. The role derives this value from inventory by default and also allows an explicit override.

  • Single-Node Single-Disk: minio_volumes points to a regular local directory derived from minio_data, defaulting to /data/minio.
  • Single-Node Multi-Disk: minio_volumes points to a sequence of local mount points derived from minio_data, for example /data{1...4}.
  • Multi-Node Single-Disk: minio_volumes points to one data path on each server, for example https://minio-{1...3}.pigsty:9000/data/minio.
  • Multi-Node Multi-Disk: minio_volumes points to mount points across multiple servers, automatically generated from two parts:
    • First, use minio_data to specify the disk mount point sequence for each cluster member /data{1...4}
    • Also use minio_node to specify the node naming pattern ${minio_cluster}-${minio_seq}.pigsty
  • Multi-Pool: Explicitly set minio_volumes to assign nodes to each storage pool.

Storage Paths and Mounts

minio_data is a filesystem directory, not a raw block device. Format and mount a local disk, cloud volume, separate partition, or LVM logical volume first, then give Silo the mount point or a directory beneath it. Do not put /dev/sdb in minio_data.

The MINIO role creates data directories and sets ownership and permissions, but it does not format or persistently mount production storage. Topologies impose different requirements on the backing filesystem:

  • Single-node single-disk may use a regular directory on the root filesystem, but only for development, testing, and demos.
  • Every path in a single-node multi-disk deployment should map to a separate filesystem. Multiple directories on one drive do not create multiple drive failure domains.
  • Multi-node distributed Silo detects and rejects data paths on the root filesystem with drive is part of root drive, will not be used.

Therefore, /data/minio may be a regular subdirectory when /data is a separately mounted persistent filesystem. If /data is merely a directory under /, it does not satisfy the distributed-storage requirement. A bind mount backed by the root filesystem does not create a new drive failure domain either.

Inspect the actual mounts before deployment:

findmnt -T /
findmnt -T /data/minio

The second command should report a separate /data or /data/minio mount rather than /. Production mounts should also be persisted in /etc/fstab or an equivalent mechanism, and drives in one storage pool should have similar capacities.

Single-Node Single-Disk

SNSD mode, compatible topology reference: MinIO Single-Node Single-Drive

In Pigsty, defining a singleton Silo instance is straightforward:

# 1 node, 1 data directory
minio: { hosts: { 10.10.10.10: { minio_seq: 1 } }, vars: { minio_cluster: minio, minio_type: silo } }

In single-node mode, the required identity parameters are minio_seq and minio_cluster, which uniquely identify each object-storage instance.

Single-node single-disk mode is for development purposes only, so you can use a regular directory as the data directory, specified by minio_data, defaulting to /data/minio.

When using Silo, we strongly recommend accessing it through a statically resolved domain name. For example, if minio_domain uses the default sss.pigsty, you can add a static resolution on all nodes to facilitate access to this service.

node_etc_hosts: ["10.10.10.10 sss.pigsty"] # domain name for accessing Silo from all nodes (required)
SNSD is for Development Only

Single-node single-disk mode should only be used for development, testing, and demo purposes, as it cannot tolerate any hardware failure and does not benefit from multi-disk performance improvements. For production, use Multi-Node Multi-Disk mode.

Single-Node Multi-Disk

SNMD mode, compatible topology reference: MinIO Single-Node Multi-Drive

To use multiple disks on a single node, the operation is similar to Single-Node Single-Disk, but you need to specify minio_data in the format {{ prefix }}{x...y}, which defines a series of disk mount points.

minio:
  hosts: { 10.10.10.10: { minio_seq: 1 } }
  vars:
    minio_cluster: minio         # required object-storage cluster identity
    minio_data: '/data{1...4}'   # minio data dir(s), use {x...y} to specify multi drivers
Use Real Disk Mount Points

Every SNMD data path must reside on a separate filesystem. If multiple paths resolve to the same filesystem, Silo refuses to treat them as separate drives. XFS is recommended for production; the Vagrant test setup can also prepare ext4 data drives when XFS tools are unavailable.

For example, the Vagrant object-storage sandbox defines a single-node Silo cluster with four disks: /data1, /data2, /data3, and /data4. Before starting Silo, mount them correctly and format them with xfs:

mkfs.xfs /dev/vdb; mkdir /data1; mount -t xfs /dev/vdb /data1;   # mount disk 1...
mkfs.xfs /dev/vdc; mkdir /data2; mount -t xfs /dev/vdc /data2;   # mount disk 2...
mkfs.xfs /dev/vdd; mkdir /data3; mount -t xfs /dev/vdd /data3;   # mount disk 3...
mkfs.xfs /dev/vde; mkdir /data4; mount -t xfs /dev/vde /data4;   # mount disk 4...

Disk mounting is part of server provisioning and beyond Pigsty’s scope. Mounted disks should be written to /etc/fstab for auto-mounting after server restart.

/dev/vdb /data1 xfs defaults,noatime,nodiratime 0 0
/dev/vdc /data2 xfs defaults,noatime,nodiratime 0 0
/dev/vdd /data3 xfs defaults,noatime,nodiratime 0 0
/dev/vde /data4 xfs defaults,noatime,nodiratime 0 0

SNMD mode can utilize multiple disks on a single machine to provide higher performance and capacity, and tolerate partial disk failures. However, single-node mode cannot tolerate entire node failure, and you cannot add new nodes at runtime, so we do not recommend using SNMD mode in production unless you have special reasons.

Multi-Node Single-Disk

MNSD uses one data drive on each of several servers. The following inventory defines a three-node, single-drive Silo cluster, which is also the storage topology used by ha/trio:

minio:
  hosts:
    10.10.10.10: { minio_seq: 1 }
    10.10.10.11: { minio_seq: 2 }
    10.10.10.12: { minio_seq: 3 }
  vars:
    minio_cluster: minio
    minio_type: silo
    minio_data: /data/minio

The role generates https://minio-{1...3}.pigsty:9000/data/minio. The three paths reside on three different servers, and /data/minio on every server must be backed by a non-root, persistent filesystem.

A three-drive set uses EC:1 by default: each object is split into two data shards and one parity shard. Read and write quorum are both two, so one node or one data drive may be unavailable. With equal-size drives, usable capacity is about two-thirds of raw capacity before filesystem and metadata overhead, and is limited by the smallest drive.

This is a resource-efficient compact HA topology that removes the single object-storage node as a failure point, but each node still has only one data drive. Use Multi-Node Multi-Disk when capacity, throughput, or per-node drive redundancy requirements are higher.

An existing single-node storage pool cannot be converted in place by adding two members. Create a new three-node cluster, migrate the objects, and switch client endpoints.

Multi-Node Multi-Disk

MNMD mode, compatible topology reference: MinIO Multi-Node Multi-Drive

In addition to using minio_data to specify disks as in Single-Node Multi-Disk mode, use minio_node to specify the multi-node naming pattern.

For example, the following configuration defines a four-node Silo cluster with four disks per node:

minio:
  hosts:
    10.10.10.10: { minio_seq: 1 }  # actual nodename: minio-1.pigsty
    10.10.10.11: { minio_seq: 2 }  # actual nodename: minio-2.pigsty
    10.10.10.12: { minio_seq: 3 }  # actual nodename: minio-3.pigsty
    10.10.10.13: { minio_seq: 4 }  # actual nodename: minio-4.pigsty
  vars:
    minio_cluster: minio
    minio_data: '/data{1...4}'                         # 4-disk per node
    minio_node: '${minio_cluster}-${minio_seq}.pigsty' # minio node name pattern

The minio_node parameter specifies the MINIO module’s internal node-name pattern, used to generate a unique name for each node. By default, the node name is ${minio_cluster}-${minio_seq}.pigsty, where ${minio_cluster} is the cluster name and ${minio_seq} is the node sequence number. Instance names are automatically written to /etc/hosts on each Silo node so cluster members can identify and reach one another.

In this case, the derived minio_volumes is https://minio-{1...4}.pigsty:9000/data{1...4}, identifying four drives on four nodes; the role writes it to Silo’s compatible environment variable. You can set minio_volumes directly on the object-storage cluster to override the automatically generated value. However, this is usually not necessary as Pigsty will automatically generate it based on the config inventory.

Multi-Pool

Silo retains the compatible ability to scale by adding new storage pools. In Pigsty, explicitly set minio_volumes to assign nodes to each pool.

For example, suppose you created the Silo cluster from the Multi-Node Multi-Disk example and now want to add another four-node storage pool.

You need to directly override the minio_volumes parameter:

minio:
  hosts:
    10.10.10.10: { minio_seq: 1 }
    10.10.10.11: { minio_seq: 2 }
    10.10.10.12: { minio_seq: 3 }
    10.10.10.13: { minio_seq: 4 }

    10.10.10.14: { minio_seq: 5 }
    10.10.10.15: { minio_seq: 6 }
    10.10.10.16: { minio_seq: 7 }
    10.10.10.17: { minio_seq: 8 }
  vars:
    minio_cluster: minio
    minio_data: "/data{1...4}"
    minio_node: '${minio_cluster}-${minio_seq}.pigsty' # minio node name pattern
    minio_volumes: 'https://minio-{1...4}.pigsty:9000/data{1...4} https://minio-{5...8}.pigsty:9000/data{1...4}'

Here, the two space-separated values represent two storage pools, each with four nodes and four disks per node. For details, see Administration: Cluster Expansion.

Multiple Clusters

You can deploy new nodes as an independent Silo cluster. The following configuration declares two object-storage clusters with different identities:

minio1:
  hosts:
    10.10.10.10: { minio_seq: 1 }
    10.10.10.11: { minio_seq: 2 }
    10.10.10.12: { minio_seq: 3 }
    10.10.10.13: { minio_seq: 4 }
  vars:
    minio_cluster: minio2
    minio_data: "/data{1...4}"

minio2:
  hosts:
    10.10.10.14: { minio_seq: 5 }
    10.10.10.15: { minio_seq: 6 }
    10.10.10.16: { minio_seq: 7 }
    10.10.10.17: { minio_seq: 8 }
  vars:
    minio_cluster: minio2
    minio_data: "/data{1...4}"
    minio_alias: sss2
    minio_domain: sss2.pigsty
    minio_endpoint: sss2.pigsty:9000

minio_cluster has no default and must be defined for every cluster. Multiple clusters must also use distinct minio_alias, minio_domain, and minio_endpoint values, or shared client aliases and domains on INFRA nodes will overwrite one another. The Ansible group name may differ from minio_cluster; roles discover members across the inventory by identity.

Expose Service

Silo serves the S3 API on port 9000 by default. A multi-node cluster can be accessed through any member.

Service access falls under the scope of the NODE module, and we’ll provide only a basic introduction here.

High-availability access to a multi-node object-storage cluster can use L2 VIP or HAProxy. For example, bind an L2 VIP with keepalived, or expose the S3 service through the haproxy component provided by the NODE module.

# object-storage cluster with 4 nodes and 4 drives per node
minio:
  hosts:
    10.10.10.10: { minio_seq: 1 , nodename: minio-1 }
    10.10.10.11: { minio_seq: 2 , nodename: minio-2 }
    10.10.10.12: { minio_seq: 3 , nodename: minio-3 }
    10.10.10.13: { minio_seq: 4 , nodename: minio-4 }
  vars:
    minio_cluster: minio
    minio_data: '/data{1...4}'
    minio_buckets: [ { name: pgsql }, { name: infra }, { name: redis } ]
    minio_users:
      - { access_key: dba , secret_key: S3User.DBA, policy: consoleAdmin }
      - { access_key: pgbackrest , secret_key: S3User.SomeNewPassWord , policy: readwrite }

    # bind a node l2 vip (10.10.10.9) to minio cluster (optional)
    node_cluster: minio
    vip_enabled: true
    vip_vrid: 128
    vip_address: 10.10.10.9
    vip_interface: eth1

    # expose minio service with haproxy on all nodes
    haproxy_services:
      - name: minio                    # [REQUIRED] service name, unique
        port: 9002                     # [REQUIRED] service port, unique
        balance: leastconn             # [OPTIONAL] load balancer algorithm
        options:                       # [OPTIONAL] minio health check
          - option httpchk
          - option http-keep-alive
          - http-check send meth OPTIONS uri /minio/health/live
          - http-check expect status 200
        servers:
          - { name: minio-1 ,ip: 10.10.10.10 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
          - { name: minio-2 ,ip: 10.10.10.11 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
          - { name: minio-3 ,ip: 10.10.10.12 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
          - { name: minio-4 ,ip: 10.10.10.13 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }

For example, the configuration above enables HAProxy on every Silo node, exposes the S3 service on port 9002, and binds a Layer 2 VIP to the cluster. Resolve sss.pigsty to VIP 10.10.10.9 and access port 9002; if a node fails, the VIP moves to another node.

In this case, also update global domain resolution and minio_endpoint so the mcli alias written to management nodes uses the new endpoint:

minio_endpoint: https://sss.pigsty:9002   # Override the default: https://sss.pigsty:9000
node_etc_hosts: ["10.10.10.9 sss.pigsty"] # Other nodes will use the sss.pigsty domain to access Silo

Dedicated Load Balancer

Pigsty allows using a dedicated load balancer server group instead of the cluster itself to run VIP and HAProxy. For example, the ha/simu template uses this approach.

proxy:
  hosts:
    10.10.10.18 : { nodename: proxy1 ,node_cluster: proxy ,vip_interface: eth1 ,vip_role: master }
    10.10.10.19 : { nodename: proxy2 ,node_cluster: proxy ,vip_interface: eth1 ,vip_role: backup }
  vars:
    vip_enabled: true
    vip_address: 10.10.10.20
    vip_vrid: 20

    haproxy_services:      # expose minio service : sss.pigsty:9000
      - name: minio        # [REQUIRED] service name, unique
        port: 9000         # [REQUIRED] service port, unique
        balance: leastconn # Use leastconn algorithm and minio health check
        options: [ "option httpchk", "option http-keep-alive", "http-check send meth OPTIONS uri /minio/health/live", "http-check expect status 200" ]
        servers:           # reload service with ./node.yml -t haproxy_config,haproxy_reload
          - { name: minio-1 ,ip: 10.10.10.21 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
          - { name: minio-2 ,ip: 10.10.10.22 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
          - { name: minio-3 ,ip: 10.10.10.23 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
          - { name: minio-4 ,ip: 10.10.10.24 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
          - { name: minio-5 ,ip: 10.10.10.25 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }

In this case, point sss.pigsty to the load balancer and update minio_endpoint so the mcli alias on management nodes uses that endpoint:

minio_endpoint: https://sss.pigsty:9002    # overwrite the defaults: https://sss.pigsty:9000
node_etc_hosts: ["10.10.10.20 sss.pigsty"] # domain name for accessing Silo from all nodes (required)

Access Service

To access the Silo service exposed through HAProxy from PGSQL, add a repository definition to pgbackrest_repo:

# New HA S3 repository definition, replacing the previous single-node configuration
minio_ha:
  type: s3
  s3_endpoint: minio-1.pigsty   # endpoint can be any load balancer or a domain pointing to one of the nodes
  s3_region: us-east-1          # you can use external domain name: sss.pigsty, which resolves to any member (`minio_domain`)
  s3_bucket: pgsql              # backup bucket name
  s3_key: pgbackrest            # use a dedicated password for Silo's pgbackrest user
  s3_key_secret: S3User.SomeNewPassWord
  s3_uri_style: path
  path: /pgbackrest
  storage_port: 9002            # Use load balancer port 9002 instead of default 9000 (direct access)
  storage_ca_file: /etc/pki/ca.crt
  bundle: y
  cipher_type: aes-256-cbc      # Better using a new cipher password for your production environment
  cipher_pass: pgBackRest.With.Some.Extra.PassWord.And.Salt.${pg_cluster}
  retention_full_type: time
  retention_full: 14

Expose Console

Silo provides an administration UI on port 9001 by default, controlled by minio_admin_port.

Exposing the administration interface externally may pose security risks. If required, add Silo to infra_portal and refresh the Nginx configuration.

# ./infra.yml -t nginx
infra_portal:
  home         : { domain: h.pigsty }
  # Object-storage administration UI requires HTTPS / WebSocket
  minio        : { domain: m.pigsty     ,endpoint: "10.10.10.10:9001" ,scheme: https ,websocket: true }
  minio10      : { domain: m10.pigsty   ,endpoint: "10.10.10.10:9001" ,scheme: https ,websocket: true }
  minio11      : { domain: m11.pigsty   ,endpoint: "10.10.10.11:9001" ,scheme: https ,websocket: true }
  minio12      : { domain: m12.pigsty   ,endpoint: "10.10.10.12:9001" ,scheme: https ,websocket: true }
  minio13      : { domain: m13.pigsty   ,endpoint: "10.10.10.13:9001" ,scheme: https ,websocket: true }

DO NOT expose an unencrypted object-storage administration UI in production.

You will usually need an m.pigsty record in DNS or local /etc/hosts to access the Silo administration page.

Meanwhile, if you are using Pigsty’s self-signed CA rather than a proper public CA, you usually need to manually trust the CA or certificate to skip the “insecure” warning in the browser.



12.3 - Parameters

The MINIO module exposes 22 parameters for deploying, configuring, and removing Silo object-storage clusters.

The MINIO module exposes 22 public parameters in two groups:

  • MINIO: 19 parameters for deploying Silo object-storage clusters
  • MINIO_REMOVE: 3 parameters controlling object-storage cluster removal
Architecture Change: Pigsty v3.6+

Since Pigsty v3.6, the minio.yml playbook no longer includes removal functionality. Removal-related parameters have been migrated to the dedicated minio_remove role and minio-rm.yml playbook.

Parameter Overview

The MINIO group configures a Silo object-storage cluster, including identity, storage paths, ports, credentials, and bucket/user provisioning.

Parameter Type Level Description
minio_type enum G/C Reserved backend selector; currently accepts only silo
minio_seq int I minio instance identifier, REQUIRED
minio_cluster string C Required object-storage cluster identity
minio_user username C minio os user, minio by default
minio_https bool G/C Enable HTTPS for object storage? true by default
minio_node string C minio node name pattern
minio_data path C minio data dir, use {x...y} for multiple disks
minio_volumes string C minio core parameter for nodes and disks, auto-gen
minio_domain string G minio external domain, sss.pigsty by default
minio_port port C minio service port, 9000 by default
minio_admin_port port C minio console port, 9001 by default
minio_access_key username C root access key, minioadmin by default
minio_secret_key password C root secret key, S3User.MinIO by default
minio_extra_vars string C extra environment variables for minio server
minio_provision bool G/C run minio provisioning tasks? true by default
minio_alias string G minio client alias for the deployment
minio_endpoint string C endpoint for the minio client alias
minio_buckets bucket[] C list of minio buckets to be created
minio_users user[] C list of minio users to be created

The MINIO_REMOVE group controls object-storage cluster removal, including safeguards, data cleanup, and package removal.

Parameter Type Level Description
minio_safeguard bool G/C/A prevent accidental removal? false by default
minio_rm_data bool G/C/A remove Silo data during removal? true by default
minio_rm_pkg bool G/C/A uninstall Silo and mcli? false by default

The minio_volumes and minio_endpoint are auto-generated parameters, but you can explicitly override them.

Defaults

MINIO: 19 public parameters, defined in roles/minio/defaults/main.yml

#-----------------------------------------------------------------
# SILO
#-----------------------------------------------------------------
minio_type: silo                  # reserved object-storage backend selector; currently accepts only silo
#minio_seq: 1                     # minio instance identifier, REQUIRED
#minio_cluster: minio             # required minio cluster identity
minio_user: minio                 # minio os user, `minio` by default
minio_https: true                 # enable HTTPS for Silo? true by default
minio_node: '${minio_cluster}-${minio_seq}.pigsty' # minio node name pattern
minio_data: '/data/minio'         # minio data dir, use `{x...y}` for multiple disks
#minio_volumes:                   # minio core parameter, auto-generated if not specified
minio_domain: sss.pigsty          # minio external domain, `sss.pigsty` by default
minio_port: 9000                  # minio service port, 9000 by default
minio_admin_port: 9001            # minio console port, 9001 by default
minio_access_key: minioadmin      # root access key, `minioadmin` by default
minio_secret_key: S3User.MinIO    # root secret key, `S3User.MinIO` by default
minio_extra_vars: ''              # extra environment variables for minio server
minio_provision: true             # run minio provisioning tasks?
minio_alias: sss                  # minio client alias for the deployment
#minio_endpoint: https://sss.pigsty:9000 # endpoint for alias, auto-generated if not specified
minio_buckets:                    # list of minio buckets to be created
  - { name: pgsql }
  - { name: meta ,versioning: true }
  - { name: data }
minio_users:                      # list of minio users to be created
  - { access_key: pgbackrest  ,secret_key: S3User.Backup ,policy: pgsql }
  - { access_key: s3user_meta ,secret_key: S3User.Meta   ,policy: meta  }
  - { access_key: s3user_data ,secret_key: S3User.Data   ,policy: data  }

MINIO_REMOVE: 3 parameters, defined in roles/minio_remove/defaults/main.yml

#-----------------------------------------------------------------
# MINIO_REMOVE
#-----------------------------------------------------------------
minio_safeguard: false            # prevent accidental removal? false by default
minio_rm_data: true               # remove minio data during removal? true by default
minio_rm_pkg: false               # uninstall minio packages during removal? false by default
# MINIO (reference)
minio_type: silo                  # object-storage engine; currently must be silo

MINIO

This section contains parameters for the minio role, used by the minio.yml playbook.

minio_type

Parameter: minio_type, Type: enum, Level: G/C

This reserved object-storage backend selector defaults to—and currently accepts only—silo. Silo retains the MinIO S3/Admin APIs, MINIO_* environment variables, and disk format.

minio and rustfs are no longer valid values and fail during role identity validation. Before upgrading a legacy MinIO cluster to v4.5, independently validate backups, MinIO-to-Silo data compatibility, and rollback; changing this parameter does not migrate data.

Both the deployment and removal roles default minio_type to silo. A minio-rm.yml run still requires the minio_cluster and minio_seq identity parameters and remains subject to minio_safeguard and the data/package cleanup switches; the engine default does not bypass these removal guards.

minio_seq

Parameter: minio_seq, Type: int, Level: I

Object-storage instance identifier, a required identity parameter. No default value—you must assign it manually.

Best practice is to start from 1, increment by 1, and never reuse previously assigned sequence numbers. The sequence number, together with the cluster name minio_cluster, uniquely identifies each object-storage instance (e.g., minio-1).

In multi-node deployments, sequence numbers are also used to generate node names, which are written to the /etc/hosts file for static resolution.

minio_cluster

Parameter: minio_cluster, Type: string, Level: C

Object-storage cluster name. This parameter is required and has no default. Use it to distinguish membership and monitoring identity when deploying multiple clusters.

The cluster name, together with the sequence number minio_seq, uniquely identifies each object-storage instance. For example, with cluster name minio and sequence 1, the instance name is minio-1.

The role finds members across the entire inventory by each host’s minio_cluster value, so the Ansible group name may differ from the cluster identity. Define this parameter explicitly in the object-storage group’s cluster variables; do not define it in all.vars, which would mark every host as a MINIO module member.

For multiple clusters, also set distinct minio_alias, minio_domain, and minio_endpoint values to avoid shared aliases and domain-name conflicts.

minio_user

Parameter: minio_user, Type: username, Level: C

Object-storage operating system user, default is minio.

Silo runs as this user, and its certificates are stored under ~/.minio/certs/.

minio_https

Parameter: minio_https, Type: bool, Level: G/C

Enable HTTPS for the object-storage service? Default is true.

Pigsty’s default pgBackRest minio repository preset uses HTTPS and validates the certificate with /etc/pki/ca.crt, so keep this parameter true when using the defaults. pgBackRest itself does not require Silo to use HTTPS; if you explicitly switch to HTTP, you must also update the storage TLS options in pgbackrest_repo rather than changing only this parameter.

When HTTPS is enabled, Pigsty automatically issues certificates for the selected server, containing the domain specified in minio_domain and the IP addresses of each node.

minio_node

Parameter: minio_node, Type: string, Level: C

Object-storage node-name pattern used for multi-node single-disk and multi-node multi-disk deployments.

Default value: ${minio_cluster}-${minio_seq}.pigsty, which uses the instance name plus .pigsty suffix as the default node name.

The domain pattern specified here generates node names, which are written to /etc/hosts on all Silo nodes.

minio_data

Parameter: minio_data, Type: path, Level: C

Silo data directory, default value: /data/minio. Set this parameter to a filesystem directory, not a raw block device such as /dev/sdb. The MINIO role creates the directory and sets its permissions, but does not format or mount production data drives.

Single-node single-disk may use a regular directory on the root filesystem for development. Multi-node single-disk, multi-node multi-disk, and single-node multi-disk deployments should use independent, persistent, non-root filesystems. Distributed Silo rejects data paths on the root filesystem.

/data/minio may be a subdirectory of a separately mounted /data filesystem. If /data is only a directory under /, it is still on the root drive. For multi-drive deployments, use {x...y} notation for multiple mount points, such as /data{1...4}/minio; every expanded path should map to a separate filesystem.

See Configuration: Storage Paths and Mounts for the complete requirements and verification commands.

minio_volumes

Parameter: minio_volumes, Type: string, Level: C

Silo core volume parameter. It is unset by default and generated with this rule:

minio_volumes: "{% if minio_cluster_size|int > 1 %}https://{{ minio_node|replace('${minio_cluster}', minio_cluster)|replace('${minio_seq}',minio_seq_range) }}:{{ minio_port|default(9000) }}{% endif %}{{ minio_data }}"
  • In single-node deployment (single or multi-drive), minio_volumes directly uses the minio_data value.
  • In multi-node deployment, minio_volumes uses minio_node, minio_port, and minio_data to generate multi-node addresses.
  • In multi-pool deployment, you typically need to explicitly specify and override minio_volumes to define multiple node pool addresses.

When specifying this parameter, ensure the values are consistent with minio_node, minio_port, and minio_data.

minio_domain

Parameter: minio_domain, Type: string, Level: G

Silo service domain name, default is sss.pigsty.

Clients can access the Silo S3 service through this domain. The name is included in the SAN (Subject Alternative Name) of certificates issued by the role, but the MINIO role does not automatically create a DNS record for minio_domain.

Add an explicit record through node_etc_hosts or dns_records, pointing it to a Silo node IP for a single-node deployment or to a load-balancer VIP for a multi-node deployment.

minio_port

Parameter: minio_port, Type: port, Level: C

Silo service port, default is 9000.

This is the Silo S3 API listening port. Clients access object storage through this port, which is also used for inter-node communication in multi-node deployments.

minio_admin_port

Parameter: minio_admin_port, Type: port, Level: C

Silo console port, default is 9001.

This is the listening port for Silo’s web management console, available at https://<minio-ip>:9001.

To expose the Silo console through Nginx, add it to infra_portal. The console requires HTTPS and WebSocket support.

minio_access_key

Parameter: minio_access_key, Type: username, Level: C

Root access key (username), default is minioadmin.

This is the Silo super-administrator username with full access to every bucket and object. Change this default in production.

minio_secret_key

Parameter: minio_secret_key, Type: password, Level: C

Root secret key (password), default is S3User.MinIO.

This is the Silo super-administrator password, used together with minio_access_key.

Security Warning: Change the default password!

Using default passwords is a high-risk behavior! Make sure to change this password in your production deployment.

Tip: ./configure -g randomizes default passwords recognized by the configuration wizard. See the Default Credentials Checklist for the complete scope.

minio_extra_vars

Parameter: minio_extra_vars, Type: string, Level: C

Extra environment variables passed to Silo. Silo retains the MINIO_* variable names.

Default is an empty string. You can use multiline strings to pass multiple environment variables:

minio_extra_vars: |
  MINIO_BROWSER_REDIRECT_URL=https://minio.example.com
  MINIO_SERVER_URL=https://s3.example.com

minio_provision

Parameter: minio_provision, Type: bool, Level: G/C

Run Silo provisioning tasks? Default is true.

When enabled, Pigsty automatically creates the buckets and users defined in minio_buckets and minio_users. Set this to false if you don’t need automatic provisioning of these resources.

minio_alias

Parameter: minio_alias, Type: string, Level: G

mcli client alias for the local Silo cluster, default value: sss.

When minio_provision is enabled, this alias is written to the mcli configuration file (~/.mcli/config.json) for the Ansible execution user on every Infra node and Silo member. Hosts in both groups are configured only once. You can then use mcli <alias> commands directly, for example mcli ls sss/.

If deploying multiple Silo clusters, specify a different alias for each cluster to avoid conflicts.

minio_endpoint

Parameter: minio_endpoint, Type: string, Level: C

Endpoint for the client alias. If specified, minio_endpoint (for example, https://sss.pigsty:9002) replaces the automatically assembled <scheme>://<minio_domain>:<minio_port> endpoint for aliases on Infra nodes and Silo members.

mcli alias set {{ minio_alias }} {% if minio_endpoint is defined and minio_endpoint != '' %}{{ minio_endpoint }}{% else %}{% if minio_https|bool %}https{% else %}http{% endif %}://{{ minio_domain }}:{{ minio_port }}{% endif %} {{ minio_access_key }} {{ minio_secret_key }}

The role runs this command as the Ansible execution user on Infra nodes and Silo members.

minio_buckets

Parameter: minio_buckets, Type: bucket[], Level: C

List of Silo buckets to create by default:

minio_buckets:
  - { name: pgsql }
  - { name: meta ,versioning: true }
  - { name: data }

Three default buckets are created with different purposes and policies:

  • pgsql bucket: Used by default for PostgreSQL pgBackREST backup storage.
  • meta bucket: Open bucket with versioning enabled, suitable for storing important metadata requiring version management.
  • data bucket: Open bucket for other purposes, e.g., Supabase templates may use this bucket for business data.

Each bucket has a corresponding access policy with the same name. For example, the pgsql policy has full access to the pgsql bucket, and so on.

You can also add a lock flag to bucket definitions to enable object locking, preventing accidental deletion of objects in the bucket.

minio_users

Parameter: minio_users, Type: user[], Level: C

List of Silo users to create, default value:

minio_users:
  - { access_key: pgbackrest  ,secret_key: S3User.Backup ,policy: pgsql }
  - { access_key: s3user_meta ,secret_key: S3User.Meta   ,policy: meta  }
  - { access_key: s3user_data ,secret_key: S3User.Data   ,policy: data  }

The default configuration creates three users corresponding to three default buckets:

  • pgbackrest: For PostgreSQL pgBackREST backups, with access to the pgsql bucket.
  • s3user_meta: For accessing the meta bucket.
  • s3user_data: For accessing the data bucket.
Using default passwords is dangerous! Make sure to change these credentials in your deployment!

Tip: ./configure -g will automatically replace these passwords in the configuration template if they appear as defaults.

MINIO_REMOVE

This section contains parameters for the minio_remove role, used by the minio-rm.yml playbook.

minio_safeguard

Parameter: minio_safeguard, Type: bool, Level: G/C/A

Safeguard switch to prevent accidental deletion, default value is false.

When enabled, the minio-rm.yml playbook aborts and refuses to remove the Silo cluster, protecting it against accidental deletion.

It’s recommended to enable this safeguard in production environments to prevent data loss from accidental operations:

minio_safeguard: true   # When enabled, minio-rm.yml will refuse to execute

minio_rm_data

Parameter: minio_rm_data, Type: bool, Level: G/C/A

Remove Silo data and configuration during removal? Default value is true.

When enabled, the minio-rm.yml playbook deletes data directories, /etc/default/silo, the .minio user directory, and /etc/systemd/system/silo.service. Setting it to false preserves data and configuration but does not prevent service deregistration, stopping, or disabling.

minio_rm_pkg

Parameter: minio_rm_pkg, Type: bool, Level: G/C/A

Uninstall Silo packages during removal? Default value is false.

When enabled, the minio-rm.yml playbook uninstalls silo and mcli. This option is disabled by default so the packages remain available for later use.

12.4 - Playbook

Deploy or remove Silo object-storage clusters with the built-in Ansible playbooks.

The MINIO module provides two built-in playbooks:

  • minio.yml: Install and configure Silo
  • minio-rm.yml: Remove Silo, its configuration, and optionally its data

minio.yml

minio.yml runs with hosts: all, but its pre-tasks skip hosts where minio_cluster is undefined. The role then validates that:

  • minio_cluster is defined and non-empty
  • minio_seq is defined and is a non-negative integer
  • minio_type must equal silo

Thus, minio_cluster is the module-membership gate, while invalid minio_seq or minio_type values fail identity validation explicitly. Do not define minio_cluster in all.vars.

The main task tags are:

  • minio-id: Validate identity and compute actual members, node names, and volume parameters from minio_cluster across the inventory
  • minio_install: Create the minio OS user, install Silo and mcli, and prepare data directories
    • minio_os_user
    • minio_pkg
    • minio_dir
  • minio_config: Render /etc/default/silo, /etc/systemd/system/silo.service, certificates, and DNS
    • minio_conf
    • minio_cert
    • minio_dns
  • minio_launch: Start or restart silo.service
  • minio_register: Write VictoriaMetrics FileSD targets
  • minio_provision: Have the cluster’s first member provision mcli aliases, buckets, and users once

Re-running minio.yml may restart a running object-storage service, but it does not proactively rebuild data. Schedule production runs according to the cluster’s failure budget.

minio-rm.yml

minio-rm.yml uses the same minio_cluster membership gate and identity validation, then runs:

  • minio_safeguard: Accidental-removal protection, default false
  • minio_pause: Pause for 3 seconds so you can abort with Ctrl+C
  • minio_deregister: Remove VictoriaMetrics targets and DNS records
  • minio_svc: Stop and disable the Silo service
  • minio_data: Delete data and configuration according to minio_rm_data
  • minio_pkg: Uninstall Silo and mcli according to minio_rm_pkg
Dangerous Operation

minio_rm_data defaults to true. A full removal run deletes every expanded minio_data directory. Before running it, verify minio_cluster, minio_seq, minio_type: silo, and all disk mount paths. To retire only the service while retaining data, explicitly pass -e minio_rm_data=false.

Both deployment and removal roles default minio_type to silo; other values are rejected. The removal examples still pass it explicitly as part of reviewing the package, service, certificate directory, and data paths; it is not an additional interactive confirmation gate.

Cheatsheet 

./minio.yml -l <group>                         # Deploy members with a minio_cluster identity in this limit
./minio.yml -l minio -t minio_install         # Install Silo and mcli; prepare directories
./minio.yml -l minio -t minio_config          # Re-render configuration, certificates, and DNS
./minio.yml -l minio -t minio_launch          # Restart the Silo service
./minio.yml -l minio -t minio_register        # Refresh monitoring targets
./minio.yml -l minio -t minio_provision       # Re-provision aliases, buckets, and users

./minio-rm.yml -l minio -e minio_type=silo                         # Remove Silo services, configuration, and data
./minio-rm.yml -l minio -e minio_type=silo -e minio_rm_data=false  # Remove services while preserving data and configuration
./minio-rm.yml -l minio -e minio_type=silo -e minio_rm_pkg=true    # Also uninstall Silo and mcli

If the configuration group name differs from minio_cluster, note that -l takes an Ansible group or host pattern, not the logical cluster name. Use a limit expression that covers every intended member.

Safeguard

For production clusters, enable accidental-removal protection in cluster variables:

minio_safeguard: true

After carefully verifying the target and backups, explicitly override it when destruction is required:

./minio-rm.yml -l minio -e minio_type=silo -e minio_safeguard=false

Demo

asciicast

12.5 - Administration

Create, remove, upgrade, expand, shrink, and recover Silo object-storage clusters.

Create Cluster

To create a cluster, define it in the config inventory and run the minio.yml playbook.

minio: { hosts: { 10.10.10.10: { minio_seq: 1 } }, vars: { minio_cluster: minio, minio_type: silo } }

The configuration above defines an SNSD Single-Node Single-Disk Silo cluster. Create it with:

./minio.yml -l minio  # Install Silo on the minio group

Remove Cluster

To destroy a cluster, run the dedicated minio-rm.yml playbook:

./minio-rm.yml -l minio -e minio_type=silo
./minio-rm.yml -l minio -e minio_type=silo -e minio_rm_data=false
./minio-rm.yml -l minio -e minio_type=silo -e minio_rm_pkg=true

The removal role also defaults minio_type to silo; other values are rejected. The examples still spell it out so the backend, cluster identity, and paths are visible during review.

Architecture Change: Pigsty v3.6+

Starting from Pigsty v3.6, cluster removal has been migrated from minio.yml playbook to the dedicated minio-rm.yml playbook. The old minio_clean task has been deprecated.

The removal playbook attempts these operations in order:

  • Deregisters object-storage targets from VictoriaMetrics monitoring
  • Removes records from the DNS service on INFRA nodes
  • Stops and disables silo.service
  • Deletes data directories and Silo configuration (minio_rm_data, enabled by default)
  • Uninstalls Silo and mcli packages (minio_rm_pkg, disabled by default)

The playbook tolerates errors. Its return status alone does not prove that the service, data, DNS records, and monitoring targets were all handled as intended; inspect each item after a real run.

Expand Cluster

This section uses the MinIO-compatible administration interfaces retained by Silo. Before a production operation, verify the constraints of the exact Silo version and complete a dedicated rehearsal.

Silo cannot directly change the node or disk count of an existing storage pool, but it can expand by adding a new pool.

Assume you have a four-node Silo cluster and want to double capacity by adding a new four-node storage pool.

minio:
  hosts:
    10.10.10.10: { minio_seq: 1 , nodename: minio-1 }
    10.10.10.11: { minio_seq: 2 , nodename: minio-2 }
    10.10.10.12: { minio_seq: 3 , nodename: minio-3 }
    10.10.10.13: { minio_seq: 4 , nodename: minio-4 }
  vars:
    minio_cluster: minio
    minio_type: silo
    minio_data: '/data{1...4}'
    minio_buckets: [ { name: pgsql }, { name: infra }, { name: redis } ]
    minio_users:
      - { access_key: dba , secret_key: S3User.DBA, policy: consoleAdmin }
      - { access_key: pgbackrest , secret_key: S3User.SomeNewPassWord , policy: readwrite }

    # bind a node l2 vip (10.10.10.9) to minio cluster (optional)
    node_cluster: minio
    vip_enabled: true
    vip_vrid: 128
    vip_address: 10.10.10.9
    vip_interface: eth1

    # expose minio service with haproxy on all nodes
    haproxy_services:
      - name: minio                    # [REQUIRED] service name, unique
        port: 9002                     # [REQUIRED] service port, unique
        balance: leastconn             # [OPTIONAL] load balancer algorithm
        options:                       # [OPTIONAL] minio health check
          - option httpchk
          - option http-keep-alive
          - http-check send meth OPTIONS uri /minio/health/live
          - http-check expect status 200
        servers:
          - { name: minio-1 ,ip: 10.10.10.10 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
          - { name: minio-2 ,ip: 10.10.10.11 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
          - { name: minio-3 ,ip: 10.10.10.12 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
          - { name: minio-4 ,ip: 10.10.10.13 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }

First, modify the Silo cluster definition to add four new nodes, assigning sequence numbers 5 to 8. The key step is to modify the minio_volumes parameter to designate the new four nodes as a new storage pool.

minio:
  hosts:
    10.10.10.10: { minio_seq: 1 , nodename: minio-1 }
    10.10.10.11: { minio_seq: 2 , nodename: minio-2 }
    10.10.10.12: { minio_seq: 3 , nodename: minio-3 }
    10.10.10.13: { minio_seq: 4 , nodename: minio-4 }
    # new nodes
    10.10.10.14: { minio_seq: 5 , nodename: minio-5 }
    10.10.10.15: { minio_seq: 6 , nodename: minio-6 }
    10.10.10.16: { minio_seq: 7 , nodename: minio-7 }
    10.10.10.17: { minio_seq: 8 , nodename: minio-8 }

  vars:
    minio_cluster: minio
    minio_type: silo
    minio_data: '/data{1...4}'
    minio_volumes: 'https://minio-{1...4}.pigsty:9000/data{1...4} https://minio-{5...8}.pigsty:9000/data{1...4}'  # new cluster config
    # ... other configs omitted

Step 2: Add these nodes to Pigsty:

./node.yml -l 10.10.10.14,10.10.10.15,10.10.10.16,10.10.10.17

Step 3: On the new nodes, use the Ansible playbook to install and prepare Silo:

./minio.yml -l 10.10.10.14,10.10.10.15,10.10.10.16,10.10.10.17 -t minio_install

Step 4: On the entire cluster, use the Ansible playbook to reconfigure Silo:

./minio.yml -l minio -t minio_config

This step updates the MINIO_VOLUMES configuration on the existing four nodes

Step 5: Restart the entire Silo cluster at once (do not use a rolling restart):

./minio.yml -l minio -t minio_launch -f 10   # Up to 10 forks; restart all 8 nodes together

Step 6 (optional): If you are using a load balancer, make sure the load balancer configuration is updated. For example, add the new four nodes to the load balancer configuration:

# expose minio service with haproxy on all nodes
haproxy_services:
  - name: minio                    # [REQUIRED] service name, unique
    port: 9002                     # [REQUIRED] service port, unique
    balance: leastconn             # [OPTIONAL] load balancer algorithm
    options:                       # [OPTIONAL] minio health check
      - option httpchk
      - option http-keep-alive
      - http-check send meth OPTIONS uri /minio/health/live
      - http-check expect status 200
    servers:
      - { name: minio-1 ,ip: 10.10.10.10 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
      - { name: minio-2 ,ip: 10.10.10.11 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
      - { name: minio-3 ,ip: 10.10.10.12 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
      - { name: minio-4 ,ip: 10.10.10.13 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }

      - { name: minio-5 ,ip: 10.10.10.14 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
      - { name: minio-6 ,ip: 10.10.10.15 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
      - { name: minio-7 ,ip: 10.10.10.16 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }
      - { name: minio-8 ,ip: 10.10.10.17 ,port: 9000 ,options: 'check-ssl ca-file /etc/pki/ca.crt check port 9000' }

Then, run the haproxy subtask of the node.yml playbook to update the load balancer configuration:

./node.yml -l minio -t haproxy_config,haproxy_reload   # Update and reload load balancer config

If you use L2 VIP for reliable load balancer access, you also need to add new nodes (if any) to the existing NODE VIP group:

./node.yml -l minio -t node_vip  # Refresh cluster L2 VIP configuration

Shrink Cluster

Silo cannot directly reduce the node or disk count of an existing storage pool. To shrink at the pool level, add a new pool, drain the old pool into it, and then retire the old pool.

Upgrade Cluster

First, download the new silo and mcli packages to the local repository on the INFRA node, then rebuild repository metadata with SOW:

./infra.yml -t repo_create

Next, upgrade the Silo server and compatible mcli client:

ansible minio -m package -b -a 'name=silo state=latest'  # Server
ansible minio -m package -b -a 'name=mcli state=latest'  # Compatible client

Finally, have the role restart the complete Silo cluster:

./minio.yml -l minio -t minio_config,minio_launch

A package upgrade and migration from legacy MinIO to Silo are different operations. The former applies to a cluster already running Silo; the latter requires separate data-compatibility validation, backups, a maintenance window, and a rehearsed rollback plan. Do not reuse this upgrade procedure for migration.

Node Failure Recovery 

# 1. Remove the failed node from the cluster
bin/node-rm <your_old_node_ip>

# 2. Replace the failed node and keep its original name (if the IP changes, update the Silo cluster definition)
bin/node-add <your_new_node_ip>

# 3. Install and configure Silo on the new node
./minio.yml -l <your_new_node_ip>

# 4. Instruct Silo to heal
mcli admin heal

Disk Failure Recovery 

# 1. Unmount the failed disk from the cluster
umount /dev/<your_disk_device>

# 2. Replace the failed disk, format with xfs
mkfs.xfs /dev/sdb -L DRIVE1

# 3. Don't forget to setup fstab for auto-mount
vi /etc/fstab
# LABEL=DRIVE1     /mnt/drive1    xfs     defaults,noatime  0       2

# 4. Remount
mount -a

# 5. Instruct Silo to heal
mcli admin heal

Manage Silo Passwords

minio_secret_key, which defaults to S3User.MinIO, is the Silo root password. It is rendered to /etc/default/silo.

After changing it, refresh configuration and restart the entire cluster:

./minio.yml -l minio -t minio_config,minio_launch,minio_alias -f 30

To change a regular Silo user password, such as pgbackrest, run this on a node that can access Silo:

set +o history
mcli admin user passwd sss pgbackrest <YOUR_NEW_PASSWORD>
set -o history

Then update every consumer of that password. For example, if pgBackRest uses the minio S3-compatible repository preset, refresh its configuration with:

./pgsql.yml -t pgbackrest_config

12.6 - Monitoring and Alerting

How Pigsty monitors Silo, including its Metrics V3 endpoint, Grafana dashboards, and alert rules.

Administration UI

Silo provides an administration UI through minio_admin_port, which defaults to 9001. Access it directly at https://<node-ip>:9001.

Some configuration templates also expose the administration entry point at m.pigsty. Login credentials come from minio_access_key and minio_secret_key.

HTTPS and Certificate Trust

Object storage uses HTTPS certificates issued by the Pigsty CA by default. Browsers and container clients must trust that CA. Do not substitute disabled certificate verification for a correct trust configuration in production.

Collection Paths

Silo retains the stable job="minio", cls, ins, ip, and instance identity labels and uses flavor="silo":

Backend Metric Path Target and Labels
Silo VictoriaMetrics scrapes https://<instance>:9000/minio/metrics/v3 job=minio, flavor=silo

Each instance’s FileSD target is written to /infra/targets/minio/<minio_cluster>-<minio_seq>.yml.

Silo registers one Metrics V3 root endpoint, which exposes cluster, system, API, and aggregated usage metrics. Pigsty drops samples with a non-empty bucket label and does not register separate per-bucket or replication endpoints, keeping time-series cardinality under control.

Grafana Dashboards

Pigsty provides the compatibility-named MinIO Overview / MinIO Instance dashboards for Silo Metrics V3 data, system logs, and instance state.

minio-overview.jpg

Alert Rules

The current files/victoria/rules/minio.yml defines five alerts for Silo:

Alert Condition Summary Severity
MinioServerDown minio_up < 1 for 1 minute CRIT
MinioNodeOffline Five-minute average offline-node count above 0 for 3 minutes WARN
MinioDiskOffline Five-minute average offline-drive count above 0 for 3 minutes WARN
MinioErasureSetUnhealthy Any erasure set’s overall health below 1 for 1 minute CRIT
MinioClusterCapacityHigh Usable capacity utilization above 90% for 15 minutes WARN

Key expressions use Metrics V3 names:

minio_up < 1

max by (cls) (
  avg_over_time(minio_cluster_health_nodes_offline_count{job="minio"}[5m])
) > 0

max by (cls) (
  avg_over_time(minio_cluster_health_drives_offline_count{job="minio"}[5m])
) > 0

min by (cls) (
  minio_cluster_erasure_set_overall_health{job="minio"}
  or (minio_cluster_erasure_set_overall_write_quorum{job="minio"} * 0)
) < 1

max by (cls) (
  1 - (
    (minio_cluster_health_capacity_usable_free_bytes{job="minio"}
     or (minio_cluster_health_capacity_usable_total_bytes{job="minio"} * 0))
    / minio_cluster_health_capacity_usable_total_bytes{job="minio"}
  )
) > 0.90

12.7 - Metrics

The Metrics V3 interface, key metrics, and stable labels used by the Pigsty MINIO module to monitor Silo.

The MINIO module collects Silo metrics through /minio/metrics/v3. The metric set varies with server versions and enabled features, so this page documents the stable interfaces used by current dashboards and alerts rather than treating a complete scrape from one version as a permanent contract.

Stable Identity Labels

All object-storage targets use these Pigsty labels:

Label Meaning Example
job Fixed module namespace minio
flavor Actual backend silo
cls minio_cluster identity minio
ins <minio_cluster>-<minio_seq> instance identity minio-1
ip Inventory management address 10.10.10.10
instance Metric target address 10.10.10.10:9000

Queries and recording rules should prefer the stable cls, ins, and ip identity labels.

Silo Metrics V3

Each Silo instance exposes only the V3 root endpoint, /minio/metrics/v3. Current key metrics are:

Category Key Metrics Meaning
Liveness minio_up Pigsty scrape/health state for the instance
Nodes minio_cluster_health_nodes_online_count, minio_cluster_health_nodes_offline_count Online and offline nodes
Drives minio_cluster_health_drives_online_count, minio_cluster_health_drives_offline_count Online and offline drives
Capacity minio_cluster_health_capacity_raw_total_bytes Raw total capacity
Capacity minio_cluster_health_capacity_usable_total_bytes, minio_cluster_health_capacity_usable_free_bytes Usable total and free capacity
Objects minio_cluster_usage_objects_count, minio_cluster_usage_objects_total_bytes Object count and used bytes
Buckets minio_cluster_usage_objects_buckets_count Aggregated bucket count
Erasure coding minio_cluster_erasure_set_overall_health, minio_cluster_erasure_set_overall_write_quorum Erasure-set health and write quorum
API minio_api_requests_total, minio_api_requests_errors_total, minio_api_requests_4xx_errors_total API requests and errors
API minio_api_requests_inflight_total, minio_api_requests_incoming_total In-flight and incoming requests
Traffic minio_api_requests_traffic_received_bytes, minio_api_requests_traffic_sent_bytes Received and sent bytes
Latency minio_api_requests_ttfb_seconds_distribution Time-to-first-byte distribution
Process minio_system_process_cpu_total_seconds, minio_system_process_resident_memory_bytes Process CPU and resident memory
System minio_system_drive_free_bytes, minio_system_drive_used_bytes, minio_system_drive_health Per-drive capacity and health
Audit minio_audit_total_messages Audit-message count

Pigsty drops samples whose bucket label is non-empty at scrape time and does not register dedicated per-bucket or replication endpoints. This is an intentional cardinality-control policy. If per-bucket metrics are required, evaluate the time-series volume before adding a custom scrape job.

12.8 - FAQ

Frequently asked questions about the Pigsty MINIO object storage module

Which backend does the MINIO module deploy by default?

In v4.5.0, the current source deploys Silo—and only Silo. The only valid value for minio_type is silo. MINIO remains the compatibility module name; it does not mean the MinIO server is running.

  • Explicitly set minio_type: silo for new clusters.
  • Both minio_type: minio and minio_type: rustfs fail during identity validation.
  • External MinIO, RustFS, or other S3 services can still serve as pgBackRest repositories, but the current MINIO role does not manage them.
  • Before upgrading a MinIO cluster managed by an older release, validate the MinIO-to-Silo data compatibility, backup, and rollback procedure.

Why does the Pigsty repository still carry MinIO or RustFS packages?

Upstream MinIO switched to source-only distribution on 2025-10-15, marked the repository as maintenance mode on 2025-12-03, and archived it on 2026-04-25. Here, “source-only distribution” means that new prebuilt community binaries stopped being published—not merely RPM and DEB packages.

Pigsty therefore previously maintained its own MinIO fork and packages. MinIO CVE-2025-62506 affects releases before RELEASE.2025-10-15T17-29-55Z and is fixed in that release; both Pigsty’s later MinIO fork and the current Silo code include the fix.

The Pigsty Infra repository still carries MinIO/RustFS RPM and DEB packages plus their build scripts, but repository availability does not mean that the v4.5 MINIO module supports those backends. The current role accepts only Silo; other services must be deployed and maintained separately.

Why is HTTPS enabled for object storage by default?

Pigsty’s default pgBackRest minio repository configuration uses HTTPS and verifies the certificate through /etc/pki/ca.crt to protect backup traffic. pgBackRest does not categorically forbid HTTP. If you explicitly choose HTTP, you must update the TLS options in pgbackrest_repo as well as disable minio_https; changing only the server-side switch is insufficient.

Getting an invalid certificate error when accessing Silo from containers?

By default, the object-storage server certificate is issued by Pigsty’s private CA. It is not a self-signed server certificate, but container images usually do not trust this private CA, so clients such as mcli, rclone, and AWS CLI report an invalid certificate chain.

For example, for a Node.js application, mount the Pigsty CA certificate into the container and specify its path through NODE_EXTRA_CA_CERTS:

    environment:
      NODE_EXTRA_CA_CERTS: /etc/pki/ca.crt
    volumes:
      - /etc/pki/ca.crt:/etc/pki/ca.crt:ro

If Silo is not used as a pgBackRest backup repository, you can disable HTTPS and use HTTP instead, but you should also assess the risk of cleartext transport.

Can a Silo data path be a regular directory?

minio_data takes a directory path, not a raw disk device. /data/minio may be a regular subdirectory, but in multi-node or multi-drive deployments its backing storage must be an independent, persistent, non-root filesystem.

  • If /data is mounted from a separate local disk, cloud volume, partition, or LVM logical volume, /data/minio is valid.
  • If /data/minio is only a directory created under the root filesystem, distributed Silo marks it as a root drive and rejects it with drive is part of root drive, will not be used.
  • Every path in a single-node multi-drive deployment should map to a separate filesystem. Multiple directories on one drive do not emulate multiple drives.
  • Only single-node single-disk may use a regular directory directly on the root filesystem, and only for development, testing, or non-critical use.

Inspect the backing mounts with:

findmnt -T /
findmnt -T /data/minio

See Configuration: Storage Paths and Mounts for details and Multi-Node Single-Disk for the three-node topology.

How do I add new members to an existing Silo cluster?

Plan Silo cluster capacity before deployment because adding a storage pool requires a global restart.

Scale the cluster by adding a group of server nodes as a new storage pool.

You cannot directly change the node or disk count of an existing storage pool; expansion requires adding a new pool.

For the procedure, see Pigsty’s Expand Cluster guide and the upstream Expand MinIO Deployment reference for the compatible interface retained by Silo.

How do I remove a Silo cluster?

Starting with Pigsty v3.6, cluster removal uses the dedicated minio-rm.yml playbook:

./minio-rm.yml -l minio -e minio_type=silo
./minio-rm.yml -l minio -e minio_type=silo -e minio_rm_data=false

The removal role also defaults minio_type to silo; other values are rejected. The examples still spell it out so the operator can review it together with cluster identity and paths before deletion.

minio_rm_data defaults to true, and the removal role tolerates some cleanup errors. Before a real run, verify the exact -l target and a recent backup. Afterwards, inspect the service, data directories, DNS records, and monitoring targets; the playbook’s return status alone does not prove that cleanup completed.

If you have enabled minio_safeguard protection, you need to explicitly override it to perform removal:

./minio-rm.yml -l minio -e minio_type=silo -e minio_safeguard=false

What’s the difference between mcli and mc commands?

Pigsty ships the compatible MinIO client under the mcli command and package name instead of upstream’s mc, avoiding a name collision with the Midnight Commander file manager.

mcli is Pigsty’s delivery name for the compatible client and retains the mc CLI, although the exact version may change with Pigsty packaging. See the MinIO Client documentation for the command reference.

How do I monitor Silo cluster status?

Pigsty provides out-of-the-box monitoring for Silo. Dashboard and metric names retain MinIO-compatible naming:

  • Grafana Dashboards: MinIO Overview and MinIO Instance
  • Alerting Rules: MinIO-compatible server-down, node-offline, and disk-offline alerts
  • Silo Built-in Console: Access via https://<minio-ip>:9001

For details, see Monitoring.

13 - Module: REDIS

Deploy Redis or Valkey through one REDIS module, with standalone replication, native cluster, and Sentinel modes.

REDIS is Pigsty’s Redis-compatible cache module. Choose Redis or Valkey with redis_type; the default remains redis. Both engines support primary-replica replication, Sentinel, and native cluster mode while sharing configuration paths, instance service names, monitoring, and log entry points.

redis_type: redis   # default; valkey is also supported

The role installs the selected engine and redis-exporter. Instance processes use redis-server / redis-cli or valkey-server / valkey-cli, respectively. Changing redis_type changes packages and binaries; it does not automatically validate data formats, replication topology, or rollback. Rehearse any existing-cluster switch, and use one engine consistently across a logical cluster.

The default Redis package remains on the 7.2 BSD branch. Patch versions can differ by operating-system repository; treat the target repository metadata as authoritative.

13.1 - Configuration

Choose the appropriate Redis mode for your use case and express your requirements through the inventory

Concept

The entity model of Redis is almost the same as that of PostgreSQL, which also includes the concepts of Cluster and Instance. Note that the Cluster here does not refer to the native Redis Cluster mode.

The core difference between the REDIS module and the PGSQL module is that Redis uses a single-node multi-instance deployment rather than the 1:1 deployment: multiple Redis instances are typically deployed on a physical/virtual machine node to utilize multi-core CPUs fully. Therefore, the ways to configure and administer Redis instances are slightly different from PGSQL.

In Redis managed by Pigsty, nodes are entirely subordinate to the cluster, which means that currently, it is not allowed to deploy Redis instances of two different clusters on one node. However, this does not affect deploying multiple independent Redis primary-replica instances on one node. Of course, there are some limitations; for example, in this case, you cannot specify different passwords for different instances on the same node.

Choose the server implementation with redis_type: redis by default, or valkey. Set it consistently at cluster level. The role switches packages and the *-server / *-cli binaries while retaining /etc/redis, /data/redis, instance systemd unit names, and the redis monitoring namespace. Validate data compatibility and rollback independently before changing an existing cluster’s engine.

Identity Parameters

Redis identity parameters are required parameters when defining a Redis cluster.

Name Attribute Description Example
redis_cluster REQUIRED, cluster level Cluster name redis-test
redis_node REQUIRED, node level Node sequence number 1,2
redis_instances REQUIRED, node level Instance definition { 6001 : {} ,6002 : {}}
  • redis_cluster: Redis cluster name, serves as the top-level namespace for cluster resources.
  • redis_node: Redis node number, an integer unique within the cluster to distinguish different nodes.
  • redis_instances: JSON object where keys are instance port numbers and values are JSON objects containing other instance configurations.

Redis Mode

There are three different working modes for Redis, specified by the redis_mode parameter:

  • standalone: Default standalone master-slave mode
  • cluster: Redis native distributed cluster mode
  • sentinel: Sentinel mode, providing high availability for standalone master-slave Redis

Here are three examples of Redis cluster definitions:

  • A 1-node, one master & one slave Redis Standalone cluster: redis-ms
  • A 1-node, 3-instance Redis Sentinel cluster: redis-sentinel
  • A 2-node, 6-instance Redis Cluster: redis-cluster
redis-ms: # redis classic primary & replica
  hosts: { 10.10.10.10: { redis_node: 1 , redis_instances: { 6379: { }, 6380: { replica_of: '10.10.10.10 6379' } } } }
  vars: { redis_cluster: redis-ms ,redis_password: 'redis.ms' ,redis_max_memory: 64MB }

redis-meta: # redis sentinel x 3
  hosts: { 10.10.10.11: { redis_node: 1 , redis_instances: { 26379: { } ,26380: { } ,26381: { } } } }
  vars:
    redis_cluster: redis-meta
    redis_password: 'redis.meta'
    redis_mode: sentinel
    redis_max_memory: 16MB
    redis_sentinel_monitor: # primary list for redis sentinel, use cls as name, primary ip:port
      - { name: redis-ms, host: 10.10.10.10, port: 6379 ,password: redis.ms, quorum: 2 }

redis-test: # redis native cluster: 3m x 3s
  hosts:
    10.10.10.12: { redis_node: 1 ,redis_instances: { 6379: { } ,6380: { } ,6381: { } } }
    10.10.10.13: { redis_node: 2 ,redis_instances: { 6379: { } ,6380: { } ,6381: { } } }
  vars: { redis_cluster: redis-test ,redis_password: 'redis.test' ,redis_mode: cluster, redis_max_memory: 32MB }

These examples omit redis_type and therefore use the default Redis engine. To deploy Valkey, add redis_type: valkey to the corresponding cluster’s vars; do not mix engines within one logical cluster.

Limitations

  • A Redis node can only belong to one Redis cluster, which means you cannot assign a node to two different Redis clusters simultaneously.
  • On each Redis node, you need to assign a unique port number to each Redis instance to avoid port conflicts.
  • Typically, the same Redis cluster will use the same password, but multiple Redis instances on a Redis node cannot have different passwords (because redis_exporter only allows one password).
  • Redis Cluster has built-in HA, while standalone master-slave HA requires additional manual configuration in Sentinel since we don’t know if you have deployed Sentinel.
  • Fortunately, configuring HA for standalone Redis is straightforward through Sentinel. For details, see Administration - Configure HA with Sentinel.

Typical Configuration Examples

Here are some common Redis configuration examples for different scenarios:

Cache Cluster (Pure In-Memory)

For pure caching scenarios with no data persistence requirements:

redis-cache:
  hosts:
    10.10.10.10: { redis_node: 1 , redis_instances: { 6379: { }, 6380: { } } }
    10.10.10.11: { redis_node: 2 , redis_instances: { 6379: { }, 6380: { } } }
  vars:
    redis_cluster: redis-cache
    redis_password: 'cache.password'
    redis_max_memory: 2GB
    redis_mem_policy: allkeys-lru    # evict LRU keys when memory is full
    redis_rdb_save: []               # disable RDB persistence
    redis_aof_enabled: false         # disable AOF persistence

Session Store Cluster

For web application session storage with some persistence needs:

redis-session:
  hosts:
    10.10.10.10: { redis_node: 1 , redis_instances: { 6379: { }, 6380: { replica_of: '10.10.10.10 6379' } } }
  vars:
    redis_cluster: redis-session
    redis_password: 'session.password'
    redis_max_memory: 1GB
    redis_mem_policy: volatile-lru   # only evict keys with expire set
    redis_rdb_save: ['300 1']        # save every 5 minutes if at least 1 change
    redis_aof_enabled: false

Message Queue Cluster

For simple message queue scenarios requiring higher data reliability:

redis-queue:
  hosts:
    10.10.10.10: { redis_node: 1 , redis_instances: { 6379: { }, 6380: { replica_of: '10.10.10.10 6379' } } }
  vars:
    redis_cluster: redis-queue
    redis_password: 'queue.password'
    redis_max_memory: 4GB
    redis_mem_policy: noeviction     # reject writes when memory full, don't evict
    redis_rdb_save: ['60 1']         # save every minute if at least 1 change
    redis_aof_enabled: true          # enable AOF for better persistence

High Availability Master-Slave Cluster

Master-slave cluster with Sentinel automatic failover:

# Master-slave cluster
redis-ha:
  hosts:
    10.10.10.10: { redis_node: 1 , redis_instances: { 6379: { } } }                              # primary
    10.10.10.11: { redis_node: 2 , redis_instances: { 6379: { replica_of: '10.10.10.10 6379' } } } # replica 1
    10.10.10.12: { redis_node: 3 , redis_instances: { 6379: { replica_of: '10.10.10.10 6379' } } } # replica 2
  vars:
    redis_cluster: redis-ha
    redis_password: 'ha.password'
    redis_max_memory: 8GB

# Sentinel cluster (manages the above master-slave cluster)
redis-sentinel:
  hosts:
    10.10.10.10: { redis_node: 1 , redis_instances: { 26379: { } } }
    10.10.10.11: { redis_node: 2 , redis_instances: { 26379: { } } }
    10.10.10.12: { redis_node: 3 , redis_instances: { 26379: { } } }
  vars:
    redis_cluster: redis-sentinel
    redis_password: 'sentinel.password'
    redis_mode: sentinel
    redis_max_memory: 64MB
    redis_sentinel_monitor:
      - { name: redis-ha, host: 10.10.10.10, port: 6379, password: 'ha.password', quorum: 2 }

Large-Scale Native Cluster

For high-volume, high-throughput scenarios using native distributed cluster:

redis-cluster:
  hosts:
    10.10.10.10: { redis_node: 1 , redis_instances: { 6379: { }, 6380: { }, 6381: { } } }
    10.10.10.11: { redis_node: 2 , redis_instances: { 6379: { }, 6380: { }, 6381: { } } }
    10.10.10.12: { redis_node: 3 , redis_instances: { 6379: { }, 6380: { }, 6381: { } } }
    10.10.10.13: { redis_node: 4 , redis_instances: { 6379: { }, 6380: { }, 6381: { } } }
  vars:
    redis_cluster: redis-cluster
    redis_password: 'cluster.password'
    redis_mode: cluster
    redis_cluster_replicas: 1        # 1 replica per primary shard
    redis_max_memory: 16GB           # max memory per instance
    redis_rdb_save: ['900 1']
    redis_aof_enabled: false

# This creates a 6-primary, 6-replica native cluster
# Total capacity ~96GB (6 * 16GB)

Security Hardening Configuration

Recommended security configuration for production environments:

redis-secure:
  hosts:
    10.10.10.10: { redis_node: 1 , redis_instances: { 6379: { } } }
  vars:
    redis_cluster: redis-secure
    redis_password: 'StrongP@ssw0rd!'  # use strong password
    redis_bind_address: ''             # bind to internal IP instead of 0.0.0.0
    redis_max_memory: 4GB
    redis_rename_commands:             # rename dangerous commands
      FLUSHDB: 'DANGEROUS_FLUSHDB'
      FLUSHALL: 'DANGEROUS_FLUSHALL'
      DEBUG: ''                        # disable command
      CONFIG: 'ADMIN_CONFIG'

13.2 - Parameters

The REDIS module provides 19 deployment and 3 removal parameters, with Redis or Valkey as the engine.

The REDIS module has 22 parameters: 19 for Redis/Valkey deployment and configuration, and 3 for removal.

Parameter Overview

The REDIS parameter group is used for Redis cluster deployment and configuration, including identity, instance definitions, operating mode, memory configuration, persistence, and monitoring.

Parameter Type Level Description
redis_cluster string C Redis cluster name, required identity parameter
redis_instances dict I Redis instance definitions on this node
redis_node int I Redis node number, unique positive integer in cluster
redis_fs_main path C Redis main data directory, /data/redis by default
redis_exporter_enabled bool C Enable Redis Exporter?
redis_exporter_port port C Redis Exporter listen port
redis_exporter_options string C/I Redis Exporter CLI arguments
redis_type enum G/C Server engine: redis (default) or valkey
redis_mode enum C Redis mode: standalone, cluster, sentinel
redis_conf string C Redis config template, except sentinel
redis_bind_address ip C Redis bind address, defaults to 0.0.0.0; empty uses host IP
redis_max_memory size C/I Max memory for each Redis instance
redis_mem_policy enum C Redis memory eviction policy
redis_password password C Redis password, empty disables password
redis_rdb_save string[] C Redis RDB save directives, empty list disables RDB
redis_aof_enabled bool C Enable Redis AOF?
redis_rename_commands dict C Rename dangerous Redis commands
redis_cluster_replicas int C Replicas per master in Redis native cluster
redis_sentinel_monitor master[] C Master list for Redis Sentinel to monitor

The REDIS_REMOVE parameter group controls Redis instance removal behavior.

Parameter Type Level Description
redis_safeguard bool G/C/A Refuse removal unconditionally when true
redis_rm_data bool G/C/A Remove Redis data directory when removing?
redis_rm_pkg bool G/C/A Uninstall the selected engine and redis-exporter?

The REDIS module contains 19 deployment parameters and 3 removal parameters.

#redis_cluster:             <CLUSTER> # Redis cluster name, required identity parameter
#redis_node: 1              <NODE>    # Redis node number, unique in cluster
#redis_instances: {}        <NODE>    # Redis instance definitions on this node
redis_fs_main: /data/redis            # Redis main data directory, `/data/redis` by default
redis_exporter_enabled: true          # Enable Redis Exporter?
redis_exporter_port: 9121             # Redis Exporter listen port
redis_exporter_options: ''            # Redis Exporter CLI arguments
redis_type: redis                     # Server engine: redis or valkey
redis_mode: standalone                # Redis mode: standalone, cluster, sentinel
redis_conf: redis.conf                # Redis config template, except sentinel
redis_bind_address: '0.0.0.0'         # Redis bind address, defaults to `0.0.0.0`; empty uses host IP
redis_max_memory: 1GB                 # Max memory for each Redis instance
redis_mem_policy: allkeys-lru         # Redis memory eviction policy
redis_password: ''                    # Redis password, empty disables password
redis_rdb_save: ['1200 1']            # Redis RDB save directives, empty disables RDB
redis_aof_enabled: false              # Enable Redis AOF?
redis_rename_commands: {}             # Rename dangerous Redis commands
redis_cluster_replicas: 1             # Replicas per master in Redis native cluster
redis_sentinel_monitor: []            # Master list for Sentinel, sentinel mode only

# REDIS_REMOVE
redis_safeguard: false                # Refuse removal unconditionally when true
redis_rm_data: true                   # Remove Redis data directory when removing?
redis_rm_pkg: false                   # Uninstall the selected engine and redis-exporter?

redis_cluster

Parameter: redis_cluster, Type: string, Level: C

Redis cluster name, a required identity parameter that must be explicitly configured at the cluster level. It serves as the namespace for resources within the cluster.

Must follow the naming pattern [a-z][a-z0-9-]* to comply with various identity constraints. Using redis- as a cluster name prefix is recommended.

redis_node

Parameter: redis_node, Type: int, Level: I

Redis node sequence number, a required identity parameter that must be explicitly configured at the node (Host) level.

A positive integer that should be unique within the cluster, used to distinguish and identify different nodes. Assign starting from 0 or 1.

redis_instances

Parameter: redis_instances, Type: dict, Level: I

Redis instance definitions on the current node, a required parameter that must be explicitly configured at the node (Host) level.

Format is a JSON key-value object where keys are numeric port numbers and values are instance-specific JSON configuration items.

redis-test: # redis native cluster: 3m x 3s
  hosts:
    10.10.10.12: { redis_node: 1 ,redis_instances: { 6379: { } ,6380: { } ,6381: { } } }
    10.10.10.13: { redis_node: 2 ,redis_instances: { 6379: { } ,6380: { } ,6381: { } } }
  vars: { redis_cluster: redis-test ,redis_password: 'redis.test' ,redis_mode: cluster, redis_max_memory: 32MB }

Each Redis instance listens on a unique port on its node. The replica_of field in instance configuration sets the upstream master address to establish replication:

redis_instances:
    6379: {}
    6380: { replica_of: '10.10.10.13 6379' }
    6381: { replica_of: '10.10.10.13 6379' }

redis_fs_main

Parameter: redis_fs_main, Type: path, Level: C

Main data directory for Redis, default is /data/redis.

Deployment does not allow the legacy value /data (redis role identity assert fails fast). For backward compatibility during removal, redis-rm.yml treats redis_fs_main=/data as /data/redis.

The data directory is owned by the redis OS user. See FHS: Redis for internal structure details.

redis_exporter_enabled

Parameter: redis_exporter_enabled, Type: bool, Level: C

Enable Redis Exporter monitoring component?

Enabled by default, deploying one exporter per Redis node, listening on redis_exporter_port 9121 by default. It scrapes metrics from all Redis instances on the node.

When set to false, roles/redis/tasks/exporter.yml still renders config files but skips starting the redis_exporter systemd service (the redis_exporter_launch task has when: redis_exporter_enabled|bool), allowing manually configured exporters to remain. redis_register still writes this node’s VictoriaMetrics file-discovery target. If you do not provide your own exporter on the same port, handle that target as well to avoid continuous scrape failures.

redis_exporter_port

Parameter: redis_exporter_port, Type: port, Level: C

Redis Exporter listen port, default value: 9121

redis_exporter_options

Parameter: redis_exporter_options, Type: string, Level: C/I

Extra CLI arguments for Redis Exporter, rendered to /etc/default/redis_exporter (see roles/redis/tasks/exporter.yml), default is empty string. REDIS_EXPORTER_OPTS is appended to the systemd service’s ExecStart=/bin/redis_exporter $REDIS_EXPORTER_OPTS, useful for configuring extra scrape targets or filtering behavior.

redis_type

Parameter: redis_type, Type: enum, Level: G/C

Select the server implementation used by the REDIS module. Allowed values are redis and valkey; the default is redis.

The role installs the package with the selected name and calls /bin/redis-server / /bin/redis-cli or /bin/valkey-server / /bin/valkey-cli from instance systemd units. Configuration paths, data directories, instance service names, exporter behavior, and monitoring labels retain the redis namespace for compatibility with existing inventories and operational entry points.

Set the same value for every member at cluster level. Changing redis_type only changes the package and binaries selected by the role; it does not validate cross-version RDB/AOF, replication, Sentinel, or Cluster compatibility. Rehearse the change and prepare a rollback before switching an existing cluster.

redis_mode

Parameter: redis_mode, Type: enum, Level: C

Redis cluster operating mode, three options: standalone, cluster, sentinel. Default: standalone

  • standalone: Default, independent Redis master-slave mode
  • cluster: Redis native cluster mode
  • sentinel: Redis high availability component: Sentinel

When using standalone mode, Pigsty sets up Redis replication based on the replica_of parameter.

When using cluster mode, Pigsty creates a native Redis cluster using all defined instances based on the redis_cluster_replicas parameter.

When redis_mode=sentinel, redis.yml runs the redis-ha phase to distribute targets from redis_sentinel_monitor to all sentinels. When redis_mode=cluster, it also runs redis-join, using the redis-cli or valkey-cli selected by the engine to execute --cluster create. Both phases run automatically during a normal ./redis.yml -l <cluster> and can also be selected with -t redis-ha or -t redis-join.

redis_conf

Parameter: redis_conf, Type: string, Level: C

Redis config template path, except for Sentinel.

Default: redis.conf, a template file at roles/redis/templates/redis.conf.

To use your own Redis config template, place it in the templates/ directory and set this parameter to the template filename.

Note: Redis Sentinel uses a different template file: roles/redis/templates/redis-sentinel.conf.

redis_bind_address

Parameter: redis_bind_address, Type: ip, Level: C

IP address Redis server binds to. Empty string uses the hostname defined in the inventory.

Default: 0.0.0.0, binding to all available IPv4 addresses on the host.

For security in production environments, bind only to internal IPs by setting this to empty string ''.

When empty, the template roles/redis/templates/redis.conf uses inventory_hostname to render bind <ip>, binding to the management address declared in the inventory.

redis_max_memory

Parameter: redis_max_memory, Type: size, Level: C/I

Maximum memory for each Redis instance, default: 1GB.

redis_mem_policy

Parameter: redis_mem_policy, Type: enum, Level: C

Redis memory eviction policy, default: allkeys-lru

  • noeviction: Don’t save new values when memory limit is reached; only applies to primary when using replication
  • allkeys-lru: Keep most recently used keys; remove least recently used (LRU) keys
  • allkeys-lfu: Keep frequently used keys; remove least frequently used (LFU) keys
  • volatile-lru: Remove least recently used keys with expire field set
  • volatile-lfu: Remove least frequently used keys with expire field set
  • allkeys-random: Randomly remove keys to make space for new data
  • volatile-random: Randomly remove keys with expire field set
  • volatile-ttl: Remove keys with expire field set and shortest remaining TTL

See Redis Eviction Policy for details.

redis_password

Parameter: redis_password, Type: password, Level: C/N

Redis password. Empty string disables password, which is the default behavior.

Note that due to redis_exporter implementation limitations, you can only set one redis_password per node. This is usually not a problem since Pigsty doesn’t allow deploying two different Redis clusters on the same node.

Pigsty automatically writes this password to /etc/default/redis_exporter (REDIS_PASSWORD=...) and passes it through REDISCLI_AUTH to the redis-cli / valkey-cli selected by redis-ha and redis-join, keeping the password out of command-line arguments.

Use a strong password in production environments

redis_rdb_save

Parameter: redis_rdb_save, Type: string[], Level: C

Redis RDB save directives. Use empty list to disable RDB.

Default is ["1200 1"]: dump dataset to disk every 20 minutes if at least 1 key changed.

See Redis Persistence for details.

redis_aof_enabled

Parameter: redis_aof_enabled, Type: bool, Level: C

Enable Redis AOF? Default is false, meaning AOF is not used.

redis_rename_commands

Parameter: redis_rename_commands, Type: dict, Level: C

Rename dangerous Redis commands. A k:v dictionary where old is the command to rename and new is the new name.

Default: {}. You can hide dangerous commands like FLUSHDB and FLUSHALL. Example:

{
  "keys": "op_keys",
  "flushdb": "op_flushdb",
  "flushall": "op_flushall",
  "config": "op_config"
}

redis_cluster_replicas

Parameter: redis_cluster_replicas, Type: int, Level: C

Number of replicas per master/primary in Redis native cluster. Default: 1, meaning one replica per master.

redis_sentinel_monitor

Parameter: redis_sentinel_monitor, Type: master[], Level: C

List of masters for Redis Sentinel to monitor, used only on sentinel clusters. Each managed master is defined as:

redis_sentinel_monitor:  # primary list for redis sentinel, use cls as name, primary ip:port
  - { name: redis-src, host: 10.10.10.45, port: 6379 ,password: redis.src, quorum: 1 }
  - { name: redis-dst, host: 10.10.10.48, port: 6379 ,password: redis.dst, quorum: 1 }

name and host are required; port, password, and quorum are optional. quorum sets the number of sentinels needed to agree on master failure, typically more than half of sentinel instances (default is 1).

Starting from Pigsty 4.0, you can add remove: true to an entry, causing the redis-ha phase to only execute SENTINEL REMOVE <name>, useful for cleaning up targets no longer needed.

REDIS_REMOVE

The following parameters are used by the redis_remove role, invoked by the redis-rm.yml playbook, controlling Redis instance removal behavior.

redis_safeguard

Parameter: redis_safeguard, Type: bool, Level: G/C/A

Redis deletion safeguard, default false. When set to true, redis-rm.yml aborts before deregistration, service shutdown, or deletion. This is a static Boolean switch and does not probe whether a Redis instance is running.

Override with CLI argument -e redis_safeguard=false to force removal.

redis_rm_data

Parameter: redis_rm_data, Type: bool, Level: G/C/A

Remove Redis data directory when removing Redis instances? Default is true.

The data directory (default /data/redis/, i.e. redis_fs_main) contains Redis RDB and AOF files. If not removed, newly deployed Redis instances will load data from these backup files.

Set to false to preserve data directories for later recovery.

redis_rm_pkg

Parameter: redis_rm_pkg, Type: bool, Level: G/C/A

When removing a Redis node, also uninstall the engine selected by redis_type and the redis-exporter package? Default is false. Removing a single instance with redis_port never uninstalls shared packages.

Typically not needed to uninstall packages; only enable when completely cleaning up a node.

13.3 - Playbook

Manage Redis clusters with Ansible playbooks and quick command reference.

The REDIS module provides two playbooks for deploying/removing Redis clusters/nodes/instances:

redis.yml

The redis.yml playbook for deploying Redis contains the following subtasks:

redis_node        : Init redis node
  - redis_install : Install Redis and the `redis-exporter` package
  - redis_user    : Create OS user redis
  - redis_dir     : Configure redis FHS directory structure
redis_exporter    : Configure redis_exporter monitoring
  - redis_exporter_config  : Generate redis_exporter config
  - redis_exporter_launch  : Launch redis_exporter
redis_instance    : Init and restart redis cluster/node/instance
  - redis_config  : Generate redis instance config
  - redis_launch  : Launch redis instance
redis_register    : Register redis to infrastructure
redis_ha          : Configure redis sentinel (sentinel mode only)
redis_join        : Join redis native cluster (cluster mode only)

Operation Levels

redis.yml supports three operation levels, controlled by -l to limit target scope and -e redis_port=<port> to specify a single instance:

Level Parameters Description
Cluster -l <cluster> Deploy all nodes and instances of the entire Redis cluster
Node -l <ip> Deploy all Redis instances on the specified node
Instance -l <ip> -e redis_port=<port> Deploy only a single instance on the specified node

Cluster-Level Operations

Deploy an entire Redis cluster, including all instances on all nodes:

./redis.yml -l redis-ms           # deploy the entire redis-ms cluster
./redis.yml -l redis-test         # deploy the entire redis-test cluster
./redis.yml -l redis-sentinel     # deploy sentinel cluster

Cluster-level operations will:

  • Install Redis or Valkey according to redis_type, plus redis-exporter, on all nodes
  • Create redis user and directory structure on all nodes
  • Start redis_exporter on all nodes
  • Deploy and start all defined Redis instances
  • Register all instances to the monitoring system
  • If sentinel mode, configure sentinel monitoring targets
  • If cluster mode, form the native cluster

Node-Level Operations

Deploy only all Redis instances on the specified node:

./redis.yml -l 10.10.10.10        # deploy all instances on this node
./redis.yml -l 10.10.10.11        # deploy another node

Node-level operations are useful for:

  • Scaling up by adding new nodes to an existing cluster
  • Redeploying all instances on a specific node
  • Reinitializing after node failure recovery

Note: Node-level commands still enter the redis-ha / redis-join mode checks. Sentinel mode refreshes managed targets. In cluster mode, the playbook first uses the selected CLI to check the seed instance for cluster_state:ok; it exits for a healthy cluster and otherwise runs --cluster create. This guard does not replace scale-out: use redis-cli / valkey-cli --cluster add-node and reshard manually for an existing native cluster.

Instance-Level Operations

Use the -e redis_port=<port> parameter to operate on a single instance:

# Deploy only the 6379 port instance on 10.10.10.10
./redis.yml -l 10.10.10.10 -e redis_port=6379

# Deploy only the 6380 port instance on 10.10.10.11
./redis.yml -l 10.10.10.11 -e redis_port=6380

Instance-level operations are useful for:

  • Adding new instances to an existing node
  • Redeploying a single failed instance
  • Updating a single instance’s configuration

When redis_port is specified:

  • Only renders the config file for that port
  • Only starts/restarts the systemd service for that port
  • Rewrites the node’s monitoring registration file (content comes from the full redis_instances definition)
  • Does not start/stop redis_exporter or reload Vector log config
  • Does not affect other Redis instance processes on the same node

Common Tags

Use the -t <tag> parameter to selectively execute certain tasks:

# Install packages only, don't start services
./redis.yml -l redis-ms -t redis_node

# Update config and restart instances only
./redis.yml -l redis-ms -t redis_config,redis_launch

# Update monitoring registration only
./redis.yml -l redis-ms -t redis_register

# Configure sentinel monitoring targets only (sentinel mode)
./redis.yml -l redis-sentinel -t redis-ha

# Form native cluster only (cluster mode, auto-runs after first deployment)
./redis.yml -l redis-cluster -t redis-join

Idempotency

Most tasks in redis.yml can be run repeatedly, but native-cluster initialization still requires attention to topology state:

  • Re-running redis_node / redis_exporter / redis_instance / redis_register overwrites config and restarts instances
  • Re-running redis-ha reapplies SENTINEL REMOVE/MONITOR based on redis_sentinel_monitor
  • redis-join first checks whether the seed instance has reached cluster_state:ok and exits for a healthy cluster. The check does not repair incomplete or damaged topologies or perform scale-out, so do not treat it as a general add-node/reshard operation.

Tip: If you only want to update configs without restarting all instances, use -t redis_config to render configs only, then manually restart the instances you need.

Redis/Valkey units use Type=notify. During startup, systemd waits up to 1800s for readiness so large RDB/AOF loads and recovery can finish. A timeout is not a reason to broaden data deletion or use a forced stop; inspect instance logs, data size, disk I/O, and memory first.

redis-rm.yml

The redis-rm.yml playbook for removing Redis contains the following subtasks:

redis_safeguard  : Safety check, abort if redis_safeguard=true
redis_deregister : Remove registration from monitoring system
  - rm_metrics   : Delete /infra/targets/redis/*.yml
  - rm_logs      : Revoke /etc/vector/redis.yaml
redis_exporter   : Stop and disable redis_exporter
redis            : Stop and disable redis instances
redis_data       : Delete data directories (when redis_rm_data=true)
redis_pkg        : Uninstall selected engine and redis-exporter (when redis_rm_pkg=true)

Tag-scoped execution follows the data/package switches. -t redis always enters the instance-stop phase; -t redis_data stops instances only when redis_rm_data=true, and -t redis_pkg stops them only when redis_rm_pkg=true. Thus -t redis_data -e redis_rm_data=false and -t redis_pkg -e redis_rm_pkg=false do not stop Redis merely because the tag was selected. Before any real removal, verify the exact same -l, tags, and extra variables.

Operation Levels

redis-rm.yml also supports three operation levels:

Level Parameters Description
Cluster -l <cluster> Remove all nodes and instances of the entire Redis cluster
Node -l <ip> Remove all Redis instances on the specified node
Instance -l <ip> -e redis_port=<port> Remove only a single instance on the specified node

Cluster-Level Removal

Remove an entire Redis cluster:

./redis-rm.yml -l redis-ms        # remove the entire redis-ms cluster
./redis-rm.yml -l redis-test      # remove the entire redis-test cluster

Cluster-level removal will:

  • Deregister all instances on all nodes from the monitoring system
  • Stop redis_exporter on all nodes
  • Stop and disable all Redis instances
  • Delete all data directories (if redis_rm_data=true)
  • Uninstall the engine selected by redis_type and redis-exporter (if redis_rm_pkg=true)

Node-Level Removal

Remove only all Redis instances on the specified node:

./redis-rm.yml -l 10.10.10.10     # remove all instances on this node
./redis-rm.yml -l 10.10.10.11     # remove another node

Node-level removal is useful for:

  • Scaling down by removing an entire node
  • Cleanup before node decommission
  • Preparation before node migration

Node-level removal will:

  • Deregister all instances on that node from the monitoring system
  • Stop redis_exporter on that node
  • Stop all Redis instances on that node
  • Delete all data directories on that node
  • Delete Vector logging config on that node

Instance-Level Removal

Use the -e redis_port=<port> parameter to remove a single instance:

# Remove only the 6379 port instance on 10.10.10.10
./redis-rm.yml -l 10.10.10.10 -e redis_port=6379

# Remove only the 6380 port instance on 10.10.10.11
./redis-rm.yml -l 10.10.10.11 -e redis_port=6380

Instance-level removal is useful for:

  • Removing a single replica from a node
  • Removing instances no longer needed
  • Removing the original primary after failover

Behavioral differences when redis_port is specified:

Component Node-Level (no redis_port) Instance-Level (with redis_port)
Monitoring registration Delete entire node’s registration file Only remove that instance from registration file
redis_exporter Stop and disable No operation (other instances still need it)
Redis instances Stop all instances Only stop the specified port’s instance
Data directory Delete entire redis_fs_main (default: /data/redis/) Only delete redis_fs_main/<cluster>-<node>-<port>/ (if redis_fs_main=/data, removal is compat-mapped to /data/redis)
Vector config Delete /etc/vector/redis.yaml No operation (other instances still need it)
Packages Optionally uninstall No operation

Control Parameters

redis-rm.yml provides the following control parameters:

Parameter Default Description
redis_safeguard false Safety guard; when true, refuses to execute removal
redis_rm_data true Whether to delete data directories (RDB/AOF files)
redis_rm_pkg false Whether to uninstall the selected engine and redis-exporter

Usage examples:

# Remove cluster but keep data directories
./redis-rm.yml -l redis-ms -e redis_rm_data=false

# Remove cluster and uninstall packages
./redis-rm.yml -l redis-ms -e redis_rm_pkg=true

# Override only after verifying the target and backups when the inventory enables the safeguard
./redis-rm.yml -l redis-ms -e redis_safeguard=false
Destructive Operation

redis_safeguard defaults to false, while redis_rm_data defaults to true. The removal playbook also tolerates several service-stop, deregistration, data-deletion, and package-removal errors. After a real run, inspect the target processes, data directories, and monitoring registration; do not treat the playbook return status alone as proof of completion.

Safeguard Mechanism

When a cluster has redis_safeguard: true configured, redis-rm.yml will refuse to execute:

redis-production:
  vars:
    redis_safeguard: true    # enable protection for production
$ ./redis-rm.yml -l redis-production
TASK [ABORT due to redis_safeguard enabled] ***
fatal: [10.10.10.10]: FAILED! => {"msg": "Abort due to redis_safeguard..."}

Explicit override is required to execute:

./redis-rm.yml -l redis-production -e redis_safeguard=false

Quick Reference

Deployment Quick Reference 

# Deploy entire cluster
./redis.yml -l <cluster>

# Scale up: deploy new node (then manually add-node in cluster mode)
./redis.yml -l <new-node-ip>

# Scale up: add new instance to existing node (add definition to config first)
./redis.yml -l <ip> -e redis_port=<new-port>

# Update config and restart
./redis.yml -l <cluster> -t redis_config,redis_launch

# Update single instance config only
./redis.yml -l <ip> -e redis_port=<port> -t redis_config,redis_launch

Removal Quick Reference 

# Remove entire cluster
./redis-rm.yml -l <cluster>

# Scale down: remove entire node
./redis-rm.yml -l <ip>

# Scale down: remove single instance
./redis-rm.yml -l <ip> -e redis_port=<port>

# Remove but keep data
./redis-rm.yml -l <cluster> -e redis_rm_data=false

# Complete cleanup (including packages)
./redis-rm.yml -l <cluster> -e redis_rm_pkg=true

Wrapper Scripts

Pigsty provides convenient wrapper scripts:

# Deploy
bin/redis-add <cluster>           # deploy cluster
bin/redis-add <ip>                # deploy node
bin/redis-add <ip> <port>         # deploy instance

# Remove
bin/redis-rm <cluster>            # remove cluster
bin/redis-rm <ip>                 # remove node
bin/redis-rm <ip> <port>          # remove instance

Demo

Initialize Redis cluster with Redis playbook:

asciicast

13.4 - Administration

Redis cluster management SOPs for creating, destroying, scaling, and configuring high availability

Here are some common Redis administration task SOPs (Standard Operating Procedures):

The REDIS module defaults to redis_type: redis. With redis_type: valkey, the server and client commands become valkey-server and valkey-cli. Examples on this page use the default redis-cli; substitute valkey-cli for Valkey clusters. Playbooks choose the correct CLI automatically.

Basic Operations

High Availability

Scaling & Migration

Troubleshooting

For more questions, please refer to FAQ: REDIS.

Initialize Redis

You can use the redis.yml playbook to initialize Redis clusters, nodes, or instances:

# Initialize all Redis instances in the cluster
./redis.yml -l <cluster>      # init redis cluster

# Initialize all Redis instances on a specific node
./redis.yml -l 10.10.10.10    # init redis node

# Initialize a specific Redis instance: 10.10.10.11:6379
./redis.yml -l 10.10.10.11 -e redis_port=6379 -t redis

You can also use wrapper scripts to initialize:

bin/redis-add redis-ms          # create redis cluster 'redis-ms'
bin/redis-add 10.10.10.10       # create redis node '10.10.10.10'
bin/redis-add 10.10.10.10 6379  # create redis instance '10.10.10.10:6379'

Remove Redis

You can use the redis-rm.yml playbook to remove Redis clusters, nodes, or instances:

redis_rm_data defaults to true. Verify RDB/AOF backups and the current primary/replica, Sentinel, or cluster topology, then have the operator confirm the exact target. The commands below perform the corresponding removal directly.

# Remove Redis cluster `redis-test`
./redis-rm.yml -l redis-test

# Remove Redis cluster `redis-test` and uninstall the selected engine and exporter
./redis-rm.yml -l redis-test -e redis_rm_pkg=true

# Remove all instances on Redis node 10.10.10.13
./redis-rm.yml -l 10.10.10.13

# Remove a specific Redis instance 10.10.10.13:6379
./redis-rm.yml -l 10.10.10.13 -e redis_port=6379

You can also use wrapper scripts to remove Redis clusters/nodes/instances:

bin/redis-rm redis-ms          # remove redis cluster 'redis-ms'
bin/redis-rm 10.10.10.10       # remove redis node '10.10.10.10'
bin/redis-rm 10.10.10.10 6379  # remove redis instance '10.10.10.10:6379'

Reconfigure Redis

You can partially run the redis.yml playbook to reconfigure Redis clusters, nodes, or instances:

./redis.yml -l <cluster> -t redis_config,redis_launch

Note that Redis cannot reload configuration online. You must restart Redis using the launch task to make configuration changes take effect.

Using Redis Client

Use redis-cli with the default Redis engine. Valkey uses valkey-cli with the same arguments:

$ redis-cli -h 10.10.10.10 -p 6379 # <--- connect with host and port
10.10.10.10:6379> auth redis.ms    # <--- authenticate with password
OK
10.10.10.10:6379> set a 10         # <--- set a key
OK
10.10.10.10:6379> get a            # <--- get the key value
"10"

Redis provides the redis-benchmark tool, which can be used for Redis performance evaluation or to generate load for testing.

redis-benchmark -h 10.10.10.13 -p 6379

Configure Redis Replica

https://redis.io/commands/replicaof/

# Promote a Redis instance to primary
> REPLICAOF NO ONE
"OK"

# Make a Redis instance a replica of another instance
> REPLICAOF 127.0.0.1 6379
"OK"

Configure HA with Sentinel

Redis standalone master-slave clusters can be configured for automatic high availability through Redis Sentinel. For detailed information, please refer to the Sentinel official documentation.

Using the four-node sandbox environment as an example, a Redis Sentinel cluster redis-meta can be used to manage multiple standalone Redis master-slave clusters.

Taking the one-master-one-slave Redis standalone cluster redis-ms as an example, you need to add the target on each Sentinel instance using SENTINEL MONITOR and provide the password using SENTINEL SET, and the high availability is configured.

# For each sentinel, add the redis master to sentinel management: (26379,26380,26381)
$ redis-cli -h 10.10.10.11 -p 26379 -a redis.meta
10.10.10.11:26379> SENTINEL MONITOR redis-ms 10.10.10.10 6379 1
10.10.10.11:26379> SENTINEL SET redis-ms auth-pass redis.ms      # if auth enabled, password needs to be configured

If you want to remove a Redis master-slave cluster managed by Sentinel, use SENTINEL REMOVE <name>.

You can use the redis_sentinel_monitor parameter defined on the Sentinel cluster to automatically configure the list of masters managed by Sentinel.

redis_sentinel_monitor:  # list of masters to be monitored, port, password, quorum (should be more than 1/2 of sentinels) are optional
  - { name: redis-src, host: 10.10.10.45, port: 6379 ,password: redis.src, quorum: 1 }
  - { name: redis-dst, host: 10.10.10.48, port: 6379 ,password: redis.dst, quorum: 1 }

The redis-ha stage in redis.yml will render /tmp/<cluster>.monitor on each sentinel instance based on this list and execute SENTINEL REMOVE and SENTINEL MONITOR commands sequentially, ensuring the sentinel management state remains consistent with the inventory. If you only want to remove a target without re-adding it, set remove: true on the monitor object, and the playbook will skip re-registration after SENTINEL REMOVE.

Use the following command to refresh the managed master list on the Redis Sentinel cluster:

./redis.yml -l redis-meta -t redis-ha   # replace redis-meta if your Sentinel cluster has a different name

Initialize Redis Native Cluster

When redis_mode is cluster, redis.yml also runs the redis-join stage. It uses the CLI selected by redis_type to execute --cluster create --cluster-yes ... --cluster-replicas {{ redis_cluster_replicas }} and assemble all inventory instances into a native cluster.

This step runs automatically during initial deployment. A later ./redis.yml -l <cluster> -t redis-join first checks the seed instance for cluster_state:ok and exits when the cluster is healthy. This guard does not perform add-node, resharding, or repair of a partially initialized topology; verify topology state before triggering the stage separately.

Scale Up Redis Nodes

Scale Up Standalone Cluster

When adding new nodes/instances to an existing Redis master-slave cluster, first add the new definition in the inventory:

redis-ms:
  hosts:
    10.10.10.10: { redis_node: 1 , redis_instances: { 6379: { }, 6380: { replica_of: '10.10.10.10 6379' } } }
    10.10.10.11: { redis_node: 2 , redis_instances: { 6379: { replica_of: '10.10.10.10 6379' } } }  # new node
  vars: { redis_cluster: redis-ms ,redis_password: 'redis.ms' ,redis_max_memory: 64MB }

Then deploy only the new node:

./redis.yml -l 10.10.10.11   # deploy only the new node

Scale Up Native Cluster

Adding new nodes to a Redis native cluster requires additional steps:

# 1. Add the new node definition in the inventory
# 2. Deploy the new node
./redis.yml -l 10.10.10.14

# 3. Add the new node to the cluster (manual execution)
redis-cli --cluster add-node 10.10.10.14:6379 10.10.10.12:6379

# 4. Reshard slots if needed
redis-cli --cluster reshard 10.10.10.12:6379

Scale Up Sentinel Cluster

After adding new instances to a Sentinel cluster, you should complete both instance deployment and target refresh:

# 1. Add new Sentinel instances to inventory, then deploy instances
./redis.yml -l <sentinel-cluster> -t redis_instance

# 2. Re-apply redis_sentinel_monitor to all sentinels
./redis.yml -l <sentinel-cluster> -t redis-ha

Scale Down Redis Nodes

Scale Down Standalone Cluster 

# 1. If removing a replica, just remove it directly
./redis-rm.yml -l 10.10.10.11 -e redis_port=6379

# 2. If removing the primary, first perform a failover
redis-cli -h 10.10.10.10 -p 6380 REPLICAOF NO ONE      # promote replica
redis-cli -h 10.10.10.10 -p 6379 REPLICAOF 10.10.10.10 6380  # demote original primary

# 3. Then remove the original primary
./redis-rm.yml -l 10.10.10.10 -e redis_port=6379

# 4. Update the inventory to remove the definition

Scale Down Native Cluster 

# 1. First migrate data slots
redis-cli --cluster reshard 10.10.10.12:6379 \
  --cluster-from <node-id> --cluster-to <target-node-id> --cluster-slots <count>

# 2. Remove node from cluster
redis-cli --cluster del-node 10.10.10.12:6379 <node-id>

# 3. Remove the instance
./redis-rm.yml -l 10.10.10.14

# 4. Update the inventory

Backup and Restore

Manual Backup 

# Trigger RDB snapshot
redis-cli -h 10.10.10.10 -p 6379 -a <password> BGSAVE

# Check snapshot status
redis-cli -h 10.10.10.10 -p 6379 -a <password> LASTSAVE

# Copy RDB file (default location)
cp /data/redis/redis-ms-1-6379/dump.rdb /backup/redis-ms-$(date +%Y%m%d).rdb

Data Restore 

# 1. Stop Redis instance
sudo systemctl stop redis-ms-1-6379

# 2. Replace RDB file
cp /backup/redis-ms-20241231.rdb /data/redis/redis-ms-1-6379/dump.rdb
chown redis:redis /data/redis/redis-ms-1-6379/dump.rdb

# 3. Start Redis instance
sudo systemctl start redis-ms-1-6379

Using AOF Persistence

If you need higher data safety, enable AOF:

redis-ms:
  vars:
    redis_aof_enabled: true
    redis_rdb_save: ['900 1', '300 10', '60 10000']  # keep RDB as well

Redeploy to apply AOF configuration:

./redis.yml -l redis-ms -t redis_config,redis_launch

Common Issue Diagnosis

Connection Troubleshooting 

# Check Redis service status
systemctl status redis-ms-1-6379

# Check port listening
ss -tlnp | grep 6379

# Check firewall
sudo iptables -L -n | grep 6379

# Test connection
redis-cli -h 10.10.10.10 -p 6379 PING

Memory Troubleshooting 

# Check memory usage
redis-cli -h 10.10.10.10 -p 6379 INFO memory

# Find big keys
redis-cli -h 10.10.10.10 -p 6379 --bigkeys

# Memory analysis report
redis-cli -h 10.10.10.10 -p 6379 MEMORY DOCTOR

Performance Troubleshooting 

# Check slow query log
redis-cli -h 10.10.10.10 -p 6379 SLOWLOG GET 10

# Real-time command monitoring
redis-cli -h 10.10.10.10 -p 6379 MONITOR

# Check client connections
redis-cli -h 10.10.10.10 -p 6379 CLIENT LIST

Replication Troubleshooting 

# Check replication status
redis-cli -h 10.10.10.10 -p 6379 INFO replication

# Check replication lag
redis-cli -h 10.10.10.10 -p 6380 INFO replication | grep lag

Performance Tuning

Memory Optimization 

redis-cache:
  vars:
    redis_max_memory: 4GB           # set based on available memory
    redis_mem_policy: allkeys-lru   # LRU recommended for cache scenarios
    redis_conf: redis.conf

Persistence Optimization 

# Pure cache scenario: disable persistence
redis-cache:
  vars:
    redis_rdb_save: []              # disable RDB
    redis_aof_enabled: false        # disable AOF

# Data safety scenario: enable both RDB and AOF
redis-data:
  vars:
    redis_rdb_save: ['900 1', '300 10', '60 10000']
    redis_aof_enabled: true

Connection Pool Recommendations

When connecting to Redis from client applications:

  • Use connection pooling to avoid frequent connection creation
  • Set reasonable timeout values (recommended 1-3 seconds)
  • Enable TCP keepalive
  • For high-concurrency scenarios, consider using Pipeline for batch operations

Key Monitoring Metrics

Monitor these metrics through Grafana dashboards:

  • Memory usage: Pay attention when redis:ins:mem_usage > 80%
  • CPU usage: Pay attention when redis:ins:cpu_usage > 70%
  • QPS: Watch for spikes and abnormal fluctuations
  • Response time: Investigate when redis:ins:rt > 1ms
  • Connection count: Monitor connection growth trends
  • Replication lag: Important for master-slave replication scenarios

13.5 - Monitoring

How to monitor Redis? What alert rules are worth paying attention to?

Dashboards

The REDIS module provides 3 monitoring dashboards:

  • Redis Overview: Overview of all Redis clusters
  • Redis Cluster: Details of a single Redis cluster
  • Redis Instance: Details of a single Redis instance

Monitoring

Pigsty provides three monitoring dashboards for the REDIS module:

Redis Overview

Redis Overview: Overview of all Redis clusters/instances

redis-overview.jpg

Redis Cluster

Redis Cluster: Details of a single Redis cluster

Redis Cluster Dashboard

redis-cluster.jpg

Redis Instance

Redis Instance: Details of a single Redis instance

Redis Instance Dashboard

redis-instance

Alert Rules

Pigsty provides the following six predefined alert rules for Redis, defined in files/victoria/rules/redis.yml:

  • RedisDown: Redis instance is down
  • RedisRejectConn: Redis instance rejecting connections
  • RedisRTHigh: Redis instance response time is too high
  • RedisCPUHigh: Redis instance CPU usage is too high
  • RedisMemHigh: Redis instance memory usage is too high
  • RedisQPSHigh: Redis instance QPS is too high

The rule expr is authoritative: response time >160µs for 1 minute, CPU and memory usage >70% for 1 minute, and QPS >32000 for 5 minutes. The source excerpt below reflects the current rule file verbatim. Its CPU, memory, and QPS descriptions still contain the old 60%, 80%, and 16000 thresholds, and the RedisRTHigh comment incorrectly names pg:ins:query_rt; these comments do not change the actual expressions.

#==============================================================#
#                         Error                                #
#==============================================================#
# redis down triggers a P0 alert
- alert: RedisDown
  expr: redis_up < 1
  for: 1m
  labels: { level: 0, severity: CRIT, category: redis }
  annotations:
    summary: "CRIT RedisDown: {{ $labels.ins }} {{ $labels.instance }} {{ $value }}"
    description: |
      redis_up[ins={{ $labels.ins }}, instance={{ $labels.instance }}] = {{ $value }} == 0
      /ui/d/redis-instance?from=now-5m&to=now&var-ins={{$labels.ins}}

# redis reject connection in last 5m
- alert: RedisRejectConn
  expr: redis:ins:conn_reject > 0
  labels: { level: 0, severity: CRIT, category: redis }
  annotations:
    summary: "CRIT RedisRejectConn: {{ $labels.ins }} {{ $labels.instance }} {{ $value }}"
    description: |
      redis:ins:conn_reject[cls={{ $labels.cls }}, ins={{ $labels.ins }}][5m] = {{ $value }} > 0
      /ui/d/redis-instance?from=now-10m&to=now&viewPanel=88&fullscreen&var-ins={{ $labels.ins }}



#==============================================================#
#                         Latency                              #
#==============================================================#
# redis avg query response time > 160 µs
- alert: RedisRTHigh
  expr: redis:ins:rt > 0.00016
  for: 1m
  labels: { level: 1, severity: WARN, category: redis }
  annotations:
    summary: "WARN RedisRTHigh: {{ $labels.cls }} {{ $labels.ins }}"
    description: |
      pg:ins:query_rt[cls={{ $labels.cls }}, ins={{ $labels.ins }}] = {{ $value }} > 160µs
      /ui/d/redis-instance?from=now-10m&to=now&viewPanel=97&fullscreen&var-ins={{ $labels.ins }}



#==============================================================#
#                        Saturation                            #
#==============================================================#
# redis cpu usage more than 70% for 1m
- alert: RedisCPUHigh
  expr: redis:ins:cpu_usage > 0.70
  for: 1m
  labels: { level: 1, severity: WARN, category: redis }
  annotations:
    summary: "WARN RedisCPUHigh: {{ $labels.cls }} {{ $labels.ins }}"
    description: |
      redis:ins:cpu_all[cls={{ $labels.cls }}, ins={{ $labels.ins }}] = {{ $value }} > 60%
      /ui/d/redis-instance?from=now-10m&to=now&viewPanel=43&fullscreen&var-ins={{ $labels.ins }}

# redis mem usage more than 70% for 1m
- alert: RedisMemHigh
  expr: redis:ins:mem_usage > 0.70
  for: 1m
  labels: { level: 1, severity: WARN, category: redis }
  annotations:
    summary: "WARN RedisMemHigh: {{ $labels.cls }} {{ $labels.ins }}"
    description: |
      redis:ins:mem_usage[cls={{ $labels.cls }}, ins={{ $labels.ins }}] = {{ $value }} > 80%
      /ui/d/redis-instance?from=now-10m&to=now&viewPanel=7&fullscreen&var-ins={{ $labels.ins }}

#==============================================================#
#                         Traffic                              #
#==============================================================#
# redis qps more than 32000 for 5m
- alert: RedisQPSHigh
  expr: redis:ins:qps > 32000
  for: 5m
  labels: { level: 2, severity: INFO, category: redis }
  annotations:
    summary: "INFO RedisQPSHigh: {{ $labels.cls }} {{ $labels.ins }}"
    description: |
      redis:ins:qps[cls={{ $labels.cls }}, ins={{ $labels.ins }}] = {{ $value }} > 16000
      /ui/d/redis-instance?from=now-10m&to=now&viewPanel=96&fullscreen&var-ins={{ $labels.ins }}

13.6 - Metrics

Complete list of monitoring metrics provided by the Pigsty REDIS module with explanations

This page is a snapshot of 275 monitoring metric categories for the REDIS module. The actual runtime metric set varies with package version, enabled collectors, and target state.

Metric Name Type Labels Description
ALERTS Unknown cls, ip, level, severity, instance, category, ins, alertname, job, alertstate N/A
ALERTS_FOR_STATE Unknown cls, ip, level, severity, instance, category, ins, alertname, job N/A
redis:cls:aof_rewrite_time Unknown cls, job N/A
redis:cls:blocked_clients Unknown cls, job N/A
redis:cls:clients Unknown cls, job N/A
redis:cls:cmd_qps Unknown cls, cmd, job N/A
redis:cls:cmd_rt Unknown cls, cmd, job N/A
redis:cls:cmd_time Unknown cls, cmd, job N/A
redis:cls:conn_rate Unknown cls, job N/A
redis:cls:conn_reject Unknown cls, job N/A
redis:cls:cpu_sys Unknown cls, job N/A
redis:cls:cpu_sys_child Unknown cls, job N/A
redis:cls:cpu_usage Unknown cls, job N/A
redis:cls:cpu_usage_child Unknown cls, job N/A
redis:cls:cpu_user Unknown cls, job N/A
redis:cls:cpu_user_child Unknown cls, job N/A
redis:cls:fork_time Unknown cls, job N/A
redis:cls:key_evict Unknown cls, job N/A
redis:cls:key_expire Unknown cls, job N/A
redis:cls:key_hit Unknown cls, job N/A
redis:cls:key_hit_rate Unknown cls, job N/A
redis:cls:key_miss Unknown cls, job N/A
redis:cls:mem_max Unknown cls, job N/A
redis:cls:mem_usage Unknown cls, job N/A
redis:cls:mem_usage_max Unknown cls, job N/A
redis:cls:mem_used Unknown cls, job N/A
redis:cls:net_traffic Unknown cls, job N/A
redis:cls:qps Unknown cls, job N/A
redis:cls:qps_mu Unknown cls, job N/A
redis:cls:qps_realtime Unknown cls, job N/A
redis:cls:qps_sigma Unknown cls, job N/A
redis:cls:rt Unknown cls, job N/A
redis:cls:rt_mu Unknown cls, job N/A
redis:cls:rt_sigma Unknown cls, job N/A
redis:cls:rx Unknown cls, job N/A
redis:cls:size Unknown cls, job N/A
redis:cls:tx Unknown cls, job N/A
redis:env:blocked_clients Unknown job N/A
redis:env:clients Unknown job N/A
redis:env:cmd_qps Unknown cmd, job N/A
redis:env:cmd_rt Unknown cmd, job N/A
redis:env:cmd_time Unknown cmd, job N/A
redis:env:conn_rate Unknown job N/A
redis:env:conn_reject Unknown job N/A
redis:env:cpu_usage Unknown job N/A
redis:env:cpu_usage_child Unknown job N/A
redis:env:key_evict Unknown job N/A
redis:env:key_expire Unknown job N/A
redis:env:key_hit Unknown job N/A
redis:env:key_hit_rate Unknown job N/A
redis:env:key_miss Unknown job N/A
redis:env:mem_usage Unknown job N/A
redis:env:net_traffic Unknown job N/A
redis:env:qps Unknown job N/A
redis:env:qps_mu Unknown job N/A
redis:env:qps_realtime Unknown job N/A
redis:env:qps_sigma Unknown job N/A
redis:env:rt Unknown job N/A
redis:env:rt_mu Unknown job N/A
redis:env:rt_sigma Unknown job N/A
redis:env:rx Unknown job N/A
redis:env:tx Unknown job N/A
redis:ins Unknown cls, id, instance, ins, job N/A
redis:ins:blocked_clients Unknown cls, ip, instance, ins, job N/A
redis:ins:clients Unknown cls, ip, instance, ins, job N/A
redis:ins:cmd_qps Unknown cls, cmd, ip, instance, ins, job N/A
redis:ins:cmd_rt Unknown cls, cmd, ip, instance, ins, job N/A
redis:ins:cmd_time Unknown cls, cmd, ip, instance, ins, job N/A
redis:ins:conn_rate Unknown cls, ip, instance, ins, job N/A
redis:ins:conn_reject Unknown cls, ip, instance, ins, job N/A
redis:ins:cpu_sys Unknown cls, ip, instance, ins, job N/A
redis:ins:cpu_sys_child Unknown cls, ip, instance, ins, job N/A
redis:ins:cpu_usage Unknown cls, ip, instance, ins, job N/A
redis:ins:cpu_usage_child Unknown cls, ip, instance, ins, job N/A
redis:ins:cpu_user Unknown cls, ip, instance, ins, job N/A
redis:ins:cpu_user_child Unknown cls, ip, instance, ins, job N/A
redis:ins:key_evict Unknown cls, ip, instance, ins, job N/A
redis:ins:key_expire Unknown cls, ip, instance, ins, job N/A
redis:ins:key_hit Unknown cls, ip, instance, ins, job N/A
redis:ins:key_hit_rate Unknown cls, ip, instance, ins, job N/A
redis:ins:key_miss Unknown cls, ip, instance, ins, job N/A
redis:ins:lsn_rate Unknown cls, ip, instance, ins, job N/A
redis:ins:mem_usage Unknown cls, ip, instance, ins, job N/A
redis:ins:net_traffic Unknown cls, ip, instance, ins, job N/A
redis:ins:qps Unknown cls, ip, instance, ins, job N/A
redis:ins:qps_mu Unknown cls, ip, instance, ins, job N/A
redis:ins:qps_realtime Unknown cls, ip, instance, ins, job N/A
redis:ins:qps_sigma Unknown cls, ip, instance, ins, job N/A
redis:ins:rt Unknown cls, ip, instance, ins, job N/A
redis:ins:rt_mu Unknown cls, ip, instance, ins, job N/A
redis:ins:rt_sigma Unknown cls, ip, instance, ins, job N/A
redis:ins:rx Unknown cls, ip, instance, ins, job N/A
redis:ins:tx Unknown cls, ip, instance, ins, job N/A
redis:node:ip Unknown cls, ip, instance, ins, job N/A
redis:node:mem_alloc Unknown cls, ip, job N/A
redis:node:mem_total Unknown cls, ip, job N/A
redis:node:mem_used Unknown cls, ip, job N/A
redis:node:qps Unknown cls, ip, job N/A
redis_active_defrag_running gauge cls, ip, instance, ins, job active_defrag_running metric
redis_allocator_active_bytes gauge cls, ip, instance, ins, job allocator_active_bytes metric
redis_allocator_allocated_bytes gauge cls, ip, instance, ins, job allocator_allocated_bytes metric
redis_allocator_frag_bytes gauge cls, ip, instance, ins, job allocator_frag_bytes metric
redis_allocator_frag_ratio gauge cls, ip, instance, ins, job allocator_frag_ratio metric
redis_allocator_resident_bytes gauge cls, ip, instance, ins, job allocator_resident_bytes metric
redis_allocator_rss_bytes gauge cls, ip, instance, ins, job allocator_rss_bytes metric
redis_allocator_rss_ratio gauge cls, ip, instance, ins, job allocator_rss_ratio metric
redis_aof_current_rewrite_duration_sec gauge cls, ip, instance, ins, job aof_current_rewrite_duration_sec metric
redis_aof_enabled gauge cls, ip, instance, ins, job aof_enabled metric
redis_aof_last_bgrewrite_status gauge cls, ip, instance, ins, job aof_last_bgrewrite_status metric
redis_aof_last_cow_size_bytes gauge cls, ip, instance, ins, job aof_last_cow_size_bytes metric
redis_aof_last_rewrite_duration_sec gauge cls, ip, instance, ins, job aof_last_rewrite_duration_sec metric
redis_aof_last_write_status gauge cls, ip, instance, ins, job aof_last_write_status metric
redis_aof_rewrite_in_progress gauge cls, ip, instance, ins, job aof_rewrite_in_progress metric
redis_aof_rewrite_scheduled gauge cls, ip, instance, ins, job aof_rewrite_scheduled metric
redis_blocked_clients gauge cls, ip, instance, ins, job blocked_clients metric
redis_client_recent_max_input_buffer_bytes gauge cls, ip, instance, ins, job client_recent_max_input_buffer_bytes metric
redis_client_recent_max_output_buffer_bytes gauge cls, ip, instance, ins, job client_recent_max_output_buffer_bytes metric
redis_clients_in_timeout_table gauge cls, ip, instance, ins, job clients_in_timeout_table metric
redis_cluster_connections gauge cls, ip, instance, ins, job cluster_connections metric
redis_cluster_current_epoch gauge cls, ip, instance, ins, job cluster_current_epoch metric
redis_cluster_enabled gauge cls, ip, instance, ins, job cluster_enabled metric
redis_cluster_known_nodes gauge cls, ip, instance, ins, job cluster_known_nodes metric
redis_cluster_messages_received_total gauge cls, ip, instance, ins, job cluster_messages_received_total metric
redis_cluster_messages_sent_total gauge cls, ip, instance, ins, job cluster_messages_sent_total metric
redis_cluster_my_epoch gauge cls, ip, instance, ins, job cluster_my_epoch metric
redis_cluster_size gauge cls, ip, instance, ins, job cluster_size metric
redis_cluster_slots_assigned gauge cls, ip, instance, ins, job cluster_slots_assigned metric
redis_cluster_slots_fail gauge cls, ip, instance, ins, job cluster_slots_fail metric
redis_cluster_slots_ok gauge cls, ip, instance, ins, job cluster_slots_ok metric
redis_cluster_slots_pfail gauge cls, ip, instance, ins, job cluster_slots_pfail metric
redis_cluster_state gauge cls, ip, instance, ins, job cluster_state metric
redis_cluster_stats_messages_meet_received gauge cls, ip, instance, ins, job cluster_stats_messages_meet_received metric
redis_cluster_stats_messages_meet_sent gauge cls, ip, instance, ins, job cluster_stats_messages_meet_sent metric
redis_cluster_stats_messages_ping_received gauge cls, ip, instance, ins, job cluster_stats_messages_ping_received metric
redis_cluster_stats_messages_ping_sent gauge cls, ip, instance, ins, job cluster_stats_messages_ping_sent metric
redis_cluster_stats_messages_pong_received gauge cls, ip, instance, ins, job cluster_stats_messages_pong_received metric
redis_cluster_stats_messages_pong_sent gauge cls, ip, instance, ins, job cluster_stats_messages_pong_sent metric
redis_commands_duration_seconds_total counter cls, cmd, ip, instance, ins, job Total amount of time in seconds spent per command
redis_commands_failed_calls_total counter cls, cmd, ip, instance, ins, job Total number of errors prior command execution per command
redis_commands_latencies_usec_bucket Unknown cls, cmd, ip, le, instance, ins, job N/A
redis_commands_latencies_usec_count Unknown cls, cmd, ip, instance, ins, job N/A
redis_commands_latencies_usec_sum Unknown cls, cmd, ip, instance, ins, job N/A
redis_commands_processed_total counter cls, ip, instance, ins, job commands_processed_total metric
redis_commands_rejected_calls_total counter cls, cmd, ip, instance, ins, job Total number of errors within command execution per command
redis_commands_total counter cls, cmd, ip, instance, ins, job Total number of calls per command
redis_config_io_threads gauge cls, ip, instance, ins, job config_io_threads metric
redis_config_maxclients gauge cls, ip, instance, ins, job config_maxclients metric
redis_config_maxmemory gauge cls, ip, instance, ins, job config_maxmemory metric
redis_connected_clients gauge cls, ip, instance, ins, job connected_clients metric
redis_connected_slave_lag_seconds gauge cls, ip, slave_ip, instance, slave_state, ins, slave_port, job Lag of connected slave
redis_connected_slave_offset_bytes gauge cls, ip, slave_ip, instance, slave_state, ins, slave_port, job Offset of connected slave
redis_connected_slaves gauge cls, ip, instance, ins, job connected_slaves metric
redis_connections_received_total counter cls, ip, instance, ins, job connections_received_total metric
redis_cpu_sys_children_seconds_total counter cls, ip, instance, ins, job cpu_sys_children_seconds_total metric
redis_cpu_sys_main_thread_seconds_total counter cls, ip, instance, ins, job cpu_sys_main_thread_seconds_total metric
redis_cpu_sys_seconds_total counter cls, ip, instance, ins, job cpu_sys_seconds_total metric
redis_cpu_user_children_seconds_total counter cls, ip, instance, ins, job cpu_user_children_seconds_total metric
redis_cpu_user_main_thread_seconds_total counter cls, ip, instance, ins, job cpu_user_main_thread_seconds_total metric
redis_cpu_user_seconds_total counter cls, ip, instance, ins, job cpu_user_seconds_total metric
redis_db_keys gauge cls, ip, instance, ins, db, job Total number of keys by DB
redis_db_keys_expiring gauge cls, ip, instance, ins, db, job Total number of expiring keys by DB
redis_defrag_hits gauge cls, ip, instance, ins, job defrag_hits metric
redis_defrag_key_hits gauge cls, ip, instance, ins, job defrag_key_hits metric
redis_defrag_key_misses gauge cls, ip, instance, ins, job defrag_key_misses metric
redis_defrag_misses gauge cls, ip, instance, ins, job defrag_misses metric
redis_dump_payload_sanitizations counter cls, ip, instance, ins, job dump_payload_sanitizations metric
redis_errors_total counter cls, ip, err, instance, ins, job Total number of errors per error type
redis_evicted_keys_total counter cls, ip, instance, ins, job evicted_keys_total metric
redis_expired_keys_total counter cls, ip, instance, ins, job expired_keys_total metric
redis_expired_stale_percentage gauge cls, ip, instance, ins, job expired_stale_percentage metric
redis_expired_time_cap_reached_total gauge cls, ip, instance, ins, job expired_time_cap_reached_total metric
redis_exporter_build_info gauge cls, golang_version, ip, commit_sha, instance, version, ins, job, build_date redis exporter build_info
redis_exporter_last_scrape_connect_time_seconds gauge cls, ip, instance, ins, job exporter_last_scrape_connect_time_seconds metric
redis_exporter_last_scrape_duration_seconds gauge cls, ip, instance, ins, job exporter_last_scrape_duration_seconds metric
redis_exporter_last_scrape_error gauge cls, ip, instance, ins, job The last scrape error status.
redis_exporter_scrape_duration_seconds_count Unknown cls, ip, instance, ins, job N/A
redis_exporter_scrape_duration_seconds_sum Unknown cls, ip, instance, ins, job N/A
redis_exporter_scrapes_total counter cls, ip, instance, ins, job Current total redis scrapes.
redis_instance_info gauge cls, ip, os, role, instance, run_id, redis_version, tcp_port, process_id, ins, redis_mode, maxmemory_policy, redis_build_id, job Information about the Redis instance
redis_io_threaded_reads_processed counter cls, ip, instance, ins, job io_threaded_reads_processed metric
redis_io_threaded_writes_processed counter cls, ip, instance, ins, job io_threaded_writes_processed metric
redis_io_threads_active gauge cls, ip, instance, ins, job io_threads_active metric
redis_keyspace_hits_total counter cls, ip, instance, ins, job keyspace_hits_total metric
redis_keyspace_misses_total counter cls, ip, instance, ins, job keyspace_misses_total metric
redis_last_key_groups_scrape_duration_milliseconds gauge cls, ip, instance, ins, job Duration of the last key group metrics scrape in milliseconds
redis_last_slow_execution_duration_seconds gauge cls, ip, instance, ins, job The amount of time needed for last slow execution, in seconds
redis_latency_percentiles_usec summary cls, cmd, ip, instance, quantile, ins, job A summary of latency percentile distribution per command
redis_latency_percentiles_usec_count Unknown cls, cmd, ip, instance, ins, job N/A
redis_latency_percentiles_usec_sum Unknown cls, cmd, ip, instance, ins, job N/A
redis_latest_fork_seconds gauge cls, ip, instance, ins, job latest_fork_seconds metric
redis_lazyfree_pending_objects gauge cls, ip, instance, ins, job lazyfree_pending_objects metric
redis_loading_dump_file gauge cls, ip, instance, ins, job loading_dump_file metric
redis_master_last_io_seconds_ago gauge cls, ip, master_host, instance, ins, job, master_port Master last io seconds ago
redis_master_link_up gauge cls, ip, master_host, instance, ins, job, master_port Master link status on Redis slave
redis_master_repl_offset gauge cls, ip, instance, ins, job master_repl_offset metric
redis_master_sync_in_progress gauge cls, ip, master_host, instance, ins, job, master_port Master sync in progress
redis_mem_clients_normal gauge cls, ip, instance, ins, job mem_clients_normal metric
redis_mem_clients_slaves gauge cls, ip, instance, ins, job mem_clients_slaves metric
redis_mem_fragmentation_bytes gauge cls, ip, instance, ins, job mem_fragmentation_bytes metric
redis_mem_fragmentation_ratio gauge cls, ip, instance, ins, job mem_fragmentation_ratio metric
redis_mem_not_counted_for_eviction_bytes gauge cls, ip, instance, ins, job mem_not_counted_for_eviction_bytes metric
redis_memory_max_bytes gauge cls, ip, instance, ins, job memory_max_bytes metric
redis_memory_used_bytes gauge cls, ip, instance, ins, job memory_used_bytes metric
redis_memory_used_dataset_bytes gauge cls, ip, instance, ins, job memory_used_dataset_bytes metric
redis_memory_used_lua_bytes gauge cls, ip, instance, ins, job memory_used_lua_bytes metric
redis_memory_used_overhead_bytes gauge cls, ip, instance, ins, job memory_used_overhead_bytes metric
redis_memory_used_peak_bytes gauge cls, ip, instance, ins, job memory_used_peak_bytes metric
redis_memory_used_rss_bytes gauge cls, ip, instance, ins, job memory_used_rss_bytes metric
redis_memory_used_scripts_bytes gauge cls, ip, instance, ins, job memory_used_scripts_bytes metric
redis_memory_used_startup_bytes gauge cls, ip, instance, ins, job memory_used_startup_bytes metric
redis_migrate_cached_sockets_total gauge cls, ip, instance, ins, job migrate_cached_sockets_total metric
redis_module_fork_in_progress gauge cls, ip, instance, ins, job module_fork_in_progress metric
redis_module_fork_last_cow_size gauge cls, ip, instance, ins, job module_fork_last_cow_size metric
redis_net_input_bytes_total counter cls, ip, instance, ins, job net_input_bytes_total metric
redis_net_output_bytes_total counter cls, ip, instance, ins, job net_output_bytes_total metric
redis_number_of_cached_scripts gauge cls, ip, instance, ins, job number_of_cached_scripts metric
redis_process_id gauge cls, ip, instance, ins, job process_id metric
redis_pubsub_channels gauge cls, ip, instance, ins, job pubsub_channels metric
redis_pubsub_patterns gauge cls, ip, instance, ins, job pubsub_patterns metric
redis_pubsubshard_channels gauge cls, ip, instance, ins, job pubsubshard_channels metric
redis_rdb_bgsave_in_progress gauge cls, ip, instance, ins, job rdb_bgsave_in_progress metric
redis_rdb_changes_since_last_save gauge cls, ip, instance, ins, job rdb_changes_since_last_save metric
redis_rdb_current_bgsave_duration_sec gauge cls, ip, instance, ins, job rdb_current_bgsave_duration_sec metric
redis_rdb_last_bgsave_duration_sec gauge cls, ip, instance, ins, job rdb_last_bgsave_duration_sec metric
redis_rdb_last_bgsave_status gauge cls, ip, instance, ins, job rdb_last_bgsave_status metric
redis_rdb_last_cow_size_bytes gauge cls, ip, instance, ins, job rdb_last_cow_size_bytes metric
redis_rdb_last_save_timestamp_seconds gauge cls, ip, instance, ins, job rdb_last_save_timestamp_seconds metric
redis_rejected_connections_total counter cls, ip, instance, ins, job rejected_connections_total metric
redis_repl_backlog_first_byte_offset gauge cls, ip, instance, ins, job repl_backlog_first_byte_offset metric
redis_repl_backlog_history_bytes gauge cls, ip, instance, ins, job repl_backlog_history_bytes metric
redis_repl_backlog_is_active gauge cls, ip, instance, ins, job repl_backlog_is_active metric
redis_replica_partial_resync_accepted gauge cls, ip, instance, ins, job replica_partial_resync_accepted metric
redis_replica_partial_resync_denied gauge cls, ip, instance, ins, job replica_partial_resync_denied metric
redis_replica_resyncs_full gauge cls, ip, instance, ins, job replica_resyncs_full metric
redis_replication_backlog_bytes gauge cls, ip, instance, ins, job replication_backlog_bytes metric
redis_second_repl_offset gauge cls, ip, instance, ins, job second_repl_offset metric
redis_sentinel_master_ckquorum_status gauge cls, ip, message, instance, ins, master_name, job Master ckquorum status
redis_sentinel_master_ok_sentinels gauge cls, ip, instance, ins, master_address, master_name, job The number of okay sentinels monitoring this master
redis_sentinel_master_ok_slaves gauge cls, ip, instance, ins, master_address, master_name, job The number of okay slaves of the master
redis_sentinel_master_sentinels gauge cls, ip, instance, ins, master_address, master_name, job The number of sentinels monitoring this master
redis_sentinel_master_setting_ckquorum gauge cls, ip, instance, ins, master_address, master_name, job Show the current ckquorum config for each master
redis_sentinel_master_setting_down_after_milliseconds gauge cls, ip, instance, ins, master_address, master_name, job Show the current down-after-milliseconds config for each master
redis_sentinel_master_setting_failover_timeout gauge cls, ip, instance, ins, master_address, master_name, job Show the current failover-timeout config for each master
redis_sentinel_master_setting_parallel_syncs gauge cls, ip, instance, ins, master_address, master_name, job Show the current parallel-syncs config for each master
redis_sentinel_master_slaves gauge cls, ip, instance, ins, master_address, master_name, job The number of slaves of the master
redis_sentinel_master_status gauge cls, ip, master_status, instance, ins, master_address, master_name, job Master status on Sentinel
redis_sentinel_masters gauge cls, ip, instance, ins, job The number of masters this sentinel is watching
redis_sentinel_running_scripts gauge cls, ip, instance, ins, job Number of scripts in execution right now
redis_sentinel_scripts_queue_length gauge cls, ip, instance, ins, job Queue of user scripts to execute
redis_sentinel_simulate_failure_flags gauge cls, ip, instance, ins, job Failures simulations
redis_sentinel_tilt gauge cls, ip, instance, ins, job Sentinel is in TILT mode
redis_slave_expires_tracked_keys gauge cls, ip, instance, ins, job slave_expires_tracked_keys metric
redis_slave_info gauge cls, ip, master_host, instance, read_only, ins, job, master_port Information about the Redis slave
redis_slave_priority gauge cls, ip, instance, ins, job slave_priority metric
redis_slave_repl_offset gauge cls, ip, master_host, instance, ins, job, master_port Slave replication offset
redis_slowlog_last_id gauge cls, ip, instance, ins, job Last id of slowlog
redis_slowlog_length gauge cls, ip, instance, ins, job Total slowlog
redis_start_time_seconds gauge cls, ip, instance, ins, job Start time of the Redis instance since unix epoch in seconds.
redis_target_scrape_request_errors_total counter cls, ip, instance, ins, job Errors in requests to the exporter
redis_total_error_replies counter cls, ip, instance, ins, job total_error_replies metric
redis_total_reads_processed counter cls, ip, instance, ins, job total_reads_processed metric
redis_total_system_memory_bytes gauge cls, ip, instance, ins, job total_system_memory_bytes metric
redis_total_writes_processed counter cls, ip, instance, ins, job total_writes_processed metric
redis_tracking_clients gauge cls, ip, instance, ins, job tracking_clients metric
redis_tracking_total_items gauge cls, ip, instance, ins, job tracking_total_items metric
redis_tracking_total_keys gauge cls, ip, instance, ins, job tracking_total_keys metric
redis_tracking_total_prefixes gauge cls, ip, instance, ins, job tracking_total_prefixes metric
redis_unexpected_error_replies counter cls, ip, instance, ins, job unexpected_error_replies metric
redis_up gauge cls, ip, instance, ins, job Information about the Redis instance
redis_uptime_in_seconds gauge cls, ip, instance, ins, job uptime_in_seconds metric
scrape_duration_seconds Unknown cls, ip, instance, ins, job N/A
scrape_samples_post_metric_relabeling Unknown cls, ip, instance, ins, job N/A
scrape_samples_scraped Unknown cls, ip, instance, ins, job N/A
scrape_series_added Unknown cls, ip, instance, ins, job N/A
up Unknown cls, ip, instance, ins, job N/A

13.7 - FAQ

Frequently asked questions about the Pigsty REDIS module

ABORT due to redis_safeguard enabled

This means the Redis instance you are trying to remove has the safeguard enabled. When redis_safeguard is true, redis-rm.yml refuses to run unconditionally; the switch does not probe whether an instance is running.

After confirming the exact -l/redis_port target, a recent backup, and the redis_rm_data setting, override the protection with -e redis_safeguard=false and run the removal. This switch only releases the guard; it does not verify the target or recoverability for you.

How to add a new Redis instance on a node?

Use bin/redis-add <ip> <port> to deploy a new Redis instance on the node.

How to remove a specific instance from a node?

Use bin/redis-rm <ip> <port> to remove a single Redis instance from the node.

How do I choose Redis or Valkey?

Current source defaults to redis_type: redis and also supports explicit redis_type: valkey. The role installs the corresponding redis or valkey package and calls the matching redis-server / valkey-server and CLI binaries from instance units. Configuration paths, instance service names, monitoring job, and parameter prefixes retain the redis namespace.

The default Redis package remains on the 7.2 BSD branch; patch versions vary by operating-system channel, so use the actual repository metadata as the source of truth. Switching an existing cluster to Valkey is not an automatic migration: first verify target-version RDB/AOF compatibility, replication and Sentinel/Cluster behavior, and a rollback path.

14 - Module: DOCKER

Docker daemon service that enables one-click deployment of containerized stateless software templates and additional functionality.

Docker is the most popular containerization platform, providing standardized software delivery capabilities.

Pigsty does not rely on Docker to deploy any of its components; instead, it provides the ability to deploy and install Docker — this is an optional module.

Pigsty offers a series of Docker software/tool/application templates for you to choose from as needed. This allows users to quickly spin up various containerized stateless software templates, adding extra functionality. You can use external, Pigsty-managed highly available database clusters while placing stateless applications inside containers.

When running configure, Pigsty automatically selects suitable upstream repositories and mirror acceleration settings based on region (for example, mainland China network environments), to improve image pull speed and availability. You can easily configure Registry and Proxy settings to flexibly access different image sources.

14.1 - Usage

Docker module quick start guide - installation, removal, download, repository, mirrors, proxy, and image pulling.

Pigsty has built-in Docker support, which you can use to quickly deploy containerized applications.

Getting Started

Docker is an optional module. In Pigsty, whether Docker is installed is controlled by docker_enabled, which is disabled by default.

The docker-ce upstream repository belongs to the infra module. If you need to explicitly include Docker packages in the offline repository, use repo_extra_packages with the docker package alias (mapped to docker-ce and docker-compose-plugin).

repo_modules: infra,node,pgsql     # <--- Keep infra module (Docker upstream belongs to infra)
repo_extra_packages:
  - pgsql-main
  - docker                         # <--- Download Docker (docker-ce + docker-compose-plugin)

After Docker is downloaded, you need to set the docker_enabled: true flag on the nodes where you want to install Docker, and configure other parameters as needed.

infra:
  hosts:
    10.10.10.10: { infra_seq: 1 ,nodename: infra-1 }
    10.10.10.11: { infra_seq: 2 ,nodename: infra-2 }
  vars:
    docker_enabled: true  # Install Docker on this group!

Finally, you can use the docker.yml playbook to install it on the nodes:

./docker.yml -l infra    # Install Docker on the infra group

Installation

If you want to temporarily install Docker directly from the internet on certain nodes, you can use the following command:

./node.yml -e '{"node_repo_modules":"node,infra","node_packages":["docker-ce","docker-compose-plugin"]}' -t node_repo,node_pkg -l <select_group_ip>

This command will first enable the upstream software sources for the node,infra modules on the target nodes, then install the docker-ce and docker-compose-plugin packages (same package names on EL/Debian).

If you want Docker-related packages to be automatically downloaded during Pigsty initialization, refer to the instructions below.

Removal

Because it’s so simple, Pigsty doesn’t provide an uninstall playbook for the Docker module. You can directly remove Docker using an Ansible command:

ansible <selector> -m package -b -a 'name=docker-ce,docker-compose-plugin state=absent'  # Remove docker

Download

To download Docker during Pigsty installation, confirm that repo_modules includes infra (the module containing Docker upstream repositories), then specify Docker packages in repo_packages or repo_extra_packages.

repo_modules: infra,node,pgsql         # <--- Docker upstream repo belongs to infra
repo_packages:
  - node-bootstrap, infra-package, infra-addons, node-package1, node-package2, pgsql-common, docker
repo_extra_packages:
  - pgsql-main
  - docker  # <--- Can also be specified here

The docker specified here (which actually corresponds to the docker-ce and docker-compose-plugin packages) will be automatically downloaded to the local repository during the default deploy.yml process. After downloading, the Docker packages will be available to all nodes via the local repository.

If you’ve already completed Pigsty installation and the local repository is initialized, you can run ./infra.yml -t repo_build after modifying the configuration to re-download and rebuild the offline repository.

Installing Docker requires the Docker YUM/APT repository. In the v4.x default repo_upstream, this repository belongs to the infra module and is usually available out of the box.

Repository

Downloading Docker requires upstream internet software repositories, which are defined in the default repo_upstream with module name infra:

- { name: docker-ce ,description: 'Docker CE' ,module: infra  ,releases: [8,9,10] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://download.docker.com/linux/centos/$releasever/$basearch/stable'    ,china: 'https://mirrors.aliyun.com/docker-ce/linux/centos/$releasever/$basearch/stable'  ,europe: 'https://mirrors.xtom.de/docker-ce/linux/centos/$releasever/$basearch/stable' }}
- { name: docker-ce ,description: 'Docker'    ,module: infra  ,releases: [11,12,13,22,24,26] ,arch: [x86_64, aarch64] ,baseurl: { default: 'https://download.docker.com/linux/${distro_name} ${distro_codename} stable' ,china: 'https://mirrors.aliyun.com/docker-ce/linux/${distro_name} ${distro_codename} stable' }}

You can reference this repository using the infra module name in repo_modules and node_repo_modules.

Docker repository access in mainland China

Docker’s official software repository is blocked by default in mainland China. Use a mainland mirror to complete the download.

If you’re in mainland China and encounter Docker download failures, check whether region is set to default in your configuration inventory. The automatically configured region: china can resolve this issue.

Proxy

If your network environment requires a proxy server to access the internet, you can configure the proxy_env parameter in Pigsty’s configuration inventory. This parameter will be written to the proxy related configuration in Docker’s configuration file.

proxy_env:
  no_proxy: "localhost,127.0.0.1,10.0.0.0/8,192.168.0.0/16,*.pigsty,*.aliyun.com,mirrors.aliyuncs.com,mirrors.tuna.tsinghua.edu.cn,mirrors.zju.edu.cn"
  #http_proxy: 'http://username:[email protected]'
  #https_proxy: 'http://username:[email protected]'
  #all_proxy: 'http://username:[email protected]'

When running configure with the -x parameter, the proxy server configuration from your current environment will be automatically generated into Pigsty’s configuration file under proxy_env.

In addition to using a proxy server, you can also configure Docker Registry Mirrors to bypass blocks.

Registry Mirrors

You can use the docker_registry_mirrors parameter to specify Docker Registry Mirrors:

For users outside the firewall, in addition to the official DockerHub site, you can also consider using the quay.io mirror site. If your internal network environment already has mature image infrastructure, you can use your internal Docker registry mirrors to avoid being affected by external mirror sites and improve download speeds.

Users of public cloud providers can consider using free internal Docker mirrors. For example, if you’re using Alibaba Cloud, you can use Alibaba Cloud’s internal Docker mirror site (requires login):

["https://registry.cn-hangzhou.aliyuncs.com"]   # Alibaba Cloud mirror, requires explicit login

If you’re using Tencent Cloud, you can use Tencent Cloud’s internal Docker mirror site (requires internal network):

["https://ccr.ccs.tencentyun.com"]   # Tencent Cloud mirror, internal network only

Additionally, you can use CF-Workers-docker.io to quickly set up your own Docker image proxy. You can also consider using free Docker proxy mirrors (use at your own risk!)

Pulling Images

The docker_image and docker_image_cache parameters can be used to directly specify a list of images to pull during Docker installation.

Using this feature, Docker will come with the specified images after installation (provided they can be successfully pulled; this task will be automatically ignored and skipped on failure).

For example, you can specify images to pull in the configuration inventory:

infra:
  hosts:
    10.10.10.10: { infra_seq: 1 }
  vars:
    docker_enabled: true  # Install Docker on this group!
    docker_image:
      - redis:latest      # Pull the latest Redis image

Another way to preload images is to use locally save d tgz archives: if you’ve previously exported Docker images using docker save xxx | gzip -c > /tmp/docker/xxx.tgz. These exported image files can be automatically loaded via the glob specified by the docker_image_cache parameter. The default location is: /tmp/docker/*.tgz.

This means you can place images in the /tmp/docker directory beforehand, and after running docker.yml to install Docker, these image packages will be automatically loaded.

For example, in the self-hosted Supabase tutorial, this technique is used. Before spinning up Supabase and installing Docker, the *.tgz image archives from the local /tmp/supabase directory are copied to the target node’s /tmp/docker directory.

- name: copy local docker images
  copy: src="{{ item }}" dest="/tmp/docker/"
  with_fileglob: "{{ supa_images }}"
  vars: # you can override this with -e cli args
    supa_images: /tmp/supabase/*.tgz

Applications

Pigsty provides a series of ready-to-use, Docker Compose-based software templates, which you can use to spin up business software that uses external Pigsty-managed database clusters.



14.2 - Parameters

DOCKER module provides 8 configuration parameters

The DOCKER module provides 8 configuration parameters.

Parameter Overview

The DOCKER parameter group is used for Docker container engine deployment and configuration, including enable switch, data directory, storage driver, registry mirrors, and monitoring.

Parameter Type Level Description
docker_enabled bool G/C/I Enable Docker on current node? disabled by default
docker_data path G/C/I Docker data directory, /data/docker by default
docker_storage_driver enum G/C/I Docker storage driver, overlay2 by default
docker_cgroups_driver enum G/C/I Docker cgroup driver: cgroupfs or systemd
docker_registry_mirrors string[] G/C/I Docker registry mirror list
docker_exporter_port port G Docker metrics exporter port, 9323 by default
docker_image string[] G/C/I Docker images to pull, empty list by default
docker_image_cache path G/C/I Docker image cache tarball path, /tmp/docker/*.tgz

You can use the docker.yml playbook to install and enable Docker on nodes.

Default parameters are defined in roles/docker/defaults/main.yml

docker_enabled: false             # Enable Docker on current node?
docker_data: /data/docker         # Docker data directory, /data/docker by default
docker_storage_driver: overlay2   # Docker storage driver, overlay2/zfs/btrfs...
docker_cgroups_driver: systemd    # Docker cgroup driver: cgroupfs or systemd
docker_registry_mirrors: []       # Docker registry mirror list
docker_exporter_port: 9323        # Docker metrics exporter port, 9323 by default
docker_image: []                  # Docker images to pull after startup
docker_image_cache: /tmp/docker/*.tgz # Docker image cache tarball glob pattern

docker_enabled

Parameter: docker_enabled, Type: bool, Level: G/C/I

Enable Docker on current node? Default: false, meaning Docker is not enabled.

docker_data

Parameter: docker_data, Type: path, Level: G/C/I

Docker data directory, default is /data/docker.

This directory stores Docker images, containers, volumes, and other data. If you have a dedicated data disk, it’s recommended to point this directory to that disk’s mount point.

docker_storage_driver

Parameter: docker_storage_driver, Type: enum, Level: G/C/I

Docker storage driver, default is overlay2.

See official documentation: https://docs.docker.com/engine/storage/drivers/select-storage-driver/

Available storage drivers include:

  • overlay2: Recommended default driver, suitable for most scenarios
  • fuse-overlayfs: For rootless container scenarios
  • btrfs: When using Btrfs filesystem
  • zfs: When using ZFS filesystem
  • vfs: For testing purposes, not recommended for production

docker_cgroups_driver

Parameter: docker_cgroups_driver, Type: enum, Level: G/C/I

Docker cgroup filesystem driver, can be cgroupfs or systemd, default: systemd

docker_registry_mirrors

Parameter: docker_registry_mirrors, Type: string[], Level: G/C/I

Docker registry mirror list, default: [] empty array.

You can use Docker mirror sites to accelerate image pulls. Here are some examples:

docker_registry_mirrors:                        # Choose one or more
  - https://docker.m.daocloud.io                # DaoCloud mirror
  - https://docker.1ms.run                      # 1ms mirror
  - https://mirror.ccs.tencentyun.com           # Tencent Cloud internal mirror
  - https://registry.cn-hangzhou.aliyuncs.com   # Alibaba Cloud mirror (requires login)

You can also consider using a Cloudflare Worker to set up a Docker Proxy for faster access.

If pull speeds are still too slow, consider using alternative registries: docker login quay.io

docker_exporter_port

Parameter: docker_exporter_port, Type: port, Level: G

Docker metrics exporter port, default is 9323.

The Docker daemon exposes Prometheus-format monitoring metrics on this port for collection by monitoring infrastructure.

docker_image

Parameter: docker_image, Type: string[], Level: G/C/I

List of Docker images to pull, default is empty list [].

Docker image names specified here will be automatically pulled during the installation phase.

docker_image_cache

Parameter: docker_image_cache, Type: path, Level: G/C/I

Local Docker image cache tarball glob pattern, default is /tmp/docker/*.tgz.

You can use docker save | gzip to package images and automatically import them during Docker installation via this parameter.

.tgz tarball files matching this pattern will be imported into Docker one by one using:

cat *.tgz | gzip -d -c - | docker load

14.3 - Playbooks

How to use the built-in Ansible playbook to manage Docker and quick reference for common management commands.

The Docker module provides a default playbook docker.yml for installing Docker Daemon and Docker Compose.

docker.yml

Playbook source file: docker.yml

Running this playbook will install docker-ce and docker-compose-plugin on target nodes with the docker_enabled: true flag, and enable the dockerd service.

The following are the available task subsets in the docker.yml playbook:

  • docker_install : Install Docker and Docker Compose packages on the node
  • docker_admin : Add specified users to the Docker admin user group
  • docker_dir : Create Docker related directories
  • docker_config : Generate Docker daemon service configuration file
  • docker_launch : Start the Docker daemon service
  • docker_register : Register Docker daemon as a monitoring target (alias tags: register / add_metrics)
  • docker_image : Attempt to load pre-cached image tarballs from /tmp/docker/*.tgz (if they exist)

The Docker module does not provide a dedicated uninstall playbook. If you need to uninstall Docker, you can manually stop Docker and then remove it:

systemctl stop docker                        # Stop Docker daemon service
yum remove docker-ce docker-compose-plugin   # Uninstall Docker on EL systems
apt remove docker-ce docker-compose-plugin   # Uninstall Docker on Debian systems

Changing docker_enabled to false only makes docker.yml skip the entire Docker role. It does not stop or uninstall an existing Docker deployment, nor delete /data/docker. The manual commands above also leave the data directory in place. Docker’s VictoriaMetrics file-discovery target can be deregistered together with the node through the node_deregister task in node-rm.yml.



14.4 - Metrics

Complete list of monitoring metrics provided by the Pigsty Docker module

This snapshot records 123 monitoring metric families for the DOCKER module. The metrics present at runtime vary with package version, enabled collectors, and target state.

Metric Name Type Labels Description
builder_builds_failed_total counter ip, cls, reason, ins, job, instance Number of failed image builds
builder_builds_triggered_total counter ip, cls, ins, job, instance Number of triggered image builds
docker_up Unknown ip, cls, ins, job, instance N/A
engine_daemon_container_actions_seconds_bucket Unknown ip, cls, ins, job, instance, le, action N/A
engine_daemon_container_actions_seconds_count Unknown ip, cls, ins, job, instance, action N/A
engine_daemon_container_actions_seconds_sum Unknown ip, cls, ins, job, instance, action N/A
engine_daemon_container_states_containers gauge ip, cls, ins, job, instance, state The count of containers in various states
engine_daemon_engine_cpus_cpus gauge ip, cls, ins, job, instance The number of cpus that the host system of the engine has
engine_daemon_engine_info gauge ip, cls, architecture, ins, job, instance, os_version, kernel, version, graphdriver, os, daemon_id, commit, os_type The information related to the engine and the OS it is running on
engine_daemon_engine_memory_bytes gauge ip, cls, ins, job, instance The number of bytes of memory that the host system of the engine has
engine_daemon_events_subscribers_total gauge ip, cls, ins, job, instance The number of current subscribers to events
engine_daemon_events_total counter ip, cls, ins, job, instance The number of events logged
engine_daemon_health_checks_failed_total counter ip, cls, ins, job, instance The total number of failed health checks
engine_daemon_health_check_start_duration_seconds_bucket Unknown ip, cls, ins, job, instance, le N/A
engine_daemon_health_check_start_duration_seconds_count Unknown ip, cls, ins, job, instance N/A
engine_daemon_health_check_start_duration_seconds_sum Unknown ip, cls, ins, job, instance N/A
engine_daemon_health_checks_total counter ip, cls, ins, job, instance The total number of health checks
engine_daemon_host_info_functions_seconds_bucket Unknown ip, cls, ins, job, instance, le, function N/A
engine_daemon_host_info_functions_seconds_count Unknown ip, cls, ins, job, instance, function N/A
engine_daemon_host_info_functions_seconds_sum Unknown ip, cls, ins, job, instance, function N/A
engine_daemon_image_actions_seconds_bucket Unknown ip, cls, ins, job, instance, le, action N/A
engine_daemon_image_actions_seconds_count Unknown ip, cls, ins, job, instance, action N/A
engine_daemon_image_actions_seconds_sum Unknown ip, cls, ins, job, instance, action N/A
engine_daemon_network_actions_seconds_bucket Unknown ip, cls, ins, job, instance, le, action N/A
engine_daemon_network_actions_seconds_count Unknown ip, cls, ins, job, instance, action N/A
engine_daemon_network_actions_seconds_sum Unknown ip, cls, ins, job, instance, action N/A
etcd_debugging_snap_save_marshalling_duration_seconds_bucket Unknown ip, cls, ins, job, instance, le N/A
etcd_debugging_snap_save_marshalling_duration_seconds_count Unknown ip, cls, ins, job, instance N/A
etcd_debugging_snap_save_marshalling_duration_seconds_sum Unknown ip, cls, ins, job, instance N/A
etcd_debugging_snap_save_total_duration_seconds_bucket Unknown ip, cls, ins, job, instance, le N/A
etcd_debugging_snap_save_total_duration_seconds_count Unknown ip, cls, ins, job, instance N/A
etcd_debugging_snap_save_total_duration_seconds_sum Unknown ip, cls, ins, job, instance N/A
etcd_disk_wal_fsync_duration_seconds_bucket Unknown ip, cls, ins, job, instance, le N/A
etcd_disk_wal_fsync_duration_seconds_count Unknown ip, cls, ins, job, instance N/A
etcd_disk_wal_fsync_duration_seconds_sum Unknown ip, cls, ins, job, instance N/A
etcd_disk_wal_write_bytes_total gauge ip, cls, ins, job, instance Total number of bytes written in WAL.
etcd_snap_db_fsync_duration_seconds_bucket Unknown ip, cls, ins, job, instance, le N/A
etcd_snap_db_fsync_duration_seconds_count Unknown ip, cls, ins, job, instance N/A
etcd_snap_db_fsync_duration_seconds_sum Unknown ip, cls, ins, job, instance N/A
etcd_snap_db_save_total_duration_seconds_bucket Unknown ip, cls, ins, job, instance, le N/A
etcd_snap_db_save_total_duration_seconds_count Unknown ip, cls, ins, job, instance N/A
etcd_snap_db_save_total_duration_seconds_sum Unknown ip, cls, ins, job, instance N/A
etcd_snap_fsync_duration_seconds_bucket Unknown ip, cls, ins, job, instance, le N/A
etcd_snap_fsync_duration_seconds_count Unknown ip, cls, ins, job, instance N/A
etcd_snap_fsync_duration_seconds_sum Unknown ip, cls, ins, job, instance N/A
go_gc_duration_seconds summary ip, cls, ins, job, instance, quantile A summary of the pause duration of garbage collection cycles.
go_gc_duration_seconds_count Unknown ip, cls, ins, job, instance N/A
go_gc_duration_seconds_sum Unknown ip, cls, ins, job, instance N/A
go_goroutines gauge ip, cls, ins, job, instance Number of goroutines that currently exist.
go_info gauge ip, cls, ins, job, version, instance Information about the Go environment.
go_memstats_alloc_bytes counter ip, cls, ins, job, instance Total number of bytes allocated, even if freed.
go_memstats_alloc_bytes_total counter ip, cls, ins, job, instance Total number of bytes allocated, even if freed.
go_memstats_buck_hash_sys_bytes gauge ip, cls, ins, job, instance Number of bytes used by the profiling bucket hash table.
go_memstats_frees_total counter ip, cls, ins, job, instance Total number of frees.
go_memstats_gc_sys_bytes gauge ip, cls, ins, job, instance Number of bytes used for garbage collection system metadata.
go_memstats_heap_alloc_bytes gauge ip, cls, ins, job, instance Number of heap bytes allocated and still in use.
go_memstats_heap_idle_bytes gauge ip, cls, ins, job, instance Number of heap bytes waiting to be used.
go_memstats_heap_inuse_bytes gauge ip, cls, ins, job, instance Number of heap bytes that are in use.
go_memstats_heap_objects gauge ip, cls, ins, job, instance Number of allocated objects.
go_memstats_heap_released_bytes gauge ip, cls, ins, job, instance Number of heap bytes released to OS.
go_memstats_heap_sys_bytes gauge ip, cls, ins, job, instance Number of heap bytes obtained from system.
go_memstats_last_gc_time_seconds gauge ip, cls, ins, job, instance Number of seconds since 1970 of last garbage collection.
go_memstats_lookups_total counter ip, cls, ins, job, instance Total number of pointer lookups.
go_memstats_mallocs_total counter ip, cls, ins, job, instance Total number of mallocs.
go_memstats_mcache_inuse_bytes gauge ip, cls, ins, job, instance Number of bytes in use by mcache structures.
go_memstats_mcache_sys_bytes gauge ip, cls, ins, job, instance Number of bytes used for mcache structures obtained from system.
go_memstats_mspan_inuse_bytes gauge ip, cls, ins, job, instance Number of bytes in use by mspan structures.
go_memstats_mspan_sys_bytes gauge ip, cls, ins, job, instance Number of bytes used for mspan structures obtained from system.
go_memstats_next_gc_bytes gauge ip, cls, ins, job, instance Number of heap bytes when next garbage collection will take place.
go_memstats_other_sys_bytes gauge ip, cls, ins, job, instance Number of bytes used for other system allocations.
go_memstats_stack_inuse_bytes gauge ip, cls, ins, job, instance Number of bytes in use by the stack allocator.
go_memstats_stack_sys_bytes gauge ip, cls, ins, job, instance Number of bytes obtained from system for stack allocator.
go_memstats_sys_bytes gauge ip, cls, ins, job, instance Number of bytes obtained from system.
go_threads gauge ip, cls, ins, job, instance Number of OS threads created.
logger_log_entries_size_greater_than_buffer_total counter ip, cls, ins, job, instance Number of log entries which are larger than the log buffer
logger_log_read_operations_failed_total counter ip, cls, ins, job, instance Number of log reads from container stdio that failed
logger_log_write_operations_failed_total counter ip, cls, ins, job, instance Number of log write operations that failed
process_cpu_seconds_total counter ip, cls, ins, job, instance Total user and system CPU time spent in seconds.
process_max_fds gauge ip, cls, ins, job, instance Maximum number of open file descriptors.
process_open_fds gauge ip, cls, ins, job, instance Number of open file descriptors.
process_resident_memory_bytes gauge ip, cls, ins, job, instance Resident memory size in bytes.
process_start_time_seconds gauge ip, cls, ins, job, instance Start time of the process since unix epoch in seconds.
process_virtual_memory_bytes gauge ip, cls, ins, job, instance Virtual memory size in bytes.
process_virtual_memory_max_bytes gauge ip, cls, ins, job, instance Maximum amount of virtual memory available in bytes.
promhttp_metric_handler_requests_in_flight gauge ip, cls, ins, job, instance Current number of scrapes being served.
promhttp_metric_handler_requests_total counter ip, cls, ins, job, instance, code Total number of scrapes by HTTP status code.
scrape_duration_seconds Unknown ip, cls, ins, job, instance N/A
scrape_samples_post_metric_relabeling Unknown ip, cls, ins, job, instance N/A
scrape_samples_scraped Unknown ip, cls, ins, job, instance N/A
scrape_series_added Unknown ip, cls, ins, job, instance N/A
swarm_dispatcher_scheduling_delay_seconds_bucket Unknown ip, cls, ins, job, instance, le N/A
swarm_dispatcher_scheduling_delay_seconds_count Unknown ip, cls, ins, job, instance N/A
swarm_dispatcher_scheduling_delay_seconds_sum Unknown ip, cls, ins, job, instance N/A
swarm_manager_configs_total gauge ip, cls, ins, job, instance The number of configs in the cluster object store
swarm_manager_leader gauge ip, cls, ins, job, instance Indicates if this manager node is a leader
swarm_manager_networks_total gauge ip, cls, ins, job, instance The number of networks in the cluster object store
swarm_manager_nodes gauge ip, cls, ins, job, instance, state The number of nodes
swarm_manager_secrets_total gauge ip, cls, ins, job, instance The number of secrets in the cluster object store
swarm_manager_services_total gauge ip, cls, ins, job, instance The number of services in the cluster object store
swarm_manager_tasks_total gauge ip, cls, ins, job, instance, state The number of tasks in the cluster object store
swarm_node_manager gauge ip, cls, ins, job, instance Whether this node is a manager or not
swarm_raft_snapshot_latency_seconds_bucket Unknown ip, cls, ins, job, instance, le N/A
swarm_raft_snapshot_latency_seconds_count Unknown ip, cls, ins, job, instance N/A
swarm_raft_snapshot_latency_seconds_sum Unknown ip, cls, ins, job, instance N/A
swarm_raft_transaction_latency_seconds_bucket Unknown ip, cls, ins, job, instance, le N/A
swarm_raft_transaction_latency_seconds_count Unknown ip, cls, ins, job, instance N/A
swarm_raft_transaction_latency_seconds_sum Unknown ip, cls, ins, job, instance N/A
swarm_store_batch_latency_seconds_bucket Unknown ip, cls, ins, job, instance, le N/A
swarm_store_batch_latency_seconds_count Unknown ip, cls, ins, job, instance N/A
swarm_store_batch_latency_seconds_sum Unknown ip, cls, ins, job, instance N/A
swarm_store_lookup_latency_seconds_bucket Unknown ip, cls, ins, job, instance, le N/A
swarm_store_lookup_latency_seconds_count Unknown ip, cls, ins, job, instance N/A
swarm_store_lookup_latency_seconds_sum Unknown ip, cls, ins, job, instance N/A
swarm_store_memory_store_lock_duration_seconds_bucket Unknown ip, cls, ins, job, instance, le N/A
swarm_store_memory_store_lock_duration_seconds_count Unknown ip, cls, ins, job, instance N/A
swarm_store_memory_store_lock_duration_seconds_sum Unknown ip, cls, ins, job, instance N/A
swarm_store_read_tx_latency_seconds_bucket Unknown ip, cls, ins, job, instance, le N/A
swarm_store_read_tx_latency_seconds_count Unknown ip, cls, ins, job, instance N/A
swarm_store_read_tx_latency_seconds_sum Unknown ip, cls, ins, job, instance N/A
swarm_store_write_tx_latency_seconds_bucket Unknown ip, cls, ins, job, instance, le N/A
swarm_store_write_tx_latency_seconds_count Unknown ip, cls, ins, job, instance N/A
swarm_store_write_tx_latency_seconds_sum Unknown ip, cls, ins, job, instance N/A
up Unknown ip, cls, ins, job, instance N/A

14.5 - FAQ

Frequently asked questions about the Pigsty Docker module

Who Can Run Docker Commands?

By default, Pigsty adds both the management user running the playbook on the remote node (i.e., the SSH login user on the target node) and the admin user specified in the node_admin_username parameter to the Docker operating system group. All users in this group (docker) can manage Docker using the docker CLI command.

If you want other users to be able to run Docker commands, add that OS user to the docker group:

usermod -aG docker <username>

Working Through a Proxy

During Docker installation, if the proxy_env parameter exists, the HTTP proxy server configuration will be written to the /etc/docker/daemon.json configuration file.

Docker will use this proxy server when pulling images from upstream registries.

Tip: Running configure with the -x flag will write the proxy server configuration from your current environment into proxy_env.

Using Mirror Registries

If DockerHub access is slow in mainland China network environments, you can prioritize:

For example:

docker login quay.io    # Enter username and password to log in

Adding Docker to Monitoring

During Docker module installation, you can register Docker as a monitoring target by running the docker_register subtask (or alias tag add_metrics) for specific nodes:

./docker.yml -l <your-node-selector> -t docker_register

Using Software Templates

Pigsty provides a collection of software templates that can be launched using Docker Compose, ready to use out of the box.

But you need to install the Docker module first.

15 - Module: JUICE

Use JuiceFS distributed filesystem with PostgreSQL metadata to provide shared POSIX storage.

JuiceFS is a high-performance POSIX-compatible distributed filesystem that can mount object storage or databases as a local filesystem.

The JUICE module depends on NODE for infrastructure and package repo, and typically uses PGSQL as the metadata engine. Data storage can be PostgreSQL (in a jfs_blob table) or Silo / S3-compatible object storage provided by the MINIO module. Monitoring relies on INFRA VictoriaMetrics.

flowchart LR
    subgraph Client["App/User"]
        app["POSIX Access"]
    end

    subgraph JUICE["JUICE"]
        jfs["JuiceFS Mount"]
    end

    subgraph PGSQL["PGSQL"]
        meta["Metadata DB"]
        blob["Data DB / jfs_blob (optional)"]
    end

    subgraph Object["Object Storage (optional)"]
        s3["Silo / S3"]
    end

    subgraph INFRA["INFRA (optional)"]
        vm["VictoriaMetrics"]
    end

    app --> jfs
    jfs --> meta
    jfs -.->|alternative data backend| blob
    jfs -.->|alternative data backend| s3
    jfs -->|/metrics| vm

    style JUICE fill:#5B9CD5,stroke:#4178a8,color:#fff
    style PGSQL fill:#3E668F,stroke:#2d4a66,color:#fff
    style Object fill:#FCDB72,stroke:#d4b85e,color:#333
    style INFRA fill:#999,stroke:#666,color:#fff

Features

  • PostgreSQL metadata: Metadata stored in PostgreSQL for easy management and backup
  • Multi-instance: One node can mount multiple independent filesystem instances
  • Multiple data backends: PostgreSQL, Silo/MinIO, S3, and more; metadata and file data remain separate roles
  • Monitoring integration: Each instance exposes Prometheus / Victoria-format metrics port
  • Simple config: Describe instances with the juice_instances dict

Quick Start

Minimal config example (single instance):

juice_instances:
  jfs:
    path: /fs
    meta: postgres://dbuser_meta:[email protected]:5432/meta
    data: --storage postgres --bucket 10.10.10.10:5432/meta --access-key dbuser_meta --secret-key DBUser.Meta
    port: 9567

Deploy:

./juice.yml -l <host>

15.1 - Configuration

JUICE module configuration, instance definition, storage backends, and mount options.

Concepts and Implementation

JuiceFS consists of a metadata engine and data storage. In the current version, meta is passed through to juicefs as the metadata engine URL, and PostgreSQL is typically used in production. Data storage is defined by data options passed to juicefs format.

JUICE module core commands:

# Format (only effective on first creation)
juicefs format --no-update <data> "<meta>" "<name>"

# Mount
juicefs mount <mount_opts> --cache-dir <juice_cache> --metrics 0.0.0.0:<port> <meta> <path>

Notes:

  • --no-update ensures existing filesystems are not overwritten.
  • data is only used for initial format; it does not affect existing filesystems.
  • mount is only used during mount, you can pass cache and concurrency options.

Module Parameters

JUICE module has only two parameters:

Parameter Type Level Description
juice_cache path C JuiceFS shared cache directory
juice_instances dict I JuiceFS instance dict (can be empty)
  • juice_cache: shared local cache directory for all instances, default /data/juice
  • juice_instances: instance-level dict, key is filesystem name; an empty dict means no instances are managed

Instance Configuration

Each entry in juice_instances represents a JuiceFS instance:

Field Required Default Description
path Yes - Mount point path, e.g. /fs
meta Yes - Metadata engine URL (PostgreSQL recommended)
data No '' juicefs format options (storage backend)
unit No juicefs-<name> systemd service name
mount No '' Extra juicefs mount options
port No 9567 Metrics port (unique per node)
owner No root Mount point owner
group No root Mount point group
mode No 0755 Mount point permissions
state No create create / absent
Important
  • It’s recommended to explicitly set data on first format to make the storage backend clear.
  • Multiple instances on the same node must use different port values.

Example:

juice_instances:
  jfs:
    path: /fs
    meta: postgres://dbuser_meta:[email protected]:5432/meta
    data: --storage postgres --bucket 10.10.10.10:5432/meta --access-key dbuser_meta --secret-key DBUser.Meta
    port: 9567

Storage Backends

data is appended to juicefs format, any supported backend works. Common examples:

PostgreSQL Data Backend 

juice_instances:
  jfs:
    path: /fs
    meta: postgres://dbuser_meta:[email protected]:5432/meta
    data: --storage postgres --bucket 10.10.10.10:5432/meta --access-key dbuser_meta --secret-key DBUser.Meta

JuiceFS creates a jfs_blob table in the database selected by --bucket for file data. This PostgreSQL data backend and the meta metadata engine are separate roles; they may use one database or be deployed separately. The database and a user with read/write privileges must already exist.

Silo / MinIO-Compatible Object Storage 

juice_instances:
  jfs:
    path: /fs
    meta: postgres://dbuser_meta:[email protected]:5432/meta
    data: --storage minio --bucket https://sss.pigsty:9000/juice --access-key <s3_access_key> --secret-key <s3_secret_key>

S3-Compatible Storage 

juice_instances:
  jfs:
    path: /fs
    meta: postgres://dbuser_meta:[email protected]:5432/meta
    data: --storage s3 --bucket https://s3.amazonaws.com/my-bucket --access-key AKIAXXXXXXXX --secret-key XXXXXXXXXX

Typical Configurations

Multi-Instance (Same Node) 

juice_instances:
  pgfs:
    path: /pgfs
    meta: postgres://dbuser_meta:[email protected]:5432/meta
    data: --storage postgres --bucket 10.10.10.10:5432/meta --access-key dbuser_meta --secret-key DBUser.Meta
    port: 9567
  shared:
    path: /shared
    meta: postgres://dbuser_meta:[email protected]:5432/shared
    data: --storage minio --bucket https://sss.pigsty:9000/shared --access-key <s3_access_key> --secret-key <s3_secret_key>
    port: 9568
    owner: postgres
    group: postgres

Shared Mount Across Nodes

Mount the same JuiceFS on multiple nodes:

app:
  hosts:
    10.10.10.11: { juice_instances: { shared: { path: /shared, meta: "postgres://...", port: 9567 } } }
    10.10.10.12: { juice_instances: { shared: { path: /shared, meta: "postgres://...", port: 9567 } } }

Only one node needs to format the filesystem; others will skip via --no-update.

Notes

  • port is exposed on 0.0.0.0. Use firewall or security group to restrict access.
  • Changing data will not update an existing filesystem; handle migration manually.
  • meta and data may contain database or object-storage credentials. Restrict access to pigsty.yml, use dedicated least-privilege accounts, and never keep example passwords in production.

15.2 - Parameters

JUICE module parameters (2 total).

JUICE module has 2 parameters:

Parameter Overview

Parameter Type Level Description
juice_cache path C JuiceFS shared cache directory
juice_instances dict I JuiceFS instance definition dict (can be empty)

Level: C = cluster level, I = instance level.

Default Parameters

Defined in roles/juice/defaults/main.yml:

#-----------------------------------------------------------------
# JUICE
#-----------------------------------------------------------------
juice_cache: /data/juice
juice_instances: {}

juice_cache

Parameter: juice_cache, type: path, level: C

Shared local cache directory for all JuiceFS instances, default /data/juice. JuiceFS isolates caches by filesystem UUID under this directory.

juice_cache: /data/juice

juice_instances

Parameter: juice_instances, type: dict, level: I

Instance definition dict, usually defined at instance level. Default is an empty dict (meaning no instances are deployed). Key is filesystem name, value is instance config object.

juice_instances:
  jfs:
    path: /fs
    meta: postgres://u:p@h:5432/db
    data: --storage postgres --bucket ...
    port: 9567

Instance fields:

Field Required Default Description
path Yes - Mount point path
meta Yes - Metadata engine URL (PostgreSQL recommended)
data No '' juicefs format options (only effective on first creation)
unit No juicefs-<name> systemd service name
mount No '' Extra juicefs mount options
port No 9567 Metrics port (unique per node)
owner No root Mount point owner
group No root Mount point group
mode No 0755 Mount point permissions
state No create create / absent
Note
  • data is only used by juicefs format, it will not update an existing filesystem.
  • Multiple instances on the same node must use different port values.

15.3 - Playbook

JUICE module playbook guide.

JUICE module provides juice.yml playbook to deploy and remove JuiceFS instances.

juice.yml

Task structure in juice.yml:

juice_id        : validate config, check port conflicts
juice_install   : install juicefs package
juice_cache     : create shared cache dir
juice_clean     : remove instance (state=absent)
juice_instance  : create instance (state=create)
  - juice_init  : format filesystem (--no-update)
  - juice_dir   : create mount dir
  - juice_config: render env file and systemd unit
  - juice_launch: start service and wait for metrics port
juice_register  : register to VictoriaMetrics targets

Scope

Scope Limit Description
Node -l <host> Deploy all instances on the node
Instance -l <host> -e fsname=<name> Only handle specified instance

Examples:

./juice.yml -l 10.10.10.10                 # deploy all instances on the node
./juice.yml -l 10.10.10.10 -e fsname=jfs   # only deploy jfs instance

Common Tags

Tag Description
juice_id Validate juice_instances and port conflicts
juice_install Install juicefs package
juice_cache Create shared cache dir
juice_clean Remove instance (state=absent)
juice_instance Create instance (umbrella tag)
juice_init Format filesystem
juice_dir Create mount dir
juice_config Render config files
juice_launch Start service
juice_register Write VictoriaMetrics target file

Config Updates

Render config only (no restart):

./juice.yml -l <host> -t juice_config

Update config and ensure service is online (without force restart):

./juice.yml -l <host> -t juice_config,juice_launch

If you need new mount options to take effect immediately, manually restart the instance service:

systemctl restart juicefs-<name>

Remove Instance

Removal flow:

  1. Set instance state to absent
  2. Run juice_clean
juice_instances:
  jfs:
    path: /fs
    meta: postgres://...
    state: absent
./juice.yml -l <host> -t juice_clean,juice_register
./juice.yml -l <host> -e fsname=jfs -t juice_clean,juice_register

Removal includes stopping the service, lazy unmounting, removing systemd unit and environment files, and reloading systemd. juice_register then rewrites the node’s target file and removes stale scrape endpoints. Running only juice_clean does not update monitoring targets. PostgreSQL metadata, PostgreSQL jfs_blob data tables, and object-storage data are not deleted.

Monitoring Registration

juice_register writes target file on infra node:

/infra/targets/juice/<hostname>.yml

To re-register manually:

./juice.yml -l <host> -t juice_register

15.4 - Administration

JUICE module operations and troubleshooting guide.

Common operations:

See FAQ for more.

Initialize Instance 

./juice.yml -l <host>
./juice.yml -l <host> -e fsname=<name>

Initialization steps:

  • Install juicefs package
  • Create shared cache dir (default /data/juice)
  • Run juicefs format --no-update (only effective on first creation)
  • Create mount point and set permissions
  • Render systemd unit and env files
  • Start service and wait for metrics port
  • Register to VictoriaMetrics (if infra node exists)

Reconfigure

After changing config, it’s recommended to run (update config and ensure service is online):

./juice.yml -l <host> -t juice_config,juice_launch

Render config without touching service state:

./juice.yml -l <host> -t juice_config

Notes:

  • juice_config,juice_launch ensures service is started, but does not force-restart an already running instance
  • data only takes effect on the first format
  • After changing mount options, manually restart the instance service (systemctl restart juicefs-<name>)

Remove Instance

  1. Set instance state to absent
  2. Run juice_clean
juice_instances:
  jfs:
    path: /fs
    meta: postgres://...
    state: absent
./juice.yml -l <host> -t juice_clean,juice_register
./juice.yml -l <host> -e fsname=jfs -t juice_clean,juice_register

Removal actions:

  • Stop systemd service
  • umount -l lazy unmount
  • Remove unit and env files
  • Reload systemd
  • Rewrite this node’s VictoriaMetrics target file, removing instances with state=absent

PostgreSQL metadata, PostgreSQL jfs_blob data tables, and object-storage data are not deleted.

Running only -t juice_clean does not update monitoring targets and temporarily leaves stale scrape endpoints for removed instances. The commands above therefore run juice_register as well.

Add New Instance

Add a new instance in config, ensure unique port:

juice_instances:
  newfs:
    path: /newfs
    meta: postgres://...
    data: --storage minio --bucket https://sss.pigsty:9000/newfs --access-key <s3_access_key> --secret-key <s3_secret_key>
    port: 9568

Deploy:

./juice.yml -l <host> -e fsname=newfs

Shared Mount Across Nodes

Configure the same meta and instance name on multiple nodes:

app:
  hosts:
    10.10.10.11: { juice_instances: { shared: { path: /shared, meta: "postgres://...", port: 9567 } } }
    10.10.10.12: { juice_instances: { shared: { path: /shared, meta: "postgres://...", port: 9567 } } }

Only one node needs to format the filesystem; others will skip via --no-update.

PITR Recovery

JuiceFS metadata and data must be restored to a mutually consistent state. Before any restore, stop every writer and unmount/stop the corresponding JuiceFS service on every client, identify the exact PostgreSQL cluster and target time, and confirm an available backup:

# Inspect Patroni members; inspect the target stanza's backup on a database node
pig pt list <cluster>
pig pb info -s <stanza>

# On the target database node, run the restore command as postgres
sudo -iu postgres pg-pitr -s <stanza> -t "2026-08-14 10:30:00+08"
PITR overwrites the PostgreSQL data directory

Only after confirming the exact cluster name, a recent backup, the recovery target, and a rollback plan should you follow the PostgreSQL PITR tutorial to stop Patroni/PostgreSQL and perform the restore. pg-pitr does not stop services, repair Patroni/DCS, validate data, or rebuild replicas; the command above is not a complete recovery procedure.

If metadata and the --storage postgres jfs_blob table are in the same restored PostgreSQL database, database PITR can return both to one point in time. If they reside in different databases or clusters, design a coordinated recovery point for both.

If file data is in Silo/S3, PostgreSQL PITR rolls back metadata only, not objects. Newer objects may remain, while old objects that were deleted or collected may be unavailable. Recoverability depends on object versioning, trash, and lifecycle policies; do not run garbage collection until validation is complete.

Troubleshooting

Mount Fails 

systemctl status juicefs-jfs
journalctl -u juicefs-jfs -f
mountpoint /fs

Metadata Connection Issues 

psql "postgres://dbuser_meta:[email protected]:5432/meta" -c "SELECT 1"

Metrics Port Check 

ss -tlnp | grep 9567
curl http://localhost:9567/metrics

Performance Tuning

Pass juicefs mount options via mount:

juice_instances:
  jfs:
    path: /fs
    meta: postgres://...
    mount: --cache-size 102400 --prefetch 3 --max-uploads 50

Key metrics to watch:

  • juicefs_blockcache_hits/juicefs_blockcache_miss: cache hit ratio
  • juicefs_object_request_durations_histogram_seconds: object storage latency
  • juicefs_transaction_durations_histogram_seconds: metadata transaction latency

15.5 - Monitoring

JUICE module monitoring and metrics.

JuiceFS instances expose Prometheus metrics via juicefs mount --metrics. In JUICE, metrics listen on 0.0.0.0:<port>, default port 9567.

Monitoring Architecture 

JuiceFS Mount (metrics: 0.0.0.0:<port>)
    ↓
VictoriaMetrics (scrape)
    ↓
Grafana Dashboard

If INFRA is deployed, juice_register writes scrape targets to:

/infra/targets/juice/<hostname>.yml

The current source includes the Node JuiceFS dashboard (UID: node-juice) for capacity, cache, object-storage, metadata-transaction, and client-resource metrics across JuiceFS mounts on a node.

Target File Example 

- labels: { ip: 10.10.10.10, ins: "node-jfs", cls: "jfs" }
  targets: [ 10.10.10.10:9567 ]

To register manually:

./juice.yml -l <host> -t juice_register

Key Metrics

Object Storage

Metric Type Description
juicefs_object_request_durations_histogram_seconds histogram Object storage request latency
juicefs_object_request_errors counter Object storage errors

Cache

Metric Type Description
juicefs_blockcache_hits counter Cache hits
juicefs_blockcache_miss counter Cache misses

Metadata Transactions

Metric Type Description
juicefs_transaction_durations_histogram_seconds histogram Metadata transaction latency (histogram)
juicefs_transaction_durations_histogram_seconds_count counter Metadata transaction request count

Common PromQL

Cache hit ratio:

rate(juicefs_blockcache_hits[5m]) /
(rate(juicefs_blockcache_hits[5m]) + rate(juicefs_blockcache_miss[5m]))

Object storage P99 latency:

histogram_quantile(0.99, rate(juicefs_object_request_durations_histogram_seconds_bucket[5m]))

15.6 - FAQ

JUICE module frequently asked questions.

Port Conflicts?

Multiple instances on the same node must use different port values. Example:

juice_instances:
  fs1:
    path: /fs1
    meta: postgres://...
    port: 9567
  fs2:
    path: /fs2
    meta: postgres://...
    port: 9568

Why does changing data not take effect?

data is only used by juicefs format --no-update. After filesystem creation it will not change. To switch backend, migrate data and reformat manually.

How to add a new instance?

  1. Add instance definition in config
  2. Run:
./juice.yml -l <host> -e fsname=<name>

How to remove an instance?

  1. Set instance state to absent
  2. Run:
./juice.yml -l <host> -t juice_clean,juice_register

Removal does not delete PostgreSQL metadata or object storage data. juice_register refreshes the target file; running only juice_clean leaves a stale monitoring scrape endpoint.

Where is file data stored?

Depends on data:

  • --storage postgres: JuiceFS creates a jfs_blob table in the PostgreSQL database selected by --bucket
  • --storage minio/s3: data is stored in a Silo/S3-compatible object-storage bucket

Metadata is stored in the metadata engine defined by meta (in Pigsty production scenarios, this is usually PostgreSQL).

Multi-node mount notes?

  • Use the same meta and instance name on all nodes
  • Only one node needs to format; others will skip
  • Ensure port does not conflict on each node

Monitoring target not generated?

juice_register only writes /infra/targets/juice/ when infra group exists. You can run manually:

./juice.yml -l <host> -t juice_register

How to change mount options?

After updating mount in the instance, refresh config first and then manually restart the service:

./juice.yml -l <host> -t juice_config,juice_launch
systemctl restart juicefs-<name>

16 - Module: VIBE

Deploy an AI coding sandbox with Pigsty: Code-Server, JupyterLab, Node.js, Claude Code, and Codex CLI.

The VIBE module provides a browser-based dev environment with Code-Server, JupyterLab, Node.js, Claude Code, and Codex CLI, and can work with JUICE shared storage and PGSQL database capabilities.

VIBE depends on NODE and INFRA:

  • NODE provides base software and Python uv environment
  • INFRA provides Nginx reverse proxy, Grafana and portal entry

Components

Component Description Local Port Access Path
Code-Server VS Code in browser 8443 /code/
JupyterLab Interactive notebooks 8888 /jupyter/
Node.js Runtime and npm - CLI
Claude Code CLI + observability config - CLI / Grafana
Codex CLI CLI installation; configuration is not managed - CLI

Notes:

  • Code-Server listens on 127.0.0.1:8443, exposed via Nginx
  • JupyterLab listens on 0.0.0.0:8888, base path /jupyter/; the current template allows any Origin and disables XSRF checks, so it is for trusted development networks only
  • Module default is jupyter_enabled: false, while conf/vibe.yml template explicitly enables Jupyter
Development sandbox, not a production security baseline

The VIBE Jupyter template relies on its token as the default authentication barrier, and conf/vibe.yml does not enable additional Basic Auth on infra_portal.home. Set a strong random token, restrict port 8888 and portal sources, and use a trusted TLS endpoint before deployment. Never expose the default template directly to the Internet.

Quick Start 

./configure -c vibe
./deploy.yml --check          # preview NODE, INFRA, ETCD, MINIO, and PGSQL targets
./deploy.yml                  # deploy after confirming the target set
./juice.yml -l <host> --check # optional shared storage preview
./juice.yml -l <host>         # deploy shared storage after review
./vibe.yml -l <host> --check  # preview VIBE changes
./vibe.yml -l <host>          # deploy VIBE after review

Default entry points (via infra_portal.home):

  • Code-Server: https://<domain>/code/
  • JupyterLab: https://<domain>/jupyter/
  • Claude Dashboard: https://<domain>/ui/d/claude-code

Features

  • Unified workspace: vibe_data as root for Code-Server and Jupyter
  • Optional shared storage: work with JUICE for multi-node sharing
  • Observability: Claude Code OpenTelemetry integrates with VictoriaMetrics/VictoriaLogs
  • Composable: enable Code/Jupyter/Node.js/Claude/Codex as needed

Documentation

16.1 - Configuration

VIBE module configuration for Code-Server, JupyterLab, Node.js, Claude Code, and Codex CLI.

VIBE supports enabling components on demand and exposes services via a unified workspace and Nginx portal.

Overview

Component Enable Param Default Description
Code-Server code_enabled Enabled VS Code in browser
JupyterLab jupyter_enabled Disabled Notebook / terminal / editor
Node.js nodejs_enabled Enabled Node.js runtime and npm
Claude Code claude_enabled Enabled CLI installation, config, and observability
Codex CLI codex_enabled Enabled CLI installation only; configuration is not managed

Note: module default is jupyter_enabled: false, while conf/vibe.yml explicitly sets it to true.

Config usually lives in cluster vars, and can be overridden at instance level:

all:
  children:
    infra:
      hosts:
        10.10.10.10:
          vibe_data: /fs
          code_enabled: true
          jupyter_enabled: true
          claude_enabled: true
          codex_enabled: true

Workspace

vibe_data is the unified workspace for VIBE:

  • Code-Server default open directory
  • JupyterLab root_dir
  • Claude Code working dir
  • Render the AGENTS.md context file and create a CLAUDE.md symlink to it

The vibe_dir task creates the directory and context files, owned by node_user.

vibe_data: /fs

Code-Server 

code_enabled: true
code_port: 8443
code_data: /data/code
code_password: Vibe.Coding
code_gallery: openvsx

Notes:

  • Service listens on 127.0.0.1:<code_port> (default 8443), accessed via Nginx /code/
  • Config file: code_data/code-server/config.yaml (default /data/code/code-server/config.yaml)
  • Env file: /etc/default/code, used to configure extension marketplace

Extension marketplace:

  • code_gallery: microsoft uses Microsoft marketplace
  • When region=china, Open VSX defaults to Tsinghua mirror

JupyterLab 

jupyter_enabled: true
jupyter_port: 8888
jupyter_data: /data/jupyter
jupyter_password: Vibe.Coding
jupyter_venv: /data/venv

Notes:

  • Service listens on 0.0.0.0:<jupyter_port> (default 8888), base path /jupyter/
  • Config file: jupyter_data/jupyter_config.py (default /data/jupyter/jupyter_config.py)
  • Login token: c.IdentityProvider.token
  • The v4.5 template also sets allow_origin = '*', disable_check_xsrf = True, and trust_xheaders = True; these relax the security boundary for a reverse-proxied development sandbox
  • Venv is not created automatically, use node_uv_env in NODE module beforehand

Use a high-entropy random token and restrict TCP/8888 to trusted networks or the reverse proxy. infra_portal.home does not enable Basic Auth by default. For a second authentication layer, configure nginx_users and set auth: true on the portal. If you tighten the Jupyter settings, validate WebSockets, origin handling, and login before rollout.

Create venv example:

uv venv /data/venv

Node.js 

nodejs_enabled: true
nodejs_registry: ''
npm_packages: []

Notes:

  • When nodejs_registry is empty and region=china, default registry is https://registry.npmmirror.com
  • npm_packages installs additional global npm packages and is empty by default
  • Claude Code and Codex CLI are installed by their own dedicated tasks

Claude Code

The claude task installs the CLI (claude_install) and writes its configuration (claude_config).

claude_enabled: true
claude_package: '@anthropic-ai/claude-code'
claude_env:
  ANTHROPIC_API_KEY: sk-ant-xxx

When Claude or Codex is enabled, VIBE ensures that the Node.js runtime is installed. Override claude_package to use a different Claude npm package.

Generated files:

  • ~/.claude.json
  • ~/.claude/settings.json

claude_env is merged with default OpenTelemetry env vars, sending telemetry to VictoriaMetrics / VictoriaLogs.

Codex CLI 

codex_enabled: true

The codex task runs npm install -g @openai/codex. VIBE installs Codex CLI only; it does not write Codex configuration or connect it to VIBE’s Claude Code observability.

Nginx Portal

VIBE exposes services through infra_portal. By default, home domain includes /code/ and /jupyter/ paths.

For dedicated domains:

infra_portal:
  code: { domain: code.pigsty, endpoint: "127.0.0.1:8443", websocket: true, auth: true }
  jupyter: { domain: jupyter.pigsty, endpoint: "127.0.0.1:8888", websocket: true, auth: true }
nginx_users:
  devadmin: '<strong-password>'

16.2 - Parameters

VIBE module parameters (18 total).

VIBE module has 18 parameters, grouped as:

  • Common
  • Code-Server
  • JupyterLab
  • Node.js
  • Claude Code
  • Codex CLI

Overview

Parameter Type Level Default Description
vibe_data path C /fs Workspace dir
code_enabled bool C true Enable Code-Server
code_port port C 8443 Code-Server port
code_data path C /data/code Code-Server data dir
code_password string C Vibe.Coding Code-Server password
code_gallery enum C openvsx Extension marketplace
jupyter_enabled bool C false Enable JupyterLab
jupyter_port port C 8888 JupyterLab port
jupyter_data path C /data/jupyter JupyterLab data dir
jupyter_password string C Vibe.Coding JupyterLab token
jupyter_venv path C /data/venv Python venv path
nodejs_enabled bool C true Enable Node.js
nodejs_registry url C '' npm registry mirror
npm_packages string[] C [] Additional global npm packages
claude_enabled bool C true Install and configure Claude Code
claude_package string C @anthropic-ai/claude-code Claude Code npm package
claude_env dict C {} Claude env vars
codex_enabled bool C true Install Codex CLI

Default Parameters

Defined in roles/vibe/defaults/main.yml:

vibe_data: /fs

code_enabled: true
code_port: 8443
code_data: /data/code
code_password: Vibe.Coding
code_gallery: 'openvsx'

jupyter_enabled: false
jupyter_port: 8888
jupyter_data: /data/jupyter
jupyter_password: Vibe.Coding
jupyter_venv: /data/venv

nodejs_enabled: true
nodejs_registry: ''
npm_packages: []

claude_enabled: true
claude_package: '@anthropic-ai/claude-code'
claude_env: {}

codex_enabled: true

Common

vibe_data

Workspace directory, default /fs. Code-Server and JupyterLab use it as their workspace root; vibe_dir renders AGENTS.md here and creates a CLAUDE.md symlink to it.

Code-Server

code_enabled

Enable Code-Server, default true.

code_port

Listen port, default 8443; bound to 127.0.0.1 and forwarded by Nginx /code/.

code_data

Data dir, config file at code_data/code-server/config.yaml (default /data/code/code-server/config.yaml).

code_password

Login password, default Vibe.Coding; it must be changed in production.

code_gallery

Extension marketplace: openvsx / microsoft. When region=china and openvsx, Tsinghua mirror is used.

JupyterLab

jupyter_enabled

Enable JupyterLab. Module default is false; conf/vibe.yml explicitly sets it to true to enable a full sandbox.

jupyter_port

Listen port, default 0.0.0.0:8888.

jupyter_data

Data dir, config file at jupyter_data/jupyter_config.py (default /data/jupyter/jupyter_config.py).

jupyter_password

Access token, default Vibe.Coding, written to c.IdentityProvider.token.

jupyter_venv

Python venv path for JupyterLab, default /data/venv; it must be created beforehand (usually by the NODE module).

Node.js

nodejs_enabled

Enable the standalone Node.js installation task, default true.

nodejs_registry

npm registry mirror; when empty and region=china, defaults to https://registry.npmmirror.com.

npm_packages

Additional global npm packages, tagged nodejs_pkg; empty by default. Claude Code and Codex CLI are installed by their own dedicated tasks and do not need to be added here.

Claude Code

claude_enabled

Enable Claude Code installation and configuration, default true. claude_install installs the CLI, while claude_config writes its configuration.

claude_package

The npm package used for Claude Code; defaults to @anthropic-ai/claude-code.

claude_env

Extra env vars merged into default OpenTelemetry config.

Default env vars include:

  • CLAUDE_CODE_ENABLE_TELEMETRY=1
  • OTEL_METRICS_EXPORTER=otlp
  • OTEL_LOGS_EXPORTER=otlp
  • OTEL_EXPORTER_OTLP_METRICS_PROTOCOL=http/protobuf
  • OTEL_EXPORTER_OTLP_LOGS_PROTOCOL=http/protobuf
  • OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=http://127.0.0.1:8428/opentelemetry/v1/metrics
  • OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=http://127.0.0.1:9428/insert/opentelemetry/v1/logs
  • OTEL_RESOURCE_ATTRIBUTES=ip=<inventory_hostname>,job=claude

Codex CLI

codex_enabled

Whether to install Codex CLI; defaults to true. When enabled, codex_install runs npm install -g @openai/codex. VIBE installs Codex CLI only; it does not manage Codex configuration or configure OpenTelemetry for it.

16.3 - Playbook

VIBE module Ansible playbook guide.

VIBE provides the vibe.yml playbook to deploy Code-Server, JupyterLab, Node.js, Claude Code, and Codex CLI.

vibe.yml includes only node_id and vibe roles, it does not include node/infra. Run deploy.yml first, or explicitly run node.yml and infra.yml.

vibe.yml

vibe.yml:

- name: VIBE
  hosts: all
  become: true
  gather_facts: no
  roles:
    - { role: node_id, tags: id }
    - { role: vibe,    tags: vibe }

Task Structure 

vibe
├── vibe_dir          # create workspace and context files
├── code              # Code-Server
│   ├── code_install
│   ├── code_dir
│   ├── code_config
│   └── code_launch
├── jupyter           # JupyterLab
│   ├── jupyter_install
│   ├── jupyter_dir
│   ├── jupyter_config
│   └── jupyter_launch
├── nodejs            # Node.js runtime and additional npm packages
│   ├── nodejs_install
│   ├── nodejs_config
│   └── nodejs_pkg
├── codex             # Codex CLI
│   └── codex_install
└── claude            # Claude Code
    ├── claude_install
    └── claude_config

Notes:

  • jupyter_install uses uv pip, it does not create venv
  • nodejs_pkg installs only the additional packages declared in npm_packages; the list is empty by default
  • claude_install installs Claude CLI with claude_package, while claude_config writes the ~/.claude configuration
  • codex_install installs @openai/codex and does not manage Codex configuration

Common Commands

Full deploy:

./vibe.yml -l <host> --check   # Preview first
./vibe.yml -l <host>           # Apply after confirming the target

Component-level:

./vibe.yml -l <host> -t code
./vibe.yml -l <host> -t jupyter
./vibe.yml -l <host> -t nodejs
./vibe.yml -l <host> -t claude
./vibe.yml -l <host> -t codex

Config updates:

./vibe.yml -l <host> -t code_config,code_launch
./vibe.yml -l <host> -t jupyter_config
ssh <host> sudo systemctl restart jupyter
./vibe.yml -l <host> -t claude_config

Skip components for this run:

./vibe.yml -l <host> -e code_enabled=false
./vibe.yml -l <host> -e jupyter_enabled=false
./vibe.yml -l <host> -e nodejs_enabled=false
./vibe.yml -l <host> -e claude_enabled=false
./vibe.yml -l <host> -e codex_enabled=false

These switches are task conditions. Setting one to false only skips the corresponding installation and configuration tasks; it does not stop, disable, or uninstall a service or package deployed earlier. To retire Code-Server or JupyterLab, run systemctl disable --now code-server or systemctl disable --now jupyter separately. VIBE currently has no dedicated removal playbook.

Node.js is a runtime dependency of Claude Code and Codex CLI. If nodejs_enabled=false but either claude_enabled or codex_enabled remains true, the nodejs phase still runs. It is skipped only when all three switches are false.

Deployment Order 

./deploy.yml --check          # Preview inventory-defined base components
./deploy.yml                  # Deploy after review
./juice.yml -l <host> --check # Optional shared-storage preview
./juice.yml -l <host>
./vibe.yml -l <host> --check  # VIBE preview
./vibe.yml -l <host>

Idempotency

vibe.yml is idempotent. Re-run after config changes.

16.4 - Administration

VIBE module operations and common admin tasks.

Service Management 

systemctl status code-server
systemctl restart code-server
systemctl status jupyter
systemctl restart jupyter

Logs:

journalctl -u code-server -f
journalctl -u jupyter -f

Workspace and Context

vibe_dir creates these under vibe_data:

  • AGENTS.md: Context file rendered from the role template
  • CLAUDE.md: Symlink to AGENTS.md

Default locations (adjustable via vibe_data):

/fs/CLAUDE.md
/fs/AGENTS.md

Password and Auth

Code-Server

Edit config:

vi /data/code/code-server/config.yaml
systemctl restart code-server

Or via Ansible:

# Persist the new password in inventory first so it does not enter shell history
./vibe.yml -l <host> -t code_config,code_launch --check
./vibe.yml -l <host> -t code_config,code_launch

JupyterLab

Config file: /data/jupyter/jupyter_config.py

Field: c.IdentityProvider.token

The current v4.5 template also allows any Origin and disables XSRF checks, so the token is its primary authentication barrier. Use a strong random token, restrict TCP/8888, and access it through a trusted TLS reverse proxy. Persist a new token in inventory, then preview and apply it:

./vibe.yml -l <host> -t jupyter_config --check
./vibe.yml -l <host> -t jupyter_config
ssh <host> sudo systemctl restart jupyter

Code-Server Extensions 

code-server --install-extension ms-python.python
code-server --list-extensions
code-server --uninstall-extension ms-python.python

Switch extension marketplace:

code_gallery: microsoft

Redeploy:

./vibe.yml -l <host> -t code_config,code_launch

JupyterLab Environment

VIBE does not create venv automatically, ensure jupyter_venv exists:

uv venv /data/venv

Install/upgrade JupyterLab:

uv pip install --python /data/venv/bin/python jupyterlab ipykernel
systemctl restart jupyter

Install extensions (in venv):

source /data/venv/bin/activate
pip install jupyterlab-git
systemctl restart jupyter

Claude Code

The claude_install subtask installs Claude CLI, while claude_config writes the configuration files.

which claude
claude --version

Config files:

  • ~/.claude.json
  • ~/.claude/settings.json

Update config:

./vibe.yml -l <host> -t claude_config

Reinstall/install Claude CLI:

./vibe.yml -l <host> -t claude_install
# or install manually
npm install -g @anthropic-ai/claude-code

Override claude_package if you need a different npm package.

Codex CLI

VIBE installs @openai/codex through codex_install, but does not manage Codex configuration:

which codex
codex --version
./vibe.yml -l <host> -t codex_install

To configure for another user, run as that user or copy the files manually.

File Locations

Component Key Files
Code-Server /data/code/code-server/config.yaml
Code-Server /etc/default/code
Code-Server /etc/systemd/system/code-server.service
JupyterLab /data/jupyter/jupyter_config.py
JupyterLab /etc/default/jupyter
JupyterLab /etc/systemd/system/jupyter.service
Claude Code ~/.claude.json / ~/.claude/settings.json

Troubleshooting

Port checks:

ss -tlnp | grep 8443
ss -tlnp | grep 8888

Nginx entry:

nginx -t
systemctl status nginx

16.5 - Monitoring

VIBE monitoring, focusing on Claude Code observability.

VIBE monitoring mainly focuses on Claude Code OpenTelemetry data. Code-Server and JupyterLab do not expose Prometheus metrics; use systemd and logs for health checks.

Claude Code Observability

VIBE writes default OpenTelemetry env vars into ~/.claude/settings.json:

{
  "env": {
    "CLAUDE_CODE_ENABLE_TELEMETRY": "1",
    "OTEL_METRICS_EXPORTER": "otlp",
    "OTEL_LOGS_EXPORTER": "otlp",
    "OTEL_EXPORTER_OTLP_METRICS_PROTOCOL": "http/protobuf",
    "OTEL_EXPORTER_OTLP_LOGS_PROTOCOL": "http/protobuf",
    "OTEL_EXPORTER_OTLP_METRICS_ENDPOINT": "http://127.0.0.1:8428/opentelemetry/v1/metrics",
    "OTEL_EXPORTER_OTLP_LOGS_ENDPOINT": "http://127.0.0.1:9428/insert/opentelemetry/v1/logs",
    "OTEL_RESOURCE_ATTRIBUTES": "ip=<host>,job=claude"
  }
}

claude_env is merged with the defaults, and can be used for API keys or custom endpoints.

Grafana Dashboard

Grafana includes claude-code dashboard by default:

  • Portal: https://<domain>/ui/d/claude-code
  • Direct: http://<ip>:3000/d/claude-code

Runtime Checks 

systemctl status code-server
systemctl status jupyter
journalctl -u code-server -f
journalctl -u jupyter -f

Port checks:

ss -tlnp | grep 8443
ss -tlnp | grep 8888

Claude Logs Query

Via VictoriaLogs:

curl -G 'http://127.0.0.1:9428/select/logsql/query' \
  --data-urlencode 'query=job:claude'

16.6 - FAQ

VIBE module frequently asked questions.

Deployment

code-server package not found

Ensure NODE and repo config are in place:

yum repolist    # EL
apt-cache policy   # Debian/Ubuntu, read-only inspection
./infra.yml -l infra -t repo --check

JupyterLab installation failed

jupyter_venv must exist:

uv venv /data/venv
./vibe.yml -l <host> -t jupyter --check
./vibe.yml -l <host> -t jupyter

Access

Cannot access /code/ or /jupyter/

  1. Check service status
  2. Check port listening
  3. Check Nginx config
systemctl status code-server
systemctl status jupyter
ss -tlnp | grep 8443
ss -tlnp | grep 8888
nginx -t

WebSocket connection fails

Ensure Nginx enables WebSocket (default is enabled). If using custom infra_portal, set websocket: true.

Password and Token

Change Code-Server password

Persist the new password in inventory first so it does not enter shell history:

./vibe.yml -l <host> -t code_config,code_launch --check
./vibe.yml -l <host> -t code_config,code_launch

Change JupyterLab token

Persist a high-entropy random token in inventory first. The current template allows any Origin and disables XSRF checks, so never use an example token or expose TCP/8888 directly:

./vibe.yml -l <host> -t jupyter_config --check
./vibe.yml -l <host> -t jupyter_config
ssh <host> sudo systemctl restart jupyter

Claude Code

CLI not found

First check whether claude_install completed:

which claude
npm list -g --depth=0 | grep '@anthropic-ai/claude-code'
./vibe.yml -l <host> -t claude_install

If claude_enabled is disabled, install manually:

npm install -g @anthropic-ai/claude-code

Use claude_package to select a different npm package.

Codex CLI not found 

which codex
npm list -g --depth=0 | grep '@openai/codex'
./vibe.yml -l <host> -t codex_install

Confirm that codex_enabled: true. VIBE installs Codex CLI only and does not generate Codex configuration.

API key not set 

export ANTHROPIC_API_KEY=sk-ant-xxx
# or set in claude_env

Telemetry not showing

Check local VictoriaMetrics/VictoriaLogs:

curl http://127.0.0.1:8428/api/v1/status/buildinfo
curl http://127.0.0.1:9428/select/logsql/stats_query

Ensure OTEL endpoints in ~/.claude/settings.json are correct.

Extensions and Plugins

Code-Server extension install fails

  • Check network
  • Try switching code_gallery
  • Or install VSIX manually
code-server --install-extension /path/to/extension.vsix

JupyterLab extension install fails 

source /data/venv/bin/activate
pip install jupyterlab-git
systemctl restart jupyter

17 - Module: KAFKA

Deploy, secure, and monitor Apache Kafka 4.1+ dynamic KRaft clusters with Pigsty.

Kafka is a distributed event-streaming platform. Pigsty’s KAFKA module deploys Apache Kafka 4.1+ dynamic KRaft clusters on managed nodes from RPM/DEB packages, with unified management of security, resources, lifecycle, and observability.

Current status: Beta module

The Kafka module is currently in Beta. Test it thoroughly and confirm it meets your requirements before using it in serious production. That includes dynamic KRaft, strict rolling restart, TLS/SCRAM/ACL, declarative topics/users, credential and certificate rotation, and the full monitoring pipeline.

Module Capabilities

The KAFKA module currently provides:

  • Native dynamic KRaft: no ZooKeeper installed, and no static controller.quorum.voters rendered
  • Three native roles combined / broker / controller, in both combined and separated control-plane/data-plane topologies
  • New clusters get a random Cluster ID and Controller Directory IDs, frozen by a minimal bootstrap manifest that fails closed on conflict
  • Automatic path selection from live health: cold start/repair, serial broker admission, dynamic controller join, or strict single-node rolling restart
  • Checks before and after each rolling step for controller majority and voter catch-up, offline partitions, under-min-ISR, and ISR catch-up
  • Playbook-orchestrated member retirement and failed-node replacement: kafka-rm.yml strict-subset retirement (dead nodes included), three commands to replace a node
  • Two security profiles, plaintext and the production scram (TLS, SCRAM-SHA-512, controller mTLS, ACLs, and default-deny authorization)
  • Declarative convergence of topics, user credentials, ACLs, and quotas without implicit deletion; protected rotation for internal credentials and certificates
  • Full observability: JMX and protocol exporters, 19 recording rules, 15 alert rules, 4 Grafana dashboards, and logs in VictoriaLogs

Module Architecture

The KAFKA module depends on NODE for node management, the package repository, and base monitoring, and on INFRA for VictoriaMetrics, VictoriaLogs, Grafana, and Alertmanager.

flowchart LR
    admin["Pigsty admin node"] -->|"kafka.yml / exact cluster"| kafka["Kafka 4.1+ / dynamic KRaft"]
    kafka --> jmx["Each Kafka JVM / JMX :9404"]
    kafka --> exporter["Up to two brokers / kafka_exporter :9308"]
    kafka --> journal["Journald"]
    jmx --> vm["VictoriaMetrics"]
    exporter --> vm
    journal --> vector["Vector"] --> vl["VictoriaLogs"]
    vm --> grafana["Grafana"]
    vl --> grafana
    vm --> alert["Alertmanager"]

    style kafka fill:#70C1B3,stroke:#4f968b,color:#fff
    style vm fill:#E66B7A,stroke:#b84e5c,color:#fff
    style vl fill:#C98367,stroke:#9e634e,color:#fff
    style grafana fill:#F29C64,stroke:#c77845,color:#fff

Every Kafka JVM has a JMX Exporter injected and is registered as job=kafka. The protocol-level kafka_exporter runs only on the first two broker-capable nodes ordered by kafka_seq; a single-broker cluster runs only one, and pure controllers run none. They return the same logical-cluster view, so recording rules deduplicate before aggregating.

Documentation

Document Contents
Quickstart From a single node to a three-node secure cluster: client access, parameter changes, and go-live checks
Cluster Config Topology, dynamic KRaft, network, storage, security, and resource declarations
Parameters 15 persistent public parameters plus transient operational variables
Administration Status checks, topics, messages, consumer groups, and topology changes
Playbook kafka.yml lifecycle, task tags, rotation, and teardown safeguards
Monitoring Metrics pipeline, dashboards, log queries, and alert rules
Metrics Metric dictionary for JMX, the protocol exporter, and recording rules
FAQ Questions on roles, identity, security, exporters, and scaling

First Time

The Quickstart offers a complete path from scratch, building up step by step: single-node development cluster → three-node TLS/SCRAM/ACL secure cluster → application client access → parameter and resource changes → go-live checks.

If you are already familiar with Kafka and Pigsty, jump straight to Cluster Config or Parameters.

Default Ports

Port Service Deployment scope plaintext scram
9092 Kafka Broker Broker-capable nodes PLAINTEXT SASL_SSL + SCRAM-SHA-512
9093 KRaft Controller Controller-capable nodes PLAINTEXT Mutual TLS
9308 kafka_exporter Up to two broker-capable nodes HTTP metrics HTTP metrics, TLS/SCRAM on the backend
9404 JMX Exporter All Kafka nodes HTTP metrics HTTP metrics

All four ports must differ from one another, and all are adjustable via parameters. The HTTP ports of the JMX and protocol exporters should still be restricted to the monitoring network by firewall.

Current Boundaries

The current role provides a core deployment baseline for Kafka, not a replacement for a full streaming platform or a managed service. The following capabilities still require an explicit runbook or a separate component:

  • Reassignment of existing partitions after broker scale-out and replica rebalancing (member join/retirement/replacement is orchestrated by the playbooks; data movement still needs an explicit plan)
  • Raising the frozen default.replication.factor after scale-out: Kafka 4.3 requires an explicit data-migration and static-config maintenance window
  • Changing the replication factor of an existing topic, deleting topics, and deleting users
  • Online migration of a formatted cluster from plaintext to scram
  • Kafka version upgrades, feature-level finalization, data backup, restore, and disaster drills
  • Multiple listeners, NAT/public addresses, multi-client networks on the same broker, and tiered storage
  • Kafka Connect, Schema Registry, MirrorMaker 2, Cruise Control, and web UIs

These boundaries should be documented explicitly in your production plan, approval process, and drills; they cannot be substituted by rerunning an ordinary inventory.

17.1 - Quick Start

Deploy single-node and 3-node Kafka clusters from scratch, with secure access, parameter tuning, and launch checklist.

This tutorial starts from a minimal single-node cluster, walking through topic creation and message read/write; it then deploys a separate, secured three-node cluster with application users, ACLs, quotas, and production topics; finally, it demonstrates core parameter changes, client access, monitoring verification, and pre-launch checks.

Tutorial Scope

Here, “from scratch” means starting before any Kafka is deployed. You will need a working Pigsty admin node with the base INFRA services already deployed; if you don’t have one yet, complete the Pigsty quick install first. The target nodes require SSH/sudo access and must be manageable by the NODE module.

Learning Path

Stage Goal End Result
1 Deploy a single-node dev cluster One combined node, PLAINTEXT, an RF=1 topic, CLI read/write
2 Deploy a three-node secure HA demo baseline Three combined nodes, dynamic KRaft, TLS/SCRAM/ACL, RF=3/minISR=2
3 Connect application clients Produce/consume using an application principal, the Pigsty CA, and SASL_SSL
4 Change core parameters Walk through heap, broker parameters, topic partitions/retention, and a secure rolling restart
5 Launch acceptance Check quorum, ISR, end-to-end read/write, monitoring, capacity, and runbooks
The Two Examples Are Separate Clusters

The kf-dev and kf-main below are two separate, brand-new clusters. If you do need it, you can also grow the single-node kf-dev into a three-controller cluster in place: declare two new combined nodes and rerun ./kafka.yml -l kf-dev, and the role formats each one, catches it up as an observer, and promotes it with add-controller, one at a time — but for a demo it is still simpler to build a fresh cluster. See Expand Cluster for the semantics.

Before You Begin

Unless noted otherwise, the following commands are run from the project directory on the Pigsty admin node:

cd ~/pigsty

Before you start, confirm that:

  • pigsty.yml is the source of configuration for the current environment — back it up and review its existing contents first;
  • each Kafka node’s inventory_hostname can be resolved and routed directly by every Kafka member and client;
  • the admin node and the Kafka nodes have synchronized clocks;
  • ports 9092, 9093, 9308, and 9404 are free of conflicts;
  • /data/kafka maps to a dedicated data disk or directory that holds nothing else;
  • every kafka.yml run uses -l to select exactly all members of one Kafka cluster;
  • before any real change, you run --check first, review the output, and obtain approval for the change.

The inventory must keep the all.children hierarchy. Merge the groups below into your existing pigsty.yml; do not let the examples overwrite your existing all.vars, infra, etcd, pgsql, or other configuration.

Part 1: Deploy a Single-Node Kafka

1. Define the Cluster

Add the following kf-dev group to all.children. This node omits kafka_role, so it uses the default combined and serves as both broker and controller:

all:
  children:
    # keep your existing infra, etcd, pgsql, and other groups

    kf-dev:
      hosts:
        10.10.10.10: { kafka_seq: 1 }
      vars:
        kafka_cluster: kf-dev
        kafka_data: /data/kafka
        kafka_security: plaintext
        kafka_topics:
          - name: quickstart.events
            partitions: 1
            replication_factor: 1
            config:
              retention.ms: 86400000   # 1 day, tutorial only

This configuration yields:

  • a random cluster ID;
  • a single dynamic KRaft combined node;
  • default RF=1 and minISR=1;
  • a single-partition topic named quickstart.events;
  • a JMX exporter on :9404 and a protocol exporter on :9308.

plaintext has no transport encryption, authentication, or ACLs, so use it only for local development or a trusted, isolated network.

2. Bring the Node Under Management

If this host has not yet been initialized by NODE, run check mode first:

./node.yml --check -l kf-dev

After reviewing the results and getting approval, bring the node under management:

./node.yml -l kf-dev

You can skip this step for a node that Pigsty already manages and whose package repository and time synchronization are healthy. For full NODE preparation and day-to-day management, see Node Administration.

3. Deploy Kafka

Run a check against the full cluster first:

./kafka.yml --check -l kf-dev

Confirm the target is exactly the full membership of kf-dev, review the data path, packages, ports, and configuration changes, then run:

./kafka.yml -l kf-dev

The role installs Java and kafka-stack, generates a random identity and bootstrap manifest, formats the KRaft storage, starts the service, creates the topics, and registers the monitoring targets.

4. Verify the Service and Quorum

Log in to the Kafka node and check the services:

systemctl is-active kafka kafka_exporter
journalctl -u kafka --since '-10 min' --no-pager

Run the role’s own health check:

sudo -u kafka /usr/local/bin/pigsty-kafka-health cluster \
  --bootstrap-server 10.10.10.10:9092 \
  --command-config /etc/kafka/admin.properties

The returned JSON should contain "healthy": true. Continue by checking the dynamic quorum and the topic:

/opt/kafka/bin/kafka-metadata-quorum.sh \
  --bootstrap-server 10.10.10.10:9092 \
  --command-config /etc/kafka/admin.properties \
  describe --status

/opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server 10.10.10.10:9092 \
  --command-config /etc/kafka/admin.properties \
  --describe --topic quickstart.events

You should see a valid LeaderId, a CurrentVoters list that includes this node, and quickstart.events with RF=1 and ISR=1.

5. Produce and Consume Messages

Start a console producer:

/opt/kafka/bin/kafka-console-producer.sh \
  --bootstrap-server 10.10.10.10:9092 \
  --command-config /etc/kafka/admin.properties \
  --topic quickstart.events

Type a few lines of messages, then press Ctrl-D to finish. Consume them from another terminal:

/opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server 10.10.10.10:9092 \
  --command-config /etc/kafka/admin.properties \
  --topic quickstart.events \
  --group quickstart.demo \
  --from-beginning

At this point, the single-node deployment, topic convergence, and message read/write are complete. For further status checks, see Day-to-Day Administration.

Part 2: Deploy a Three-Node Production Baseline

The three-node example is a brand-new kf-main cluster built from three combined nodes. It can tolerate the loss of one controller; business topics use RF=3/minISR=2, and the scram production security profile is enabled.

1. Define the Secured Cluster and Its Resources

Add the following group to your existing all.children:

all:
  children:
    # keep your existing groups

    kf-main:
      hosts:
        10.10.10.11: { kafka_seq: 1 }
        10.10.10.12: { kafka_seq: 2 }
        10.10.10.13: { kafka_seq: 3 }
      vars:
        kafka_cluster: kf-main
        kafka_data: /data/kafka
        kafka_heap_opts: '-Xms4G -Xmx4G'
        kafka_security: scram

        kafka_parameters:
          num.partitions: 12
          num.network.threads: 6
          num.io.threads: 16
          log.retention.hours: 168
          log.segment.bytes: 1073741824

        kafka_users:
          - name: quickstart-app
            password: "{{ vault_kafka_quickstart_password }}"
            acls:
              - resource: topic
                name: quickstart.
                pattern: prefixed
                operations: [Read, Write, Describe]
              - resource: group
                name: quickstart.
                pattern: prefixed
                operations: [Read]
              - resource: cluster
                name: kafka-cluster
                operations: [Describe, IdempotentWrite]
            quota:
              producer_byte_rate: 10485760
              consumer_byte_rate: 20971520

        kafka_topics:
          - name: quickstart.events
            partitions: 12
            replication_factor: 3
            config:
              min.insync.replicas: 2
              cleanup.policy: delete
              retention.ms: 604800000

vault_kafka_quickstart_password must be supplied by your existing Ansible Vault, KMS, or another secret-injection mechanism, and must be at least 12 characters. Never commit a real password directly to Git, logs, or tickets.

The key semantics of this configuration:

  • all three nodes omit kafka_role, so they consistently use combined;
  • the new cluster is bootstrapped directly as dynamic KRaft;
  • scram enables node TLS, controller mTLS, SCRAM-SHA-512, ACLs, and deny-by-default all at once;
  • with three brokers, the initial replication policy is automatically derived as RF=3 and minISR=2;
  • quickstart.events is created explicitly with 12 partitions and 3 replicas;
  • quickstart-app can read and write quickstart.* topics, read quickstart.* groups, and use the idempotent producer;
  • at most two brokers run kafka_exporter, while all three Kafka JVMs run the JMX exporter.

If the three brokers truly reside in different failure domains, you may add kafka_rack: az-a/az-b/az-c to all nodes respectively. Do not use fictitious rack labels to manufacture a disaster-recovery guarantee that does not exist; for the detailed rules, see Cluster Configuration: Rack.

2. Bring Under Management and Deploy

If the nodes are not yet under management:

./node.yml --check -l kf-main
./node.yml -l kf-main

When deploying Kafka, you must select all three members:

./kafka.yml --check -l kf-main
./kafka.yml -l kf-main

You cannot use -l 10.10.10.11 alone: every selected cluster must be complete, and a partial selection is refused. Selecting several complete clusters at once (-l kf-dev,kf-main) or running bare against all clusters is allowed.

3. Verify the Health of All Three Nodes

From the admin node, check the three Kafka services:

ansible kf-main -b -m command -a 'systemctl is-active kafka'

Run the full health check on any broker:

sudo -u kafka /usr/local/bin/pigsty-kafka-health cluster \
  --bootstrap-server 10.10.10.11:9092 \
  --command-config /etc/kafka/admin.properties

Query the quorum and the topic:

/opt/kafka/bin/kafka-metadata-quorum.sh \
  --bootstrap-server 10.10.10.11:9092 \
  --command-config /etc/kafka/admin.properties \
  describe --status

/opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server 10.10.10.11:9092 \
  --command-config /etc/kafka/admin.properties \
  --describe --topic quickstart.events

Before launch, you should see one active controller, three current voters, and three available brokers; every quickstart.events partition has three replicas with ISR=3, and there are no offline, under-replicated, or under-min-ISR partitions.

Part 3: Connect Application Clients

1. Distribute the CA Public Certificate

Securely copy the public CA certificate from the admin node to the application host:

files/pki/ca/ca.crt  ->  /etc/kafka-client/pigsty-ca.crt

ca.crt is a public certificate that is safe to distribute. Never copy, expose, or distribute files/pki/ca/ca.key. On the application host, the CA file should be owned by root and set read-only. An application host already managed by Pigsty needs no copy: the NODE module has installed the same CA at /etc/pki/ca.crt, which the client can reference directly.

2. Create the Client Configuration

On the application host, create /etc/kafka-client/client.properties:

bootstrap.servers=10.10.10.11:9092,10.10.10.12:9092,10.10.10.13:9092

security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="quickstart-app" password="<secret-from-vault>";

ssl.truststore.type=PEM
ssl.truststore.location=/etc/kafka-client/pigsty-ca.crt
ssl.endpoint.identification.algorithm=https

The Kafka Java client supports SASL_SSL + SCRAM and a PEM truststore. A real application should inject the password from a secret manager at runtime rather than committing a file containing the password to the repository. For the complete set of fields, see the Kafka 4.3 SASL/SCRAM and producer configuration references.

3. Why Applications Should Connect Directly to Multiple Brokers

The Kafka client is itself cluster-aware. bootstrap.servers is used only to obtain the initial metadata; once connected, the client uses that metadata to connect directly to the leader broker of each partition and refreshes its routing whenever a leader changes. The standard practice in production is therefore to:

  • configure at least two, and usually three, broker addresses in bootstrap.servers, located in different failure domains;
  • allow the application to reach 9092 on all brokers, and ensure that each broker’s advertised inventory_hostname is resolvable and routable;
  • let producers and consumers rely on the Kafka client’s own retry, metadata-refresh, idempotence, and consumer-group protocols;
  • not place HAProxy, a Keepalived VIP, a layer-4 load balancer, or a layer-7 reverse proxy in front of the Kafka data plane.

A single VIP or LB can neither substitute for the broker addresses in Kafka’s metadata nor transparently forward a connection to the correct partition leader; it only adds complexity around long-lived connection state, fault localization, and capacity planning. If your platform must provide a unified discovery entry point, a DNS name or TCP load balancer can serve as a bootstrap-only entry point, but each broker’s advertised.listeners must still return an address the client can reach directly, and the application must not be permitted to reach only the LB. Scenarios that span NAT, the public internet, Kubernetes, or multiple networks require a dedicated externally reachable address and an additional listener per broker; the current module fixes the inventory address as advertised.listeners and does not support such mappings.

4. Authenticate and Read/Write with the Application Identity

On an application host with the Kafka 4.3 CLI installed, run:

kafka-console-producer.sh \
  --bootstrap-server 10.10.10.11:9092,10.10.10.12:9092,10.10.10.13:9092 \
  --command-config /etc/kafka-client/client.properties \
  --topic quickstart.events

When consuming, use a group prefix permitted by the ACL:

kafka-console-consumer.sh \
  --bootstrap-server 10.10.10.11:9092,10.10.10.12:9092,10.10.10.13:9092 \
  --command-config /etc/kafka-client/client.properties \
  --topic quickstart.events \
  --group quickstart.demo \
  --from-beginning

A production application should also explicitly review its client semantics:

Client Setting Suggested Starting Point Notes
acks all Pairs with RF=3/minISR=2 to avoid waiting on the leader alone
enable.idempotence true Reduces the risk of duplicate writes from retries; requires the IdempotentWrite ACL
group.id A distinct, stable name Do not reuse a group across different business or consumption semantics
Offset commit Choose per workload Auto-commit is simple; manual commit ties commits to business processing results more reliably
client.id An identifiable instance name Helps with logs, quotas, and client diagnostics

Client-side acks, retries, idempotence, batching, compression, and offset strategy are application configuration and should not be written into the broker’s kafka_parameters.

Part 4: Change Core Parameters

Always express persistent intent for Kafka by editing pigsty.yml, never by editing /etc/kafka/server.properties directly. Common intents map as follows:

Goal Parameter Behavior
Adjust the JVM heap kafka_heap_opts A static change; a healthy cluster enters a strict one-node-at-a-time rolling restart
Adjust threads, retention, segments kafka_parameters Broker parameters not owned by the role; static changes require a rolling restart
Adjust topic partitions/retention kafka_topics Online resource convergence; partitions can only increase, never decrease
Adjust application passwords/ACLs/quotas kafka_users Online resource convergence; passwords come from a secret system
Declare failure domains kafka_rack All-or-nothing across every broker-capable node; a change triggers a rolling restart but does not relocate data
Choose a security profile kafka_security Decided only when bootstrapping a new cluster; it cannot be switched online by an ordinary rerun

Example: Adjust the Heap and Broker Default Parameters

Suppose that after load testing you decide to raise the heap to 6G, increase the thread counts, and change the default retention for new topics to 72 hours:

kf-main:
  vars:
    kafka_cluster: kf-main
    kafka_heap_opts: '-Xms6G -Xmx6G'
    kafka_parameters:
      num.partitions: 12
      num.network.threads: 8
      num.io.threads: 24
      log.retention.hours: 72
      log.segment.bytes: 1073741824

Do not copy 6G/8/24 verbatim; these values must be determined by load-testing against your CPU, memory, connection count, message size, partition count, disk, and page cache.

Example: Add Partitions and Shorten Topic Retention

Increase quickstart.events from 12 partitions to 24 and change the retention to three days:

kafka_topics:
  - name: quickstart.events
    partitions: 24
    replication_factor: 3
    config:
      min.insync.replicas: 2
      cleanup.policy: delete
      retention.ms: 259200000

Partitions cannot be reduced. When replication_factor differs from what is live, the role refuses ordinary convergence and requires an explicit partition reassignment; it never relocates existing replicas automatically.

Apply the Change

Whether you change static parameters or dynamic resources, run the full state machine:

./kafka.yml --check -l kf-main
./kafka.yml -l kf-main

Do not run -t kafka_config alone. The role decides automatically: a static change triggers a strict node-by-node rolling restart; when only dynamic resources such as topics or users change, Kafka is not restarted.

The following keys belong to the role itself and must not be placed in kafka_parameters:

kafka_parameters:
  min.insync.replicas: 2          # wrong: owned by the role
  default.replication.factor: 3   # wrong: owned by the role
  listeners: ...                  # wrong: owned by the role

For all 15 public parameters, their defaults, and the reserved keys, see the Parameter Reference.

Part 5: Key Pre-Launch Checks

Topology and Data Safety

  • Production uses at least three brokers and an odd number of controllers; for critical or large clusters, consider a separated 3-controller + N-broker topology;
  • topic RF, minISR, and the producer’s acks form a consistent failure model;
  • kafka_rack expresses only real failure domains, and replica placement has been verified;
  • data-disk capacity, throughput, latency, retention, peak write rate, and recovery time have been load-tested;
  • there is an explicit reassignment plan for after a new broker joins, since existing topics’ RF is not raised automatically;
  • the procedures for Kafka data backup/rebuild and disaster recovery are defined, and failed-node replacement and member retirement have been rehearsed.

Security and Network

  • a new production cluster uses kafka_security: scram from bootstrap onward;
  • application passwords are injected by Vault/KMS/a secret manager and never reach Git or logs;
  • only the CA public certificate is distributed to clients, never the CA private key;
  • clients can resolve and reach the inventory_hostname of every broker directly;
  • 9092/9093 are open only to the principals that need them, and 9308/9404 only to the monitoring network;
  • a review checklist for application principals, topic/group/cluster ACLs, and quotas is in place;
  • a protected rotation of internal credentials and certificates is scheduled.

Operations and Monitoring

  • /usr/local/bin/pigsty-kafka-health cluster reports healthy;
  • the dynamic quorum has exactly one leader, and every expected controller is in the current voters;
  • there are no offline, under-replicated, or under-min-ISR partitions;
  • produce/consume verification is done over the real application network with a real principal;
  • the Kafka Overview, Kafka Instance, Kafka Topic, and Kafka Consumer dashboards show healthy data;
  • alert routing, log search, capacity thresholds, on-call responsibilities, and rollback conditions are confirmed;
  • upgrades, feature-level changes, topic deletion, user deletion, and cluster teardown each have a separate approval process.

For detailed alerts and PromQL, see Monitoring and Alerting; for metric semantics, see Metric Definitions.

Documentation Index and Next Steps

We recommend continuing along the following path:

What You Want to Do Next Document
Plan a combined or separated controller/broker topology, network, rack, storage, and security Cluster Configuration
Look up the 15 public parameters, their defaults, schema, and reserved keys Parameter Reference
Look up quorum, topic, user, message, consumer-group, and scaling operations Day-to-Day Administration
Understand the kafka.yml lifecycle, strict rolling restart, rotation, and cluster teardown Playbook
Use the dashboards, alerts, PromQL, and VictoriaLogs Monitoring and Alerting
Understand every JMX/exporter/recording-rule metric Metric Definitions
Troubleshoot identity conflicts, connectivity, SCRAM, exporters, lag, and scaling issues FAQ
Return to the overview of module capabilities, default ports, and boundaries Kafka Module Home

One recommended reading path is: Quick Start → Cluster Configuration → Parameter Reference → Day-to-Day Administration → Playbook → Monitoring and Alerting → FAQ.

17.2 - Configuration

Plan Kafka dynamic KRaft topology, identity, network, storage, security, and declarative resources.

The KAFKA module expresses cluster intent through 15 persistent public parameters; everything else — topology, listeners, storage subdirectories, replication safety, authorization, and Exporter placement — is derived by the role in a single, consistent way. For a first deployment, start with the Quickstart; for the full field reference, see Parameters.

Plan First, Then Format

kafka_seq becomes the KRaft node.id. For a new cluster, the randomly generated Cluster ID, the initial controller identities, the security profile, and the initial replication policy are all written into the bootstrap manifest. Once storage has been formatted, do not casually change identities, the security profile, or the controller set. The role validates the live state against the manifest and fails closed on any conflict; it never silently overwrites or reformats data.

Pre-Deployment Checklist

Before filling in the inventory, confirm at least the following:

  • The target hosts are already commissioned by NODE, the software repository is reachable, and inventory_hostname is directly routable by every Kafka member and client.
  • A single operation will use -l to select precisely all members of one kafka_cluster — not a single node, a subset of members, or multiple clusters.
  • kafka_seq is unique within the cluster, the controller count is odd, and the broker count matches your fault domains and capacity targets.
  • Ports 9092, 9093, 9308, and 9404 do not collide, and Infra nodes can reach both metrics ports.
  • kafka_data maps to a dedicated filesystem, sized for retention, write peaks, replication traffic, recovery time, and growth headroom.
  • Production uses kafka_security: scram; node and admin clocks are synchronized, the Pigsty CA is available, and application passwords come from a secret source such as Vault.
  • Topic partitions, replicas, min.insync.replicas, and retention policy — along with client acks, retries, and consumer recovery strategy — have been reviewed.
  • Scale-out and scale-in, partition reassignment, upgrades, backup, restore, and controller membership changes each have their own runbook.

Roles and Topology

kafka_role accepts only three values:

Role Kafka process.roles Broker Port Controller Port JMX kafka_exporter
combined broker,controller ✓ ✓ ✓ eligible
broker broker ✓ － ✓ eligible
controller controller － ✓ ✓ －

kafka_role is all-or-nothing: cluster members either all omit it (and consistently use combined) or all declare it explicitly — a mix is refused during the identity precheck. A cluster must contain at least one controller-capable node and at least one broker-capable node; an even number of controllers produces a warning, and production typically uses 3 controllers.

Single-Node Development Cluster

A single node serves as both broker and controller. It cannot tolerate a node failure and is suitable only for development, testing, and feature validation:

kf-dev:
  hosts:
    10.10.10.10: { kafka_seq: 1 }
  vars:
    kafka_cluster: kf-dev

The role derives RF=1 and minISR=1 from the initial broker count. Do not use a single-node topology or the default plaintext security profile directly in production.

Three-Node Combined Deployment

All three nodes serve as both broker and controller — a compact production starting point. Omit every role field to use the default combined:

kf-main:
  hosts:
    10.10.10.11: { kafka_seq: 1 }
    10.10.10.12: { kafka_seq: 2 }
    10.10.10.13: { kafka_seq: 3 }
  vars:
    kafka_cluster: kf-main
    kafka_heap_opts: '-Xms4G -Xmx4G'
    kafka_security: scram
    kafka_parameters:
      num.partitions: 3
      num.network.threads: 6
      num.io.threads: 16
    kafka_topics:
      - name: order.events
        partitions: 12
        replication_factor: 3
        config:
          min.insync.replicas: 2
          cleanup.policy: delete

The initial three brokers automatically get the role-owned replication policy of RF=3 and minISR=2. You neither need nor are allowed to override the internal-topic RF, default.replication.factor, or min.insync.replicas in kafka_parameters. The 4G heap in the example is only illustrative; in production, balance the JVM heap, the operating-system page cache, and any other processes on the same host through load testing.

Separating Controllers and Brokers

Critical or larger clusters can separate the control plane from the data plane. Because explicit roles are present, every member must declare its role:

kf-main:
  hosts:
    10.10.10.11: { kafka_seq: 1, kafka_role: controller }
    10.10.10.12: { kafka_seq: 2, kafka_role: controller }
    10.10.10.13: { kafka_seq: 3, kafka_role: controller }
    10.10.10.21: { kafka_seq: 4, kafka_role: broker }
    10.10.10.22: { kafka_seq: 5, kafka_role: broker }
    10.10.10.23: { kafka_seq: 6, kafka_role: broker }
  vars:
    kafka_cluster: kf-main
    kafka_security: scram

A controller-only node does not listen on 9092 and does not run the protocol Exporter; it still exposes KRaft and JVM state through JMX. At most two kafka_exporter instances are placed on the broker-capable nodes with the lowest kafka_seq.

Dynamic KRaft and the Bootstrap Manifest

A new cluster uses the dynamic quorum directly: every node renders controller.quorum.bootstrap.servers, and no static controller.quorum.voters is generated. At the first format:

  • The Cluster ID is generated randomly, not hashed from the cluster name;
  • The Directory IDs of the initial controllers are generated randomly and frozen;
  • Each node is formatted explicitly with either --initial-controllers or --no-initial-controllers mode;
  • After the first bootstrap startup, the role waits for the dynamic quorum to elect a leader, and verifies that every initial controller’s Directory ID has entered the live quorum.

The bootstrap-only facts live on every cluster member:

/etc/kafka/manifest.yml

Each member of a scram cluster also holds /etc/kafka/secrets.yml. The admin node keeps no kafka state at all: the manifest and secrets are resolved from any member copy on every run, and the issued node certificates live in the shared PKI tree (files/pki/kafka/, with CSRs under files/pki/csr/) and are simply re-signed from the Pigsty CA when absent. The manifest records only the cluster identity, the initial controller identities, the security profile, and the initial RF/minISR. The live cluster is always the authority on runtime facts:

  • When the manifest conflicts with the live identity or security profile, ordinary playbooks fail closed;
  • When an old manifest exists but all data disks are empty, reviving the old cluster is refused;
  • When no member holds a manifest copy while storage is already formatted, the role fails closed and asks you to restore the file on any member first;
  • An already-formatted scram cluster likewise fails closed when no member holds the secret material.

The manifest is the cluster’s birth certificate: after the first commission, membership is authoritative in the live Raft state. Combined/controller nodes newly declared in the inventory are then joined to the dynamic quorum by the playbook (fresh format → observer catch-up → add-controller promotion), and retirement is a kafka-rm.yml strict-subset run (automatic remove-controller and broker unregistration) — see Expand Cluster and Shrink Cluster.

Identity Parameters

Identity Source Example Constraint
Cluster name kafka_cluster kf-main Starts with a letter or digit; only letters, digits, underscores, and hyphens
Node number kafka_seq 1 Non-negative integer, unique within the cluster
Instance name Auto-generated kf-main-1 ${kafka_cluster}-${kafka_seq}
Node role kafka_role combined One of the three native roles
KRaft Cluster ID Randomly generated at bootstrap 22-character Kafka UUID kafka_cluster_id is only a takeover/recovery assertion

An already-formatted node reads cluster.id and node.id from ${kafka_data}/metadata/meta.properties and cross-checks them against the manifest and the inventory; the initial controllers’ Directory IDs are compared against the live quorum after startup. An identity mismatch is a protective failure and must not be worked around by deleting meta.properties or wiping data.

Network and Listeners

The role exposes only ports — not bind addresses, advertised addresses, or the listener map:

Parameter Default Purpose
kafka_port 9092 Broker, client, and inter-broker communication
kafka_controller_port 9093 KRaft controller quorum
kafka_exporter_port 9308 Protocol Exporter HTTP metrics
kafka_jmx_exporter_port 9404 JMX Exporter HTTP metrics

The fixed listener conventions are as follows:

  • The broker listener binds 0.0.0.0, and the controller listener binds inventory_hostname;
  • The broker’s advertised.listeners uses inventory_hostname;
  • The controller bootstrap address also uses inventory_hostname;
  • plaintext: both BROKER and CONTROLLER use PLAINTEXT;
  • scram: BROKER uses SASL_SSL + SCRAM-SHA-512, and CONTROLLER uses mutual TLS.

Clients must therefore be able to resolve and directly reach every broker’s inventory_hostname. The current v1 does not support NAT, public-address mapping, multiple client networks for the same broker, or arbitrary raw listener overrides; these scenarios cannot be assembled around through kafka_parameters.

Kafka’s standard access model is a smart client connecting directly to brokers: bootstrap.servers is configured with several seed addresses, and once the client fetches cluster metadata it connects directly to the partition leaders. HAProxy, a Keepalived VIP, or a cloud LB should not be used as the regular Kafka data-plane entry point, because they are unaware of Kafka metadata and partition leaders and cannot exempt clients from reaching every advertised.listeners address. A DNS or TCP LB can at most serve as an optional bootstrap discovery entry point; even then, the application network must still reach all brokers directly. See Quickstart: Connecting Application Clients for details.

The minimum required network flows:

Source Destination Port Purpose
Kafka clients, other brokers All brokers 9092 Produce, Fetch, metadata, and inter-broker communication
All Kafka members All controllers 9093 KRaft metadata quorum
Infra/VictoriaMetrics All Kafka nodes 9404 JVM/Kafka metrics
Infra/VictoriaMetrics Selected Exporter nodes 9308 Cluster/Topic/Consumer metrics

The metrics ports are HTTP, and even when Kafka uses scram they should be restricted to the monitoring network by firewall.

Storage, Heap, and Rack

You set only the root directory:

kafka_data: /data/kafka

The role derives the topic data directory ${kafka_data}/data and the KRaft metadata directory ${kafka_data}/metadata in a fixed way. kafka_data must be a dedicated absolute path, and cannot be /, /data, /var, /etc, /opt, /usr, /home, /root, or /pg.

Production planning should account for at least retention, message peaks, replication traffic, partition/segment counts, disk latency and throughput, file descriptors, recovery time, JVM heap, and page cache. The current role generates only a single log.dirs; multi-disk JBOD, disk replacement, and automatic data migration require separate runbooks.

For deployments spanning fault domains, declare kafka_rack consistently on all broker-capable nodes:

10.10.10.21: { kafka_seq: 4, kafka_role: broker, kafka_rack: az-a }
10.10.10.22: { kafka_seq: 5, kafka_role: broker, kafka_rack: az-b }
10.10.10.23: { kafka_seq: 6, kafka_role: broker, kafka_rack: az-c }

Broker-capable nodes must either all set the rack or all omit it. Changing the rack triggers a safe rolling restart but does not automatically migrate existing replicas.

Replication Policy

At the first bootstrap, the policy is derived from the initial broker count:

replication_factor = min(3, broker_count)
min_insync_replicas = max(1, replication_factor - 1)

The initial default RF for future topics, the internal-topic RF, and the cluster minISR are all written into the manifest and frozen. After scaling out:

  • default.replication.factor keeps its initial value; Kafka 4.3 does not allow changing it online through dynamic broker configuration;
  • The RF of existing internal/business topics is not raised automatically;
  • The role does not report “brokers have joined” as “data is balanced”;
  • An RF change must use a reviewed kafka-reassign-partitions.sh plan; raising the static default additionally requires controller high availability or an explicit maintenance window, and takes effect through a full-cluster safe rolling restart.

Producer acks, idempotence, retries, batching, and compression are client policy, not parameters of the Kafka broker role.

kafka_parameters

kafka_parameters is the only broker-parameter escape hatch. It defaults to {} and is rendered only onto broker-capable nodes. It is suited to non-role-owned keys such as num.partitions, thread counts, buffers, retention, and segments.

The following patterns are owned by the role and must not be overridden:

process.roles
node.id
controller.quorum.*
listeners
advertised.listeners
listener.security.protocol.map
inter.broker.listener.name
controller.listener.names
log.dirs
metadata.log.dir
min.insync.replicas
default.replication.factor
offsets.topic.replication.factor
transaction.state.log.replication.factor
transaction.state.log.min.isr
share.coordinator.state.topic.replication.factor
share.coordinator.state.topic.min.isr
broker.rack
authorizer.class.name
super.users
allow.everyone.if.no.acl.found
sasl.*
ssl.*
listener.*

If any reserved key appears, the identity preflight fails outright before any file is written.

Security and Declarative Resources

kafka_security: scram is a complete production profile, not a set of switches to be combined at will. It automatically enables:

  • Per-node certificates issued by the Pigsty CA;
  • Mutual TLS on the controller listener;
  • SASL_SSL + SCRAM-SHA-512 for broker/client and inter-broker traffic;
  • StandardAuthorizer, deny-by-default, and role-owned admin/monitoring identities;
  • Convergence of the minimal monitoring ACL before the protocol Exporter starts.

Application resources are declared through two domain objects:

kafka_security: scram
kafka_users:
  - name: order-service
    password: "{{ vault_kafka_order_password }}"
    acls:
      - resource: topic
        name: order.
        pattern: prefixed
        operations: [Read, Write, Describe]
      - resource: group
        name: order.
        pattern: prefixed
        operations: [Read]
    quota:
      producer_byte_rate: 10485760
      consumer_byte_rate: 20971520
kafka_topics:
  - name: order.events
    partitions: 12
    replication_factor: 3
    config:
      min.insync.replicas: 2
      cleanup.policy: delete

Resource convergence semantics: topic creation is idempotent, partitions only increase, and only explicitly declared config is updated; an RF change is refused with a prompt to run reassignment. A declared user’s password, ACLs, and the quota fields you provide converge idempotently. Removing a topic or user entry is not an implicit deletion procedure.

The security profile cannot be switched by an ordinary playbook after bootstrap. Internal credentials and certificates can use protected rotation, but an online migration from plaintext to scram still requires an explicit state machine that is planned for the future.

Packages and File Layout

The role installs java-runtime and kafka-stack through platform mappings. The payload verified on 2026-07-16 is Kafka 4.3.1, kafka_exporter 1.9.0, and JMX Exporter 1.6.0; the actual versions still depend on the target platform’s repository and the installed packages.

Path Purpose
/opt/kafka/ Kafka programs and CLI
/etc/kafka/server.properties Role-generated service configuration
/etc/kafka/admin.properties Role-generated broker admin channel; the CLI should always use it
/etc/kafka/controller.properties Role-generated controller admin channel
/etc/kafka/log4j2.yaml Journald logging configuration
/etc/kafka/jmx_exporter.yml Bounded JMX metrics rules
/etc/kafka/manifest.yml Authoritative copy of the bootstrap manifest on the node
/etc/kafka/secrets.yml Copy of the internal secrets on a scram node
/etc/kafka/.pigsty-applied-static.sha256 Fingerprint of the static config proven live; the rolling-restart trigger
/etc/kafka/pki/kafka.pem PEM private key and certificate on a scram node; the trust anchor uses the system /etc/pki/ca.crt
${kafka_data}/data/ Topic log data
${kafka_data}/metadata/ KRaft metadata and meta.properties
files/pki/kafka/ Issued node certs on the admin node (<cluster>-<seq>.key/.crt, CSRs under files/pki/csr/)

These files are managed by the role. Persistent intent belongs in pigsty.yml. Do not edit generated files directly on the nodes, and do not copy passwords, private keys, or role-owned secret contents into inventory, logs, or tickets.

17.3 - Parameters

15 persistent public parameters and transient protected operational variables of the KAFKA module.

The KAFKA role deliberately exposes only 15 persistent parameters. Details such as topology, listeners, security implementation, storage subdirectories, replication safety, and Exporter placement are derived by the role in a single, consistent way, and cannot be overridden as additional persistent variables.

Parameter Overview

Parameter Level Default Description
kafka_cluster Cluster required Kafka cluster identity
kafka_seq Instance required Cluster-unique KRaft node.id
kafka_role Instance combined combined, broker, or controller
kafka_cluster_id Cluster unset Takeover/recovery assertion; randomly generated for a new cluster
kafka_data Instance /data/kafka Role-owned data root directory
kafka_heap_opts Instance -Xms1G -Xmx1G Kafka JVM heap
kafka_port Instance 9092 Broker/client port
kafka_controller_port Instance 9093 KRaft controller port
kafka_rack Instance unset Broker fault-domain label
kafka_parameters Cluster/Instance {} Non-role-owned broker parameters
kafka_jmx_exporter_port Instance 9404 JMX Exporter HTTP port
kafka_exporter_port Instance 9308 Protocol Exporter HTTP port
kafka_security Cluster plaintext plaintext or the production scram profile
kafka_users Cluster [] User credentials, ACLs, and quotas
kafka_topics Cluster [] Declarative topics

kafka_cluster and kafka_seq must be defined; kafka_role has a real default. The cluster’s roles are either all omitted or all declared explicitly.

Identity and Topology

kafka_cluster

The required cluster identity. It must start with a letter or digit and contain only letters, digits, underscores, and hyphens:

kafka_cluster: kf-main

It is used to discover the complete cluster membership, generate instance names, and locate the bootstrap manifest. Every kafka.yml lifecycle operation must use a precise -l to select all members of this cluster.

kafka_seq

A required non-negative integer, unique within the same kafka_cluster, which becomes the KRaft node.id directly:

10.10.10.11: { kafka_seq: 1 }

The instance name is derived as ${kafka_cluster}-${kafka_seq}. Once a node has been formatted, do not change or reuse a sequence number that still has associated data.

kafka_role

Defaults to combined, and accepts only:

Value Kafka process.roles Semantics
combined broker,controller Broker and controller co-located
broker broker Broker only
controller controller Controller only

When all cluster members omit it, they consistently use combined; as soon as any member sets it explicitly, all members must set it explicitly. No legacy role aliases are provided.

kafka_cluster_id

Unset by default, used only to assert the identity of an existing cluster during takeover or recovery. It must be a 22-character Kafka UUID:

kafka_cluster_id: MkU3OEVBNTcwNTJENDM2Qk

Do not set it for an ordinary new cluster. The role generates the Cluster ID randomly and writes it into every member’s /etc/kafka/manifest.yml. This parameter does not relabel existing data; it fails closed when it conflicts with the manifest or meta.properties.

kafka_rack

An optional broker fault-domain label, rendered as broker.rack:

10.10.10.21: { kafka_seq: 4, kafka_role: broker, kafka_rack: az-a }

All broker-capable nodes must either all declare it or all omit it. Controller-only nodes do not use this value. Changing the rack is a static change that goes through a strict rolling restart, but does not reassign existing replicas.

Storage, JVM, and Network

kafka_data

The data root directory, defaulting to /data/kafka:

kafka_data: /data/kafka

The role derives ${kafka_data}/data and ${kafka_data}/metadata in a fixed way. This path must be a dedicated absolute path, and cannot be /, /data, /var, /etc, /opt, /usr, /home, /root, or /pg. kafka-rm.yml deletes the entire root directory by default, so do not mix other services or business files into it.

kafka_heap_opts

The Kafka JVM heap, defaulting to:

kafka_heap_opts: '-Xms1G -Xmx1G'

In production, set it according to load and memory load testing. Typically keep Xms and Xmx equal, and leave enough memory for the operating-system page cache and other processes.

kafka_port

The broker/client listener port, defaulting to 9092, and listening only on broker-capable nodes. plaintext mode uses PLAINTEXT; scram mode uses SASL_SSL + SCRAM-SHA-512.

kafka_controller_port

The KRaft controller listener port, defaulting to 9093 (the conventional Kafka KRaft port), and listening only on controller-capable nodes. When sharing a node with other services, verify yourself that the ports do not collide; the role does not automatically detect cross-service port usage.

The four public ports must all differ from one another. The broker listener binds 0.0.0.0, while the controller listener, the broker advertised address, and the controller bootstrap address all use inventory_hostname in a fixed way, with no separate address parameters.

kafka_parameters

Defaults to {}. It is the only Kafka broker-parameter escape hatch, rendered only onto broker-capable nodes:

kafka_parameters:
  num.partitions: 12
  num.network.threads: 6
  num.io.threads: 16
  log.retention.hours: 168
  log.segment.bytes: 1073741824

The following keys or patterns are owned by the role and cannot be overridden through this mapping:

process.roles
node.id
controller.quorum.*
listeners
advertised.listeners
listener.security.protocol.map
inter.broker.listener.name
controller.listener.names
log.dirs
metadata.log.dir
min.insync.replicas
default.replication.factor
offsets.topic.replication.factor
transaction.state.log.replication.factor
transaction.state.log.min.isr
share.coordinator.state.topic.replication.factor
share.coordinator.state.topic.min.isr
broker.rack
authorizer.class.name
super.users
allow.everyone.if.no.acl.found
sasl.*
ssl.*
listener.*

Identity, listeners, security, storage, and replication policy must remain single-authority; when a reserved key is included, the preflight fails outright.

Observability

kafka_jmx_exporter_port

The JMX Exporter HTTP port, defaulting to 9404. The role injects the JMX Exporter Java agent unconditionally into every Kafka JVM and registers it as job=kafka; there is no separate toggle parameter. The lifecycle health gate uses the role-owned Kafka CLI/metadata channel and does not depend on JMX. Infra monitoring nodes must be able to reach this port; the endpoint does not automatically enable HTTPS because of kafka_security: scram, so it should be protected by the monitoring network and firewall.

kafka_exporter_port

The HTTP port of the protocol-level kafka_exporter, defaulting to 9308. The role configures, starts, and registers it only on the first two broker-capable nodes after sorting by kafka_seq; a single-broker cluster runs only one. The monitoring target file is refreshed on every full run according to the current placement, but a stale Exporter service on a node that was previously selected is not stopped automatically by an ordinary playbook.

The Kafka protocol version, TLS/SCRAM parameters, and replica placement used by the Exporter are all internal role conventions, with no additional public toggles or options parameters.

Security and Resources

kafka_security

Defaults to plaintext, and accepts only:

Value Broker/client Controller Authorization Purpose
plaintext PLAINTEXT PLAINTEXT none Development or a trusted, isolated network
scram SASL_SSL + SCRAM-SHA-512 mutual TLS StandardAuthorizer, deny-by-default Production security baseline

scram simultaneously configures the Pigsty CA-issued node certificates, the role-owned admin/monitoring/internal identities, and the ordering in which TLS/SCRAM and ACLs are enabled. The security profile is written into the bootstrap manifest; once the cluster is formatted, an ordinary rerun can neither switch plaintext to scram nor switch back.

Node certificate validity follows Pigsty’s shared CA parameter cert_validity (7300d by default); the KAFKA module has no separate certificate validity parameter.

kafka_users

Defaults to [], and may only be declared in scram mode. The collection must be a list of mappings, and each mapping accepts only name, password, acls, and quota; a non-mapping item or unknown top-level key fails before resource convergence:

kafka_users:
  - name: order-service
    password: "{{ vault_kafka_order_password }}"
    acls:
      - resource: topic
        name: order.
        pattern: prefixed
        operations: [Read, Write, Describe]
      - resource: group
        name: order.
        pattern: prefixed
        operations: [Read]
      - resource: transactional_id
        name: order.
        pattern: prefixed
        operations: [Write, Describe]
    quota:
      producer_byte_rate: 10485760
      consumer_byte_rate: 20971520

Constraints:

  • name is unique within the list; password is required, must be at least 12 characters, and should reference a secret management system;
  • ACL resource is one of topic, group, transactional_id, cluster;
  • pattern is literal (default) or prefixed;
  • Operations are Read, Write, Create, Delete, Alter, Describe, ClusterAction, DescribeConfigs, AlterConfigs, IdempotentWrite;
  • Quota keys are producer_byte_rate, consumer_byte_rate, request_percentage, controller_mutation_rate.

The role converges the SCRAM password, the complete ACL set, and the explicitly provided quota fields for a declared user. Removing a user entry does not implicitly delete the principal or credentials; deletion/revocation requires a separate, audited operation.

kafka_topics

Defaults to []. The collection must be a list of mappings, and each mapping accepts only name, partitions, replication_factor, and config; a non-mapping item or unknown top-level key fails before resource convergence:

kafka_topics:
  - name: order.events
    partitions: 12
    replication_factor: 3
    config:
      min.insync.replicas: 2
      cleanup.policy: delete
      retention.ms: 604800000

The identity precheck only validates that name is unique within the list. Partition and RF validity (at least 1, and RF not exceeding the current broker count) is decided by Kafka at creation time, so such errors surface during the resource convergence stage rather than under --check. The convergence semantics are:

  • The topic is created idempotently when it does not exist;
  • Partitions may only increase; decreasing fails;
  • When RF differs from the live state, ordinary convergence is refused and an explicit reassignment is required;
  • Only the keys declared in config are updated;
  • Removing a topic from the list never deletes the topic.

Transient Protected Operational Variables

The following variables are used only via the command-line -e for one-off operational actions. They are not part of the 15 persistent API parameters and should not be written into pigsty.yml:

Action Playbook Transient Variable Protection Condition
Rotate internal credentials kafka.yml kafka_rotate_credentials=true, kafka_rotate_confirm=<cluster> A healthy, fully-formatted scram cluster
Rotate certificates kafka.yml kafka_rotate_certificates=true, kafka_rotate_confirm=<cluster> A healthy, fully-formatted scram cluster
Tear down the cluster kafka-rm.yml kafka_rm_data (default true), kafka_rm_pkg (default false), kafka_safeguard (default false) Explicit -l required; kafka_safeguard=true aborts all deletion

The two rotation actions are mutually exclusive, and must target a precise, complete cluster. kafka-rm.yml deletes the data directory and node-local /etc/kafka recovery state by default; kafka_rm_data=false retains both. Before running it, explicitly confirm the target cluster and your backup/rebuild intent. For the commands and full semantics, see Playbooks.

kafka_safeguard

For kafka-rm.yml only, false by default. When set to true, the removal role aborts before deregistration, retirement, service shutdown, and deletion. This is a boolean safety switch; it does not probe whether the cluster is alive.

kafka_rm_data

For kafka-rm.yml only, true by default. When enabled it deletes the entire kafka_data directory and /etc/kafka, the latter holding the manifest, credential copies, and the recovery state required to re-adopt retained storage. Setting it to false keeps both, but monitoring targets are still deregistered, services still stopped, and runtime integration config still removed.

kafka_rm_pkg

For kafka-rm.yml only, false by default. When set to true it uninstalls the kafka-stack packages from the platform mapping (the Kafka, Kafka Exporter, and JMX Exporter payload); the shared Java runtime is never uninstalled.

17.4 - Administration

Kafka status checks, topic and user management, config changes, scaling, failed-node replacement, and security rotation.

The KAFKA module installs Kafka under /opt/kafka, manages the service with Systemd, and keeps its persistent intent in pigsty.yml. The files generated on the nodes are not meant to be edited by hand.

All of the Kafka CLI examples below use the role-generated /etc/kafka/admin.properties. Even when the current profile is plaintext, keep --command-config on every command: that way the command structure stays the same when you switch the admin channel to scram. Replace <broker>:9092 with a reachable inventory_hostname and port.

–command-config on the console tools requires a Kafka 4.2+ CLI

KIP-1147 standardized the config-file argument as --command-config and the key-value argument as --command-property across every CLI, starting with Kafka 4.2. The CLI under /opt/kafka/bin comes from the Pigsty repository (currently a 4.3.x payload) and works as shown; if you run these from an external 4.1 or older CLI, the console producer/consumer still need the old names --producer.config / --consumer.config. The admin tools (kafka-topics.sh, kafka-configs.sh, kafka-acls.sh, kafka-consumer-groups.sh, kafka-metadata-quorum.sh, and friends) have always used --command-config and are unaffected.

Quick Reference

Operation Command Description
Create Cluster ./kafka.yml -l <cls> Create or converge kafka clusters; a bare run covers all
Expand Cluster ./kafka.yml -l <cls> Declare new members: broker admission, controller join
Shrink Cluster ./kafka-rm.yml -l <ip> Retire a member: remove voter entry & broker registration
Remove Cluster ./kafka-rm.yml -l <cls> Tear down a whole cluster; deletes data by default
Replace Failed Node retire → provision → rejoin Three commands; replicas are re-inherited automatically
Config Cluster ./kafka.yml -l <cls> Edit the inventory, then roll under safety gates
Manage Topics ./kafka.yml -l <cls> Declaratively create topics, grow partitions, set configs
Manage Users ./kafka.yml -l <cls> Declaratively converge users, ACLs, and quotas
Rotate Credentials ./kafka.yml -e kafka_rotate_... Protected internal credential / certificate rotation

Cluster definition and parameters are covered in Configuration, playbook semantics in Playbooks, and monitoring in Monitoring.

Status Check

Check the service and recent logs on any Kafka node:

systemctl status kafka
systemctl is-enabled kafka
journalctl -u kafka --since '-30 min' --no-pager

The protocol exporter runs only on at most two broker-capable nodes with the lowest kafka_seq. On the selected nodes, also check:

systemctl status kafka_exporter
journalctl -u kafka_exporter --since '-30 min' --no-pager

Check listeners and metric endpoints:

ss -lntp | grep -E ':9092|:9093|:9308|:9404'
curl -fsS http://<kafka-ip>:9404/metrics | grep -E '^(jmx_scrape_error|kafka_server_raft_state|kafka_server_broker_messages_in_total)'
curl -fsS http://<exporter-ip>:9308/metrics | grep -E '^(kafka_brokers|kafka_topic_partitions)'

kafka_up and kafka_exporter_up are recording metrics on the VictoriaMetrics side and do not necessarily appear on the raw endpoints. The JMX endpoint should contain jmx_scrape_error 0.0, JVM metrics, and kafka_ metrics matching the node’s role.

Health Check

The role’s lifecycle gates do not rely on JMX. They check the dynamic quorum, unavailable partitions, under-replication, and under-min-ISR through the same admin channel:

sudo -u kafka /usr/local/bin/pigsty-kafka-health cluster \
  --bootstrap-server <broker>:9092 \
  --command-config /etc/kafka/admin.properties

Only healthy: true in the returned JSON means the gate passes. It is suitable for read-only diagnostics but is no substitute for end-to-end business validation.

The script also ships a built-in parser regression suite (pigsty-kafka-health selftest) that every playbook run executes right after installation; if the selftest fails, the health predicate itself cannot be trusted — stop making changes and investigate.

KRaft Quorum Status

Query the dynamic quorum from any available broker:

/opt/kafka/bin/kafka-metadata-quorum.sh \
  --bootstrap-server <broker>:9092 \
  --command-config /etc/kafka/admin.properties \
  describe --status

Key things to verify:

  • LeaderId exists and matches an expected controller;
  • CurrentVoters matches the expected membership (a joining node appears under CurrentObservers first);
  • MaxFollowerLag and MaxFollowerLagTimeMs are not growing continuously;
  • Exactly one active controller shows on the dashboards.

To confirm the dynamic quorum (KIP-853) feature level, inspect kraft.version with /opt/kafka/bin/kafka-features.sh ... describe.

Inspect controller replication state:

/opt/kafka/bin/kafka-metadata-quorum.sh \
  --bootstrap-server <broker>:9092 \
  --command-config /etc/kafka/admin.properties \
  describe --replication

If there is no leader, a member lags persistently, or the voter set differs from expectation, stop other changes first and preserve the logs, the manifest, and meta.properties as evidence before analyzing. Remove a dead voter through the shrink or replace flows; never rewrite quorum state by hand.

Manage Topics

Production topics should be declared in kafka_topics in pigsty.yml:

kafka_topics:
  - name: orders
    partitions: 12
    replication_factor: 3
    config:
      min.insync.replicas: 2
      retention.ms: 604800000

Converge after editing the declaration:

./kafka.yml --check -l kf-main
./kafka.yml -l kf-main

The role creates topics idempotently, only grows partitions, and only modifies the declared config keys. An RF change fails and demands an explicit partition reassignment; removing an entry from the inventory does not delete the topic.

Read-only topic inspection:

/opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server <broker>:9092 \
  --command-config /etc/kafka/admin.properties \
  --list

/opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server <broker>:9092 \
  --command-config /etc/kafka/admin.properties \
  --describe --topic orders

Ad-hoc or externally managed topics can be created with the Kafka CLI, but they are not written back into pigsty.yml. Never let declarative and manual management own the same topic. Topic deletion is a business-data deletion: it requires separate approval, exact-name confirmation, and a recovery plan, so no generic delete command is given here.

Manage Users and ACLs

With kafka_security: scram, application identities should be managed through kafka_users:

kafka_users:
  - name: order-service
    password: "{{ vault_kafka_order_password }}"
    acls:
      - resource: topic
        name: orders
        operations: [Read, Write, Describe]
      - resource: group
        name: order-worker
        operations: [Read]
    quota:
      producer_byte_rate: 10485760
      consumer_byte_rate: 20971520

A full playbook run idempotently converges the password, the user’s ACL set, and the explicitly given quota fields. Never commit passwords in plaintext or print them into logs. Removing a user entry does not delete the principal/credentials automatically; deletion or full revocation needs a separately reviewed procedure.

Verify Read/Write

Use a test topic for end-to-end validation. The console producer/consumer share the same client config file:

/opt/kafka/bin/kafka-console-producer.sh \
  --bootstrap-server <broker>:9092 \
  --command-config /etc/kafka/admin.properties \
  --topic ops-smoke

Consume in another terminal:

/opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server <broker>:9092 \
  --command-config /etc/kafka/admin.properties \
  --topic ops-smoke \
  --from-beginning \
  --group ops-smoke-check

Production acceptance should run from the real client network, covering DNS/advertised.listeners, certificate validation, ACLs, producer ACKs, consumer commits, and end-to-end latency — not just the broker-local path.

Manage Consumer Groups

List and inspect consumer groups:

/opt/kafka/bin/kafka-consumer-groups.sh \
  --bootstrap-server <broker>:9092 \
  --command-config /etc/kafka/admin.properties \
  --list

/opt/kafka/bin/kafka-consumer-groups.sh \
  --bootstrap-server <broker>:9092 \
  --command-config /etc/kafka/admin.properties \
  --describe --group order-worker

Judge lag against the consumption rate and business SLO: a short backlog can be batch-processing behavior, while sustained growth with consumption slower than production means the group cannot catch up. Resetting offsets may duplicate or skip messages, so it requires separate approval, exact group/topic confirmation, and a replay plan.

Config Cluster

After editing pigsty.yml, run against complete clusters:

./kafka.yml --check -l kf-main
./kafka.yml -l kf-main

The role picks the path from live health and the static fingerprint:

  • Cluster unhealthy or stopped: start only the stopped controllers, restore and catch up the quorum, then start the brokers; if static changes coexist, the still-online members proceed into the strict rolling afterwards;
  • Controller-capable nodes awaiting quorum join: each catches up as an observer and is promoted with add-controller, one at a time;
  • Healthy cluster with new pure brokers: format, start, and confirm registration one at a time;
  • Healthy cluster with static changes: strict node-by-node rolling, with pre/post gates on controller majority and voter catch-up, offline partitions, under-min-ISR, and ISR catch-up around every restart;
  • No static change: Kafka is not restarted.

Do not bypass the full state machine with -t kafka_config. Dynamic topic/user/ACL/quota convergence lives in the kafka_provision resource stage; whether a static change restarts anything is decided by the role.

Expand Cluster

A healthy cluster takes new members declared directly in the inventory: kafka_role: broker, combined, or controller all work. Give each new node a never-used kafka_seq (a host can belong to only one Kafka cluster at a time), make sure the node is managed by Pigsty, then still target complete clusters:

./node.yml  --check -l 10.10.10.14    # manage the new node
./kafka.yml --check -l kf-main        # dry run first
./kafka.yml -l kf-main                # admit / join new members one at a time

The role picks the path per member type and handles one new node at a time:

  • Pure broker: format, start, and verify the broker is registered and not fenced (admit);
  • Combined / controller: format fresh with --no-initial-controllers, start as an observer and catch up on metadata, promote with add-controller, then verify it entered the voter set with the cluster fully healthy (join).

The quorum-join-hosts / broker-admission-hosts summary at the end of the run lists the nodes actually processed. Two reminders:

  • Adding a controller-capable node changes controller.quorum.bootstrap.servers on every member, so the existing nodes go through one gated strict rolling round afterwards — this is expected;
  • Scaling out to an even controller count prints a warning: an even quorum adds no fault tolerance, keep the count odd.

Joining does not migrate existing partitions onto the new broker. Generate, review, and monitor a kafka-reassign-partitions.sh plan separately, control the disk/network load, and prepare a rollback. “The service is registered” is not “the expansion is complete.”

The replication policy does not scale up with the broker count either. In particular, Kafka 4.3’s default.replication.factor cannot be changed dynamically: after scaling from 1 broker to 3, it remains the RF=1 set at initial build, and any future topic without an explicit RF is still created with RF=1. First complete the reassignment of existing partitions, then plan for controller high availability or a maintenance window, and finally let the new static default take effect through a safe full-cluster rolling restart. Do not bypass the downtime gates just to change a default.

Shrink Cluster

Selecting a strict subset of a cluster with kafka-rm.yml retires members (selecting the whole cluster is a teardown). Retirement removes the leaving node from live metadata through a surviving member:

./kafka-rm.yml -l 10.10.10.13     # retire one member: remove voter entry, unregister broker, clean the host

The execution order is: deregister monitoring targets → stop services → remove-controller to remove the KRaft voter entry (if the member is a voter; strictly serialized when retiring several members) → kafka-cluster.sh unregister to drop the broker registration → clean local config and data (controlled by kafka_rm_data). Broker unregistration tolerates failure so the workflow is re-entrant and can handle an unreachable member. Delete the member from pigsty.yml only after checking the live quorum, broker registrations, replica health, and the target’s local state.

Before retiring, confirm yourself that: the remaining controllers still form a majority, the controller count stays odd, and the remaining broker count is not below the highest topic RF. If the retiring broker still hosts partition replicas, the role prints a warning: those partitions stay under-replicated until a replacement with the same kafka_seq rejoins (it re-inherits the assignment and resyncs automatically), or until you reassign them explicitly. A planned shrink should drain with a reassignment first, then retire.

Replace Failed Node

When a node is permanently lost (disk gone, machine scrapped), keep its IP and kafka_seq and replace it in three steps:

./kafka-rm.yml -l 10.10.10.13     # 1. retire the dead member: remove voter entry & broker registration (works while unreachable)
./node.yml     -l 10.10.10.13     # 2. provision the replacement machine (repaired or new, same IP)
./kafka.yml    -l kf-main         # 3. rejoin: format, catch up, admit/promote; replica assignment is re-inherited and resynced

All metadata operations in step 1 are delegated to a surviving member, so it works even when the node itself is unreachable; it also cleans up the monitoring target, so the dead node stops firing KafkaDown. In step 3, a broker with the same kafka_seq automatically re-inherits its former partition assignment and resynchronizes from the surviving replicas — no manual reassignment needed.

If you skip step 1 and rerun kafka.yml against a re-imaged node directly, the role fails fast in the config phase, reporting the stale voter entry’s directory ID together with the exact kafka-rm.yml command — run it and retry. The join flow is safely re-entrant: if any step is interrupted, rerunning kafka.yml continues from live state.

Change Address or Port

The role always uses inventory_hostname as the broker’s advertised address and the controller’s bootstrap address. Changing an inventory address, kafka_port, or kafka_controller_port affects client metadata, broker communication, or the quorum, and counts as a high-risk static change: check DNS, certificate SANs, routing, firewalls, bootstrap addresses, monitoring targets, and all cluster members in lockstep.

Rotate Credentials and Certificates

A formatted, healthy scram cluster supports two mutually exclusive protected actions: internal credential rotation and certificate rotation. Both require exact complete clusters and a matching kafka_rotate_confirm confirmation string, and running --check first is recommended. Certificates are re-issued by the same Pigsty CA, old and new certificates trust each other, and the rotation takes effect node by node through the strict rolling restart.

For the exact commands and failure semantics, see Playbook: Protected Rotation. The security profile itself is a bootstrap-only property; these actions do not imply support for online migration from plaintext to scram.

Data Protection and Recovery

Kafka’s data protection relies on replicas across failure domains, correct min-ISR, producer ACKs, and a rehearsed recovery procedure. The current role does not provide Kafka data backup, automatic broker drain (a planned shrink needs a manual reassignment first), or cross-region disaster recovery.

When a disk or node fails:

  1. First look at the Kafka Overview/Node dashboards, the quorum, ISR, offline partitions, and under-min-ISR partitions;
  2. Preserve the evidence: journalctl -u kafka, node metrics, the manifest, server.properties, and meta.properties;
  3. Confirm the node’s role, node.id, cluster ID, directory ID, and the availability of the remaining replicas;
  4. Once the node is confirmed unrecoverable, follow Replace Failed Node: retire with kafka-rm.yml → provision with node.yml → rejoin with kafka.yml; if the disk survives and only the service misbehaves, do not rush to retire or delete meta.properties — try an ordinary converge first;
  5. Data-movement operations such as reassignment or RF changes still deserve a separately reviewed runbook.

Log Diagnostics 

journalctl -u kafka -f
journalctl -u kafka_exporter -f
journalctl SYSLOG_IDENTIFIER=kafka --since today
journalctl SYSLOG_IDENTIFIER=kafka_exporter --since today

VictoriaLogs/Grafana queries:

job:syslog unit:kafka
job:syslog app:kafka
job:syslog unit:kafka_exporter

The usual diagnostic order is: service logs → listening ports → admin-channel health → dynamic quorum → broker/partition/ISR → client addresses and certificates/ACLs → consumer lag. For dashboards and alert mappings, see Monitoring.

17.5 - Playbook

Run dynamic KRaft lifecycle, strict rolling, resource convergence, rotation, and removal with kafka.yml and kafka-rm.yml.

The KAFKA module ships two playbooks: kafka.yml deploys an Apache Kafka 4.1+ dynamic KRaft cluster and converges its security, resource, and monitoring state; kafka-rm.yml tears down a cluster or removes a member.

Cluster Completeness Constraint

Every selected kafka_cluster must include all of its members: a partial selection fails before anything is written, while one cluster, several complete clusters, or a bare run over the whole inventory are all allowed. Run --check first against the exact same target; before the real run, still verify the backup/rebuild intent, capacity, business window, rollback plan, and change approval by hand.

kafka.yml 

./kafka.yml --check -l kf-main   # dry run first
./kafka.yml -l kf-main           # create or converge one cluster
./kafka.yml                      # bare run: create / converge ALL kafka clusters

The limit rule is: every selected cluster must be complete. You may select one cluster, several clusters, or run bare against the whole inventory (strictly serial within a cluster, concurrent across clusters); a partial selection of a cluster’s members is refused outright.

Check mode validates the public API, the full cluster, roles, racks, ports, the manifest, and any inspectable file changes, but it skips formatting, service startup, and live health acceptance. A successful --check is therefore not a guarantee of a successful runtime.

Execution Stages

kafka.yml is itself a thin wrapper: a single play runs the node_id and kafka roles in sequence, mirroring the structure of pgsql.yml. Inside the role, the lifecycle is split into six task stages; all cross-node ordering (parallel bootstrap, one-controller-at-a-time join, one-broker-at-a-time admission, strict node-by-node rolling) is handled centrally by the launch stage:

Stage Tag Purpose
Identity kafka-id Derive and assert identity, cluster completeness, roles, racks, ports, and reserved keys
Install kafka_install Create the kafka system user, install the java-runtime and kafka-stack packages
Config kafka_config Read/restore/create the manifest, issue security material, render config, compute the static fingerprint, format empty storage, decide the lifecycle path
Launch kafka_launch Converge an unhealthy cluster, join controllers and admit brokers one at a time, strict rolling, commit the manifest and applied static state
Provision kafka_provision Converge dynamic min-ISR, user credentials, ACLs, quotas, and declarative topics; report internal-topic RF drift
Monitor kafka_monitor Configure the protocol exporter and register VictoriaMetrics targets

The play uses any_errors_fatal: true. When a stage fails, dangerous forward progress stops; once you fix the cause, you can re-run against the full cluster, and the role recovers from live state and the persistent fingerprint instead of blindly reformatting.

Lifecycle Paths

The config stage uses the role’s own admin channel to judge cluster health and select exactly one downstream path:

Cold Start, First Deployment, or Repair

When the cluster is stopped or the health predicate does not hold, it enters converge:

  1. Start all controller-capable nodes;
  2. Wait for the controller listener and a dynamic quorum leader;
  3. On the first bootstrap, verify that the initial controller directory IDs have entered the live quorum;
  4. Start the pure brokers;
  5. Wait for the broker listener and require the full cluster to be healthy;
  6. Persist the static fingerprint only after the config has been proven to run successfully.

JMX plays no part in the lifecycle gates: the decisions for startup, admission, and rolling are made entirely on the role’s own Kafka CLI/metadata admin channel.

Adding Brokers or Controllers to a Healthy Cluster

Newly formatted kafka_role: broker nodes are admitted one at a time (admit): after starting, each must be registered and not fenced before the next one proceeds.

New combined/controller nodes join the dynamic quorum one at a time instead (join): on a commissioned cluster the node is formatted fresh with --no-initial-controllers, starts as an observer and catches up on metadata, then the role promotes it with add-controller and verifies through the health post-check that it entered the voter set with the cluster fully healthy. The join flow is re-entrant: an interrupted run continues from live state on the next rerun, and if the node’s node.id still has a stale voter entry from a dead predecessor, the config phase fails fast with the exact kafka-rm.yml retirement command.

Admission/join only proves membership; existing partitions are not migrated onto the new broker automatically, so you must run an explicit reassignment separately.

Static Changes on a Healthy Cluster

When the rendered static fingerprint changes, the strict rolling restart handles one node at a time:

  • Before restarting, it checks the controller majority, that all voters have zero lag and recently completed catch-up, offline partitions, under-replicated, under-min-ISR, and the effective ISR of each partition once the target is excluded;
  • After restarting, it requires the target controller to be back as a voter and re-caught-up, the target broker to be registered and not fenced, and its replicas to re-enter the ISR;
  • Any failed gate immediately stops the remaining nodes.

If a fault to repair and a static change coexist, converge only starts the stopped members and does not restart the still-online members in parallel; once the quorum recovers and catches up, the static changes that have not yet been loaded proceed into the strict rolling restart.

If the static fingerprint is unchanged, Kafka is not restarted. Dynamic resource changes still take effect online during the resource-convergence stage.

Task Tags

Tag Stage / Purpose
kafka-id The identity, full-cluster, and topology-derivation assertions that always run
kafka_install The overall entry point for the install stage
kafka_user Create the kafka system user and group
kafka_pkg Install the java-runtime and kafka-stack packages per platform mapping
kafka_config Manifest, security material, config rendering, static fingerprint, storage formatting, and path decision
kafka_launch Converge, serialized controller join and broker admission, strict rolling, and manifest commission
kafka_provision Convergence of dynamic min-ISR, topics, users, ACLs, and quotas
kafka_monitor / monitor The overall entry point for protocol-exporter configuration and monitoring registration
kafka_register / register / add_metrics Refresh only the VictoriaMetrics file-discovery targets

An ordinary configuration change should run the full kafka.yml and let the role choose its own lifecycle path. Stage tags are meant primarily for development, diagnostics, and controlled repair; you cannot bypass the full state machine with -t kafka_config or by limiting to a single node.

Identity, Formatting, and Manifest

Before writing any config, the role validates that:

  • Every selected cluster contains all of its members;
  • kafka_seq is unique, and the roles are either all omitted or all explicit;
  • There is at least one controller and one broker;
  • Racks are either all present or all absent across the broker-capable nodes;
  • Ports are valid and non-conflicting, and the role-owned keys are not overridden by kafka_parameters;
  • The manifest, security profile, meta.properties, and the live cluster identity are consistent.

A new cluster randomly generates the cluster ID and the initial controller directory IDs and formats each node in explicit dynamic-quorum mode. When ${kafka_data}/metadata/meta.properties already exists, it validates the cluster ID and node ID locally; initial controller directory IDs are compared against the live quorum only on the first bootstrap — after commissioning, membership is authoritative in the live Raft state. The role never reformats existing storage automatically.

The authoritative bootstrap manifest lives on every cluster member:

/etc/kafka/manifest.yml

Each member of a scram cluster additionally has /etc/kafka/secrets.yml; the admin node keeps no kafka state and resolves both from any member copy on every run. The live cluster is the authoritative runtime fact, but an ordinary playbook will not silently rewrite either side on conflict:

  • When no member has a manifest copy but the storage is already formatted, it fails closed and asks you to restore the file on any member first;
  • When a manifest exists but all data disks are empty, it fails closed;
  • When the cluster ID, security profile, or controller identity conflicts, it fails closed;
  • A new node whose node.id still has a stale predecessor voter entry in the quorum fails fast and asks you to retire it with kafka-rm.yml first.

Do not delete meta.properties, the manifest, or the secrets to bypass these protections.

Static Fingerprint and Recoverable Re-runs

The role computes an expected fingerprint over the static files that affect the Kafka process, and writes /etc/kafka/.pigsty-applied-static.sha256 only after one of the following holds:

  • Converge has successfully started and passed the global health check;
  • The strict rolling restart has restarted this node, let it catch up, and passed the post-restart gates.

If the run is interrupted, changes that have not been proven to take effect are not recorded as “applied.” The next full re-run can still recognize the pending static restart.

Resource Convergence and Monitoring Registration

Once the cluster is fully healthy, the resource-convergence and monitoring stages run in order:

  1. Converge the role-owned dynamic cluster min-ISR;
  2. Idempotently process the credentials, ACLs, and declared quotas of kafka_users;
  3. Idempotently process the creation, partition growth, and explicit config of kafka_topics;
  4. Check internal-topic RF drift, but do not reassign automatically;
  5. Configure and start the protocol exporter on the first two broker-capable nodes, ordered by kafka_seq;
  6. Refresh the file-discovery targets on all infra nodes.

Each instance maps to one target file, and both the JMX target and the (selected nodes’) protocol-exporter target live under the same kafka scrape job:

/infra/targets/kafka/<kafka_instance>.yml

The target files are refreshed on every full run to match the current exporter placement; target deletion is handled by the deregistration step of kafka-rm.yml.

Protected Rotation

The rotation variables are one-shot extra-vars and should not be written into pigsty.yml. The two actions are mutually exclusive and only one may run at a time; the prerequisites are that all members are formatted, the cluster is healthy, the security profile is scram, the role-owned secret material exists, and kafka_rotate_confirm matches the cluster name exactly.

Internal Credential Rotation 

./kafka.yml --check -l kf-main \
  -e kafka_rotate_credentials=true \
  -e kafka_rotate_confirm=kf-main

./kafka.yml -l kf-main \
  -e kafka_rotate_credentials=true \
  -e kafka_rotate_confirm=kf-main

The role uses active/standby internal identities: it first updates the inactive credential through the live admin channel, then atomically switches the local protected record, and enters the normal strict rolling restart. The old active identity is kept as the next round’s standby, so a re-run after an interruption is recoverable.

Certificate Rotation 

./kafka.yml --check -l kf-main \
  -e kafka_rotate_certificates=true \
  -e kafka_rotate_confirm=kf-main

./kafka.yml -l kf-main \
  -e kafka_rotate_certificates=true \
  -e kafka_rotate_confirm=kf-main

The role discards the node certificates already issued in the shared PKI tree, re-issues a private key and certificate for each node from the same Pigsty CA, updates the PEM certificate bundle on the nodes, and enters the strict rolling restart. Because the old and new certificates are issued by the same CA and trust each other, no staged trust swap is needed; if the health precheck fails, the rotation does not begin and the existing certificates on the nodes are left unchanged.

kafka-rm.yml

Removal is not in kafka.yml; it uses the separate kafka-rm.yml playbook. It requires a non-empty -l/--limit and fails before entering any role when run without one. Selecting all members of a cluster with -l is a teardown, selecting a strict subset is member retirement; both share the same execution order:

Deregister the VictoriaMetrics targets (kafka_deregister) → stop and disable the kafka/kafka_exporter services (kafka) → remove the KRaft voter entry and broker registration through a surviving member (kafka_retire, which only has a surviving member to work through when a strict subset is selected) → delete exporter config, Systemd environment/units, and helper scripts (kafka_config) → delete the data directories and node-local /etc/kafka recovery state (kafka_data, controlled by kafka_rm_data) → optionally uninstall the packages (kafka_pkg, controlled by kafka_rm_pkg).

Before deregistration or service stop, the role also checks that kafka_data is a dedicated safe absolute path: it must contain no ./.. path segment and cannot be /, /data, /var, /etc, /opt, /usr, /home, /root, or /pg. The safeguard switch is kafka_safeguard: when set to true (on the command line or in the inventory), the playbook aborts immediately and deletes nothing. An identity conflict, an exporter anomaly, or an ordinary startup failure is not a reason to delete data — converge with kafka.yml first and read the failure reason.

Cluster Teardown 

./kafka-rm.yml -l kf-main                          # Remove the cluster: deregister monitoring and stop services; deletes data and /etc/kafka recovery state by default
./kafka-rm.yml -l kf-main -e kafka_rm_data=false   # Keep data and /etc/kafka recovery state; remove only service integration
./kafka-rm.yml -l kf-main -e kafka_rm_pkg=true     # Also uninstall the kafka-stack packages (the shared Java runtime is not removed)
Permanent Deletion

kafka_rm_data defaults to true: a single default-parameter run of kafka-rm.yml deletes the selected nodes’ data/KRaft metadata and /etc/kafka recovery state. The playbook has no extra gate such as a confirmation string, so before running it you must verify the -l target, the backup or an explicit rebuild intent, and the impact on producers/consumers by hand.

Member Retirement 

./kafka-rm.yml -l 10.10.10.13                      # Retire one member: drop its voter entry and broker registration, then clean the node

Partial retirement requires at least one broker-capable (combined/broker) survivor and one controller-capable (combined/controller) survivor outside -l; one combined node can satisfy both roles. If either anchor is missing, the playbook fails before deregistration or service stop. It then uses those survivors to remove each target’s KRaft voter entry (remove-controller, strictly serialized for several members) and broker registration (unregister) before local cleanup. Metadata actions are delegated to survivors, so this also works for nodes that are already dead and unreachable—step one of Replace Failed Node. Broker unregistration is deliberately re-entrant and tolerates failure; after a real run, inspect the live quorum, broker registrations, and replica health instead of treating the playbook status alone as proof of retirement.

Automated retirement does not remove the need for planning: after the shrink, the remaining controllers should stay odd-numbered and keep a live majority, and the remaining broker count must not fall below the highest topic RF; when the retiring broker still hosts partition replicas, the playbook prints a warning — a planned shrink should drain with a reassignment first.

Playbook Boundaries

Neither playbook performs partition reassignment and data balancing, topic/user deletion, online plaintext → scram migration, version upgrades and feature-level finalization, or data backup and disaster recovery, and neither deploys ecosystem components such as Connect, Schema Registry, MirrorMaker, or Cruise Control. For the full list see Module Boundaries; for day-to-day read-only checks and resource management, see Administration.

17.6 - Monitoring

Kafka metrics collection, Grafana dashboards, log queries, and alerting rules.

Pigsty gives the KAFKA module a unified observability stack that combines metrics, logs, dashboards, and alerts. Monitoring covers both the Kafka JVM internals and the Kafka protocol view, so you never end up seeing only that the process is alive without visibility into partitions, ISR, and consumer lag, nor seeing only cluster metadata without visibility into the JVM, request queues, and KRaft controller health.

Scrape Architecture

The KAFKA module uses two complementary exporters:

Scrape Surface Service / Method Job Node Scope Main Content
JVM and Kafka internals JMX Exporter Java agent :9404 kafka (with role label) All Kafka nodes JVM, broker throughput, replication, request path, KRaft, controller
Kafka protocol view kafka_exporter :9308 kafka (no role label) The one or two broker-capable nodes with the smallest kafka_seq Broker, topic, partition, offset, consumer group, lag
Host resources node_exporter node Managed nodes CPU, memory, disk, network, filesystem
Logs Journald → Vector → VictoriaLogs syslog All Kafka nodes Structured, searchable Kafka and exporter logs

On each Infra node, the role generates one file-discovery target per instance. The JMX target and the protocol exporter target (on the selected nodes) both live in the same file, under the same kafka scrape job:

/infra/targets/kafka/<kafka_instance>.yml

A single-broker cluster runs only one protocol exporter; a multi-broker cluster runs at most two. Controller-only nodes register only the JMX target; brokers that were not selected and controller-only nodes have no protocol exporter target, which is expected behavior. The target file is refreshed to match the current placement on every full run; deletion of an instance target is handled by the deregistration step in kafka-rm.yml.

Label Model

Both target types are registered under the same job=kafka scrape job, and are distinguished by the presence or absence of the role label.

JMX Target

Label Meaning Example
job Scrape job kafka
cls Kafka cluster name kf-main
ins Kafka instance name kf-main-1
ip Inventory host address 10.10.10.11
instance JMX scrape endpoint 10.10.10.11:9404
role Pigsty Kafka role combined, broker, or controller
node_id KRaft node ID 1

Protocol Exporter Target

A protocol exporter target carries only cls, ins, ip, and instance (10.10.10.11:9308); it has no role or node_id labels. The recording rules on the vmagent side use this to distinguish the two availability types: kafka_up is up{job="kafka",role=~".+"}, and kafka_exporter_up is up{job="kafka",role=""}.

The exporter queries the entire Kafka cluster through a broker, so the two exporters of the same cluster may return an identical view of topics, partitions, and consumer groups. The cluster-level recording rules first deduplicate across exporter instances, then aggregate the logical cluster rates. In scram mode, the TLS/SCRAM parameters the exporter needs to connect to Kafka are generated automatically from the role’s own monitoring identity.

Grafana Dashboards

Pigsty ships four complementary dashboards:

Kafka Overview

Cluster and global overview. cls=All is the overview across all Kafka clusters; once you select a specific cls, the same dashboard becomes the overview for that Kafka cluster, rather than a separate set of panels.

Main content:

  • Inventory of clusters, brokers, topics, partitions, and consumer groups
  • Broker availability, exporter health, and cluster workload
  • Leaderless, under-replicated, ISR deficit, and non-preferred replica
  • Topic offset progress, consumer commit progress, and total lag
  • Consumer group members, lag ranking, and topic/group drill-down
  • Kafka/exporter log volume, firing alerts, and log detail

Common variables: cls, members, topic, group, topk.

Kafka Instance

Use the ins variable to select any Kafka broker/controller JVM, including controller-only nodes, correlated with host resources.

Main content:

  • Instance identity, role, JMX availability, and scrape quality
  • JVM heap, GC, threads, buffer pool, CPU, FD, and uptime
  • Broker throughput, replication state, request errors/latency/queues, and handler/network idle
  • KRaft member state, metadata log, controller health, and event latency
  • Node CPU/memory, disk I/O, network, filesystem, and Kafka logs

Common variables: cls, ins, ip.

Kafka Topic

Use cls and topic to select a logical topic and inspect topic/partition state from the protocol view.

Main content:

  • Topic and partition inventory, leader, replicas, ISR, and preferred leader
  • Current offset, retention span, and message append rate
  • Leaderless, ISR deficit, and non-preferred replica
  • Associated consumer groups, commit progress, and lag

Common variables: cls, topic, topk.

Kafka Consumer

Use cls and group to select a consumer group and inspect members, committed offsets, consumption progress, and backlog.

Main content:

  • Consumer group inventory and member count
  • Committed offsets by group/topic/partition
  • Commit rate, total lag, maximum partition lag, and backlog trend
  • Drill-down from group to topic/partition

Common variables: cls, group, topic, topk.

Choosing a Dashboard

Question Preferred Dashboard Drill-Down Path
Which cluster or topic is misbehaving? Kafka Overview Select cls, topic, group
Why is a consumer group falling behind? Kafka Consumer Group → Topic → Partition offset
Is a particular topic/partition unhealthy? Kafka Topic Topic → Partition → Consumer
Is a particular broker overloaded? Kafka Instance Request path → JVM → Node resources
Is the KRaft controller healthy? Kafka Instance KRaft metadata plane → Controller health
Are there leaderless/URP/ISR problems? Kafka Overview Cluster → Kafka Instance / Topic
Is the exporter missing data, or is Kafka itself unhealthy? Overview + Instance Compare kafka_exporter_up with kafka_up

Recording Rules

The Kafka rule file lives at /infra/rules/kafka.yml. The main recorded metrics are:

Metric Meaning
kafka:topic:msg_rate1m/5m 1m/5m forward change rate of a topic’s current offset
kafka:cls:msg_rate1m/5m Deduplicated cluster message append rate
kafka:csg_topic:commit_rate5m 5-minute commit progress rate per consumer group/topic
kafka:csg_topic:lag Total lag per consumer group/topic
kafka:csg:lag Total lag of a consumer group across topics
kafka:cls:lag Total lag of all consumer groups in a Kafka cluster
kafka:ins:jvm_heap_used_ratio Kafka JVM heap usage ratio
kafka:ins:jvm_cpu_cores Number of CPU cores consumed by the Kafka JVM
kafka:ins:load / kafka:cls:load Saturation of the busiest request thread pool, and the cluster average
kafka:ins:jvm_gc_time_rate5m 5-minute GC time rate
kafka:ins:messages_in_rate5m Broker 5-minute message receive rate
kafka:ins:bytes_in_rate5m Broker 5-minute inbound client byte rate
kafka:ins:bytes_out_rate5m Broker 5-minute outbound client byte rate
kafka:ins:request_error_rate5m Broker 5-minute request error rate
kafka:cls:under_replicated_partitions Total under-replicated partitions in the cluster
kafka:cls:offline_partitions Offline partitions in the cluster

Rates derived from offset changes represent progress, not client request counts. Log truncation, offset rollback, or an exporter restart can produce a transient negative change; the rules use clamp_min(..., 0) to keep only forward progress.

Alert Rules

Alert Condition Duration Severity Preferred Drill-Down
KafkaDown up{job="kafka",role=~".+"} < 1 1m CRIT Kafka Instance / ins
KafkaExporterDown up{job="kafka",role=""} < 1 1m CRIT Kafka Instance / ins
KafkaJmxScrapeError jmx_scrape_error{job="kafka"} > 0 3m WARN Kafka Instance / JMX Collector
KafkaJvmHeapHigh Heap usage > 90% 15m WARN Kafka Instance / JVM Memory
KafkaJvmDeadlock JVM deadlocked threads > 0 1m CRIT Kafka Instance / JVM Threads
KafkaRequestHandlerSaturated Handler idle < 10% 10m WARN Kafka Instance / Request Path
KafkaNetworkProcessorSaturated Network processor idle < 10% 10m WARN Kafka Instance / Request Path
KafkaUnderReplicatedPartitions URP > 0 5m WARN Kafka Instance / Replication
KafkaUnderMinISR Under min ISR > 0 1m CRIT Kafka Instance / Replication
KafkaOfflineLogDirectory Offline log directory > 0 1m CRIT Kafka Instance / Disk Pressure
KafkaOfflinePartitions Controller offline partitions > 0 1m CRIT Kafka Overview / cls
KafkaControllerCountMismatch Active controller count is not 1 1m CRIT Kafka Overview / cls
KafkaFencedBrokers Fenced brokers > 0 5m WARN Kafka Overview / cls
KafkaUncleanLeaderElection An unclean leader election in the last 5 minutes immediate CRIT Kafka Overview / cls
KafkaConsumerLagGrowing Group lag > 100000 and still growing after 30m 30m WARN Kafka Consumer / group

An unclean leader election can mean data loss. Immediately preserve the controller/broker logs, confirm the affected topics and replicas, and only then decide on a recovery action.

Common PromQL

Check scrape targets:

kafka_up
kafka_exporter_up
up{job="kafka"}

Check the replication health of a cluster:

sum by (cls) (kafka_server_replica_manager_under_replicated_partitions{job="kafka"})
sum by (cls) (kafka_server_replica_manager_under_min_isr_partitions{job="kafka"})
max by (cls) (kafka_controller_offline_partition_count{job="kafka"})

Check consumer lag:

topk(20, kafka_consumergroup_lag_sum{cls="kf-main"})

Check request saturation and latency:

kafka_server_request_handler_idle_ratio{job="kafka",cls="kf-main"}
max by (ins,request,quantile) (
  kafka_network_request_total_time_seconds{job="kafka",cls="kf-main",quantile=~"0.95|0.99"}
)

Log Queries

Kafka services write stdout and stderr to Journald; the node’s Vector Journald source forwards them to VictoriaLogs, all under job:syslog.

job:syslog unit:kafka
job:syslog app:kafka
job:syslog unit:kafka_exporter
ip:10.10.10.11 job:syslog (unit:kafka OR app:kafka)

The log panel on the Kafka Instance dashboard uses similar queries and shows time, level, systemd unit, and message. When diagnosing, align the logs with the KRaft, ISR, request queue, GC, disk I/O, and network metrics from the same time window.

Verifying the Monitoring Chain

Verify the raw endpoints on a Kafka node:

curl -fsS http://<kafka-ip>:9404/metrics | grep '^jmx_scrape_error'
curl -fsS http://127.0.0.1:9308/metrics | grep '^kafka_brokers'

Check file discovery on an Infra node (one file per instance; the file for a selected node contains both the JMX and the protocol exporter targets):

ls -l /infra/targets/kafka/
cat /infra/targets/kafka/kf-main-1.yml

Then query up{job="kafka"} in VictoriaMetrics (or the recorded metrics kafka_up and kafka_exporter_up). After a failed scrape, custom exporter metrics may briefly retain stale samples, so endpoint liveness should be judged by Prometheus’s native up. If the raw endpoints are fine but the recorded metrics are missing, check file discovery, the VictoriaMetrics target, network reachability, rule loading, and labels, in that order. If the JMX HTTP endpoint is fine but jmx_scrape_error is 1, check the Kafka logs and the MBean matching in /etc/kafka/jmx_exporter.yml.

For complete metric semantics, see Metric Definitions.

17.7 - Metrics

Kafka JMX, protocol exporter, and recording rule metrics dictionary.

The KAFKA module uses two kinds of metric sources, both registered under the same job=kafka scrape job: the JMX target (with a role label) collects the internal state of each JVM, while the protocol exporter target (without a role label) collects the state of the logical cluster, topics, partitions, and consumer groups over the Kafka protocol. The protocol exporter is placed only on the one or two broker-capable nodes with the smallest kafka_seq, and a single-broker cluster runs just one.

The JMX configuration is an allow-list: it exports only the JVM baseline and a bounded set of broker, replication, request-path, and KRaft metrics. High-cardinality per-client and per-partition JMX MBeans are deliberately excluded; partition detail is supplied by the protocol exporter.

Common Labels

Metric Source Common Labels
JMX target (:9404) job, cls, ins, ip, instance, role, node_id
Protocol exporter target (:9308) job, cls, ins, ip, instance

The job for both target types is kafka; whether a series carries the role label is what distinguishes the two.

Some metrics also carry dimensions such as topic, partition, broker, consumergroup, request, version, error, quantile, state, or operation.

Availability and Scrape Metrics

Metric Type Meaning
kafka_up Gauge/Recording JMX target scrape availability: up{job="kafka",role=~".+"}
kafka_exporter_up Gauge/Recording Protocol exporter target scrape availability: up{job="kafka",role=""}
up Gauge VictoriaMetrics scrape status for the raw target
jmx_scrape_error Gauge Whether the JMX Exporter’s last scrape errored; healthy value is 0
jmx_scrape_duration_seconds Gauge JMX scrape duration
jmx_scrape_cached_beans Gauge Number of MBeans cached by the JMX Exporter
scrape_duration_seconds Gauge Time VictoriaMetrics took to scrape the exporter
scrape_samples_scraped Gauge Number of samples in this scrape

Protocol Exporter Metrics

The following metrics come from the protocol exporter target. Multiple exporters of the same cluster see the same logical cluster state, so any direct cluster aggregation must deduplicate by semantics rather than simply summing across all ins.

Broker and Topic

Metric Type Key Dimensions Meaning
kafka_brokers Gauge Cluster Number of brokers discovered by the exporter
kafka_broker_info Gauge id, address, etc. Broker info, carried on labels with value 1
kafka_topic_partitions Gauge topic Number of partitions in a topic
kafka_topic_partition_current_offset Gauge topic, partition Partition’s current log end offset
kafka_topic_partition_oldest_offset Gauge topic, partition Partition’s current earliest readable offset
kafka_topic_partition_leader Gauge topic, partition Current leader broker ID; used to spot anomalies when there is no leader
kafka_topic_partition_replicas Gauge topic, partition, broker The replica set assigned to a partition
kafka_topic_partition_in_sync_replica Gauge topic, partition, broker Current ISR members
kafka_topic_partition_under_replicated_partition Gauge topic, partition Whether the partition is under-replicated
kafka_topic_partition_leader_is_preferred Gauge topic, partition Whether the current leader is the preferred replica

current_offset - oldest_offset estimates the currently retained offset span, but an offset count is not a byte count, and for a compacted topic it is not an exact message count either.

Consumer Group

Metric Type Key Dimensions Meaning
kafka_consumergroup_members Gauge consumergroup Current member count of the group
kafka_consumergroup_current_offset Gauge consumergroup, topic, partition Group’s committed offset
kafka_consumergroup_current_offset_sum Gauge consumergroup, topic Sum of committed offsets
kafka_consumergroup_lag Gauge consumergroup, topic, partition Partition-level consumer lag
kafka_consumergroup_lag_sum Gauge consumergroup, topic Consumer lag aggregated per group/topic

Ephemeral consumers that never commit an offset, clients that use external offset storage, and groups that have not yet consumed a topic will not necessarily produce these time series.

Exporter Itself

Metric Type Meaning
kafka_exporter_build_info Gauge Exporter version, revision, and build info
process_* Gauge/Counter Exporter process CPU, memory, FD, start time, etc.
go_* Gauge/Counter Exporter Go runtime, GC, goroutine, and memory state
promhttp_metric_handler_* Counter /metrics request handling status

JMX: JVM Baseline

excludeJvmMetrics: false makes the JMX Exporter expose the standard JVM/process metrics. The Kafka Instance dashboard mainly uses:

Metric Meaning
jvm_memory_used_bytes Used memory, split by heap/non-heap and memory pool
jvm_memory_committed_bytes JVM committed memory
jvm_memory_max_bytes Maximum memory available to the JVM
jvm_gc_collection_seconds_count GC count
jvm_gc_collection_seconds_sum Cumulative GC time
jvm_threads_state Thread count by thread state
jvm_threads_deadlocked Number of detected deadlocked thread cycles
jvm_buffer_pool_used_bytes Direct/mapped buffer pool usage
process_cpu_seconds_total Cumulative CPU time of the Kafka JVM
process_open_fds / process_max_fds Open and maximum file descriptors
process_start_time_seconds Kafka JVM start time

JMX: Broker Traffic

Metric Type Meaning
kafka_server_broker_messages_in_total Counter Total messages received by the broker
kafka_server_broker_bytes_in_total Counter Total client bytes received by the broker
kafka_server_broker_bytes_out_total Counter Total client bytes sent by the broker
kafka_server_broker_replication_bytes_in_total Counter Total replication bytes received by the broker
kafka_server_broker_replication_bytes_out_total Counter Total replication bytes sent by the broker
kafka_server_broker_produce_requests_total Counter Total produce requests
kafka_server_broker_failed_produce_requests_total Counter Total failed produce requests
kafka_server_broker_fetch_requests_total Counter Total fetch requests
kafka_server_broker_failed_fetch_requests_total Counter Total failed fetch requests

These are broker-wide totals with no topic dimension, which keeps the JMX series count from growing with the number of topics. Topic-level offsets and progress come from the protocol exporter.

JMX: Replication and Storage

Metric Type Meaning
kafka_server_replica_manager_under_replicated_partitions Gauge Number of partitions whose ISR is smaller than the assigned replica set
kafka_server_replica_manager_under_min_isr_partitions Gauge Number of partitions whose ISR is below min.insync.replicas
kafka_server_replica_manager_at_min_isr_partitions Gauge Number of partitions whose ISR is exactly min.insync.replicas
kafka_server_replica_manager_offline_replicas Gauge Number of offline replicas on the current broker
kafka_server_replica_manager_partitions Gauge Number of replicas hosted by the current broker
kafka_server_replica_manager_leaders Gauge Number of partitions led by the current broker
kafka_server_replica_manager_isr_shrinks_total Counter Total ISR shrink events
kafka_server_replica_manager_isr_expands_total Counter Total ISR expand events
kafka_server_replica_manager_failed_isr_updates_total Counter Total failed ISR updates
kafka_server_replica_manager_reassigning_partitions Gauge Number of leader partitions currently being reassigned
kafka_server_delayed_operation_purgatory_size Gauge Number of delayed operations waiting, split by operation
kafka_log_manager_offline_log_directories Gauge Number of log directories Kafka has marked offline

Under Replicated means the replicas are not all in sync. Under Min ISR is more serious: the write-availability or durability condition has fallen below the configured minimum ISR. At Min ISR has not crossed the line yet, but there is no remaining replica headroom.

JMX: Request Path

Metric Type Extra Labels Meaning
kafka_network_request_total Counter request, version Total requests per Kafka API
kafka_network_request_errors_total Counter request, error Total error responses per API/error code
kafka_network_request_total_time_seconds Gauge request, version, quantile Total API time at P50/P95/P99
kafka_network_request_queue_size Gauge — Number of requests waiting for a request handler
kafka_network_response_queue_size Gauge — Number of responses waiting for a network processor
kafka_server_request_handler_idle_ratio Gauge — Average request-handler idle ratio
kafka_network_processor_idle_ratio Gauge — Average network-processor idle ratio

When investigating high latency, look at request volume, error codes, P95/P99, both queues, handler/processor idle, GC, CPU, disk I/O, and network together. A low idle ratio alone is not enough to pinpoint where the bottleneck is.

JMX: KRaft and Broker Metadata

Metric Type Meaning
kafka_server_raft_state Gauge The current member’s KRaft state, expressed via the state label
kafka_server_raft_current_leader Gauge Current KRaft leader node ID; -1 means unknown
kafka_server_raft_current_epoch Gauge Current KRaft epoch
kafka_server_raft_high_watermark Gauge Metadata log high watermark
kafka_server_raft_log_end_offset Gauge Metadata log end offset
kafka_server_broker_metadata_last_applied_record_lag_seconds Gauge Time lag of the broker applying metadata records
kafka_server_broker_metadata_load_errors_total Counter Total broker metadata load errors
kafka_server_broker_metadata_apply_errors_total Counter Total broker metadata image apply errors
kafka_server_metadata_snapshot_bytes Gauge Size of the most recently generated or loaded metadata snapshot
kafka_server_metadata_snapshot_age_seconds Gauge Age of the most recent metadata snapshot

log_end_offset - high_watermark helps gauge metadata commit lag; also factor in the member role, current leader, epoch, and controller event latency.

JMX: Controller

These MBeans exist only in Kafka processes that carry the controller role:

Metric Type Meaning
kafka_controller_active_controller_count Gauge 1 on the active controller, 0 on the others
kafka_controller_fenced_broker_count Gauge Number of fenced brokers observed by the active controller
kafka_controller_active_broker_count Gauge Number of active brokers
kafka_controller_global_topic_count Gauge Number of topics observed by the controller
kafka_controller_global_partition_count Gauge Number of partitions observed by the controller
kafka_controller_offline_partition_count Gauge Number of offline non-internal partitions
kafka_controller_preferred_replica_imbalance_count Gauge Number of partitions whose leader is not the preferred replica
kafka_controller_metadata_errors_total Counter Total controller metadata processing errors
kafka_controller_last_applied_record_lag_seconds Gauge Time lag of the controller applying metadata records
kafka_controller_timed_out_broker_heartbeats_total Counter Total broker heartbeat timeouts
kafka_controller_elections_total Counter Total new active-controller elections observed by this node
kafka_controller_unclean_leader_elections_total Counter Total unclean leader elections
kafka_controller_event_queue_time_seconds Gauge Controller event queue time at P50/P95/P99
kafka_controller_event_processing_time_seconds Gauge Controller event processing time at P50/P95/P99

A healthy cluster should have exactly one active controller. Any increase in offline_partition_count, metadata_errors_total, or unclean_leader_elections_total should be treated as a priority.

Recording Rule Metrics

Offset Progress

Metric Aggregation Level Window Meaning
kafka:topic:msg_rate1m Topic 1m Forward growth rate of current offset, deduplicated across exporters
kafka:topic:msg_rate5m Topic 5m Forward growth rate of current offset, deduplicated across exporters
kafka:cls:msg_rate1m Logical cluster 1m Message append rate, deduplicated across exporters
kafka:cls:msg_rate5m Logical cluster 5m Message append rate, deduplicated across exporters
kafka:csg_topic:commit_rate5m Group/Topic 5m Forward growth rate of commit offset
kafka:csg_topic:lag Group/Topic current Partition lag, deduplicated and summed
kafka:csg:lag Consumer group current Total group lag across topics
kafka:cls:lag Logical cluster current Total cluster lag across consumer groups

JVM and Broker

Metric Meaning
kafka:ins:jvm_heap_used_ratio Heap used / heap max
kafka:ins:jvm_cpu_cores 5-minute JVM CPU core consumption
kafka:ins:load Saturation of the instance’s busiest request thread pool
kafka:cls:load Average load across the cluster’s instances
kafka:ins:jvm_gc_time_rate5m 5-minute GC time rate
kafka:ins:messages_in_rate5m Broker 5-minute message receive rate
kafka:ins:bytes_in_rate5m Broker 5-minute inbound client byte rate
kafka:ins:bytes_out_rate5m Broker 5-minute outbound client byte rate
kafka:ins:request_error_rate5m 5-minute rate of non-NONE request errors
kafka:cls:under_replicated_partitions Total under-replicated partitions in the cluster
kafka:cls:offline_partitions Offline partitions in the cluster

Cardinality and Interpretation Notes

  • Do not directly sum multiple kafka_exporter results for the same cls; they may be duplicate views of the same cluster.
  • kafka_topic_partition_current_offset is an offset, not an exact count of bytes, requests, or business events.
  • Consumer lag covers only groups that are visible in Kafka and have committed an offset.
  • A controller-only node lacking broker metrics and protocol exporter metrics is a normal role difference; a broker that was not selected lacking protocol exporter metrics is a normal result of placement.
  • When a given MBean does not exist in a specific Kafka version/role, the corresponding JMX series will not appear either; interpret this together with role.
  • Per-client/per-partition JMX metrics are excluded by the allow-list to avoid unpredictable time-series cardinality.

For how to use the dashboards and alerts, see Monitoring.

17.8 - FAQ

Frequently asked questions about the Pigsty Kafka 4.1+ dynamic KRaft module.

How mature is the current KAFKA module?

The current role implements a production-grade v1 baseline: dynamic KRaft, full cluster guardrails, cold-start/repair, serial broker admission and dynamic controller join, member retirement (including dead nodes), three-command failed-node replacement, strict rolling restart, TLS/SCRAM/ACL, declarative convergence of topics/users, internal credential and certificate rotation, and the full monitoring pipeline.

It is not a managed Kafka product. Production still requires kafka_security: scram, an odd number of controllers, sufficient brokers/RF/minISR, plus your own capacity planning, reassignment/data balancing, upgrade, backup, restore, and failure drills. The default plaintext is only suitable for development or a trusted, isolated network.

Why is there no ZooKeeper and no controller.quorum.voters?

This module targets Kafka 4.1+ and uses native dynamic KRaft, with no ZooKeeper installed and no static quorum created. All members render controller.quorum.bootstrap.servers; new clusters are formatted explicitly with --initial-controllers/--no-initial-controllers, and after startup the role verifies that the directory IDs of the initial controllers have joined the live quorum.

The initial controller identity is written into the bootstrap manifest, but it is only a birth certificate: after the cluster’s first commission, live quorum membership is authoritative in Raft itself. Later controller additions and removals are orchestrated by the playbooks — additions go through kafka.yml’s observer catch-up + add-controller join flow, removals through a kafka-rm.yml strict-subset retirement (automatic remove-controller) — you only edit the inventory and run the matching playbook.

What is the difference between combined, broker, and controller?

  • combined: acts as both broker and controller, listening on 9092 and 9093; this is the default;
  • broker: pure data plane, listening only on 9092;
  • controller: pure control plane, listening only on 9093.

The cluster roles must either be omitted entirely and consistently use combined, or be declared explicitly for every member. The old role aliases are no longer provided.

Will controller port 9093 collide with Alertmanager?

No. Pigsty’s Alertmanager listens on alertmanager_port 9059 with cluster port 9094, clear of the KRaft controller’s conventional port 9093. If you changed those ports and created a clash, adjust kafka_controller_port for that cluster — the role only enforces that the four Kafka ports 9092, 9093, 9308, and 9404 differ from one another, and does not detect port conflicts with other services.

The service is up, but a remote client cannot connect?

A broker’s advertised.listeners always uses inventory_hostname. After connecting to the bootstrap server, a client must also resolve and reach every broker address returned in the metadata.

Check in order:

grep '^advertised.listeners' /etc/kafka/server.properties
ss -lntp | grep ':9092'
getent hosts <inventory-hostname>

A scram client must additionally check the CA, SASL mechanism, username/password, and ACLs. The current v1 does not offer custom advertised addresses, multiple listeners, or NAT/public mapping; if a client cannot route directly to inventory_hostname, that network model is outside the current core contract and cannot be worked around by overriding the raw listener via kafka_parameters.

Why does it report a Cluster ID, Node ID, or Directory ID mismatch?

The role cross-checks the bootstrap manifest, ${kafka_data}/metadata/meta.properties, the inventory, and the live dynamic quorum. Common causes include:

  • kafka_cluster or kafka_seq was changed;
  • another cluster’s data disk was mounted on the current node;
  • an incorrect kafka_cluster_id was given during restore/takeover;
  • the controller data directory or directory ID does not match the live voter records;
  • the wrong target cluster was selected, or a stale manifest was used.

This is a protective failure. Do not delete meta.properties, the manifest, or run kafka-rm.yml. First confirm data ownership, the remaining replicas, the true Cluster/Node/Directory identity, and the recovery target.

What happens if the manifest is lost or only an old one remains?

Every cluster member keeps an authoritative copy of the manifest at /etc/kafka/manifest.yml (a scram cluster also has /etc/kafka/secrets.yml). The admin node keeps no kafka state and resolves both from any member copy on every run, so replacing the admin node or losing the local checkout does not affect cluster management. Only when all member copies are lost while the storage has already been formatted does the role fail closed and prompt you to restore the file on any member first; a formatted scram cluster likewise fails closed when no member holds the secret material. Issued node certificates are cached under files/pki/kafka/ and are simply re-signed from the Pigsty CA when absent.

Conversely, if the manifest exists but all Kafka data disks are empty, the role fails closed to avoid accidentally reviving a vanished cluster under an old identity. If you genuinely intend to rebuild, you must first run kafka-rm.yml and follow an explicit rebuild procedure.

Why are some keys in kafka_parameters rejected?

Identity, the dynamic quorum, listeners, storage, replication, rack, and security must have a single source of authority, so those keys are owned by the role: if any one of them appears, the identity precheck fails before anything is written. For the complete reserved list, see kafka_parameters.

Use the corresponding public parameters instead. The role provides no variables for advertised addresses, path subdirectories, the listener map, or exporter options.

How do I enable TLS, SCRAM, and ACLs?

Set on a new cluster:

kafka_security: scram

This enables the Pigsty CA node certificates, controller mTLS, broker/client SASL_SSL + SCRAM-SHA-512, StandardAuthorizer, and default-deny all at once. Application users declare their passwords, ACLs, and optional quotas via kafka_users.

The security mode is a bootstrap-only property. A formatted cluster cannot switch online from plaintext to scram via ordinary playbooks; that requires a separate migration state machine. A healthy scram cluster can rotate internal credentials or certificates through protected actions.

Do kafka_topics and kafka_users delete resources?

No. Removing an entry from the inventory never implicitly deletes a topic or a user.

Topics are created idempotently, partitions only increase, and only the declared configs are updated; an RF change requires an explicit reassignment. A declared user has its password, complete ACL set, and the given quota fields converged. Topic deletion, user deletion, and full privilege revocation are all separate, audited operations.

What is the difference between the JMX Exporter and kafka_exporter?

The JMX Exporter is injected into every Kafka JVM and collects JVM, broker, replication, request-path, and KRaft internal metrics, registered as a job=kafka target with a role label.

kafka_exporter queries the logical cluster, topics, partitions, offsets, consumer groups, and lag over the Kafka protocol, registered as a target under the same job=kafka but without a role label. The role runs it only on the first two broker-capable nodes ordered by kafka_seq; a single-broker cluster runs one, and pure controllers run none.

The two are complementary. The lifecycle health gate uses the role’s own Kafka CLI/metadata channel and does not depend on either exporter.

Why does a particular broker or pure controller have no kafka_exporter?

This is the expected derived placement. The protocol exporter returns a view of the entire logical cluster, not node metrics; capping it at two replicas avoids a monitoring single point of failure while keeping the cost of duplicate scraping in check.

Check the current targets (one file per instance; the files of selected nodes contain the :9308 protocol exporter target):

ls -l /infra/targets/kafka/
grep 9308 /infra/targets/kafka/*.yml

A full run refreshes each instance’s target file according to the current placement, so you should not run against a single node just to register labels. Note: if the exporter placement moves due to a topology change, the old kafka_exporter service on a formerly selected node is not stopped automatically by ordinary playbooks and must be cleaned up manually or via kafka-rm.yml.

Why is the JMX endpoint reachable but jmx_scrape_error=1?

HTTP reachability only means the Java agent is loaded; jmx_scrape_error=1 means the MBean scrape failed this round:

journalctl -u kafka --since '-30 min' --no-pager
curl -fsS http://<kafka-ip>:9404/metrics | head -n 40

Check whether /etc/kafka/jmx_exporter.yml matches the currently installed Kafka/JMX Exporter packages, and whether the JVM has passed startDelaySeconds. Real startup acceptance requires jmx_scrape_error 0.0, JVM metrics, and at least one kafka_ metric matching the role.

Why is there no Consumer Lag data?

Common causes: the consumer does not use a group, does not commit offsets to Kafka, stores offsets in an external system, the group has not yet consumed the target topic, or the protocol exporter has a TLS/SCRAM/ACL/network problem.

/opt/kafka/bin/kafka-consumer-groups.sh \
  --bootstrap-server <broker>:9092 \
  --command-config /etc/kafka/admin.properties \
  --describe --group <group>

Then check kafka_exporter_up, the exporter logs, the dashboard variables, and the raw kafka_consumergroup_* metrics. Endpoint liveness is judged by Prometheus’s native up; do not substitute a custom metric that may briefly linger after a scrape failure.

Why can’t the cluster metrics from two kafka_exporters be added together?

Both exporters query the same logical cluster and may return the same topic/partition/consumer-group state; summing them directly double-counts. Pigsty’s kafka:cls:* recording rules first deduplicate across the exporter replicas, then aggregate to the cluster.

Should applications go through HAProxy, a Keepalived VIP, or an LB?

No. Kafka producers and consumers are cluster-aware clients: once they reach any seed in bootstrap.servers and fetch metadata, they connect directly to each partition leader. A VIP or generic TCP LB neither understands partition leaders nor rewrites the broker addresses in metadata; putting one in the data plane only adds long-connection state, an extra point of failure, and troubleshooting complexity.

If a platform mandates a single discovery entry point, DNS or a TCP LB may serve bootstrap only, but advertised.listeners still returns a client-reachable address for each broker, and the application network must reach every broker. Exposure across NAT, the public internet, multiple networks, or Kubernetes requires a dedicated external address and an additional listener per broker; the current module always advertises the inventory address and does not support such mapping.

See Quickstart: why applications should connect directly to multiple brokers and Cluster Config: network and listeners.

Can I just add or remove a broker or controller?

Yes. Edit the inventory and let the playbooks orchestrate every step of the KRaft membership change:

  • Add: declare the new member in the inventory (broker, combined, and controller all work) and run ./kafka.yml -l <cls> against the complete cluster (you cannot limit the run to the new node only). Pure brokers are formatted, started, and verified as registered one at a time; combined/controller nodes are formatted with --no-initial-controllers, catch up as observers, then get promoted with add-controller. One node at a time, with health gates throughout.
  • Remove: ./kafka-rm.yml -l <ip> (a strict subset of the cluster) performs remove-controller and the broker unregistration through a surviving member — it works even when the node is unreachable — then delete the member from the inventory.

You still own the planning: keep the controller count odd with a live majority after the change, make one membership change at a time, and drain the partitions off a removed broker first (or let a same-kafka_seq replacement take them over). After a node joins, existing partitions are not migrated automatically; you must run and monitor reassignment separately — “the broker is registered” does not mean “capacity is already balanced.”

Which parameter controls the package version?

The role uses package_map['java-runtime'] and package_map['kafka-stack']; there is no kafka_version, scala_version, or exporter version parameter. The actual versions are determined by the Pigsty repository for the target platform and the installed packages.

The payload verified on 2026-07-16 is Kafka 4.3.1, kafka_exporter 1.9.0, and JMX Exporter 1.6.0. An upgrade still requires a separate review of compatibility, backup/rollback, rolling order, and feature level; you cannot simply swap the packages.

How do I safely wipe Kafka data?

kafka.yml never performs cleanup; deletion lives only in the separate kafka-rm.yml playbook. Selecting a whole cluster with -l (or running bare for all clusters) is a teardown; selecting a strict subset is member retirement. By default kafka_rm_data=true permanently deletes the data/KRaft metadata, node-local /etc/kafka recovery state, and monitoring targets; kafka_rm_data=false keeps the data and recovery state, and kafka_safeguard=true aborts any deletion.

The playbook has no extra gate such as a confirmation string. Commands delete immediately; before running one, manually confirm the exact -l target, a recoverable backup or clear rebuild intent, and the business-decommissioned status. Broker unregistration during member retirement tolerates failure, so after a real run also inspect quorum, broker registrations, and replica health. For the full semantics, see Playbook: kafka-rm.yml.

18 - Module: MYSQL

Deploy native MySQL 8.4 LTS as a standalone instance or a three-node InnoDB Cluster, with TLS, daily backups, and full observability.

MySQL is one of the world’s most popular open-source relational databases. Pigsty’s MYSQL module deploys a fixed, native MySQL 8.4 LTS platform on managed nodes: either a standalone instance or a three-node single-primary InnoDB Cluster built on Group Replication, with TLS, backups, monitoring, and lifecycle handled for you.

Current status: Pilot module

MYSQL is a supplementary pilot module. It aims to be a simple, inexpensive, good-enough MySQL cluster — not a peer of the PGSQL module. The core capabilities (deployment and convergence, HA failover, daily backups, monitoring and alerting) have been tested systematically; destructive procedures such as complete-outage recovery and physical restore are deliberately kept manual, with runbooks provided in Administration.

Module Capabilities

The MYSQL module currently provides:

  • A fixed native MySQL 8.4 LTS platform: server, client, Shell, Router, and XtraBackup at matching versions, working out of the box
  • Two topologies: a standalone instance, or a three-node single-primary InnoDB Cluster created and reconciled through MySQL Shell AdminAPI
  • MySQL Router on every HA member, providing topology-aware read-write (6446) and read-only (6447) endpoints
  • TLS everywhere: leaf certificates issued from the shared Pigsty CA; non-TLS connections are rejected
  • Declarative business objects: mysql_databases and mysql_users converge additively and never delete data implicitly
  • mysql_parameters overrides for key settings such as max_connections, with orchestrated rolling restarts on configuration change
  • A daily full physical backup: XtraBackup backup plus prepare, with retention, concurrency locking, and atomic commit
  • Full observability: mysqld_exporter metrics, 68 recording rules, 27 alert rules, 5 Grafana dashboards, and error logs shipped to VictoriaLogs
  • sql_require_primary_key enabled by default, blocking PK-less tables that would break MGR replication and disaster recovery
  • Convergent operations: for a dropped member or drifted AdminAPI state, rerunning mysql.yml heals the cluster; destructive paths are fenced by guardrails

Module Architecture

The MYSQL module depends on NODE for node management, package repositories, and the shared CA, and on INFRA for VictoriaMetrics, VictoriaLogs, Grafana, and Alertmanager. It does not require ETCD or PGSQL.

flowchart LR
    admin["Pigsty admin node"] -->|"mysql.yml"| mysqld["mysqld ×3 / single-primary MGR<br>3306 · TLS"]
    client["Application clients"] -->|"RW 6446 / RO 6447"| router["MySQL Router<br>(on every HA member)"]
    router --> mysqld
    mysqld --> backup["XtraBackup daily full<br>(current primary only)"]
    mysqld --> exporter["mysqld_exporter :9104"]
    mysqld --> journal["Error log → Journald"]
    exporter --> vm["VictoriaMetrics"]
    journal --> vector["Vector"] --> vl["VictoriaLogs"]
    vm --> grafana["Grafana"]
    vl --> grafana
    vm --> alertmanager["Alertmanager"]

    style mysqld fill:#4479A1,stroke:#33618a,color:#fff
    style router fill:#70C1B3,stroke:#4f968b,color:#fff
    style vm fill:#E66B7A,stroke:#b84e5c,color:#fff
    style vl fill:#C98367,stroke:#9e634e,color:#fff

In the three-node topology, mysql_seq=1 is only the bootstrap coordinator. The runtime PRIMARY is elected, and reruns never force the primary back to node 1.

Components and Ports

Component Purpose Fixed endpoint
mysqld Standalone server or MGR member Classic 3306, X Protocol 33060
Group Replication Three-member replication and consensus (XCOM) 33061
MySQL Router Topology-aware entry point on every HA member RW 6446, RO 6447
MySQL Shell AdminAPI cluster lifecycle Local control plane
XtraBackup Daily full physical backup Local backup repository
mysqld_exporter Server and MGR metrics 9104

The role creates and manages three platform identities:

  • dbuser_cluster@'%': TLS-only AdminAPI and Router bootstrap identity (created on HA clusters only);
  • dbuser_monitor@'127.0.0.1': least-privilege exporter identity;
  • dbuser_backup@'localhost': local XtraBackup identity.

Supported Platforms

The native-package platform gate admits:

Arch Supported systems
x86_64 EL 8/9/10, Debian 12/13, Ubuntu 22/24
aarch64 EL 9/10

Debian/Ubuntu ARM64 is rejected at preflight: Oracle’s APT repository publishes no arm64 payload for MySQL 8.4. On ARM, use EL 9/10 (e.g. Rocky Linux).

Scope and Boundaries

MYSQL is a fixed platform, not a general-purpose MySQL installer. The following are deliberate non-goals — confirm they are acceptable before adopting:

  • Topology is fixed at 1 or 3 nodes: no in-place 1→3 upgrade, no 3→5 scale-out, no persistent two-node operation. Capacity upgrades go through logical migration; hardware refresh goes through same-address replacement
  • Versions, ports, directories, and charset are fixed: no parameters expose them. Memory sizing is derived from node specs and can be overridden per key via mysql_parameters
  • Backups are daily local fulls: no incremental chain, no continuous binlog archiving, no PITR. Physical restore is a manual procedure with a runbook
  • Complete-outage recovery stays manual to rule out split-brain from automated guessing; playbook failures print the recovery instructions
  • No VIP / DNS / HAProxy access layer: clients connect through any member’s Router ports, preferably with a multi-host DSN

Documentation

Page Content
Configuration Topology planning, identity, databases, users, parameter overrides, backup settings
Parameters The 11 public parameters and fixed platform conventions
Administration Status checks, client access, config changes, failure handling, and three recovery runbooks
Playbook mysql.yml and mysql-rm.yml usage, tags, and guardrails
Monitoring Dashboards, recording rules, alert rules, log queries
Metrics Label model and the derived-metric dictionary
FAQ Platform limits, primary-key policy, recovery, troubleshooting

Quick Start

Declare a cluster in the inventory (full template: conf/demo/mysql.yml):

all:
  children:
    my-test:
      hosts:
        10.10.10.11: { mysql_seq: 1 }
        10.10.10.12: { mysql_seq: 2 }
        10.10.10.13: { mysql_seq: 3 }
      vars:
        mysql_cluster: my-test
        mysql_databases: [ { name: app } ]
        mysql_users: [ { name: app, password: DBUser.App, priv: { 'app.*': 'ALL PRIVILEGES' } } ]

  vars:
    node_repo_modules: node,infra,mysql   # repos must include the mysql module
    mysql_root_password: MySQL.Root       # change all sample passwords in production
    mysql_monitor_password: MySQL.Monitor
    mysql_cluster_password: MySQL.Cluster

After NODE provisioning, deploy:

./node.yml  -l my-test             # node provisioning: repo, shared CA, monitoring agents
./mysql.yml -l my-test --check     # preflight the complete three-node cluster
./mysql.yml -l my-test             # real run; a three-node cluster takes ~2 minutes

mysql -h 10.10.10.11 -P 6446 -u app -pDBUser.App \
  --ssl-mode=VERIFY_CA --ssl-ca=/etc/pki/ca.crt app   # connect through the Router RW endpoint

Then open the Grafana MySQL Overview dashboard to inspect the cluster.

18.1 - Configuration

Plan MySQL topology and identity; declare databases, users, parameter overrides, and backup policy.

The MYSQL module is driven by the inventory: you declare the desired cluster, and mysql.yml converges the live state to match. This page covers topology planning and every configuration block; see Parameters for the full reference.

Before You Deploy

  • Target nodes are NODE-managed, with the shared CA installed at /etc/pki/ca.crt (managed by the node_ca role; the MySQL role only issues leaf certificates);
  • Package repositories include the mysql module: node_repo_modules: node,infra,mysql, or a local repo cached with repo_extra_packages: [mysql];
  • The platform is in the support matrix: x86_64 on EL 8/9/10, Debian 12/13, Ubuntu 22/24; or aarch64 on EL 9/10;
  • The three platform passwords (mysql_root_password, mysql_monitor_password, mysql_cluster_password) are set to production values — preflight rejects CHANGE_ME placeholders.

Identity

Each cluster is an inventory group with two required identity parameters:

Parameter Level Description
mysql_cluster Cluster Cluster name; must match the inventory group holding the members. Also the backup directory and the cls monitoring label
mysql_seq Instance 1 for standalone; sequential 1..3 for HA; doubles as server_id

Topology is inferred from member count: 1 member is a standalone, 3 members form an InnoDB Cluster; any other count is rejected at preflight. mysql_seq=1 is only the bootstrap coordinator, not the runtime primary.

Instance names follow {{ mysql_cluster }}-{{ mysql_seq }} (e.g. my-test-1). The inventory host address (IP or resolvable hostname) is the advertised MySQL and MGR address and cannot be changed by an ordinary rerun.

Standalone Instance

The minimal standalone declaration:

my-meta:
  hosts:
    10.10.10.10: { mysql_seq: 1 }
  vars:
    mysql_cluster: my-meta

Standalone instances have no Router (6446/6447 do not exist); clients connect to 3306 directly. Backups, monitoring, and TLS behave exactly as in HA mode.

Three-Node InnoDB Cluster 

my-test:
  hosts:
    10.10.10.11: { mysql_seq: 1 }
    10.10.10.12: { mysql_seq: 2 }
    10.10.10.13: { mysql_seq: 3 }
  vars:
    mysql_cluster: my-test
    mysql_databases:
      - { name: app }
    mysql_users:
      - name: app
        host: '%'
        password: DBUser.App
        connlimit: 20
        priv: { 'app.*': 'ALL PRIVILEGES' }

This yields a single-primary MGR cluster: one writable PRIMARY, two read-only SECONDARY members, tolerating one node failure. Every member runs a Router, so port 6446 on any member reaches the current primary.

Always select the complete cluster

Every mysql.yml run must select all members of the cluster with -l (or omit -l to converge every MySQL cluster). Partial member selection is rejected at preflight — a deliberate guard against topology divergence.

Databases

mysql_databases declares databases additively:

mysql_databases:
  - { name: app }                                              # utf8mb4 / utf8mb4_0900_ai_ci by default
  - { name: app2, encoding: utf8mb4, collate: utf8mb4_general_ci }
Field Default Description
name required Database name, [A-Za-z0-9_$-]; system schema names are rejected
encoding utf8mb4 Character set
collate utf8mb4_0900_ai_ci Collation

Each entry accepts only these three fields; preflight validation rejects additional keys.

Convergence is additive: reruns create missing databases, but removing an entry never drops one. Deleting data is a manual operation by design.

Every table needs a primary key

The platform enables sql_require_primary_key=ON by default, so creating a PK-less table fails with ERROR 3750. This is not pedantry: PK-less tables are read-only under MGR and block AdminAPI cluster rebuilds during disaster recovery. Define a primary key on every table (invisible-column PKs work too); override via mysql_parameters only if you truly must.

Users

mysql_users declares users and grants additively:

mysql_users:
  - name: app                        # username
    host: '%'                        # grant source, defaults to '%'
    password: DBUser.App             # required; special characters are handled
    connlimit: 20                    # MAX_USER_CONNECTIONS, 0 = unlimited
    priv:                            # grant map: 'db.table' -> privilege list
      'app.*': 'ALL PRIVILEGES'
      'app2.*': 'SELECT, INSERT, UPDATE, DELETE'

Grant scopes are written as 'db.table', with * wildcards on either side ('*.*', 'app.*'); values are comma-separated privilege names. Preflight validates usernames, hosts, scopes, and privilege words, rejecting malformed declarations.

Semantics:

  • Missing users are created; existing users get their password and connection limit updated;
  • Grants in priv are applied, but removing a mapping does not REVOKE;
  • The platform identities (root, dbuser_monitor, dbuser_cluster, dbuser_backup) cannot be declared;
  • The server enforces TLS: the client default PREFERRED mode negotiates encryption automatically, and plaintext (DISABLED) connections are rejected; prefer an explicit VERIFY_CA.

Parameter Overrides

mysql_parameters overrides [mysqld] options, rendered at the end of the managed config so the last value wins:

my-test:
  vars:
    mysql_cluster: my-test
    mysql_parameters:
      max_connections: 500
      long_query_time: 2
      innodb_print_all_deadlocks: true   # booleans render as ON/OFF

Rules and safety:

  • Keys must be plain option names (letter first; ._- allowed); values must be single-line scalars;
  • The rendered config still passes mysqld --validate-config, so a bad option fails at deploy time without touching the running service;
  • Platform-reserved options cannot be overridden: identity and protocol (user, pid_file, server_id, datadir, socket, port, bind_address, mysqlx_bind_address, report_host, mysqlx, …), replication and plugins (gtid_mode, enforce_gtid_consistency, log_bin, relay_log, plugin_load*, clone, plugin_clone, plugin_mysqlx, group_replication_*, …), and TLS (require_secure_transport, ssl_*) are role-managed and rejected if declared;
  • Parameter changes trigger an orchestrated rolling restart: secondaries first, primary last.

Memory needs no configuration: the buffer pool is 25% of node memory (256MB floor), redo capacity is half the buffer pool (128MB–4GB), and replica parallelism follows CPU count. For precise control, override innodb_buffer_pool_size and friends via mysql_parameters.

Backup Settings 

mysql_backup_enabled: true            # daily backup timer, on by default
mysql_backup_repo:
  local:
    path: /data/backups/mysql         # local backup root
    retention: 7                      # keep the last 7 fulls

The backup contract (details in Administration):

  • One XtraBackup full physical backup per day, prepared immediately after — the output directory is directly restorable;
  • Standalone backs up locally; in HA every member’s timer fires, but only the current PRIMARY actually runs — other members skip cleanly;
  • Layout is <path>/<cluster>/<UTC timestamp>/, with an atomic latest symlink and retention-based pruning;
  • No incremental chain, no binlog archiving, no PITR: for a standalone the recovery point is the most recent backup.
Backups follow the primary

After a failover, new backups land on the new primary’s local disk. Before restoring, check the latest timestamp on all members and take the newest. For off-site protection, sync the backup directory yourself (e.g. a scheduled rclone/rsync job).

Platform Credentials 

mysql_root_password: MySQL.Root          # local root (root@localhost, socket only)
mysql_monitor_password: MySQL.Monitor    # exporter identity
mysql_cluster_password: MySQL.Cluster    # AdminAPI / Router / backup identity

Credential lifecycle rules:

  • Passwords must be single-line and must not keep the CHANGE_ME prefix — enforced at preflight;
  • On HA clusters, mysql_cluster_password cannot be rotated by an ordinary rerun: it is embedded in cluster metadata and Router keyrings, so implicit rotation is rejected (standalone instances have no such binding and rotate normally);
  • mysql_root_password cannot be silently reset either: if the live root password differs from the declaration, the task fails explicitly instead of overwriting it.

Credential material lives in /etc/mysql/pigsty/ (root-owned: directory 0700, files 0600), including ready-to-use client configs for local operations:

mysql --defaults-extra-file=/etc/mysql/pigsty/root.cnf        # local root session
mysql --defaults-extra-file=/etc/mysql/pigsty/cluster.cnf     # cluster session via the local Router (HA members only)

Full Example

Standalone plus three-node HA, matching the four-node sandbox:

all:
  children:
    infra:
      hosts:
        10.10.10.10: { infra_seq: 1 }

    my-meta:
      hosts:
        10.10.10.10: { mysql_seq: 1 }
      vars: { mysql_cluster: my-meta, node_cluster: my-meta }

    my-test:
      hosts:
        10.10.10.11: { mysql_seq: 1 }
        10.10.10.12: { mysql_seq: 2 }
        10.10.10.13: { mysql_seq: 3 }
      vars:
        mysql_cluster: my-test
        node_cluster: my-test
        mysql_databases:
          - { name: app }
        mysql_users:
          - { name: app, password: DBUser.App, priv: { 'app.*': 'ALL PRIVILEGES' } }
        mysql_parameters:
          max_connections: 500

  vars:
    version: v4.5.0
    admin_ip: 10.10.10.10
    node_repo_modules: node,infra,mysql
    node_tune: oltp

    mysql_root_password: MySQL.Root
    mysql_monitor_password: MySQL.Monitor
    mysql_cluster_password: MySQL.Cluster

See conf/demo/mysql.yml for the full template. Note that conf/mysql.yml is the OpenHalo template (a MySQL-compatible PostgreSQL kernel) and is unrelated to this module.

18.2 - Parameters

The MYSQL module’s 13 public parameters: 11 deployment parameters, 2 protected-removal parameters, and fixed platform conventions.

The MYSQL deployment role deliberately exposes only 11 parameters; the removal role adds 2 protected operations parameters. Software versions, ports, directories, charset, TLS paths, and timer schedules are fixed by the role; memory sizing is derived from node specs. To adjust server behavior, use mysql_parameters.

Quick Reference

Parameter Level Default Description
mysql_cluster Cluster required Cluster name and identity
mysql_seq Instance required 1 for standalone; sequential 1..3 for HA
mysql_root_password Cluster DBUser.Root Local root password
mysql_monitor_password Cluster DBUser.Monitor Exporter identity password
mysql_cluster_password Cluster DBUser.Cluster AdminAPI/Router/backup identity password
mysql_databases Cluster [] Additive database declarations
mysql_users Cluster [] Additive user and grant declarations
mysql_parameters Cluster/Instance {} [mysqld] option overrides
mysql_backup_enabled Cluster true Daily full-backup timer
mysql_backup_repo Cluster see below Local backup path and retention
mysql_exporter_enabled Cluster true Exporter and monitoring target

The removal parameters are used by mysql-rm.yml:

Parameter Level Default Description
mysql_safeguard Global/Cluster/CLI true Refuse removal by default
mysql_rm_confirm CLI '' Must exactly match the instance or cluster name

Variables that appeared on earlier versions of this page — mysql_role, mysql_services, mysql_packages, mysql_data, mysql_port, mysql_replication_*, mysql_*_username — are no longer part of the interface. Do not use them.

Identity

mysql_cluster

Required cluster identity; must match an inventory group containing the member hosts (enforced at preflight). Starts with a letter, digit, or underscore; ._- allowed; up to 63 characters:

mysql_cluster: my-test

Used to derive instance names (my-test-1), the deterministic MGR group UUID, the backup directory (<repo>/my-test/), and the cls monitoring label.

mysql_seq

Required instance sequence. 1 for standalone; a consecutive 1, 2, 3 for HA. Doubles as server_id:

10.10.10.11: { mysql_seq: 1 }

mysql_seq=1 only marks the bootstrap coordinator. The runtime primary is elected, and reruns never move it back.

Credentials

mysql_root_password

Password for root@'localhost', usable only locally (socket or loopback). Single-line, and must not keep the CHANGE_ME prefix:

The default is DBUser.Root:

mysql_root_password: DBUser.Root

Set at first launch. Afterwards, if the live password differs from the declaration, the run fails explicitly rather than resetting it — rotate manually with ALTER USER, then update the inventory.

mysql_monitor_password

Password for dbuser_monitor@'127.0.0.1', used by mysqld_exporter: loopback-only, capped at 3 connections, read-only privileges:

The default is DBUser.Monitor:

mysql_monitor_password: DBUser.Monitor

mysql_cluster_password

Shared password for dbuser_cluster@'%' (TLS-required) and dbuser_backup@'localhost', covering AdminAPI cluster management, Router bootstrap, and XtraBackup:

The default is DBUser.Cluster:

mysql_cluster_password: DBUser.Cluster

On HA clusters this password is embedded in cluster metadata and Router keyrings, so it cannot be rotated by an ordinary rerun: a mismatch between the live value and the declaration is rejected at preflight. Standalone instances have no such binding — update the inventory and rerun.

Business Objects

mysql_databases

Additive database list accepting only name / encoding / collate:

mysql_databases:
  - { name: app }
  - { name: app2, encoding: utf8mb4, collate: utf8mb4_general_ci }

Creates and updates only; removing an entry never drops a database. Syntax and validation rules: Configuration.

mysql_users

Additive user list with fields name / host / password / connlimit / priv:

mysql_users:
  - name: app
    host: '%'
    password: DBUser.App
    connlimit: 20
    priv: { 'app.*': 'ALL PRIVILEGES' }

Grants are applied but never revoked implicitly; platform identities cannot be declared. Syntax and validation rules: Configuration.

mysql_parameters

A dictionary of [mysqld] overrides, rendered at the end of the managed config (last value wins):

mysql_parameters:
  max_connections: 500
  long_query_time: 2
  innodb_buffer_pool_size: 2G
  innodb_print_all_deadlocks: true    # true/false render as ON/OFF

Constraints and behavior:

  • Keys match [A-Za-z][A-Za-z0-9_.-]{0,63}; values are single-line scalars. The rendered config still passes mysqld --validate-config, so typos fail at deploy time without touching the running instance;
  • Reserved options are rejected (- and _ spellings are treated alike): user, pid_file, server_id, datadir, socket, port, bind_address, mysqlx_bind_address, report_host, gtid_mode, enforce_gtid_consistency, log_bin, relay_log, require_secure_transport, ssl_ca, ssl_cert, ssl_key, plugin_load, plugin_load_add, clone, plugin_clone, mysqlx, and plugin_mysqlx, plus all group_replication_*, plugin_group_replication*, plugin_mysqlx_bind_address, and ssl_* options;
  • Applying a change reruns mysql.yml, which orchestrates a rolling restart (secondaries first, primary last) — expect one brief write interruption when the primary restarts;
  • Commonly overridden defaults: sql_require_primary_key (default ON), long_query_time (default 1), binlog_expire_logs_seconds (default 7 days), and the memory settings.

A note on dynamic variables: the few replication settings AdminAPI manages via SET PERSIST are authoritative at runtime; on every converge the role pins group_replication_group_seeds back to the declared member list to prevent persisted drift.

Backup

mysql_backup_enabled

Whether the daily backup timer runs (mysql-backup.timer, daily with up to 30 minutes of randomized delay):

mysql_backup_enabled: true

Setting false stops the timer but keeps the backup script and config. Note that if backups were never enabled, the repository directory does not exist and a manual trigger exits immediately.

mysql_backup_repo

The local backup repository — local is the only supported method:

mysql_backup_repo:
  local:
    path: /data/backups/mysql     # absolute path; must not overlap the datadir
    retention: 7                  # keep the last N committed fulls (1-9999)

Layout and restore procedure: Administration.

Monitoring

mysql_exporter_enabled

Whether mysqld_exporter runs and the VictoriaMetrics target is registered:

mysql_exporter_enabled: true

Setting false stops the exporter and converges /infra/targets/mysql/<instance>.yml to an empty list (the file itself is only removed by mysql-rm.yml).

Removal Parameters

mysql_safeguard

Protected-removal safety switch, defaulting to true. You must explicitly set it to false when running mysql-rm.yml, or the role refuses to continue:

./mysql-rm.yml -l my-test -e mysql_safeguard=false -e mysql_rm_confirm=my-test

mysql_rm_confirm

Target-name confirmation string, empty by default. When removing a single member, it must exactly equal the instance name, such as my-test-3; when removing a complete cluster or standalone instance, it must exactly equal mysql_cluster. Both this value and mysql_safeguard=false are required.

Fixed Platform Conventions

The following are fixed or derived by the role — not inventory parameters — listed here for operators’ reference:

Item Value
Versions MySQL Server/Client/Shell/Router 8.4 LTS, Percona XtraBackup 8.4
Ports 3306 (classic), 33060 (X Protocol; loopback-only on standalone), 33061 (MGR), 6446/6447 (Router RW/RO), 9104 (exporter); INFRA carries the read-only reference constant mysql_exporter_port: 9104 for monitoring config, not as a public MYSQL parameter
Data directory /var/lib/mysql (binlogs under binlog/, 7-day expiry)
Config file EL: /etc/my.cnf.d/pigsty.cnf; Debian/Ubuntu: /etc/mysql/mysql.conf.d/pigsty.cnf
Service units MySQL: mysqld on EL, mysql on Debian/Ubuntu; Router: mysqlrouter; Exporter: mysqld_exporter
Secrets and scripts /etc/mysql/pigsty/ (root-owned: directory 0700, files 0600)
Logs Error log at /var/log/mysql/error.log, mirrored to Journald; slow log at /var/log/mysql/slow.log (1s threshold)
TLS Enforced (require_secure_transport=ON); CA at /etc/pki/ca.crt, leaf certs under /etc/mysql/pki/
Charset utf8mb4 / utf8mb4_0900_ai_ci
Memory Buffer pool = max(25% of node memory, 256MB); redo = clamp(50% of buffer pool, 128MB, 4GB)
Replication GTID enforced, sql_require_primary_key=ON, single-primary MGR, BEFORE_ON_PRIMARY_FAILOVER consistency
Datadir markers .pigsty-mysql-initialized (ownership check) and .pigsty-mysql-retired (retirement guard)

18.3 - Administration

Status checks, client access, configuration changes, failure handling, and the three recovery runbooks for MySQL clusters.

This page covers day-to-day operations for the MYSQL module. The governing principle: declare state in the inventory, converge with the playbook. Most anomalies — a dropped member, drifted AdminAPI state — heal with a single ./mysql.yml -l <cluster> rerun. Only three destructive scenarios (member replacement, physical restore, complete-outage recovery) require the manual runbooks below.

Quick Reference

Operation Command
Deploy / converge a cluster ./mysql.yml -l <cluster>
Preflight without changes ./mysql.yml -l <cluster> --check
Local root session mysql --defaults-extra-file=/etc/mysql/pigsty/root.cnf
Inspect MGR topology SELECT MEMBER_HOST,MEMBER_STATE,MEMBER_ROLE FROM performance_schema.replication_group_members;
AdminAPI status dba.getCluster().status() in mysqlsh
Trigger a backup systemctl start mysql-backup (in HA, only the primary runs it)
Retire a secondary ./mysql-rm.yml -l <IP> -e mysql_safeguard=false -e mysql_rm_confirm=<instance>
Retire a whole cluster ./mysql-rm.yml -l <cluster> -e mysql_safeguard=false -e mysql_rm_confirm=<cluster>

Status Checks

Run the commands in this page on a cluster member as root: the client configs and secrets under /etc/mysql/pigsty/ are readable by root only. Examples use EL unit names — on Debian/Ubuntu the MySQL service unit is mysql, not mysqld.

On any member, confirm services and topology:

systemctl status mysqld mysqlrouter mysqld_exporter mysql-backup.timer

mysql --defaults-extra-file=/etc/mysql/pigsty/root.cnf -e "
  SELECT MEMBER_HOST, MEMBER_STATE, MEMBER_ROLE, MEMBER_VERSION
  FROM performance_schema.replication_group_members ORDER BY MEMBER_HOST;"

A healthy three-node cluster shows three ONLINE rows with exactly one PRIMARY. For the AdminAPI view:

mysqlsh --js -e '
shell.options.useWizards=false;
var pw = os.loadTextFile("/etc/mysql/pigsty/cluster-password").replace(/[\r\n]+$/, "");
shell.connect({user:"dbuser_cluster", password:pw, host:"127.0.0.1", port:3306,
  "ssl-mode":"VERIFY_CA", "ssl-ca":"/etc/pki/ca.crt"});
print(dba.getCluster().status());'

For a fleet-level view, use the Grafana MySQL Overview dashboard or the derived metric mysql:cls:health (2 healthy / 1 degraded / 0 critical).

Client Access

HA clients connect through any member’s Router, which follows failovers automatically:

# Read-write endpoint (current primary)
mysql -h <any-member> -P 6446 -u app -pDBUser.App --ssl-mode=VERIFY_CA --ssl-ca=/etc/pki/ca.crt app

# Read-only endpoint (round-robin over secondaries)
mysql -h <any-member> -P 6447 -u app -pDBUser.App --ssl-mode=VERIFY_CA --ssl-ca=/etc/pki/ca.crt app

Guidance:

  • TLS is enforced server-side and plaintext connections are rejected; the client default PREFERRED mode negotiates encryption automatically, but prefer an explicit VERIFY_CA (JDBC: sslMode=VERIFY_CA) trusting the Pigsty CA;
  • There is no VIP/DNS layer. To avoid a single Router node becoming a point of failure, configure a multi-host DSN, e.g. jdbc:mysql://10.10.10.11:6446,10.10.10.12:6446,10.10.10.13:6446/app, or list all members in your application-side load balancer;
  • Standalone clusters have no Router — connect to 3306 directly;
  • A member that is partitioned or has lost quorum makes its local Router refuse both RW and RO connections (fail-safe): no stale reads through the Router.

Measured expectations: a graceful primary stop interrupts writes for ~3–4 seconds; a primary crash (kill -9) for ~20 seconds with default eviction settings; rolling restarts of secondaries are invisible to clients.

Manage Databases and Users

Edit mysql_databases / mysql_users in the inventory, then converge:

./mysql.yml -l my-test                      # full converge
./mysql.yml -l my-test -t mysql_provision   # business objects only (faster)

In HA, object changes execute on the current primary and replicate out. Declarations are additive: nothing is dropped or revoked implicitly — do those by hand, then update the inventory to match.

Change Cluster Parameters

All tuning goes through mysql_parameters:

mysql_parameters:
  max_connections: 500
  long_query_time: 2
./mysql.yml -l my-test --check    # preview the pending change
./mysql.yml -l my-test            # apply with an orchestrated rolling restart

Rolling-restart semantics (verified by testing):

  1. The rendered config passes mysqld --validate-config first — a bad option fails the run without touching the service;
  2. Cluster health is checked up front: a degraded cluster (fewer than 3 ONLINE) refuses a rolling restart — repair first, then change;
  3. Secondaries restart one at a time, each waiting to return ONLINE; the primary restarts last;
  4. The primary restart triggers one automatic failover with a write pause of a few seconds — schedule a change window if that matters.

Standalone instances restart in place.

Switchover

The module does not orchestrate planned switchovers; use AdminAPI when you need one:

mysqlsh --js -e '
shell.options.useWizards=false;
var pw = os.loadTextFile("/etc/mysql/pigsty/cluster-password").replace(/[\r\n]+$/, "");
shell.connect({user:"dbuser_cluster", password:pw, host:"127.0.0.1", port:3306,
  "ssl-mode":"VERIFY_CA", "ssl-ca":"/etc/pki/ca.crt"});
dba.getCluster().setPrimaryInstance("10.10.10.12:3306");   // the new primary
'

Routers follow automatically. Rerun ./mysql.yml -l <cluster> afterwards to confirm convergence — primary placement is runtime state, not declared state, so the playbook will not move it back.

Member Failures and Self-Healing

No action is needed during a failure: after a primary crash, MGR elects a new primary within ~20 seconds and Routers re-route; the crashed member is restarted by systemd and rejoins on its own. Intervene only in these cases:

Symptom Action
A member stays OFFLINE (process up, GR stopped) Rerun ./mysql.yml -l <cluster> — it rejoins the member
A member repeatedly fails to join, logging peers not configured Same: the converge pins group_replication_group_seeds back to the declared list
A member has not returned after a network partition heals Wait ~1 minute for auto-rejoin; rerun the playbook if it still has not rejoined
All members OFFLINE Complete outage — see Recover from a Complete Outage
Hardware is unrecoverable See Replace a Failed Member

Matching alerts: MySQLClusterMemberOffline (WARN), MySQLClusterNoPrimary / MySQLClusterQuorumLost (CRIT).

Replace a Failed Member

The replacement contract: the new machine reuses the failed member’s service address (the inventory does not change). Three steps, assuming my-test-3 (10.10.10.13) died:

# 1. Remove the failed member. If the machine is still reachable, use the retirement playbook:
./mysql-rm.yml -l 10.10.10.13 -e mysql_safeguard=false -e mysql_rm_confirm=my-test-3

# 1b. If the machine is truly dead (unreachable over SSH), the playbook cannot run on it;
#     force-remove it from any healthy member via AdminAPI instead:
mysqlsh --js -e '
shell.options.useWizards=false;
var pw = os.loadTextFile("/etc/mysql/pigsty/cluster-password").replace(/[\r\n]+$/, "");
shell.connect({user:"dbuser_cluster", password:pw, host:"127.0.0.1", port:3306,
  "ssl-mode":"VERIFY_CA", "ssl-ca":"/etc/pki/ca.crt"});
dba.getCluster("my-test").removeInstance("10.10.10.13:3306", {force: true});'

# 2. Provision a fresh machine at the same address (reinstall the OS), then manage it
./node.yml -l 10.10.10.13

# 3. Reconverge the complete cluster: the new member is cloned and joined automatically
./mysql.yml -l my-test --check
./mysql.yml -l my-test

Notes:

  • Step 1’s real job is evicting the address from cluster metadata — only an address absent from metadata takes the fresh-clone path. The retirement playbook requires a reachable target (an ONLINE SECONDARY or an already-detached member); for a dead machine, use the force removal in 1b instead;
  • The replacement must be a truly fresh machine (empty datadir, no leftover Router keyring) — an OS reinstall guarantees that. Half-clean machines are rejected by preflight or the Router bootstrap;
  • Clone copies the full dataset; duration scales with data size. The cluster stays available throughout (one primary, one secondary online);
  • Changing a member’s address during replacement is not supported, nor is running two nodes long-term.

Retire and Resurrect a Cluster

Retire a whole cluster (stop services, deregister monitoring, keep all data):

./mysql-rm.yml -l my-test -e mysql_safeguard=false -e mysql_rm_confirm=my-test

Retirement writes /var/lib/mysql/.pigsty-mysql-retired on every member, which blocks ordinary mysql.yml reruns so a retired instance cannot be revived by accident. To deliberately resurrect:

ansible my-test -b -a 'rm -f /var/lib/mysql/.pigsty-mysql-retired'
./mysql.yml -l my-test

Two commands suffice for a standalone. HA clusters need one more step: the rerun brings services up, but all three members return with Group Replication OFFLINE (split-brain protection — nobody self-bootstraps) and the playbook exits with the complete-outage error. Continue with steps 3–4 of Recover from a Complete Outage to rebuild quorum.

Actual destruction (removing datadirs, backups, packages) is never done by playbooks — that is a manual decision made after verifying backups.

Manage Backups 

systemctl list-timers mysql-backup.timer          # next scheduled run
systemctl start mysql-backup                      # run now (in HA the primary executes, secondaries skip)
journalctl -u mysql-backup --since today          # backup logs

Backup layout, on the current primary’s local disk:

/data/backups/mysql/<cluster>/
├── 20260729T053900Z/          # one prepared full backup (directly restorable)
│   ├── backup.ok              # commit marker: present only for fully successful backups
│   ├── backup.log             # XtraBackup output
│   └── ...                    # InnoDB data files
├── ...                        # last N kept per retention
└── latest -> 20260729T053900Z # atomic pointer to the newest backup

Check backup freshness — on all members for HA, since backups follow the primary:

ansible my-test -b -a 'ls -l /data/backups/mysql/my-test/latest'
Backup alerting gap

This version exports no backup-freshness metric and ships no backup alerts: a failed backup is only visible in the mysql-backup logs (queryable in VictoriaLogs and on the Instance dashboard’s Router / Backup Logs panel). For important environments, add external log checks and rehearse the restore runbook below periodically.

Restore from Physical Backup

This runbook restores a standalone instance to its most recent backup. It is destructive: writes after the backup are lost — check the latest timestamp first. Rebuilding an HA cluster works the same way: restore one node as the primary, then let the others rejoin via clone.

# 0. Verify the backup is committed: backup.ok must exist
BK=/data/backups/mysql/my-meta/latest
sudo test -f $BK/backup.ok && sudo cat $BK/backup.ok

# 1. Stop the server; keep the wreckage for forensics until verified
sudo systemctl stop mysqld
sudo mv /var/lib/mysql /var/lib/mysql.destroyed

# 2. Copy the backup back (already prepared — no --prepare step needed)
sudo mkdir -p /var/lib/mysql && sudo chown mysql:mysql /var/lib/mysql && sudo chmod 750 /var/lib/mysql
sudo xtrabackup --copy-back --target-dir=$BK
sudo rm -f /var/lib/mysql/backup.ok /var/lib/mysql/backup.log   # bookkeeping files carried over by copy-back

# 3. Recreate runtime directories the backup does not contain
sudo mkdir -p /var/lib/mysql/binlog /var/lib/mysql/tmp
sudo chown -R mysql:mysql /var/lib/mysql
sudo chmod 750 /var/lib/mysql/binlog /var/lib/mysql/tmp

# 4. Recreate the Pigsty ownership marker (set cluster/instance/topology for your instance)
echo '{"version": 1, "cluster": "my-meta", "instance": "my-meta-1", "topology": "standalone"}' | \
  sudo tee /var/lib/mysql/.pigsty-mysql-initialized > /dev/null
sudo chown mysql:mysql /var/lib/mysql/.pigsty-mysql-initialized
sudo chmod 600 /var/lib/mysql/.pigsty-mysql-initialized

# 5. Restore SELinux context on EL, then start
sudo restorecon -RF /var/lib/mysql 2>/dev/null || true
sudo systemctl start mysqld

# 6. Verify data and GTID position; confirm the playbook still converges
sudo mysql --defaults-extra-file=/etc/mysql/pigsty/root.cnf -e 'SELECT @@gtid_executed; SHOW DATABASES;'
./mysql.yml -l my-meta        # expect a green run (changed=0 or routine items only)

The step-4 marker is Pigsty’s proof of datadir ownership: without it (or with mismatched content), mysql.yml refuses to manage the restored datadir. For HA members, use "topology": "innodb_cluster" and the member’s own instance name.

Recover from a Complete Outage

When all three members are OFFLINE (power loss, cascading failure), MGR deliberately does not rebuild quorum on its own — that is split-brain protection — and mysql.yml refuses with instructions. The procedure:

# 1. Confirm mysqld is running everywhere (systemd usually restarted it) and GR is OFFLINE
ansible my-test -b -a "mysql --defaults-extra-file=/etc/mysql/pigsty/root.cnf -NBe \
  \"SELECT COALESCE((SELECT MEMBER_STATE FROM performance_schema.replication_group_members \
  WHERE MEMBER_ID=@@server_uuid),'OFFLINE')\""

# 2. Find the most advanced member: compare GTID sets, pick the superset (ties: any)
ansible my-test -b -a "mysql --defaults-extra-file=/etc/mysql/pigsty/root.cnf -NBe 'SELECT @@gtid_executed'"

# 3. Reboot the cluster from that member via AdminAPI (replace 10.10.10.12 accordingly)
mysqlsh --js -e '
shell.options.useWizards=false;
var pw = os.loadTextFile("/etc/mysql/pigsty/cluster-password").replace(/[\r\n]+$/, "");
shell.connect({user:"dbuser_cluster", password:pw, host:"10.10.10.12", port:3306,
  "ssl-mode":"VERIFY_CA", "ssl-ca":"/etc/pki/ca.crt"});
var c = dba.rebootClusterFromCompleteOutage("my-test");
print(c.status().defaultReplicaSet.status);'

# 4. Rerun the playbook: members still OFFLINE are rejoined automatically
./mysql.yml -l my-test

Notes:

  • Step 3 usually brings every reachable member back at once; stragglers are rejoined by step 4 — no per-node manual work;
  • If only a minority of machines survived, complete the reboot first to restore writes, then follow Replace a Failed Member for the rest;
  • No writes are possible until step 3 completes (super_read_only); members usually remain readable, though a member that was expelled earlier may sit in offline_mode and refuse ordinary connections;
  • The default sql_require_primary_key=ON prevents the PK-less tables that would otherwise block this procedure.

Platform Password Boundaries

Operational boundaries for the three platform passwords (details: Parameters):

  • mysql_monitor_password: update the inventory and rerun — rotates cleanly;
  • mysql_root_password: implicit resets are refused. Rotate manually — ALTER USER 'root'@'localhost' IDENTIFIED BY '...'; on the primary — then update the inventory and rerun to refresh credential files;
  • mysql_cluster_password: on HA clusters, bound to cluster metadata and Router keyrings — ordinary reruns reject rotation, and no automated HA procedure ships yet (standalone instances rotate normally via inventory + rerun). If HA rotation is unavoidable, do it manually via AdminAPI, sync every member’s credential files, then update the inventory.

18.4 - Playbook

Deploy, converge, tune, and retire MySQL clusters with mysql.yml and mysql-rm.yml.

The MYSQL module ships two playbooks: mysql.yml deploys and converges, while mysql-rm.yml performs protected member retirement and cluster teardown. Re-running the former converges toward declared state; the latter is a separate lifecycle operation whose target, backups, and exact confirmation value must be checked again before every real run.

mysql.yml

Runs the full check → install → bootstrap → access → provision → backup → monitor convergence on the selected clusters:

./mysql.yml -l my-test --check     # preflight: validate declarations against the live state
./mysql.yml -l my-test             # converge one cluster (must select all members)
./mysql.yml                        # converge every MySQL cluster in the inventory

Usage rules:

  • HA clusters must be selected whole: a -l that covers only some members is rejected at preflight (guarding against topology divergence). Multiple complete clusters, or no -l at all, are fine;
  • Idempotent: a converged cluster reruns as changed=0 in seconds. The run immediately after an AdminAPI membership operation (rejoin/clone) may report one convergence changed — the replication seed list being pinned back to the declaration — which is expected;
  • Check mode: on fresh nodes it can only preview up to package installation (later steps need the installed platform); on deployed clusters it previews fully;
  • Initial three-node deployment takes ~2 minutes: certificates → config and initialization → AdminAPI cluster creation → cloning two members → Router bootstrap on each → business objects → backup and monitoring.

Stages and Tags 

mysql
├── mysql_check       # identity, platform, credentials, parameters, datadir ownership (always)
├── mysql_install     # install the fixed MySQL 8.4 package set
├── mysql_bootstrap
│   ├── mysql_cert    # issue and install node TLS leaf certificates
│   ├── mysql_config  # render config (incl. mysql_parameters); initialize empty datadirs only
│   ├── mysql_launch  # start / rolling-restart mysqld; converge root and AdminAPI identities
│   └── mysql_cluster # create or reconcile the InnoDB Cluster (rejoin / clone)
├── mysql_access
│   └── mysql_router  # bootstrap and verify Router on HA members
├── mysql_provision   # converge platform identities and declared databases/users
├── mysql_backup      # install the backup script and daily timer
├── mysql_monitor     # configure the exporter and register monitoring targets
└── mysql_done        # print the instance summary

Common tag-scoped runs:

./mysql.yml -l my-test -t mysql_provision    # business databases and users only
./mysql.yml -l my-test -t mysql_backup       # backup config and timer only
./mysql.yml -l my-test -t mysql_monitor      # exporter and target registration only

For parameter and configuration changes, run the full playbook — they involve the rolling-restart orchestration described below.

Config Changes and Rolling Restarts

The mysql_launch stage orchestrates restarts whenever the config file, certificates, or systemd units change:

  1. Health precondition: an HA cluster must have all 3 members ONLINE before a rolling restart; degraded clusters are refused (repair first, then change);
  2. Secondaries first: ordered by live runtime role (not mysql_seq), each secondary restarts and must return ONLINE before the next;
  3. Primary last: the final restart triggers one automatic failover with a seconds-long write pause.

Standalone instances restart in place. mysqld --validate-config at render time guarantees invalid options fail before any service is touched.

Guardrails

mysql.yml refuses to act in the following situations, with errors that state the reason and the way forward:

Refused scenario Rationale
Partial member selection HA operations must cover the whole cluster
Invalid topology Member count must be 1 or 3, with consecutive mysql_seq
Unsupported platform Arch/OS outside the support matrix (e.g. Ubuntu ARM64)
Placeholder passwords CHANGE_ME credentials left in place
Foreign datadir Datadir lacks the Pigsty marker, or the marker names another cluster/instance/topology
Retirement marker present Instance was retired by mysql-rm.yml; resurrection must be explicit
Implicit password changes mysql_cluster_password or live root password differs from the declaration
Illegal parameter overrides Reserved keys, malformed names, or multi-line values in mysql_parameters
Degraded-cluster restart Config-driven restarts require all members ONLINE
Non-fresh clone target Replacement members must be brand-new machines with empty datadirs
Complete outage Quorum is never rebuilt automatically; the error prints the manual recovery steps

These guardrails significantly reduce operational risk, but they are not a guarantee that data can never be lost. Every bypass—such as deleting markers or wiping data directories—must be a deliberate decision made after verifying backups and the exact scope.

mysql-rm.yml

The retirement playbook accepts three scopes, each requiring double confirmation (mysql_safeguard=false plus mysql_rm_confirm exactly matching the target):

# Retire one member of an HA cluster (target must be reachable: an ONLINE SECONDARY or an already-detached member)
./mysql-rm.yml -l 10.10.10.13 --check -e mysql_safeguard=false -e mysql_rm_confirm=my-test-3
./mysql-rm.yml -l 10.10.10.13         -e mysql_safeguard=false -e mysql_rm_confirm=my-test-3

# Retire a whole HA cluster
./mysql-rm.yml -l my-test -e mysql_safeguard=false -e mysql_rm_confirm=my-test

# Retire a standalone instance
./mysql-rm.yml -l my-meta -e mysql_safeguard=false -e mysql_rm_confirm=my-meta

What it does — and does not do:

  • Single-member retirement removes an ONLINE SECONDARY via AdminAPI (force: false), or verifies an already-detached member, then stops local services. The removal script runs on the target itself, so the target must be reachable — for a dead machine, use manual force removal instead (Replace a Failed Member). Retiring the primary directly is refused (switch it away first with setPrimaryInstance); so is retiring 2 of 3 members at once;
  • Whole-cluster retirement stops the Router and backup timer, stops secondaries before the primary, and deregisters exporters and monitoring targets;
  • Every datadir gets the retirement marker .pigsty-mysql-retired, blocking ordinary mysql.yml reruns;
  • All data is preserved: datadirs, backups, config, certificates, packages, metadata, and Router identities stay untouched. Actual destruction is a separate, manual, backup-verified decision.

--check previews the complete plan without touching anything.

Playbook Boundaries

The following are out of playbook scope by design; manual procedures live in Administration:

  • Planned switchover (setPrimaryInstance);
  • Force removal of dead, unreachable members (removeInstance with force: true);
  • Quorum rebuild after a complete outage (rebootClusterFromCompleteOutage);
  • Physical restore (XtraBackup copy-back runbook);
  • Destruction: removing datadirs, backups, or retirement markers;
  • Topology changes (1→3, 3→5) and member re-addressing.

18.5 - Monitoring

MySQL metric collection, Grafana dashboards, alert rules, and log queries.

The MYSQL module plugs into Pigsty’s observability stack: metrics flow through mysqld_exporter into VictoriaMetrics, error logs flow through Journald/Vector into VictoriaLogs, Grafana ships 5 dashboards, and vmalert loads 68 recording rules plus 27 alert rules.

Collection Architecture

Each MySQL node runs one mysqld_exporter (port 9104) using the least-privilege monitor account (dbuser_monitor@'127.0.0.1'). Deployment writes a file-based service-discovery target on the Infra node:

/infra/targets/mysql/<instance>.yml     # e.g. my-test-1.yml

The VictoriaMetrics mysql scrape job consumes this directory. mysql_exporter_enabled: false converges the target to an empty list; only mysql-rm.yml deletes target files.

Enabled collectors include global status/variables, binlog size, InnoDB metrics, the process list, performance-schema statement digests (top 50), table/index I/O waits, and MGR membership plus replication statistics.

Label Model

All MySQL metrics carry a consistent label set:

Label Meaning Example
job Scrape job mysql
cls Cluster name my-test
ins Instance name my-test-1
ip Member address 10.10.10.11
topology Topology type innodb_cluster / standalone

Derived rules are named mysql:ins:* (instance level) and mysql:cls:* (cluster level); the full dictionary is in Metrics.

Grafana Dashboards

Dashboard Purpose
MySQL Overview Fleet view: cluster inventory, health, QPS/TPS, active alerts, instance list
MySQL Cluster One cluster: member states, workload, node resources, cluster logs
MySQL Instance One instance: connections, statements, InnoDB, temp tables, locks, logs
MySQL Group Replication MGR deep dive: roles, certification/applier queues, flow control, read-only safety, GR logs
MySQL Alert Alert summary and key platform logs

Cluster health at a glance: mysql:cls:health is 2 (healthy) / 1 (degraded but writable) / 0 (critical or unwritable) — the Overview’s Healthy Clusters stat and Cluster Health timeline are built on it.

The Group Replication dashboard is only meaningful for innodb_cluster topologies; selecting a standalone cluster legitimately shows No data on MGR panels.

Alert Rules

The 27 alert rules are tiered by severity (CRIT / WARN / INFO). The ones to page on:

Availability and Cluster State

Alert Severity Condition
MySQLInstanceDown CRIT Connection probe failing for 1m
MySQLClusterNoPrimary CRIT No ONLINE primary for 1m
MySQLClusterQuorumLost CRIT ONLINE members below majority for 1m
MySQLClusterMultiplePrimary CRIT More than one primary for 30s (split-brain signal)
MySQLSecondaryWritable CRIT A secondary writable for 2m (divergence risk)
MySQLClusterMemberOffline WARN A declared member out of the group for 5m
MySQLPrimaryReadOnly WARN Primary read-only for 5m
MySQLExporterDown WARN Scrape failing for 2m

Capacity and Performance

Connection pressure (MySQLConnectionsHigh WARN at 80% / MySQLConnectionsCritical CRIT at 95%), replication queues (MySQLGRQueueHigh WARN / MySQLGRQueueCritical CRIT), flow control (MySQLGRFlowControlHigh), InnoDB signals (MySQLBufferPoolWaits, MySQLInnoDBLogWaits, MySQLRedoCapacityHigh, MySQLDeadlocksHigh, MySQLHistoryListLarge), and INFO-level hints for slow queries, disk temp tables, full joins, buffer-pool hit ratio, and recent restarts.

Observed behavior from testing: a primary crash-failover (~20s) only produces pending alerts, no false pages; a genuine complete outage drives ClusterNoPrimary and QuorumLost to firing within 2 minutes.

Log Queries

MySQL error logs are written twice: to /var/log/mysql/error.log and via syslog → Journald → Vector → VictoriaLogs. Entries carry app=mysqld-<instance>, so the dashboard log panels work out of the box, and LogsQL queries are straightforward:

# Recent errors from one instance
curl -s http://<infra>:9428/select/logsql/query \
  -d 'query=app:mysqld-my-test-1 level:err _time:1h'

# All MySQL-related logs for a cluster, including backup runs
curl -s http://<infra>:9428/select/logsql/query \
  -d 'query=job:syslog cls:my-test (app:~"mysqld-" OR unit:mysql-backup) _time:1h | limit 100'

Note the log cls label is the node cluster name (node_cluster) — keep node_cluster aligned with mysql_cluster, as the configuration examples do, so metric and log labels agree.

Known boundaries:

  • The slow query log (slow.log, 1s threshold) stays on local disk and is not shipped to VictoriaLogs — inspect it on the instance, or use the statement-digest metrics (mysql:ins:statement_latency and friends);
  • Router runtime logs live in /var/log/mysqlrouter/ and are also local-only.

Verify the Pipeline

Self-check every hop after deployment:

# Exporter itself
curl -s http://<member>:9104/metrics | grep -E '^mysql_up '

# VictoriaMetrics scrape and recording rules
curl -s 'http://<infra>:8428/api/v1/query?query=mysql_up'
curl -s 'http://<infra>:8428/api/v1/query?query=mysql:cls:health'

# vmalert rule groups (expect mysql-rules and mysql-alerts)
curl -s 'http://<infra>:8880/api/v1/rules' | grep -o '"name":"mysql-[a-z]*"'

# Log ingestion
curl -s 'http://<infra>:9428/select/logsql/query' -d 'query=app:~"mysqld-" _time:1h | stats by (app) count()'

18.6 - Metrics

Label model, derived-metric dictionary, and raw metric families of the MYSQL module.

MYSQL metrics come from mysqld_exporter (raw metrics, mysql_ prefix) and vmalert recording rules (mysql:ins:* / mysql:cls:*). Dashboards and alerts are built on the derived metrics; this page is their dictionary.

Common Labels

Every metric carries job=mysql and the identity labels cls / ins / ip / topology (standalone or innodb_cluster). Instance-level derived metrics keep all identity labels; cluster-level metrics aggregate to cls + topology.

Availability

Metric Meaning
mysql:ins:exporter_up Scrape success (transport health)
mysql:ins:up MySQL connection probe success (database health)
mysql:ins:uptime Instance uptime in seconds
mysql:cls:instances Declared instance count
mysql:cls:up Online instance count
mysql:cls:health Cluster health: 2 healthy / 1 degraded-writable / 0 critical

For HA clusters, mysql:cls:health combines quorum, single-primary, and full-membership status; for standalones it is 2 × mysql:cls:up.

Workload

Metric Meaning
mysql:ins:qps Questions per second
mysql:ins:tps Transactions per second (commit + rollback)
mysql:ins:read_qps / mysql:ins:write_qps Read-class / write-class command rate
mysql:ins:row_ops InnoDB row operations (read/insert/update/delete dimensions)
mysql:ins:statement_rate Performance-schema statement rate
mysql:ins:statement_latency Mean statement latency (seconds)
mysql:ins:rows_examined_per_query Average rows examined per query
mysql:ins:statement_errors Statement error rate
mysql:ins:slow_queries / mysql:ins:slow_query_ratio Slow query rate and ratio
mysql:ins:no_index_queries Rate of queries using no index

Connections and Sessions

Metric Meaning
mysql:ins:connections Current connections (Threads_connected)
mysql:ins:connection_usage Connections / max_connections
mysql:ins:connection_rate New connection rate
mysql:ins:threads_running / mysql:ins:threads_cached Active / cached threads
mysql:ins:aborted_connects / mysql:ins:aborted_clients Failed handshakes / abnormal disconnects
mysql:ins:connection_errors Total connection error rate
mysql:ins:rx_bytes / mysql:ins:tx_bytes Network receive / transmit rate

Temp Tables, Scans, and Caches

Metric Meaning
mysql:ins:tmp_tables / mysql:ins:tmp_disk_tables In-memory / on-disk temp table creation rate
mysql:ins:tmp_disk_ratio Share of temp tables spilling to disk
mysql:ins:full_joins / mysql:ins:full_scans Index-less join / full scan rate
mysql:ins:sort_merge_passes Sort merge passes (undersized sort buffer signal)
mysql:ins:table_open_cache_hit_ratio Table open cache hit ratio
mysql:ins:open_files_usage Open file usage ratio

InnoDB

Metric Meaning
mysql:ins:buffer_pool_hit_ratio Buffer pool hit ratio
mysql:ins:buffer_pool_usage / mysql:ins:buffer_pool_dirty_ratio Buffer pool usage / dirty page ratio
mysql:ins:buffer_pool_waits Free-page wait rate (memory pressure signal)
mysql:ins:data_read_bytes / mysql:ins:data_write_bytes Data file read / write throughput
mysql:ins:data_reads / mysql:ins:data_writes / mysql:ins:data_fsyncs Data file I/O and fsync rates
mysql:ins:redo_bytes Redo write throughput
mysql:ins:redo_utilization Redo capacity utilization (checkpoint lag)
mysql:ins:log_waits Redo buffer wait rate
mysql:ins:row_lock_waits / mysql:ins:row_lock_time Row lock wait rate / time
mysql:ins:deadlocks Deadlock rate
mysql:ins:history_list_length Purge lag (history list length)
mysql:ins:binlog_bytes Total on-disk binlog size (bytes)

Group Replication

Instance-level membership flags (value 1 or absent — the series does not exist when the condition is false, which is why alerts use unless):

Metric Meaning
mysql:ins:gr_member Instance is in any MGR member state
mysql:ins:gr_online Instance is ONLINE
mysql:ins:gr_primary / mysql:ins:gr_secondary Instance is the ONLINE primary / a secondary

Cluster-level quorum and topology:

Metric Meaning
mysql:cls:gr_online_members ONLINE member count
mysql:cls:gr_primary_members ONLINE primary count
mysql:cls:gr_quorum Majority held (0/1)
mysql:cls:gr_single_primary Exactly one primary (0/1)

Replication pipeline (certification and apply):

Metric Meaning
mysql:ins:gr_certifier_queue / mysql:ins:gr_applier_queue Transactions backed up in certification / applier queues
mysql:ins:gr_certifier_queue_ratio / mysql:ins:gr_applier_queue_ratio Queue depth relative to flow-control thresholds
mysql:ins:gr_checked_rate / mysql:ins:gr_applied_rate Certification / apply throughput
mysql:ins:gr_conflict_rate Certification conflict rate (should be 0 under single-primary)

Raw Metric Families

For anything not covered by derived metrics, query the exporter’s raw families:

Prefix Content
mysql_up / up Database probe / scrape status
mysql_global_status_* Full SHOW GLOBAL STATUS counters
mysql_global_variables_* Key system variables (e.g. max_connections)
mysql_perf_schema_events_statements_* Statement digests (top 50 by digest)
mysql_perf_schema_table_io_waits_* / ..._index_io_waits_* Table / index I/O waits
mysql_perf_schema_replication_group_member_info MGR membership (member_state / member_role dimensions)
mysql_perf_schema_transactions_* / mysql_perf_schema_conflicts_detected_total MGR certification, applier queues, and conflicts
mysql_binlog_* Binlog file count and size
mysql_info_schema_processlist_* Session distribution by state

Browse the complete list with the mysql_ prefix in VictoriaMetrics vmui (/select/vmui).

18.7 - FAQ

Frequently asked questions and troubleshooting for the Pigsty MySQL pilot module.

How mature is the MYSQL module?

It is a pilot module aiming for a simple, inexpensive, good-enough MySQL cluster. The four core capabilities — deployment and convergence, HA failover, daily backups, monitoring and alerting — have been tested systematically, including fault injection and complete-outage drills. Destructive recovery flows are deliberately manual, with runbooks provided. It does not aim for PGSQL-module completeness: no PITR, no VIP/DNS access layer, no automatic scaling. Validate and rehearse recovery against your own requirements before serious production use.

Why is MySQL fixed at 8.4? Can I pick a version?

MYSQL is a “fixed platform”, not a general installer: server, client, Shell, Router, and XtraBackup are all pinned to the 8.4 LTS line, which keeps component compatibility and behavior predictable and eliminates a version-matrix testing burden. That is the core trade-off keeping this pilot simple. If you need other versions or deep customization, this module is not the right tool.

Why only 1 or 3 nodes? How do I scale?

Topology is fixed: standalone or three-node single-primary InnoDB Cluster. Preflight rejects other member counts, and the datadir identity marker blocks in-place 1→3 conversion. Dynamic membership would drag in quorum management, Router re-bootstrap, and convergence-path complexity beyond what a pilot should carry.

Scaling paths:

  • Vertical: move to bigger machines one at a time via same-address replacement (rolling hardware refresh);
  • Standalone → HA: build a new three-node cluster and migrate logically (mysqldump or mysqlsh util.dumpInstance);
  • Read scaling: send read-only traffic to 6447, shared by the two secondaries.

Why does CREATE TABLE fail with ERROR 3750 (primary key required)?

The platform defaults to sql_require_primary_key=ON. PK-less tables are read-only under Group Replication and, worse, block AdminAPI cluster rebuilds during disaster recovery — better to fail at CREATE than to explode mid-recovery. Give every table a primary key; if you are onboarding a legacy system that truly cannot change, override:

mysql_parameters: { sql_require_primary_key: false }

Standalone instances keep the same default so they stay HA-portable.

Why is Ubuntu/Debian ARM64 rejected?

Oracle’s APT repository ships no arm64 packages for MySQL 8.4 — nothing Pigsty can work around. On ARM (including VMs on Apple Silicon), use EL 9/10 (Rocky/Alma): Oracle’s YUM repository has full aarch64 support.

Which port should clients use? Is TLS mandatory?

For HA, connect to 6446 (read-write) or 6447 (read-only) on any member; the Router follows failovers. Standalone connects to 3306 directly. TLS is mandatory: the server sets require_secure_transport=ON and rejects plaintext connections (ERROR 3159). The client default PREFERRED mode negotiates TLS automatically (only an explicit DISABLED is refused); prefer VERIFY_CA trusting /etc/pki/ca.crt.

Routers are per-node with no shared VIP — use a multi-host DSN listing all members’ 6446 to survive the loss of any single node.

The primary moved. Will it move back automatically?

No — and it does not need to. mysql_seq=1 is only the bootstrap order; the runtime primary is wherever MGR elected it, which is a fully legitimate state after failovers or rolling restarts. Reruns never relocate the primary. To place it deliberately, use setPrimaryInstance.

A member went down. What do I do?

Usually nothing: systemd restarts a crashed mysqld and the member rejoins on its own (a primary crash completes failover and self-healing in ~20 seconds). If a member stays OFFLINE — after a healed network partition, or a STOP GROUP_REPLICATION — rerun ./mysql.yml -l <cluster> and it will be rejoined. If that fails, read the error: it states the cause and the next step.

All three nodes are down. How do I recover?

This is the one availability scenario requiring manual action (deliberate split-brain protection): run dba.rebootClusterFromCompleteOutage() on the most advanced member, then rerun the playbook to converge the rest. Full steps: Recover from a Complete Outage. The mysql.yml failure message in this state prints exactly these instructions.

Where are backups stored? Can I restore to a point in time?

Backups are daily full physical backups stored under /data/backups/mysql/<cluster>/ on the current primary (after a failover, new backups follow the new primary — check every member when looking for the latest). There is no incremental chain and no binlog archiving, hence no PITR: a standalone’s recovery point is its most recent backup (worst case, one day of writes); an HA cluster’s data safety rests primarily on its three synchronized replicas, with backups as the last line and for full rebuilds. Restore procedure: Restore from Physical Backup. For off-site protection, sync the backup directory yourself.

Will I be alerted if backups fail?

Not yet — a known gap. There is no backup-freshness metric or alert; backup logs are shipped to VictoriaLogs (unit:mysql-backup) and visible on the Instance dashboard’s Router / Backup Logs panel. For important environments, add external log checks and rehearse restores periodically.

Why are some mysql_parameters keys rejected?

Identity (server_id, datadir, ports, …), replication (gtid_mode, log_bin, group_replication_*), and the TLS family are part of the platform’s guarantees. Overriding them would corrupt cluster identity or security floors, so preflight rejects them (- and _ spellings alike). Everything else is allowed and still validated by mysqld --validate-config. Full reserved list: Parameters.

Does changing parameters cause downtime?

One controlled blip: parameter changes trigger an orchestrated rolling restart — secondaries first (invisible to clients), primary last with one automatic failover (measured at ~3–4 seconds of write pause). Degraded clusters refuse rolling restarts so a change can never pile onto an outage. Schedule a window if your workload is failover-sensitive.

How do I change root or platform passwords?

  • mysql_monitor_password: update the inventory and rerun — done (the exporter config is refreshed along the way);
  • mysql_root_password: implicit resets are refused (protection against silent misconfiguration). Run ALTER USER 'root'@'localhost' ... manually, then update the inventory and rerun;
  • mysql_cluster_password: on HA clusters, bound to cluster metadata and Router keyrings — ordinary reruns reject rotation and no automated HA flow ships yet (standalones rotate normally). If unavoidable, rotate manually via AdminAPI and sync each member’s credential files before updating the inventory.

How do I resurrect a retired cluster? What if the marker is deleted by mistake?

mysql-rm.yml keeps all data and writes a retirement marker. To resurrect: delete /var/lib/mysql/.pigsty-mysql-retired on each member and rerun mysql.yml (details) — that suffices for a standalone; an HA cluster additionally needs its quorum rebuilt per Recover from a Complete Outage. The marker only prevents accidental revival; datadir ownership is checked independently through .pigsty-mysql-initialized, so removing the retirement marker can never hand the data to a different cluster.

Why doesn’t conf/mysql.yml match this module?

That template is OpenHalo — a MySQL wire-compatible solution on a PostgreSQL kernel (pg_mode: mysql) — unrelated to this module. The native MySQL template is conf/demo/mysql.yml. Rule of thumb: need real MySQL ecosystem compatibility → this module; running MySQL-protocol apps on PostgreSQL infrastructure → consider OpenHalo.

The playbook fails with no ONLINE member holds the cluster?

That is the complete-outage verdict: no ONLINE member is carrying the cluster (or the members that are online cannot be reached). Follow the printed instructions — see Recover from a Complete Outage. If members are online and you still see this, check connectivity from the seq-1 (coordinator) member — where the reconciliation script runs — to every member’s port 3306, and the TLS trust chain (/etc/pki/ca.crt in place).

Monitoring shows no data / blank dashboards?

Walk the pipeline: curl http://<member>:9104/metrics | grep mysql_up (exporter) → confirm instance files under /infra/targets/mysql/ on the Infra node → query up{job="mysql"} in VictoriaMetrics. Note the Group Replication dashboard legitimately shows No data for standalone clusters. Full self-check commands: Monitoring.