Task on Pigsty v5.0

Managing PostgreSQL Clusters

Mon, 01 Jan 0001 00:00:00 +0000

Quick Reference

Action	Command	Description
Create Cluster	`bin/pgsql-add <cls>`	Create a new PostgreSQL cluster
Expand Cluster	`bin/pgsql-add <cls> <ip...>`	Add replica to existing cluster
Shrink Cluster	`bin/pgsql-rm <cls> <ip...>`	Remove instance from cluster
Remove Cluster	`bin/pgsql-rm <cls>`	Destroy entire PostgreSQL cluster
Reload Service	`bin/pgsql-svc <cls> [ip...]`	Reload cluster load balancer config
Reload HBA	`bin/pgsql-hba <cls> [ip...]`	Reload cluster HBA access rules
Clone Cluster	-	Clone via standby cluster or PITR

For other management tasks, see: HA Management, Manage Users, Manage Databases.

Managing PostgreSQL Users

Mon, 01 Jan 0001 00:00:00 +0000

Quick Start

Pigsty uses declarative management: first define users in the inventory, then use bin/pgsql-user <cls> <username> to create or modify.

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_users: [{ name: dbuser_app, password: 'DBUser.App', pgbouncer: true }]  # <--- Define user list here!

Script

bin/pgsql-user <cls> <username>    # Create/modify <username> user on <cls> cluster

Playbook

./pgsql-user.yml -l pg-meta -e username=dbuser_app    # Use playbook to create/modify user

Example

bin/pgsql-user pg-meta dbuser_app    # Create/modify dbuser_app user on pg-meta cluster

For the complete user definition reference, see User Configuration. See Access Control for roles and privileges, and Authentication for credential management.

Managing PostgreSQL Databases

Mon, 01 Jan 0001 00:00:00 +0000

Quick Start

Pigsty uses declarative management: first define databases in the inventory, then use bin/pgsql-db <cls> <dbname> to create or modify.

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_databases: [{ name: some_db }]  # <--- Define database list here!

Script

bin/pgsql-db <cls> <dbname>    # Create/modify <dbname> database on <cls> cluster

Playbook

./pgsql-db.yml -l pg-meta -e dbname=some_db    # Use playbook to create/modify database

Example

bin/pgsql-db pg-meta some_db    # Create/modify some_db database on pg-meta cluster

For the complete database definition reference, see Database Configuration. For database access permissions, see Access Control: Database Isolation.

Patroni HA Management

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Pigsty uses Patroni to manage PostgreSQL clusters. It handles config changes, status checks, switchover, restart, reinit replicas, and more.

To use Patroni for management, you need one of the following identities:

From INFRA node as admin user, managing all clusters in the environment.
From PGSQL node as pg_dbsu (default postgres), managing the current cluster only.

Patroni provides patronictl CLI for management. Pigsty provides a wrapper alias pg to simplify operations.

Managing PostgreSQL HBA Rules

Mon, 01 Jan 0001 00:00:00 +0000

Quick Start

Pigsty uses declarative management: first define HBA rules in the inventory, then use bin/pgsql-hba <cls> to refresh.

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_hba_rules:                            # <--- Define HBA rules here!
      - {user: dbuser_app, db: app, addr: intra, auth: pwd, title: 'app access'}
      - {user: dbuser_api, db: all, addr: world, auth: ssl, title: 'api ssl access'}

Script

bin/pgsql-hba <cls>              # Refresh PostgreSQL and Pgbouncer HBA rules for cluster
bin/pgsql-hba <cls> <ip>...      # Refresh HBA rules for specific instances

Playbook

./pgsql.yml -l <cls> -t pg_hba,pg_reload                 # Refresh PostgreSQL HBA only
./pgsql.yml -l <cls> -t pgbouncer_hba,pgbouncer_reload   # Refresh Pgbouncer HBA only
./pgsql.yml -l <cls> -t pg_hba,pg_reload,pgbouncer_hba,pgbouncer_reload  # Refresh both

Example

bin/pgsql-hba pg-meta                      # Refresh pg-meta cluster HBA rules
bin/pgsql-hba pg-meta 10.10.10.10          # Refresh specific instance only
bin/pgsql-hba pg-meta 10.10.10.11 10.10.10.12  # Refresh multiple instances

For rule syntax, see HBA Configuration. For authentication methods, default boundaries, and credential management, see Authentication.

Pgbouncer Connection Pooling

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Pigsty uses Pgbouncer as PostgreSQL connection pooling middleware, listening on port 6432 by default, proxying access to local PostgreSQL on port 5432.

This is an optional component. If you don’t have massive connections or need transaction pooling and query metrics, you can disable it, connect directly to the database, or keep it unused.

User & Database Management

Pgbouncer users and databases are auto-managed by Pigsty, applying database config and user config when creating databases and creating users.

Managing PostgreSQL Component Services

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Pigsty’s PGSQL module consists of multiple components, each running as a systemd service on nodes. (pgbackrest is an exception)

Understanding these components and their management is essential for maintaining production PostgreSQL clusters.

Component	Port	Service Name	Description
Patroni	`8008`	`patroni`	HA manager, manages PostgreSQL lifecycle
PostgreSQL	`5432`	`postgres`	Placeholder service, not used, for emergency
Pgbouncer	`6432`	`pgbouncer`	Connection pooling middleware, traffic entry
PgBackRest	-	-	pgBackRest has no daemon service
HAProxy	`543x`	`haproxy`	Load balancer, exposes database services
pg_exporter	`9630`	`pg_exporter`	PostgreSQL metrics exporter
pgbouncer_exporter	`9631`	`pgbouncer_exporter`	Pgbouncer metrics exporter
vip-manager	-	`vip-manager`	Optional, manages L2 VIP address floating

Important

Do NOT use systemctl directly to manage PostgreSQL service. PostgreSQL is managed by Patroni - use patronictl commands instead. Direct PostgreSQL operations may cause Patroni state inconsistency and trigger unexpected failover. The postgres service is an emergency escape hatch when Patroni fails.

Manage PostgreSQL Cron Jobs

Mon, 01 Jan 0001 00:00:00 +0000

Pigsty uses crontab to manage scheduled tasks for routine backups, freezing aging transactions, and reorganizing bloated tables and indexes.

Quick Reference

Operation	Quick Command	Description
Configure Cron Jobs	`./pgsql.yml -t pg_crontab -l <cls>`	Apply pg_crontab config
View Cron Jobs	`crontab -l`	View as postgres user
Physical Backup	`pg-backup [full\|diff\|incr]`	Execute backup with pgBackRest
Transaction Freeze	`pg-vacuum [database...]`	Freeze aging transactions, prevent XID wraparound
Bloat Maintenance	`pg-repack [database...]`	Online reorganize bloated tables and indexes

For other management tasks, see: Backup Management, Monitoring System, HA Management.

Managing PostgreSQL Extensions

Mon, 01 Jan 0001 00:00:00 +0000

Quick Start

Pigsty provides 575 extensions. Using extensions involves four steps: Download, Install, Configure, Enable.

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pg_extensions: [ postgis, timescaledb, pgvector ]           # <--- Install extension packages
    pg_libs: 'timescaledb, pg_stat_statements, auto_explain'    # <--- Configure preload extensions
    pg_databases:
      - name: meta
        extensions: [ postgis, timescaledb, vector ]            # <--- Enable in database

Script

bin/pgsql-ext <cls>           # Install extensions defined in config on <cls> cluster
bin/pgsql-ext <cls> [ext...]  # Install extensions specified on command line

Playbook

./pgsql.yml -l pg-meta -t pg_ext    # Use playbook to install extensions

Example

bin/pgsql-ext pg-meta                         # Install defined extensions on pg-meta cluster
bin/pgsql-ext pg-meta pg_duckdb pg_mooncake   # Install specified extensions

For complete extension reference, see Extensions. For available extensions, see Extension Catalog.

Upgrading PostgreSQL Major/Minor Versions

Mon, 01 Jan 0001 00:00:00 +0000

Quick Start

PostgreSQL version upgrades fall into two types: minor version upgrade and major version upgrade, with very different risk and complexity.

Type	Example	Downtime	Data Compatibility	Risk
Minor upgrade	17.2 → 17.3	Seconds (rolling)	Fully compatible	Low
Major upgrade	17 → 18	Minutes	Requires data dir upgrade	Medium

Minor

# Rolling upgrade: replicas first, then primary
ansible <cls> -b -a 'yum upgrade -y postgresql17*'
pg restart --role replica --force <cls>
pg switchover <cls>
pg restart <cls> <old-primary> --force

Major

# Recommended: Logical replication migration
bin/pgsql-add pg-new              # Create new version cluster
# Configure logical replication to sync data...
# Switch traffic to new cluster

Extension

ansible <cls> -b -a 'yum upgrade -y postgis36_17*'
psql -c 'ALTER EXTENSION postgis UPDATE;'

For detailed online migration process, see Online Migration documentation.

Administration

Mon, 01 Jan 0001 00:00:00 +0000

Backup Policy

Mon, 01 Jan 0001 00:00:00 +0000

The chart below combines the “Recovery Window” and “Backup Storage Usage” on a single timeline (0~108h) so they can be inspected together.

Under the same assumptions (database size 100GB, daily writes 10GB), it shows how both metrics evolve over 30 days with “weekly full + daily incremental” backups and 14-day full-backup retention.

When: Backup schedule
Where: Backup repository
How: Backup method

When to Backup

The first question is when to backup your database - this is a tradeoff between backup frequency and recovery time. Since you need to replay WAL logs from the last backup to the recovery target point, the more frequent the backups, the less WAL logs need to be replayed, and the faster the recovery.

Restore Operations

Mon, 01 Jan 0001 00:00:00 +0000

Pigsty provides three restore entry points. They share the same parameter semantics, but serve different scopes:

Entry point	Use case	What it controls
`pgsql-pitr.yml`	Production cluster recovery	HA pause, multiple nodes, etcd cleanup, restore, and restart
`pig pitr`	A local database node	Single-instance orchestration without the admin node
`pig pb restore`	An instance not managed by Patroni	A direct pgBackRest restore wrapper

For a hands-on sandbox drill, see Manual Recovery. To recover into another cluster without changing production, see Clone a PG Cluster.

Clone a PG Cluster

Mon, 01 Jan 0001 00:00:00 +0000

Cloning is one of the safest and most useful applications of recovery: leave production untouched and restore its historical state into another cluster. You can recover accidentally deleted data from the clone, validate backups in a drill, inspect a historical state, or reset a test environment to a production snapshot.

The target must be able to access the source backup repository, may be overwritten, and must use a compatible PostgreSQL major version. With a shared Silo/S3 repository, each cluster’s backups are isolated by a stanza and visible to targets holding the required credentials.

Backup & Restore

Mon, 01 Jan 0001 00:00:00 +0000

Pigsty uses pgBackRest for PostgreSQL backups. It supports full, differential, and incremental backups, parallel processing, encryption, and Silo/S3 object storage. Every PGSQL cluster is configured for backup and WAL archiving by default.

This chapter is the operational manual for backup configuration, management, recovery, and drills. For design concepts and tradeoffs, see Point-in-Time Recovery.

All backup and recovery operations ultimately invoke pgBackRest. Pigsty provides several wrapper layers:

Layer	Interface	Form	Scope
Cluster orchestration	`pg_pitr` + `pgsql-pitr.yml`	Ansible playbook	HA, etcd, and multi-node recovery
Instance orchestration	`pig pitr`	CLI	Local-node recovery without the admin node
Command primitives	`pig pb`, `pb`, and `pg-backup`	pgBackRest wrappers	Backup, inspection, expiry, and unmanaged restore
Engine	`pgbackrest`	Native CLI	Underlying backup, archive, and restore engine

Section	Content
Mechanism	Stanzas, repositories, retention, timelines, and Pigsty wrapper mapping
Policy	Scheduling, recovery windows, and storage planning
Repository	Local, Silo, and external S3 repositories; encryption, versioning, and locking
Administration	Backup commands, inspection, expiration, and stanza management
Restore	Recovery targets, staged PITR, and complete parameter reference
Clone	Restore production history into another cluster and perform drills
Tutorial	A sandbox restore using pgBackRest primitives

Disclaimer

Pigsty makes every effort to provide a reliable PITR solution, but accepts no liability for data loss caused by restore operations. If you need assistance, consider professional services.

Troubleshooting

Mon, 01 Jan 0001 00:00:00 +0000

This document lists potential failures in PostgreSQL and Pigsty, as well as SOPs for locating, handling, and analyzing issues.

Disk Space Exhausted

Disk space exhaustion is the most common type of failure.

Symptoms

When the disk space where the database resides is exhausted, PostgreSQL will not work normally and may exhibit the following symptoms: database logs repeatedly report “no space left on device” errors, new data cannot be written, and PostgreSQL may even trigger a PANIC and force shutdown.

Data Migration

Mon, 01 Jan 0001 00:00:00 +0000

Pigsty includes a built-in playbook pgsql-migration.yml that implements online database migration based on logical replication.

With pre-generated automation scripts, application downtime can be reduced to just a few seconds. However, note that logical replication requires PostgreSQL 10 or later to work.

Of course, if you have sufficient downtime budget, you can always use the pg_dump | psql approach for offline migration.

Defining Migration Tasks

To use Pigsty’s online migration playbook, you need to create a definition file that describes the migration task details.

Backup Mechanism

Mon, 01 Jan 0001 00:00:00 +0000

Pigsty’s backup and restore operations ultimately execute pgBackRest commands. Using them safely requires both pgBackRest’s model and the mapping from Pigsty’s orchestration layers to native options.

Core pgBackRest Concepts

Stanza: the Cluster’s Backup Identity

A stanza names one PostgreSQL backup configuration and isolates that cluster inside a repository. Pigsty maps it directly from pg_cluster: the pg-meta stanza stores data under backup/pg-meta/ and archive/pg-meta/, so several clusters can share one repository.

Backup Repository

Mon, 01 Jan 0001 00:00:00 +0000

Two parameters decide where backups are stored: pgbackrest_repo defines candidate repositories, while pgbackrest_method selects one. Repository keys are rendered deterministically as pgBackRest repo1-* options, so any supported pgBackRest repository option can be used directly. Pigsty v4.5.0 renders only the entry selected by pgbackrest_method as repo1; keeping several candidate keys does not enable multi-repository backup.

Default Repositories

Pigsty supplies two definitions: local and minio.

local is the default. /pg/backup points at pg_fs_backup, /data/backups by default.
minio uses Silo from the MINIO module or another compatible S3 service. It is supported but not selected by default.

pgbackrest_method: local
pgbackrest_repo:
  local:
    path: /pg/backup
    retention_full_type: count
    retention_full: 2
  minio:
    type: s3
    s3_endpoint: sss.pigsty
    s3_region: us-east-1
    s3_bucket: pgsql
    s3_key: pgbackrest
    s3_key_secret: S3User.Backup
    s3_uri_style: path
    path: /pgbackrest
    storage_port: 9000
    storage_ca_file: /etc/pki/ca.crt
    block: y
    bundle: y
    bundle_limit: 20MiB
    bundle_size: 128MiB
    cipher_type: aes-256-cbc
    cipher_pass: pgBackRest
    retention_full_type: time
    retention_full: 14

The presets deliberately differ. local favors simple, fast restores with count-based retention and no encryption or bundling. minio enables AES-256-CBC encryption, bundles small files, uses block incremental backup, and retains full backups by time.

Admin Commands

Mon, 01 Jan 0001 00:00:00 +0000

Run backup commands as the database superuser (pg_dbsu, postgres by default) on a database node. You can use any of these entry points:

pig pb: the PIG CLI wrapper, with automatic stanza detection, DBSU switching, and safety checks; this is the recommended interface
pb: a login-shell function that supplies --stanza and forwards arguments to pgBackRest
pgbackrest: the native command; see the pgBackRest command reference

Command Overview

PIG command	Alias	Native pgBackRest command	Purpose
`pig pb info`	`i`	`info`	Show backup and archive status
`pig pb list`	`ls`	—	List repositories, stanzas, or backup sets
`pig pb backup [full/diff/incr]`	`b`	`backup`	Create a backup after checking the primary role
`pig pb restore`	`r`	`restore`	Low-level restore primitive; see Restore Operations
`pig pb expire`	`e`	`expire`	Expire backups according to retention (`--plan` previews)
`pig pb create`	`c`	`stanza-create`	Create a stanza
`pig pb upgrade`	`u`	`stanza-upgrade`	Upgrade a stanza after a major-version change or clone
`pig pb delete`	`d`	`stanza-delete`	Delete a stanza and all of its backups
`pig pb check`	`ck`	`check`	Verify configuration, repository access, and archiving
`pig pb start`	`up`	`start`	Re-enable pgBackRest operations
`pig pb stop`	`dw`	`stop`	Stop new pgBackRest operations
`pig pb log [list/show/tail]`	`l`	—	Inspect pgBackRest logs

Enable Backup

If pgbackrest_enabled is true when the cluster is created, backup is enabled automatically. If it was disabled at creation time, or repository settings have changed, run the pg_backup subtask:

Manual PITR Drill

Mon, 01 Jan 0001 00:00:00 +0000

This tutorial drills PostgreSQL point-in-time recovery in Pigsty v4.5.0’s four-node sandbox. The main path runs pgsql-pitr.yml as down → pitr → up, giving the operator a separate validation gate before data overwrite, timeline promotion, and HA reconstruction.

For one current node, use pig pitr. For direct pgBackRest control, see the low-level pg-pitr utility.

Follow this literally only in a disposable sandbox

Recovery stops Patroni/PostgreSQL and overwrites the target PGDATA with pgbackrest --force restore; the up stage also deletes the target cluster’s etcd prefix and rebuilds Patroni state. The playbook prints a plan but has no interactive confirmation. Before production use, the operator must state and confirm the exact cluster and recovery point, verify a recent usable backup that has been independently tested, run --check with exactly the same -l, variables, and tags, and schedule a maintenance window. This tutorial does not authorize running these commands in any production environment.

Enabling HugePage for PostgreSQL

Mon, 01 Jan 0001 00:00:00 +0000

Use node_hugepage_count and node_hugepage_ratio or /pg/bin/pg-tune-hugepage

If you plan to enable HugePages, consider using node_hugepage_count and node_hugepage_ratio, and apply with ./node.yml -t node_tune.

HugePages have pros and cons for databases. The advantage is that memory is managed exclusively, eliminating concerns about being reallocated and reducing database OOM risk. The disadvantage is that it may negatively impact performance in certain scenarios.

Before PostgreSQL starts, you need to allocate enough huge pages. The wasted portion can be reclaimed using the pg-tune-hugepage script, but this script is only available for PostgreSQL 15+.

Clone and Side-Restore a PostgreSQL Instance

Mon, 01 Jan 0001 00:00:00 +0000

Pigsty v4.5.0 provides two local shell utilities:

pg-fork copies a PostgreSQL data directory and gives the copy a separate port.
pg-pitr invokes pgBackRest to restore a stopped data directory to a selected target.

They are useful for sandbox drills, side-channel investigation, and temporary testing. They are not complete Patroni-cluster recovery orchestrators. Prefer pig pitr for a managed instance and staged pgsql-pitr.yml for a multi-node cluster.

Verify paths, backups, and stopped state first

pg-fork recursively removes an existing destination directory; pg-pitr overwrites the destination with backup data. Both can execute without a prompt in a non-interactive environment. Before a real run, verify source and destination absolute paths, ports, tablespaces, exact cluster/instance identity, and an independent recent backup that has been tested. A newly created CoW clone is not an independent backup.

Accidental Deletion

Mon, 01 Jan 0001 00:00:00 +0000

Accidental Data Deletion

If it’s a small-scale DELETE misoperation, you can consider using the pg_surgery or pg_dirtyread extension for in-place surgical recovery.

-- Immediately disable Auto Vacuum on this table and abort Auto Vacuum worker processes for this table
ALTER TABLE public.some_table SET (autovacuum_enabled = off, toast.autovacuum_enabled = off);

CREATE EXTENSION pg_dirtyread;
SELECT * FROM pg_dirtyread('tablename') AS t(col1 type1, col2 type2, ...);

If the deleted data has already been reclaimed by VACUUM, then use the general accidental deletion recovery process.

Playbook

Mon, 01 Jan 0001 00:00:00 +0000

Pigsty provides a series of playbooks for cluster provisioning, scaling, user/database management, monitoring, backup & recovery, and migration.

Playbook	Function
`pgsql.yml`	Initialize PostgreSQL cluster or add new replicas
`pgsql-rm.yml`	Remove PostgreSQL cluster or specific instances
`pgsql-user.yml`	Add new business user to existing PostgreSQL cluster
`pgsql-db.yml`	Add new business database to existing PostgreSQL cluster
`pgsql-monitor.yml`	Monitor remote PostgreSQL instances
`pgsql-migration.yml`	Generate migration manual and scripts for existing PostgreSQL
`pgsql-pitr.yml`	Perform Point-In-Time Recovery (PITR)

Safeguard

Be extra cautious when using PGSQL playbooks. Misuse of pgsql.yml and pgsql-rm.yml can lead to accidental database deletion!

HA Drill: Handling 2-of-3 Node Failure

Sat, 11 Jan 2025 00:00:00 +0000

If a classic 3-node HA deployment experiences simultaneous failure of two nodes (majority), the system typically cannot complete automatic failover and requires manual intervention.

First, assess the status of the other two servers. If they can be brought up quickly, prioritize recovering those two servers. Otherwise, enter the Emergency Recovery Procedure.

The Emergency Recovery Procedure assumes your admin node has failed and only a single regular database node survives. In this case, the fastest recovery process is:

Bind a L2 VIP to PostgreSQL Primary with VIP-Manager

Mon, 01 Jan 0001 00:00:00 +0000

You can define an OPTIONAL L2 VIP on a PostgreSQL cluster, provided that all nodes in the cluster are in the same L2 network.

This VIP works on Master-Backup mode and always points to the node where the primary instance of the database cluster is located.

This VIP is managed by the VIP-Manager, which reads the Leader Key written by Patroni from DCS (etcd) to determine whether it is the master.

Deploy HA Citus Cluster

Sat, 11 Jan 2025 00:00:00 +0000

Citus is a PostgreSQL extension that transforms PostgreSQL into a distributed database, enabling horizontal scaling across multiple nodes to handle large amounts of data and queries.

Patroni v3.0+ provides native high-availability support for Citus, simplifying the setup of Citus clusters. Pigsty also provides native support for this.

Note

Citus 13.x supports PostgreSQL 18, 17, 16, 15, and 14. The Pigsty extension repository provides Citus ARM64 packages.

Playbook

Mon, 01 Jan 0001 00:00:00 +0000

Pigsty provides three playbooks related to the INFRA module:

deploy.yml: Deploy the NODE, INFRA, ETCD, MINIO, and PGSQL core modules on all nodes in one pass
infra.yml: Initialize Pigsty infrastructure on infra nodes
infra-rm.yml: Remove infrastructure components from infra nodes

`deploy.yml`

Deploy the NODE, INFRA, ETCD, MINIO, and PGSQL core modules on all nodes in one pass, resolving INFRA/NODE circular dependency issues.

This playbook interleaves subtasks from infra.yml and node.yml, completing deployment of the core components in the following order:

Administration

Mon, 01 Jan 0001 00:00:00 +0000

This section covers daily administration and operations for Pigsty deployments.

Nginx Management

Mon, 01 Jan 0001 00:00:00 +0000

Pigsty installs Nginx on INFRA nodes as the entry point for all web services, listening on standard ports 80/443.

In Pigsty, you can configure Nginx to provide various services through inventory:

Expose web interfaces for monitoring components like Grafana, VictoriaMetrics (VMUI), Alertmanager, and VictoriaLogs
Serve static files (software repos, documentation sites, websites, etc.)
Proxy custom application services (internal apps, database management UIs, Docker application interfaces, etc.)
Automatically issue self-signed HTTPS certificates, or use Certbot to obtain free Let’s Encrypt certificates
Expose services through a single port using different subdomains for unified access

Basic Configuration

Customize Nginx behavior via infra_portal parameter:

Software Repository

Mon, 01 Jan 0001 00:00:00 +0000

Pigsty’s REPO role downloads required packages and creates a local YUM/APT repository under /www/pigsty that Nginx can serve. The current package candidate is SOW 0.3.0; the source uses SOW to generate metadata for both repository types instead of calling createrepo_c, modifyrepo_c, or dpkg-scanpackages separately.

Quick Start

Add packages to repo_packages or repo_extra_packages, then run:

./infra.yml -t repo_build   # Download and build only when the repository is absent
./node.yml -t node_repo     # Refresh repository definitions and caches on each node

If /www/pigsty/repo_complete already exists, the default repo_build skips the build. To force a rebuild, override it explicitly:

Domain Management

Mon, 01 Jan 0001 00:00:00 +0000

Use domain names instead of IP addresses to access Pigsty’s various web services.

Quick Start

Add the following static resolution records to /etc/hosts:

10.10.10.10 i.pigsty

Replace IP address with your actual Pigsty node’s IP.

Why Use Domain Names

Easier to remember than IP addresses
Flexible pointing to different IPs
Unified service management through Nginx
Support for HTTPS encryption
Prevent ISP hijacking in some regions
Allow access to internally bound services via proxy

DNS Mechanism

DNS Protocol: Resolves domain names to IP addresses. Multiple domains can point to same IP.

Module Management

Mon, 01 Jan 0001 00:00:00 +0000

This document covers daily management operations for the INFRA module, including installation, uninstallation, scaling, and component maintenance.

Install INFRA Module

Use the infra.yml playbook to install the INFRA module on the infra group:

./infra.yml     # Install INFRA module on infra group

Uninstall INFRA Module

Use the infra-rm.yml playbook to uninstall the INFRA module from the infra group:

CA and Certificates

Mon, 01 Jan 0001 00:00:00 +0000

Pigsty maintains a self-signed Certificate Authority (CA) on the admin node by default. It signs certificates for PostgreSQL, Patroni, etcd, Silo, Nginx, and other internal services. Public Nginx entries can use Certbot/Let’s Encrypt certificates configured through infra_portal.

Protect the CA private key

files/pki/ca/ca.key is the trust-root private key for the entire deployment. Never print, commit, upload, or transmit it over an unprotected channel. Back it up together with ca.crt, encrypted, with tightly restricted read access.

Playbook

Mon, 01 Jan 0001 00:00:00 +0000

Pigsty provides two playbooks related to the NODE module:

node.yml: Add nodes to Pigsty and configure them to the desired state
node-rm.yml: Remove managed nodes from Pigsty

Two wrapper scripts are also provided: bin/node-add and bin/node-rm, for quickly invoking these playbooks.

`node.yml`

The node.yml playbook for adding nodes to Pigsty contains the following subtasks:

node-id       : generate node identity
node_name     : setup hostname
node_hosts    : setup /etc/hosts records
node_resolv   : setup DNS resolver /etc/resolv.conf
node_firewall : setup firewall & selinux
node_ca       : add & trust CA certificate
node_repo     : add upstream software repository
node_pkg      : install rpm/deb packages
node_uv       : setup uv Python virtual environment
node_feature  : setup numa, grub, static network
node_kernel   : enable kernel modules
node_tune     : setup tuned profile
node_sysctl   : setup additional sysctl parameters
node_profile  : write /etc/profile.d/node.sh
node_alias    : write /etc/profile.d/node.alias.sh
node_ulimit   : setup resource limits
node_data     : setup data directory
node_admin    : setup admin user and ssh key
node_timezone : setup timezone
node_ntp      : setup NTP server/client
node_crontab  : add/overwrite crontab tasks
node_vip      : setup optional L2 VIP for node cluster
haproxy       : setup haproxy on node to expose services
monitor       : setup node monitoring: node_exporter & vector

`node-rm.yml`

The node-rm.yml playbook for removing nodes from Pigsty contains the following subtasks:

Administration

Mon, 01 Jan 0001 00:00:00 +0000

Here are common administration operations for the NODE module:

For more questions, see FAQ: NODE

Add Node

To add a node to Pigsty, you need passwordless ssh/sudo access to that node.

You can also add an entire cluster at once, or use wildcards to match nodes in the inventory to add to Pigsty.

Administration

Mon, 01 Jan 0001 00:00:00 +0000

Common etcd admin SOPs:

Create Cluster: Initialize an etcd cluster
Destroy Cluster: Destroy an etcd cluster
CLI Environment: Configure etcd client to access server cluster
RBAC Authentication: Use etcd RBAC auth
Reload Config: Update etcd server member list for clients
Add Member: Add new member to existing etcd cluster
Remove Member: Remove member from etcd cluster
Utility Scripts: Simplify ops with bin/etcd-add and bin/etcd-rm

For more, refer to FAQ: ETCD.

Create Cluster

Define etcd cluster in config inventory:

Playbook

Mon, 01 Jan 0001 00:00:00 +0000

The ETCD module provides two core playbooks: etcd.yml for installing and configuring etcd clusters, and etcd-rm.yml for removing etcd clusters or members.

Architecture Change: Pigsty v3.6+

Since Pigsty v3.6, the etcd.yml playbook focuses on cluster installation and member addition. All removal operations have been moved to the dedicated etcd-rm.yml playbook using the etcd_remove role.

`etcd.yml`

Playbook source: etcd.yml

This playbook installs and configures an etcd cluster on the hardcoded etcd group, then launches the etcd service.

Playbook

Mon, 01 Jan 0001 00:00:00 +0000

The MINIO module provides two built-in playbooks:

minio.yml: Install and configure Silo
minio-rm.yml: Remove Silo, its configuration, and optionally its data

`minio.yml`

minio.yml runs with hosts: all, but its pre-tasks skip hosts where minio_cluster is undefined. The role then validates that:

minio_cluster is defined and non-empty
minio_seq is defined and is a non-negative integer
minio_type must equal silo

Thus, minio_cluster is the module-membership gate, while invalid minio_seq or minio_type values fail identity validation explicitly. Do not define minio_cluster in all.vars.

Administration

Mon, 01 Jan 0001 00:00:00 +0000

Create Cluster

To create a cluster, define it in the config inventory and run the minio.yml playbook.

minio: { hosts: { 10.10.10.10: { minio_seq: 1 } }, vars: { minio_cluster: minio, minio_type: silo } }

The configuration above defines an SNSD Single-Node Single-Disk Silo cluster. Create it with:

./minio.yml -l minio  # Install Silo on the minio group

Remove Cluster

To destroy a cluster, run the dedicated minio-rm.yml playbook:

Playbook

Mon, 01 Jan 0001 00:00:00 +0000

The REDIS module provides two playbooks for deploying/removing Redis clusters/nodes/instances:

redis.yml: Deploy Redis cluster/node/instance
redis-rm.yml: Remove Redis cluster/node/instance

`redis.yml`

The redis.yml playbook for deploying Redis contains the following subtasks:

redis_node        : Init redis node
  - redis_install : Install Redis and the `redis-exporter` package
  - redis_user    : Create OS user redis
  - redis_dir     : Configure redis FHS directory structure
redis_exporter    : Configure redis_exporter monitoring
  - redis_exporter_config  : Generate redis_exporter config
  - redis_exporter_launch  : Launch redis_exporter
redis_instance    : Init and restart redis cluster/node/instance
  - redis_config  : Generate redis instance config
  - redis_launch  : Launch redis instance
redis_register    : Register redis to infrastructure
redis_ha          : Configure redis sentinel (sentinel mode only)
redis_join        : Join redis native cluster (cluster mode only)

Operation Levels

redis.yml supports three operation levels, controlled by -l to limit target scope and -e redis_port=<port> to specify a single instance:

Administration

Mon, 01 Jan 0001 00:00:00 +0000

Here are some common Redis administration task SOPs (Standard Operating Procedures):

The REDIS module defaults to redis_type: redis. With redis_type: valkey, the server and client commands become valkey-server and valkey-cli. Examples on this page use the default redis-cli; substitute valkey-cli for Valkey clusters. Playbooks choose the correct CLI automatically.

Basic Operations

High Availability

Scaling & Migration

Grafana High Availability: Using PostgreSQL Backend

Mon, 01 Jan 0001 00:00:00 +0000

You can use PostgreSQL as Grafana’s backend database.

This is a great opportunity to understand Pigsty’s deployment system. By completing this tutorial, you’ll learn:

How to create a new database cluster
How to create new business users in an existing cluster
How to create new business databases in an existing cluster
How to access databases created by Pigsty
How to manage Grafana dashboards
How to manage PostgreSQL datasources in Grafana
How to upgrade Grafana database in one step

TL;DR

vi pigsty.yml # Uncomment DB/User definitions: dbuser_grafana  grafana
bin/pgsql-user  pg-meta  dbuser_grafana
bin/pgsql-db    pg-meta  grafana

psql postgres://dbuser_grafana:DBUser.Grafana@meta:5436/grafana -c \
  'CREATE TABLE t(); DROP TABLE t;' # Verify connection string works

vi /etc/grafana/grafana.ini # Modify [database] type url
systemctl restart grafana-server

Create Database Cluster

We can define a new database grafana on pg-meta, or create a dedicated Grafana database cluster pg-grafana on new nodes.

Playbooks

Mon, 01 Jan 0001 00:00:00 +0000

The Docker module provides a default playbook docker.yml for installing Docker Daemon and Docker Compose.

`docker.yml`

Playbook source file: docker.yml

Running this playbook will install docker-ce and docker-compose-plugin on target nodes with the docker_enabled: true flag, and enable the dockerd service.

The following are the available task subsets in the docker.yml playbook:

docker_install : Install Docker and Docker Compose packages on the node
docker_admin : Add specified users to the Docker admin user group
docker_dir : Create Docker related directories
docker_config : Generate Docker daemon service configuration file
docker_launch : Start the Docker daemon service
docker_register : Register Docker daemon as a monitoring target (alias tags: register / add_metrics)
docker_image : Attempt to load pre-cached image tarballs from /tmp/docker/*.tgz (if they exist)

The Docker module does not provide a dedicated uninstall playbook. If you need to uninstall Docker, you can manually stop Docker and then remove it:

Playbook

Mon, 01 Jan 0001 00:00:00 +0000

JUICE module provides juice.yml playbook to deploy and remove JuiceFS instances.

`juice.yml`

Task structure in juice.yml:

juice_id        : validate config, check port conflicts
juice_install   : install juicefs package
juice_cache     : create shared cache dir
juice_clean     : remove instance (state=absent)
juice_instance  : create instance (state=create)
  - juice_init  : format filesystem (--no-update)
  - juice_dir   : create mount dir
  - juice_config: render env file and systemd unit
  - juice_launch: start service and wait for metrics port
juice_register  : register to VictoriaMetrics targets

Scope

Scope	Limit	Description
Node	`-l <host>`	Deploy all instances on the node
Instance	`-l <host> -e fsname=<name>`	Only handle specified instance

Examples:

Administration

Mon, 01 Jan 0001 00:00:00 +0000

Common operations:

See FAQ for more.

Initialize Instance

./juice.yml -l <host>
./juice.yml -l <host> -e fsname=<name>

Initialization steps:

Install juicefs package
Create shared cache dir (default /data/juice)
Run juicefs format --no-update (only effective on first creation)
Create mount point and set permissions
Render systemd unit and env files
Start service and wait for metrics port
Register to VictoriaMetrics (if infra node exists)

Reconfigure

After changing config, it’s recommended to run (update config and ensure service is online):

Playbook

Mon, 01 Jan 0001 00:00:00 +0000

VIBE provides the vibe.yml playbook to deploy Code-Server, JupyterLab, Node.js, Claude Code, and Codex CLI.

vibe.yml includes only node_id and vibe roles, it does not include node/infra. Run deploy.yml first, or explicitly run node.yml and infra.yml.

`vibe.yml`

vibe.yml:

- name: VIBE
  hosts: all
  become: true
  gather_facts: no
  roles:
    - { role: node_id, tags: id }
    - { role: vibe,    tags: vibe }

Task Structure

vibe
├── vibe_dir          # create workspace and context files
├── code              # Code-Server
│   ├── code_install
│   ├── code_dir
│   ├── code_config
│   └── code_launch
├── jupyter           # JupyterLab
│   ├── jupyter_install
│   ├── jupyter_dir
│   ├── jupyter_config
│   └── jupyter_launch
├── nodejs            # Node.js runtime and additional npm packages
│   ├── nodejs_install
│   ├── nodejs_config
│   └── nodejs_pkg
├── codex             # Codex CLI
│   └── codex_install
└── claude            # Claude Code
    ├── claude_install
    └── claude_config

Notes:

Administration

Mon, 01 Jan 0001 00:00:00 +0000

Service Management

systemctl status code-server
systemctl restart code-server
systemctl status jupyter
systemctl restart jupyter

Logs:

journalctl -u code-server -f
journalctl -u jupyter -f

Workspace and Context

vibe_dir creates these under vibe_data:

AGENTS.md: Context file rendered from the role template
CLAUDE.md: Symlink to AGENTS.md

Default locations (adjustable via vibe_data):

Administration

Mon, 01 Jan 0001 00:00:00 +0000

The KAFKA module installs Kafka under /opt/kafka, manages the service with Systemd, and keeps its persistent intent in pigsty.yml. The files generated on the nodes are not meant to be edited by hand.

All of the Kafka CLI examples below use the role-generated /etc/kafka/admin.properties. Even when the current profile is plaintext, keep --command-config on every command: that way the command structure stays the same when you switch the admin channel to scram. Replace <broker>:9092 with a reachable inventory_hostname and port.

Playbook

Mon, 01 Jan 0001 00:00:00 +0000

The KAFKA module ships two playbooks: kafka.yml deploys an Apache Kafka 4.1+ dynamic KRaft cluster and converges its security, resource, and monitoring state; kafka-rm.yml tears down a cluster or removes a member.

Cluster Completeness Constraint

Every selected kafka_cluster must include all of its members: a partial selection fails before anything is written, while one cluster, several complete clusters, or a bare run over the whole inventory are all allowed. Run --check first against the exact same target; before the real run, still verify the backup/rebuild intent, capacity, business window, rollback plan, and change approval by hand.