Skip to content

This is the multi-page printable view of this section. .

Return to the regular view of this page.

Point-in-Time Recovery — A Time Machine for PostgreSQL

High availability handles machine failure; point-in-time recovery handles incorrect data. Pigsty uses pgBackRest to provide PITR out of the box, allowing a cluster to return to any recoverable point covered by its backup and WAL history.

If data, a table, or even a database is deleted accidentally, Point-in-Time Recovery (PITR) can return the cluster to an earlier state.

This capability, once treated as specialist DBA work, is enabled by Pigsty’s standard PostgreSQL configuration.


Replication Is Not Backup

High availability can fail over to another instance when hardware fails. It has a natural blind spot, however: replication is not backup.

Streaming replication faithfully sends every primary change to every replica within milliseconds, including a DELETE without a WHERE clause or a DROP TABLE issued against the wrong database. Failover handles a broken machine; when the data itself is wrong, every replica can contain the same error.

Database disasters therefore fall into two broad classes. Redundancy handles physical service failure through multiple copies and automatic failover. Logical errors require history: a base backup plus continuous WAL archives from which PostgreSQL can reconstruct a state before the mistake.

Threat High Availability Delayed Cluster PITR
Hardware or instance failure ✔ Automatic failover ✔, with a longer RTO
Accidental DML, table drop, or database drop ✘ The error is replicated ✔ Within the delay ✔ At any recoverable point
Defective software corrupts data over time ✘ The error is replicated ✔ Within the delay ✔ Try different recovery targets
Entire cluster or site is lost ✔ Only if the repository survives that failure domain

These mechanisms complement one another: HA restores service quickly, a delayed cluster provides a short undo window, and PITR is the final historical recovery path.


How the Time Machine Works

A database can be viewed as a state machine. A base backup is a complete physical snapshot at one point, while WAL (Write-Ahead Log) records every subsequent state change. With a snapshot and an unbroken WAL history starting from it, PostgreSQL can replay the database to any target covered by that history. The backup determines how far back recovery can start; the latest archived WAL determines how close to the present it can reach.

Base backup + WAL archive = point-in-time recovery

Pigsty orchestrates both inputs. Cluster initialization attempts an initial full backup by default, and the primary continuously sends completed WAL segments to the selected repository. See How PITR Works for the complete model of backups, archives, targets, and timelines.


Available Out of the Box

PITR is enabled in Pigsty’s standard PostgreSQL configuration. Each cluster is prepared with a backup repository, WAL archiving, and recovery tooling powered by pgBackRest. The policy remains declarative and can be customized with a few parameters:

pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary } }
  vars:
    pg_cluster: pg-meta
    pgbackrest_method: minio       # Silo / S3-compatible storage; local is the default
    pg_crontab: [ '00 01 * * * /pg/bin/pg-backup full' ]  # daily full backup at 01:00

The default local method stores backups under /pg/backup and retains two full backups. With one successful full backup per day, the resulting window is roughly 24–48 hours. Selecting the remote minio preset places the repository in Silo or compatible S3 storage, enables AES-256-CBC repository encryption, and uses time-based retention. With a 14-day retention setting and weekly full backups, the steady-state recovery window is roughly 14–21 days. Treat both ranges as policy estimates: actual coverage starts at the oldest usable backup and ends at the latest WAL that reached the repository.

Recovery is declarative too: specify a target, then let the playbook stop the cluster, restore files, replay WAL, and rebuild HA. An operator must still verify the recovered business state.

./pgsql-pitr.yml -l pg-meta -e '{"pg_pitr": { "time": "2026-07-11 10:00:00+08", "action": "promote" }}'

This follows Pigsty’s declarative configuration model: backup policy is part of the cluster definition, and a recovery target is another declared parameter.


Benefits and Costs

PITR materially improves data integrity and availability:

  • RPO (maximum data loss) is usually reduced to minutes, bounded by WAL that had not reached a surviving repository.
  • RTO (time to restore service) becomes tens of minutes to hours rather than permanent loss, depending on backup size, WAL replay distance, and disk or network throughput.
Standalone strategy Event RTO RPO
No backup Host and local data are lost Permanent loss All data
Base backups only Host and local data are lost Backup size and bandwidth, often hours Changes since the latest backup
Base backups + WAL archives Host and local data are lost Backup size, replay distance, and bandwidth WAL not yet present in the surviving repository

The costs fall mainly into three areas:

  • Confidentiality: backups are another copy of business data and need encryption and access control. Pigsty’s remote preset enables repository encryption, but its default password must be changed.
  • Resources: backups consume storage and archiving consumes bandwidth. Compression, bundling, and block incremental backup reduce this cost but do not eliminate capacity planning.
  • Operations: backup status must be monitored and recovery must be rehearsed. A green backup job alone is not proof that the data can be restored within the required RTO.

PITR by itself does not replace HA. A production design normally combines HA for physical failures with PITR for logical errors and site-level recovery.


Next Steps

  • How PITR Works: snapshots, WAL history, recovery windows, targets, and timelines
  • PITR Architecture: pgBackRest, repository selection, archive flow, scheduling, and failover behavior
  • PITR Tradeoffs: failure domains, capacity, retention, and backup frequency
  • Declarative Recovery: the pg_pitr parameter, pgsql-pitr.yml, and pig pitr
  • PITR Scenarios: accidental deletion, bad releases, investigation, and site loss

For the operational runbooks, see PGSQL Backup and Recovery.

1 - How PITR Works

Snapshots, WAL history, recovery windows, recovery targets, and timelines: the five concepts needed to reason accurately about PostgreSQL PITR.

If a database is a state machine, WAL (Write-Ahead Log) is its ordered change history. PostgreSQL records each modification in WAL before applying it to data files. Save a physical snapshot at one point, preserve all later WAL, and PostgreSQL can replay that history to a selected consistent state.

PITR is therefore the combination of three simple elements: a snapshot (base backup), history (WAL archive), and a target (where replay should stop).


Snapshot: Base Backup

A base backup is a physical snapshot of the whole PostgreSQL cluster and supplies a starting point for recovery. Pigsty uses pgBackRest to create and manage three backup types:

Type Contents Recovery characteristics
Full All database-cluster files Self-contained, shortest chain, largest backup
Differential Changes since the latest full backup Restore uses the full plus the differential
Incremental Changes since the latest backup of any type Smallest backup, restore depends on its complete chain

The wrapper pg-backup [full|diff|incr] triggers a backup. With no argument it requests incr; pgBackRest creates a full backup instead when no valid full exists. pg_crontab declares recurring jobs and installs them in the postgres user’s crontab.

Backup frequency affects recovery time: the newer the usable backup, the less WAL must be replayed to reach a given target. See PITR Tradeoffs.


WAL History

A snapshot reaches only its own state. WAL archiving preserves every later change needed to advance beyond it. Pigsty’s standard Patroni templates enable archiving and ask PostgreSQL to hand each completed WAL segment to pgBackRest:

archive_mode: 'on'
archive_command: 'pgbackrest --stanza=pg-meta archive-push %p'
archive_timeout: 300

Two implementation details matter:

  • archive_timeout: 300: on a low-write cluster, PostgreSQL can force a segment switch after five minutes so a partially filled segment does not wait indefinitely. This normally keeps the right edge of the recovery window within minutes when WAL is being generated; it is not a promise that every commit is already remote.
  • Asynchronous archive: pgBackRest uses /pg/spool with archive-async=y to batch transfers. Pigsty sets archive-push-queue-max=4GiB; if repository failure lets the queue cross that bound, pgBackRest can drop the queued WAL to protect local disk. That creates an archive gap, so a new full backup is required to establish a fresh recoverable chain.

Expiration is automatic. When old backups expire under the repository policy, pgBackRest also expires archived WAL that no remaining backup needs, unless archive retention is overridden explicitly.


Recovery Window

The backup and its continuous WAL history form a recovery window:

  • Left boundary: the start of the oldest usable remaining backup chain. In practical time-based descriptions, this is usually summarized by the oldest retained full backup’s time.
  • Right boundary: the latest WAL successfully archived to a repository that survives the incident.

The window moves forward as new backups arrive and old chains expire. Pigsty’s local preset keeps two full backups; with one successful full per day, coverage is roughly one to two days. The minio preset uses retention_full_type: time with retention_full: 14; with weekly full backups, the oldest retained chain normally yields roughly 14–21 days of steady-state coverage. These are estimates, not SLAs: missed backups, archive gaps, explicit archive-retention overrides, or repository loss change the actual window. Verify it with pig pb info and restore drills.

See PITR Tradeoffs and Backup Policy.


Targets: Where Replay Stops

PostgreSQL supports several ways to locate a state inside the recovery window. Pigsty exposes six target types through pg_pitr:

pg_pitr type Meaning Typical use
default Replay through all WAL available from the repository Restore the newest archived state after total loss
time Stop at a timestamp Recover from accidental DML or DDL
xid Stop at a transaction ID Exclude a precisely identified bad transaction
lsn Stop at a WAL location Low-level exact targeting
name Stop at a restore point created with pg_create_restore_point() Planned change checkpoint
immediate Stop as soon as the selected backup becomes consistent Validate or expose the selected backup state quickly

The set field is different: it chooses which backup set pgBackRest restores as the starting snapshot; it is not itself a replay stop target.

Boundary Semantics

Targets are inclusive by default: the transaction at the target is retained. To stop immediately before a known bad target, set exclusive: true, which maps to recovery_target_inclusive = false.

Transactions remain atomic. Committed transactions before the effective target survive; transactions not committed at that point are rolled back. Recovery produces a consistent database state rather than half of a transaction.


Timelines

Restoring to the past and accepting new writes creates a fork in history. PostgreSQL uses a timeline to distinguish each branch. PITR promotion, replica promotion, and failover can all create a new timeline; new WAL does not overwrite the old timeline’s files.

gitGraph
    commit id: "Full backup"
    commit id: "Normal writes"
    commit id: "Bad change"
    commit id: "More writes"
    branch Timeline-2
    checkout Timeline-2
    commit id: "PITR before bad change"
    commit id: "New writes"

Keeping the old history allows another attempt if the first target was wrong. The timeline field can select a timeline; Pigsty’s recovery declaration defaults to latest.

Continue with PITR Architecture to see how these concepts map to Pigsty components and configuration.

2 - PITR Architecture

Pigsty implements PITR with pgBackRest: repository selection, archive flow, scheduling, primary-aware backup execution, performance defaults, and observability.

The PITR principle is compact; the engineering is not. WAL archiving must not stall production writes, object-storage backups need encryption, backup jobs must follow the primary after failover, shared repositories must isolate clusters, and large numbers of small objects can limit throughput.

Pigsty uses pgBackRest as its backup engine and ships production-oriented defaults for those concerns. This page describes the engine, repository abstraction, archive path, scheduler, and primary-aware execution model.


Backup Engine: pgBackRest

Pigsty uses pgBackRest for three responsibilities: create base backups with backup, receive WAL with archive-push, and restore data with restore plus archive-get.

Relevant capabilities include:

  • Parallelism: backup, archive, and restore operations can use multiple processes.
  • Backup chains: full, differential, incremental, and block incremental backups reduce repeated transfer and storage.
  • Compression and encryption: zstd compression and AES-256-CBC repository encryption are built in.
  • Repository backends: POSIX filesystems, S3-compatible services such as Silo and MinIO, Azure, GCS, and SFTP are supported by pgBackRest.
  • Bundling: small files can be packed into larger repository objects, reducing object-storage overhead.

pgBackRest separates cluster histories using a stanza. Pigsty maps the stanza name directly to pg_cluster, allowing multiple clusters to share one storage service without sharing a backup identity:

repository
├── backup/
│   ├── pg-meta/          # base backups for pg-meta
│   └── pg-test/          # base backups for pg-test
└── archive/
    ├── pg-meta/          # archived WAL for pg-meta
    └── pg-test/          # archived WAL for pg-test

Repository Abstraction

Two parameters define repository selection. pgbackrest_method chooses one repository name, and pgbackrest_repo is a dictionary of candidate definitions. Pigsty v4.5.0 renders only the selected pgbackrest_repo[pgbackrest_method] entry as pgBackRest repo1; listing both local and minio does not enable two active repositories.

pgbackrest_method: local          # local, minio, or a custom key below
pgbackrest_repo:
  local:
    path: /pg/backup
    retention_full_type: count
    retention_full: 2             # retain two full backups; a third may exist before expiration
  minio:
    type: s3
    s3_endpoint: sss.pigsty
    s3_region: us-east-1
    s3_bucket: pgsql
    s3_key: pgbackrest
    s3_key_secret: S3User.Backup
    s3_uri_style: path
    path: /pgbackrest
    storage_port: 9000
    storage_ca_file: /etc/pki/ca.crt
    block: y
    bundle: y
    bundle_limit: 20MiB
    bundle_size: 128MiB
    cipher_type: aes-256-cbc
    cipher_pass: pgBackRest       # replace this default secret in production
    retention_full_type: time
    retention_full: 14

The presets intentionally differ. local favors simplicity and fast local restore; it is unencrypted, unbundled, and retained by full-backup count. minio targets a remote Silo or compatible S3 repository, enabling encryption, bundles, block incremental backup, and time-based retention.

Rendering is mechanical: underscores in the chosen repository’s keys become hyphens and each key gets a repo1- prefix in /etc/pgbackrest/pgbackrest.conf. A custom cloud repository can therefore use pgBackRest options directly:

pgbackrest_method: s3
pgbackrest_repo:
  s3:
    type: s3                         # repo1-type=s3
    s3_endpoint: s3.us-west-1.amazonaws.com
    s3_region: us-west-1
    s3_bucket: <your_bucket>
    s3_key: <your_access_key>
    s3_key_secret: <your_secret>
    s3_uri_style: host
    path: /pgbackrest
    cipher_type: aes-256-cbc
    cipher_pass: <your_password>
    retention_full_type: time
    retention_full: 90

See Backup Repository for Silo, external S3-compatible storage, versioning, object locking, TLS, and credential details.


Archiving and Scheduling

When pgbackrest_enabled is true, as it is by default, the Patroni templates configure:

archive_mode: 'on'
archive_timeout: 300
archive_command: 'pgbackrest --stanza=<cluster> archive-push %p'

Base backups enter the system in two ways:

  • Initial backup: after bootstrapping a top-level primary, Pigsty attempts a backup when pgbackrest_init_backup is true. The task ignores backup failure and writes /etc/pgbackrest/initial.done only after success, so the marker means “completed,” not merely “attempted.”
  • Scheduled backup: pg_crontab installs jobs in the database superuser’s crontab. Its role default is an empty list; standard example configurations usually add a daily 01:00 full backup.
pg_crontab: [ '00 01 * * * /pg/bin/pg-backup full' ]

pg-backup [full|diff|incr] is a small wrapper around pgbackrest backup. With no argument it requests an incremental backup, which pgBackRest promotes to a full backup if no usable full exists.


Backups Follow the Primary

pgBackRest and the same scheduled job are installed on every PostgreSQL node, but pg-backup checks /pg/bin/pg-role and only proceeds on the current primary. Replicas fail fast rather than writing a competing backup.

That design decouples the backup schedule from the HA topology:

  • all members receive the same repository configuration and crontab;
  • after failover, the new primary becomes eligible for subsequent backups and WAL archiving without rewriting the schedule;
  • one current primary owns the authoritative write flow to a stanza.

With a non-local repository, Pigsty also adds pgBackRest after basebackup in Patroni’s create_replica_methods. Patroni tries basebackup first; if that method fails, it can restore a replica from the repository with pgbackrest --delta restore, shifting the copy load away from the primary.


Performance Defaults

The shipped pgBackRest template favors light production overhead and aggressive restore throughput:

Setting v4.5.0 behavior Rationale
Compression compress-type=zst Balance compression ratio and throughput
Backup/archive workers One quarter of CPU, clamped to 2–4 Limit competition with the database
Restore workers All detected CPU, capped at 8 Minimize restore time
Asynchronous archive archive-async=y, spool under /pg/spool Batch transfer without synchronous object-store latency
Archive queue limit archive-push-queue-max=4GiB Bound local spool growth
Fast backup start start-fast=y Request an immediate checkpoint
Incremental restore delta=y Reuse destination files that already match

The 4 GiB queue is a safety tradeoff: if the repository remains unavailable and the queue exceeds the limit, pgBackRest can discard queued archive files. PostgreSQL continues running, but the WAL archive becomes incomplete and a new full backup is needed to establish a new recovery chain. See How PITR Works.


Observability

When both backup and exporter settings are enabled, pgbackrest_exporter runs on each PostgreSQL node and exposes metrics on port 9854. The monitoring stack uses those metrics for backup age, type, size, duration, and error visibility.

Useful diagnostic entry points include:

Entry Purpose
pb info Shell helper for pgbackrest info using the configured stanza
/pg/log/pgbackrest/ pgBackRest backup, archive, and restore logs
`pg-backup full diff

See Backup Administration for operational checks, then PITR Tradeoffs for policy design.

3 - PITR Tradeoffs

Repository location determines the failure domain, retention determines the recovery window, and backup frequency shapes restore time. Together they define a backup policy.

A backup is an insurance policy. Its premium is storage, network traffic, and operational work; its benefit is how much data can be recovered and how quickly service can return. There is no universal free policy: more history normally needs more capacity, while a shorter RTO normally needs newer backups and tested procedures.

Designing a policy means answering three questions: where is the repository, how long is history retained, and how often are backups taken?


Where: Choose the Failure Domain

Repository location is the most important decision because it defines which disasters the backup survives.

A local repository (pgbackrest_method: local) stores backups on the primary’s local filesystem. It is simple, fast, and has no remote service dependency. But data and backup normally share one host failure domain: loss of the machine, disk, or filesystem can destroy both. Local backup protects well against logical errors, but not total host loss unless /pg/backup is deliberately placed on independent storage.

An object-storage repository (pgbackrest_method: minio or a custom S3 definition) sends backups to Silo or S3. It becomes an independent disaster-recovery copy only when deployed outside the database host or site failure domain. Pigsty’s minio preset also enables AES-256-CBC repository encryption, bundling, and block incremental backup. Recovery throughput then depends on the network and storage service, and that service adds operational responsibility.

Scenario Recommended repository Reason
Development, test, demo local Minimal dependencies; rebuild is acceptable
Production Dedicated Silo or compatible S3 storage Independent failure domain and encrypted repository
Cloud deployment Managed S3-compatible or cloud object storage supported by pgBackRest Independent storage and lower operational burden
Ransomware/compliance Versioned storage plus correctly configured object lock/retention Prevent privileged database-host access from deleting protected versions

The backup repository is itself sensitive business data. Change the default access keys and cipher_pass, restrict access, protect credentials separately from the database hosts, and verify any object-lock policy. See Backup Repository.


How Long: Capacity and Recovery Window

Longer retained history generally consumes more storage, but compression, deduplication, block incremental backup, database change rate, and the mix of full/differential/incremental backups determine the actual amount. Measure real backup and WAL growth instead of relying on a fixed multiplier.

For an illustrative 100 GB database changing by 10 GB per day, before compression:

  • Daily full, retain two (local preset policy): about 200 GB of full backups plus WAL, commonly giving roughly a one-to-two-day window when every job succeeds.
  • Weekly full, daily incremental, retain full history by 14 days (minio preset policy): the oldest surviving weekly chain commonly produces roughly 14–21 days of coverage. Capacity must include multiple full backups, their incrementals, archived WAL, and transient retention-plus-one behavior during expiration.

The precise window is not the configuration number alone. It runs from the oldest usable backup chain to the newest WAL present in the surviving repository. pgBackRest’s time retention removes an old full only when another qualifying full can satisfy the period, and related incrementals and WAL follow the retained full chains. Check pig pb info, monitor archive health, and prove coverage with a restore.

Choose a window long enough to cover the delay between an error occurring and being detected. A dropped table may be noticed in minutes; slow corruption or a month-end reconciliation failure can take weeks to surface.


How Often: Backup Frequency and RTO

Restore time has two main components: restore a backup chain, then replay WAL to the target. Backup size and storage throughput shape the first; the distance between the chosen backup and target shapes the second.

WAL replay is largely serial. On a write-heavy database, restoring from a weekly full immediately before the next full can require nearly a week of replay. Daily incremental backups reduce that replay distance while transferring only changes since the previous backup. They still depend on a valid chain, so monitor and test the entire chain rather than only the newest file.

A useful rule is: within the available backup window and production load budget, take backups often enough that measured restore time meets the RTO.


Pigsty Presets

Pigsty provides two candidate repository definitions, but pgbackrest_method selects one for the generated repo1 configuration.

Standard policy: local repository and daily full backup. It is simple and restores through local I/O, making it suitable for development or environments where host-level disaster recovery is provided separately:

pgbackrest_method: local
pg_crontab: [ '00 01 * * * /pg/bin/pg-backup full' ]
# Local preset retains two full backups; actual coverage depends on successful jobs and WAL continuity.

Production policy: remote Silo/S3 repository, weekly full, daily incremental. It separates the repository failure domain and uses the encrypted minio preset:

pgbackrest_method: minio
pg_crontab:
  - '00 01 * * 1 /pg/bin/pg-backup full'
  - '00 01 * * 2,3,4,5,6,7 /pg/bin/pg-backup'
# The preset retains full history by 14 days; weekly fulls commonly yield about 14–21 days.

Do not describe the default pgbackrest_repo dictionary as a “dual-repository” setup: it contains alternative definitions, and the template renders only pgbackrest_repo[pgbackrest_method] as repo1. A real multi-repository pgBackRest design requires explicit advanced configuration and an independently tested backup, expiration, and restore workflow; the two Pigsty presets alone do not create it.

Use Backup Policy for capacity modelling and schedule details.


A Backup Is Proven by Restore

Monitoring a successful backup job is necessary but insufficient. Add clone restore drills to routine operations so you can answer:

  1. Is the chain usable? Restore it end to end and validate data.
  2. What is the measured RTO? Database size and WAL volume change over time.
  3. Can the on-call operator execute the runbook? The first full exercise should not happen during an incident.

A clone recovery leaves the source cluster online but overwrites the designated destination cluster, so verify the exact target and use disposable infrastructure. See Declarative Recovery for the recovery interface.

4 - Declarative Recovery

Declare the desired pg_pitr recovery target and let pgsql-pitr.yml or pig orchestrate the recovery workflow.

The value of a backup system is realized at restore time, often during an incident when every minute matters. A traditional PITR procedure requires a long sequence of coupled manual steps: pause HA, stop PostgreSQL, prepare recovery settings, restore the backup, replay WAL, validate the target, rebuild metadata, and start the cluster again.

Pigsty applies the same approach used by declarative configuration to recovery: declare the recovery target, then let the orchestration tools stop the cluster, restore the data, replay WAL, and return control to the operator.


Declare a Recovery Target

Describe the target with the pg_pitr parameter and execute it with pgsql-pitr.yml. The most common form restores a cluster to a specific time:

./pgsql-pitr.yml -l pg-meta -e '{"pg_pitr": { "time": "2026-07-11 10:00:00+08", "action": "promote" }}'

The six recovery target types and the rest of the recovery behavior are expressed through fields in this parameter:

pg_pitr:                           # Recovery declaration; every field is optional
  cluster: pg-meta                 # Source backup stanza; defaults to this cluster
  type: time                       # default | time | xid | lsn | name | immediate
  time: '2026-07-11 10:00:00+08'   # Mutually exclusive with xid, lsn, and name
  exclusive: false                 # Stop before the target; inclusive by default
  action: promote                  # Explicit promotion; a targeted restore defaults to pause
  timeline: latest                 # Target timeline; latest by default
  set: latest                      # Starting backup set; selected automatically by default
  repo: { ... }                    # Temporary repository definition when not using local config
  backup: false                    # Move the old data directory to /pg/data-backup first
  archive: true                    # Preserve archiving; exploratory recovery can set false
  db_include: [ ... ]              # Restore only selected databases
  data: /pg/data                   # Destination data directory

See Restore Operations for the complete field reference and examples.


What the Playbook Does

pgsql-pitr.yml turns the manual recovery workflow into six stages and supports Ansible tags for staged execution:

Stage Action
print Print the source cluster, target, and restore command; this stage reports the plan and does not prompt for confirmation
pause Run patronictl pause so Patroni does not intervene during maintenance
stop Stop Patroni and PostgreSQL on replicas, then on the primary
pitr Render recovery settings, run an incremental pgBackRest restore, start PostgreSQL to replay WAL, wait for consistency, and print control data
etcd Remove stale cluster metadata from etcd so old and new timelines are not mixed
start Start Patroni again, resume HA management, and rebuild replicas

Several details are important:

  • Incremental restore: pgBackRest uses delta, so it rewrites only files that differ from the backup. For large databases, this can reduce RTO substantially.
  • Verification, not assumption: the playbook prints checkpoint LSN, timeline, and NextXID data from pg_controldata; an operator must still verify that the recovered business state is correct.
  • Rollback copy: with backup: true, the original data directory is moved to /pg/data-backup before recovery. A later run with backup: true removes an existing /pg/data-backup, so this is not a versioned snapshot store.
  • Staged execution: run -t down, -t pitr, and -t up separately when you want an operator checkpoint between phases. Completion of the pitr phase means PostgreSQL reached a consistent recovery state; for a time, XID, LSN, or named target, also confirm WAL replay reached that target.

The action field controls what happens at the target: promote opens a new timeline, pause waits at the target for inspection, and shutdown stops PostgreSQL there. A targeted recovery defaults to pause when action is omitted. To preserve a manual gate for pause or shutdown, run the stages separately; a one-shot recovery should choose promote explicitly. The playbook performs the mechanical workflow, but it cannot decide whether the recovered data is correct.


Command-Line Recovery with pig

The pig CLI provides single-instance PITR orchestration directly on a database node, without requiring the management node or an Ansible environment:

pig pitr -t "2026-07-11 10:00:00+08"    # Recover to a point in time
pig pitr --xid 250000 -X                # Stop before transaction 250000
pig pitr -d                             # Replay through the WAL archive
pig pitr -I --no-restart                # Prepare immediate recovery and leave PostgreSQL stopped

pig pitr validates the target, stanza, and available backups; stops Patroni and PostgreSQL; performs the restore; optionally starts PostgreSQL; and prints post-recovery instructions. For a Patroni-managed data directory, Patroni remains stopped afterward. Validate the data, then use pig pt start to return the instance to HA management. This single-node workflow does not clear etcd, rebuild replicas, or automatically rejoin the cluster, and it refuses destructive forced shutdown unless --force-stop is supplied explicitly.

The lower-level pig pb commands wrap pgBackRest: pb info lists backups, pb backup creates a backup, and pb restore performs a raw restore. There is a deliberate safety boundary: pig pb restore refuses to run while Patroni still manages the instance, because Patroni could restart PostgreSQL during the restore. Use pig pitr or pgsql-pitr.yml for Patroni-managed instances.


In-Place and Clone Recovery

The same mechanism supports two different workflows:

Dimension In-place recovery Clone recovery
Method Roll the production cluster back Restore a source backup into a different cluster
Downtime Required during recovery The source production cluster remains online
Effect Discards all writes after the target Does not affect the source; the destination is overwritten and can be retried
Best for Whole-cluster corruption or disaster recovery Recovering deleted objects, audit work, and recovery drills

For a clone recovery, the cluster field names the source backup stanza. This example restores the historical state of pg-meta into pg-test without stopping the source cluster:

./pgsql-pitr.yml -l pg-test -e '{"pg_pitr": { "cluster": "pg-meta", "time": "2026-07-11 10:00:00+08", "archive": false, "action": "promote" }}'

Exporting an accidentally deleted table from the clone and importing it into production is generally safer than rolling the entire production cluster back. See Clone a Database Cluster for the complete workflow and cleanup steps.


After Recovery

Recovery completion is not the end of the incident. Include these steps in the closeout checklist:

  1. New timeline, new backup: after promotion, create a full backup with pg-backup full so a recoverable window exists on the new timeline.
  2. Archiving state: if an exploratory restore used archive: false, restore normal archiving as described in Post-Recovery.
  3. Clone cleanup: a clone’s cluster identity and source backup stanza do not match. Recreate the destination stanza before enabling its own backups; see Clone a Database Cluster.

The tools execute the procedure; operators still decide the target, whether to restore in place or into a clone, and whether the recovered data is correct. Continue with PITR Scenarios for that decision framework.

5 - PITR Scenarios

How to choose a recovery target and workflow for accidental DML, dropped objects, defective releases, investigations, and site loss — and why recovery drills must be routine.

During an incident, the most expensive resource is often decision time. Pigsty can orchestrate the mechanical recovery steps, but an operator must still answer three questions: what is the target, should recovery be in place or into a clone, and how will the result be validated?

Read and rehearse this framework before an incident.


Decision Framework

Scenario Typical problem Recommended workflow Target
Accidental DML DELETE or UPDATE affects the wrong rows Clone, validate, then copy back data time / xid
Dropped table, schema, or database DROP or an incorrect migration Clone, validate, then copy back objects time / name
Defective release or batch corruption Software writes incorrect data for a period Clone and compare before choosing repair or cutover time / xid
Audit, investigation, or forensics Inspect historical state Clone and hold at the target for inspection time / lsn
Whole-cluster or site loss Hosts or storage are gone or encrypted Recover in place on replacement infrastructure default / time

Two principles apply throughout:

  • Stop the damage first. Pause the defective application or remove its write access before choosing a target. The window is moving, but a rushed restore to the wrong cluster can cause a second incident.
  • Prefer a clone while production is usable. It leaves the source untouched, supports repeated target selection, and allows validation before export or cutover. It does overwrite the designated destination cluster. In-place recovery is appropriate when the whole cluster is unusable or the business has explicitly accepted rolling every database back.
flowchart TD
    A["Data error detected"] --> B["Contain the source of bad writes"]
    B --> C{"Can production still serve?"}
    C -->|Yes| D["Clone recovery<br/>validate and copy back or cut over"]
    C -->|No| E["In-place recovery<br/>or rebuild on new infrastructure"]
    D --> F["Validate, take a new backup, review the incident"]
    E --> F

Accidental DML

A DELETE without WHERE, an incorrect UPDATE, or a defective batch job is the most common PITR use case.

First locate the error using application logs, PostgreSQL logs, metrics, or audit records. A timestamp is usually sufficient. If the exact transaction ID is known, xid plus exclusive: true can stop immediately before that transaction.

# If the deletion occurred around 10:15, clone the state from 10:14
./pgsql-pitr.yml -l pg-test -e '{"pg_pitr": { "cluster": "pg-meta", "time": "2026-07-11 10:14:00+08", "archive": false, "action": "promote" }}'

# If the deleting transaction was 250000, stop immediately before it
./pgsql-pitr.yml -l pg-test -e '{"pg_pitr": { "cluster": "pg-meta", "xid": "250000", "exclusive": true, "archive": false, "action": "promote" }}'

Validate the recovered rows, then copy only the required data back with pg_dump, COPY, or an application-specific reconciliation procedure. If a configured delayed cluster is still inside its delay window, reading from it may be faster than PITR.


Dropped Objects

The same approach applies to DROP TABLE, DROP DATABASE, or a migration executed in the wrong environment, with an even stronger preference for a clone. Rolling the entire production cluster back to recover one object also discards every legitimate write after the target.

Restore a separate destination to before the DDL, validate the object, export it with pg_dump, and import it into production. For planned high-risk changes, create a named restore point with pg_create_restore_point() beforehand; a name target then removes timestamp ambiguity.


Defective Release or Batch Corruption

When a faulty release corrupts data for hours, the challenge is usually identifying the last clean state and the full impact. A clone provides a clean comparison set. Restore repeatedly to candidate times, compare it with production, and decide whether to copy back corrected rows or cut over to a recovered cluster.

This decision needs application-owner validation: a successful PostgreSQL restore proves consistency at a target, not that the target represents correct business state.


Audit and Investigation

Questions such as “what was this balance at month end?” require historical state. Restore into a separate destination, stop at a time, LSN, XID, or named restore point, and inspect without altering the source.

action: pause is the targeted-restore default and holds recovery at the target for inspection; it does not itself configure read-only access or create a separate cluster. The inventory limit and cluster source field determine the destination workflow. Run -t down, -t pitr, and -t up separately when you need an operator gate before promotion, and enforce read-only access explicitly if the investigation requires it. immediate means “stop at the first consistent point,” not “choose a historical timestamp.”


Site Loss

If every database host is destroyed or encrypted, HA cannot help. Recovery requires a repository and the other control-plane assets to have survived outside that failure domain. That survivor can be Silo/S3, another protected host or filesystem, or another tested pgBackRest backend; a remote object store is recommended but the essential property is independent failure-domain survival.

Rebuild hosts, restore the declarative inventory, credentials, and PKI, point the cluster at the surviving repository, then restore through the end of archived WAL:

./pgsql-pitr.yml -l pg-meta -e '{"pg_pitr": {"action": "promote"}}'

Inventory and backup data are necessary but not sufficient. Preserve installation media or package repositories, repository credentials and encryption passwords, CA material, custom files, DNS dependencies, and an independently accessible runbook. Keep secrets encrypted and separate from both the database hosts and ordinary source control.


Make Recovery a Routine Drill

The first end-to-end execution of any of these workflows should not occur during a production incident. Use a disposable destination to rehearse clone recovery regularly and after material architecture changes. Measure three outcomes:

  1. Usability: can the backup and complete WAL chain be restored and validated?
  2. RTO: how long does the actual restore and replay take now?
  3. Operator readiness: can the on-call engineer identify source and destination, select a target, and follow the safety gates?

See Restore Operations and Clone a Database Cluster for the task-level runbooks.