My nightly pgBackRest backup made Postgres slow. A memory ceiling helped more than an I/O cap

Short answer: a pgBackRest backup can slow PostgreSQL even when disk and CPU look fine, because streaming the whole database through the page cache evicts Postgres's hot pages. Running the backup in its own cgroup with MemoryHigh (plus a read-bandwidth cap) makes it recycle its own cache. On my database, kswapd scanning dropped 3–9×, statement timeouts went to zero and the worst page load fell from 28.8 s to 16.1 s. It didn't remove the slowdown completely; the next lever is delta.

This is a write-up of one week of backup tuning on the Postgres instance behind my products. Everything below comes from my ops notes; numbers are what the tools printed.

What does the setup look like?

• One PostgreSQL primary, managed by Pigsty, about 46 GB of data. No standby, so backups can only come from the primary.
• pgBackRest stanza pg-meta, scheduled by cron (times in Asia/Shanghai):

Why two full backups on Sunday? pgBackRest writes to one repository per run and can't copy between repositories, so the offsite copy needs its own read of the data. Daily diffs were 21–44 GB against a 46 GB database, so switching the offsite job to diffs would save almost nothing.

What was the symptom?

On Sunday 2026-10-18, during the backup window, page loads on Vaneform, my website traffic checker that reads from this database, went from report queries of ~12 ms to several seconds: 783 slow page loads, the worst at 28.8 s, plus 2 batch statement timeouts.

Why wasn't it the disk?

The obvious suspect was I/O. The numbers said otherwise:

• Disk utilization 13–23%, await about 1 ms.
• CPU 75–80% idle.
• kswapd scanning about 6,000 pages per second.

The backup was streaming 46 GB through roughly 6 GB of page cache. Every file it read pushed out pages Postgres had cached, so queries that normally hit memory went to disk and waited behind reclaim. The disk wasn't saturated; the cache was being thrashed.

How does a memory ceiling help?

On cgroup v2, page cache is charged to the cgroup of the process that reads the file. If the backup runs in its own cgroup with MemoryHigh, the kernel reclaims the backup's own cache first once it reaches the limit, instead of letting it grow over everything else.

I wrapped every backup job in a small runner that starts a transient systemd unit:

#!/bin/bash
# Run one scheduled pgBackRest job gently on the database host.
set -euo pipefail
job=$1   # ops-full | ops-diff | r2-full
command=(/usr/bin/pgbackrest --stanza=pg-meta --repo=2 --type=diff backup)  # per job

read_cap=${BAY_PGBACKREST_READ_MBPS:-20}M
read_iops=${BAY_PGBACKREST_READ_IOPS:-600}
memory_high=${BAY_PGBACKREST_MEMORY_HIGH:-1G}

# A later job waits for the running one (Sunday R2 full waits for the ops full).
exec /usr/bin/flock --wait 21600 /run/lock/bay-pgbackrest.lock \
  /usr/bin/systemd-run --unit="bay-pgbackrest-$job-$(date +%Y%m%d%H%M)" \
    --wait --collect --quiet \
    -p User=postgres -p Group=postgres \
    -p Nice=19 -p IOSchedulingClass=idle -p CPUWeight=20 -p IOWeight=10 \
    -p MemoryHigh="$memory_high" \
    -p "IOReadBandwidthMax=/pg/data $read_cap" -p "IOReadIOPSMax=/pg/data $read_iops" \
    -p RuntimeMaxSec=6h \
    -- "${command[@]}"

The flock keeps it to one job at a time. Limits come from environment variables, so tuning doesn't need a code change. The runner lives in Bay, the small ops console I built to run my servers, and cron calls it instead of pgBackRest directly.

Did it work?

First night on the runner, Monday 2026-10-19, differential backup:

• Backup: completed successfully, 30.8 GB, 02:30:01 to 03:24:51. That's 55 minutes, up from ~36 minutes before the cap.
• Limits live: a probe unit with IOReadBandwidthMax=5M read an uncached file at exactly 5.0 MB/s, and io.max showed rbps=5000000.

sar between 02:30 and 03:30, uncapped Sunday full vs capped Monday diff:

And on the app side: 167 page loads over 1.5 s (an undercount, since one pod's logs were lost to a restart), worst 16.1 s, zero statement timeouts, no 5xx. The night before had 783 slow loads and a 28.8 s worst case.

Verdict: clearly better, not solved. Reclaim is still far above the quiet baseline of about 300 pages per second, and because the capped backup runs longer, the slow period now peaks late in the window.

Which systemd limits did nothing?

IOSchedulingClass=idle and IOWeight did nothing on this VM. The disk's scheduler was none, and those settings need BFQ. Check /sys/block/<dev>/queue/scheduler before you count on them. Only the bandwidth/IOPS caps and the memory ceiling acted.

Also, the "memory peak" systemd logs when a unit stops isn't useful here; it reported tiny values for unrelated units too. Read memory.peak inside the unit's cgroup while it runs.

What's the next lever?

Pigsty's generated pgbackrest.conf sets delta=y in [global]. For backups, delta means pgBackRest checksums every file to find changes instead of trusting timestamps, so every differential reads the entire database.

The least invasive change I've found is an override that only affects backups:

# /etc/pgbackrest/conf.d/bay-ops.conf
[global:backup]
delta=n

Restore keeps delta=y, which is where it's actually useful. My expectation, not yet measured, is that diffs would read about 31 GB of changed files instead of 46 GB and the window would shrink from ~55 to ~30 minutes. A second option is a MemoryLow drop-in on patroni.service, so global reclaim prefers the backup's cache over Postgres's.

How do I check a backup is usable?

Run verify against a specific set, under the same limits:

sudo -u postgres pgbackrest --stanza=pg-meta --repo=2 --set=<label> verify

The 2026-10-18 full set (46.2 GB, 8.1 GB stored) verified in 77 seconds. Verifying is cheap compared to discovering a broken backup during a restore.

FAQ

Why does pgBackRest slow down PostgreSQL when disk usage is low?

Reading the whole database pushes it through the page cache, which evicts pages PostgreSQL was serving from memory. In my case disk was only 13–23% busy while kswapd scanned about 6,000 pages per second.

Does IOSchedulingClass=idle work for backups?

Only with the BFQ I/O scheduler. On a VM whose disk scheduler is none, it has no effect. Bandwidth and IOPS caps (IOReadBandwidthMax, IOReadIOPSMax) work without BFQ.

What does delta=y do for pgBackRest backups?

It makes pgBackRest checksum every file to detect changes instead of relying on timestamps and sizes. That's safer, but every differential then reads the full database. Setting delta=n under [global:backup] keeps delta for restores only.

How much slower is a capped backup?

With a 20 MB/s read cap, my differential took 55 minutes instead of about 36. The cap mostly clips peaks; before the change the backup averaged about 19 MB/s with peaks of 38 MB/s.