pgBackRest Exporter

Prometheus exporter for pgBackRest.

The metrics are collected based on result of pgbackrest info --output json command. By default, the metrics are collected for all stanzas received by command. You can specify stanzas to collect metrics. You need to run exporter on the same host where pgBackRest was installed or inside Docker.

Grafana dashboard

To get a dashboard for visualizing the collected metrics, you can use a ready-made dashboard pgBackRest Exporter Dashboard or make your own.

Collected metrics

Stanza metrics

Metric	Description	Labels	Additional Info
pgbackrest_stanza_backup_compete_bytes	completed size for backup in progress	stanza
pgbackrest_stanza_backup_total_bytes	total size for backup in progress	stanza
pgbackrest_stanza_lock_status	current stanza lock status	stanza	Values description: 0 - no active operation with stanza, 1 - one of the commands is running for stanza: backup, expire or stanza-*.
pgbackrest_stanza_status	current stanza status	stanza	Values description: 0 - ok, 1 - missing stanza path, 2 - no valid backups, 3 - missing stanza data, 4 - different across repos, 5 - database mismatch across repos, 6 - requested backup not found, 99 - other.

Repository metrics

Metric	Description	Labels	Additional Info
pgbackrest_repo_status	current repository status	cipher, repo_key, stanza	Values description: 0 - ok, 1 - missing stanza path, 2 - no valid backups, 3 - missing stanza data, 4 - different across repos, 5 - database mismatch across repos, 6 - requested backup not found, 99 - other

Backup metrics

Metric	Description	Labels	Additional Info
pgbackrest_backup_annotations	number of annotations in backup	backup_name, backup_type, database_id, block_incr, repo_key, stanza
pgbackrest_backup_databases	number of databases in backup	backup_name, backup_type, block_incr, database_id, repo_key, stanza
pgbackrest_backup_references	number of references to other backups (backup reference list)	backup_name, backup_type, block_incr, database_id, ref_backup, repo_key, stanza
pgbackrest_backup_duration_seconds	backup duration in seconds	backup_name, backup_type, block_incr, database_id, repo_key, stanza, start_time, stop_time
pgbackrest_backup_error_status	backup error status	backup_name, backup_type, block_incr, database_id, repo_key, stanza	Values description: 0 - backup doesn't contain page checksum errors, 1 - backup contains one or more page checksum errors. To display the list of errors, you need manually run the command like pgbackrest info --stanza stanza --set backup_name --repo repo_key.
pgbackrest_backup_info	backup info	backrest_ver, backup_name, backup_type, block_incr, database_id, lsn_start, lsn_stop, pg_version, prior, repo_key, stanza, wal_start, wal_stop	Values description: 1 - info about backup is exist.
pgbackrest_backup_delta_bytes	amount of data in the database to actually backup	backup_name, backup_type, block_incr, database_id, repo_key, stanza
pgbackrest_backup_size_bytes	full uncompressed size of the database	backup_name, backup_type, block_incr, database_id, repo_key, stanza
pgbackrest_backup_repo_delta_bytes	compressed files size in backup	backup_name, backup_type, block_incr, database_id, repo_key, stanza
pgbackrest_backup_repo_size_bytes	full compressed files size to restore the database from backup	backup_name, backup_type, block_incr, database_id, repo_key, stanza
pgbackrest_backup_repo_delta_map_bytes	size of block incremental delta map	backup_name, backup_type, block_incr, database_id, repo_key, stanza
pgbackrest_backup_repo_size_map_bytes	size of block incremental map	backup_name, backup_type, block_incr, database_id, repo_key, stanza

Last backup metrics

Metric	Description	Labels	Additional Info
pgbackrest_backup_since_last_completion_seconds	seconds since the last completed full, differential or incremental backup	backup_type, block_incr, stanza
pgbackrest_backup_last_annotations	number of annotations in the last full, differential or incremental backup	backup_type, block_incr, stanza
pgbackrest_backup_last_databases	number of databases in the last full, differential or incremental backup	backup_type, block_incr, stanza
pgbackrest_backup_last_references	number of references to other backups (backup reference list) in the last full, differential or incremental backup	backup_type, block_incr, ref_backup, stanza
pgbackrest_backup_last_duration_seconds	backup duration for the last full, differential or incremental backup	backup_type, block_incr, stanza
pgbackrest_backup_last_error_status	error status in the last full, differential or incremental backup	backup_type, block_incr, stanza
pgbackrest_backup_last_delta_bytes	amount of data in the database to actually backup in the last full, differential or incremental backup	backup_type, block_incr, stanza
pgbackrest_backup_last_size_bytes	full uncompressed size of the database in the last full, differential or incremental backup	backup_type, block_incr, stanza
pgbackrest_backup_last_repo_delta_bytes	compressed files size in the last full, differential or incremental backup	backup_type, block_incr, stanza
pgbackrest_backup_last_repo_size_bytes	full compressed files size to restore the database from the last full, differential or incremental backup	backup_type, block_incr, stanza
pgbackrest_backup_last_repo_size_map_bytes	size of block incremental map in the last full, differential or incremental backup	backup_type, block_incr, stanza
pgbackrest_backup_last_repo_delta_map_bytes	size of block incremental delta map in the last full, differential or incremental backup	backup_type, block_incr, stanza

WAL metrics

Metric	Description	Labels	Additional Info
pgbackrest_wal_archive_status	current WAL archive status	database_id, pg_version, repo_key, stanza, wal_max, wal_min	Values description: 0 - any one of WALMin and WALMax have empty value, there is no correct information about WAL archiving, 1 - both WALMin and WALMax have no empty values, there is correct information about WAL archiving.

Exporter metrics

Metric	Description	Labels	Additional Info
pgbackrest_exporter_build_info	information about pgBackRest exporter	branch, goarch, goos, goversion, revision, tags, version
pgbackrest_exporter_status	pgBackRest exporter get data status	stanza	Values description: 0 - errors occurred when fetching information from pgBackRest, 1 - information successfully fetched from pgBackRest.

Additional description of metrics

For pgbackrest_*_last_* metrics for differential backups (backup_type="diff") the following logic is applied:

• if the last backup was full, the metric will take full backup value;
• otherwise, the value will be set.

For pgbackrest_*_last_* metrics for incremental backups (backup_type="incr") the following logic is applied:

• if the last backup was full or differential, the metric will take full or differential backup value;
• otherwise, the value will be set.

For pgbackrest_exporter_status metric the following logic is applied:

• if the information is collected for all available stanzas, the stanza label value will be all-stanzas;
• if the information is collected for all available stanzas except excluded, the stanza label value will be all-stanzas-except-excluded;
• otherwise, the stanza name will be set.

If pgbackrest_stanza_lock_status metric is 1, than one of the commands is running for stanza: backup, expire or stanza-*. With a very high probability it is backup/expire.

For pgBackRest >= v2.48 it is possible to determine whether the backup is running:

• if pgbackrest_stanza_backup_compete_bytes and pgbackrest_stanza_backup_total_bytes metric s are different from 0 and pgbackrest_stanza_lock_status metric is equal to 1, then a backup is running;
• if pgbackrest_stanza_backup_compete_bytes and pgbackrest_stanza_backup_total_bytes metric s are equal to 0 and pgbackrest_stanza_lock_status metric is equal to 1, then one of the commands is running for stanza: expire or stanza-*. With a very high probability it is expire.

Compatibility with pgBackRest versions

The number of collected metrics may vary depending on pgBackRest version.

For different versions, some metrics may not be collected or have insignificant label values:

• pgBackRest < v2.48

  The following metrics will always be 0:

  • pgbackrest_stanza_backup_compete_bytes,
  • pgbackrest_stanza_backup_total_bytes.

• pgBackRest >= v2.45

  For pgbackrest_backup_repo_size_bytes metric the values will be 0 for block incremental backups.

• pgBackRest < v2.44

  The following metrics will always be 0:

  • pgbackrest_backup_repo_size_map_bytes,
  • pgbackrest_backup_repo_delta_map_bytes.

  For pgbackrest_backup_* metrics the label will be block_incr="n".

• pgBackRest < v2.41

  The following metrics will always be 0:

  • pgbackrest_backup_databases,
  • pgbackrest_backup_last_databases,
  • pgbackrest_backup_annotations,
  • pgbackrest_backup_last_annotations.

• pgBackRest < v2.38

  For pgbackrest_backup_info metric the labels will be lsn_start="-" and lsn_stop="-".

• pgBackRest < v2.36

  The following metrics will always be 0:

  • pgbackrest_backup_error_status.

• pgBackRest < v2.32

  The following metrics will always be 0:

  • pgbackrest_repo_status.

  For all metrics the label will be repo_key="0".

Getting Started

Building and running

git clone https://github.com/woblerr/pgbackrest_exporter.git
cd pgbackrest_exporter
make build
./pgbackrest_exporter <flags>

Available configuration flags:

./pgbackrest_exporter --help
usage: pgbackrest_exporter [<flags>]


Flags:
  -h, --[no-]help                Show context-sensitive help (also try --help-long and --help-man).
      --web.telemetry-path="/metrics"  
                                 Path under which to expose metrics.
      --web.listen-address=:9854 ...  
                                 Addresses on which to expose metrics and web interface. Repeatable for multiple addresses. Examples: `:9100` or `[::1]:9100` for http, `vsock://:9100` for vsock
      --web.config.file=""       Path to configuration file that can enable TLS or authentication. See:
                                 https://github.com/prometheus/exporter-toolkit/blob/master/docs/web-configuration.md
      --collect.interval=600     Collecting metrics interval in seconds.
      --backrest.config=""       Full path to pgBackRest configuration file.
      --backrest.config-include-path=""  
                                 Full path to additional pgBackRest configuration files.
      --backrest.stanza-include="" ...  
                                 Specific stanza for collecting metrics. Can be specified several times.
      --backrest.stanza-exclude="" ...  
                                 Specific stanza to exclude from collecting metrics. Can be specified several times.
      --backrest.backup-type=""  Specific backup type for collecting metrics. One of: [full, incr, diff].
      --[no-]backrest.database-count  
                                 Exposing the number of databases in backups.
      --backrest.database-parallel-processes=1  
                                 Number of parallel processes for collecting information about databases.
      --[no-]backrest.database-count-latest  
                                 Exposing the number of databases in the latest backups.
      --[no-]backrest.reference-count  
                                 Exposing the number of references to other backups (backup reference list).
      --[no-]backrest.verbose-wal  
                                 Exposing additional labels for WAL metrics.
      --log.level=info           Only log messages with the given severity or above. One of: [debug, info, warn, error]
      --log.format=logfmt        Output format of log messages. One of: [logfmt, json]
      --[no-]version             Show application version.

Additional description of flags

Custom config and/or custom config-include-path for pgbackrest command can be specified via --backrest.config and --backrest.config-include-path flags. Full paths must be specified.
For example, --backrest.config=/tmp/pgbackrest.conf and/or --backrest.config-include-path=/tmp/pgbackrest/conf.d.

Custom stanza for collecting metrics can be specified via --backrest.stanza-include flag. You can specify several stanzas.
For example, --backrest.stanza-include=demo1 --backrest.stanza-include=demo2.
For this case, metrics will be collected only for demo1 and demo2 stanzas.

Custom stanza to exclude from collecting metrics can be specified via --backrest.stanza-exclude flag. You can specify several stanzas.
For example, --backrest.stanza-exclude=demo1 --backrest.stanza-exclude=demo2.
For this case, metrics will not be collected for demo1 and demo2 stanzas.
If the same stanza is specified for include and exclude flags, then metrics for this stanza will not be collected. The flag --backrest.stanza-exclude has a higher priority.
For example, --backrest.stanza-include=demo1 --backrest.stanza-exclude=demo1.
For this case, metrics will not be collected for demo1 stanza.
When flag --backrest.stanza-exclude is specified, the pgbackrest_exporter_status metric will have label stanza=all-stanzas-except-excluded.

When flag --backrest.verbose-wal is specified - WALMin and WALMax are added as metric labels.
This creates new different time series on each WAL archiving.

When --log.level=debug is specified - information of values and labels for metrics is printing to the log.

The flag --web.config.file allows to specify the path to the configuration for TLS and/or basic authentication.
The description of TLS configuration and basic authentication can be found at exporter-toolkit/web.

Custom backup type for collecting metrics can be specified via --backrest.backup-type flag. Valid values: full, incr or diff.
For example, --backrest.backup-type=full.
For this case, metrics will be collected only for full backups.
This flag works for pgBackRest >= v2.38.
When parameter value is incr or diff, all pgbackrest_backup_last_* metrics will not be collected.
When parameter value is full, the metrics will be as if the last backup was full (i.e. the same for diff and incr).
For earlier pgBackRest versions there will be an error like: option 'type' not valid for command 'info'.

When flag --backrest.database-count is specified - information about the number of databases in backup is collected.
This flag works for pgBackRest >= v2.41.
For earlier pgBackRest versions there will be an error like: option 'set' is currently only valid for text output.
For a significant numbers of stanzas and backups, this may require much more additional time to collect metrics. Each stanza requires pgBackRest execution for backups to get data.

The flag --backrest.database-parallel-processes allows to increase the number of parallel processes for collecting information about databases in backups.
This flag is valid only when the flag --backrest.database-count is specified.

When flag --backrest.database-count-latest is specified - information about the number of databases in the last full, differential or incremental backup is collected.
This flag works for pgBackRest >= v2.41.
For earlier pgBackRest versions there will be an error like: option 'set' is currently only valid for text output.
For a significant number of stanzas, this may require additional time to collect metrics. Each stanza requires pgBackRest execution for the last full, differential or incremental backups to get data.

When the --backrest.reference-count flag is specified, information about the number of references to other backups (backup reference list) is collected.
The pgbackrest_backup_references metric can be a little annoying. This metric is hidden behind the flag. However, the pgbackrest_backup_last_references metric is always collected for the latest backups.

Building and running docker

By default, pgBackRest version is 2.56.0. Another version can be specified via arguments. For base image used docker-pgbackrest image.

Environment variables supported by this image:

• all environment variables from docker-pgbackrest image;
• EXPORTER_TELEMETRY_PATH - path under which to expose metrics, default /metrics;
• EXPORTER_PORT - port for prometheus metrics to listen on, default 9854;
• EXPORTER_CONFIG - path to the configuration file for TLS and/or basic authentication, default "";
• STANZA_INCLUDE - specific stanza for collecting metrics, default "";
• STANZA_EXCLUDE - specific stanza to exclude from collecting metrics, default "";
• COLLECT_INTERVAL - collecting metrics interval in seconds, default 600;
• BACKUP_TYPE - specific backup type for collecting metrics, default "";
• VERBOSE_WAL - enabling additional labels for WAL metrics, default false;
• DATABASE_COUNT - exposing the number of databases in backups, default false;
• DATABASE_PARALLEL_PROCESSES - number of parallel processes for collecting information about databases in backups, default 1;
• DATABASE_COUNT_LATEST - exposing the number of databases in the latest backups, default false.

Pull

Change tag to the release number.

• Docker Hub:

docker pull woblerr/pgbackrest_exporter:tag

docker pull woblerr/pgbackrest_exporter:tag-alpine

• GitHub Registry:

docker pull ghcr.io/woblerr/pgbackrest_exporter:tag

docker pull ghcr.io/woblerr/pgbackrest_exporter:tag-alpine

Build

make docker

make docker-alpine

or for specific pgBackRest version

docker build -f Dockerfile --build-arg BACKREST_VERSION=2.34 -t pgbackrest_exporter .

docker build -f Dockerfile --build-arg BACKREST_VERSION=2.34-alpine -t pgbackrest_exporter-alpine .

Run

You will need to mount the necessary directories or files inside the container.

Simple run:

docker run -d \
    --name pgbackrest_exporter \
    -p 9854:9854 \
    -v  /etc/pgbackrest/pgbackrest.conf:/etc/pgbackrest/pgbackrest.conf \ 
    pgbackrest_exporter

With some enviroment variables:

docker run -d \
    --name pgbackrest_exporter \
    -e BACKREST_USER=postgres \
    -e BACKREST_UID=1001 \
    -e BACKREST_GROUP=postgres \
    -e BACKREST_GID=1001 \
    -e TZ=America/Chicago \
    -e COLLECT_INTERVAL=60 \
    -p 9854:9854 \
    -v  /etc/pgbackrest/pgbackrest.conf:/etc/pgbackrest/pgbackrest.conf \
    pgbackrest_exporter

For specific stanza:

docker run -d \
    --name pgbackrest_exporter \
    -e STANZA_INCLUDE=demo \
    -p 9854:9854 \
    -v  /etc/pgbackrest/pgbackrest.conf:/etc/pgbackrest/pgbackrest.conf \
    pgbackrest_exporter

If you want to specify several stanzas for collecting metrics, you can run containers on different ports:

docker run -d \
    --name pgbackrest_exporter_demo1 \
    -e STANZA_INCLUDE=demo1 \
    -p 9854:9854 \
    -v  /etc/pgbackrest/pgbackrest.conf:/etc/pgbackrest/pgbackrest.conf \
    pgbackrest_exporter

docker run -d \
    --name pgbackrest_exporter_demo2 \
    -e STANZA_INCLUDE=demo2 \
    -p 9855:9854 \
    -v  /etc/pgbackrest/pgbackrest.conf:/etc/pgbackrest/pgbackrest.conf \
    pgbackrest_exporter

To exclude specific stanza:

docker run -d \
    --name pgbackrest_exporter \
    -e STANZA_EXCLUDE=demo \
    -p 9854:9854 \
    -v  /etc/pgbackrest/pgbackrest.conf:/etc/pgbackrest/pgbackrest.conf \
    pgbackrest_exporter

For specific backup type:

docker run -d \
    --name pgbackrest_exporter \
    -e BACKUP_TYPE=full \
    -p 9854:9854 \
    -v  /etc/pgbackrest/pgbackrest.conf:/etc/pgbackrest/pgbackrest.conf \
    pgbackrest_exporter

With exposing the number of databases in backups:

docker run -d \
    --name pgbackrest_exporter \
    -e DATABASE_COUNT=true \
    -p 9854:9854 \
    -v  /etc/pgbackrest/pgbackrest.conf:/etc/pgbackrest/pgbackrest.conf \
    pgbackrest_exporter

With exposing the number of databases in the latest backups:

docker run -d \
    --name pgbackrest_exporter \
    -e DATABASE_COUNT_LATEST=true \
    -p 9854:9854 \
    -v  /etc/pgbackrest/pgbackrest.conf:/etc/pgbackrest/pgbackrest.conf \
    pgbackrest_exporter

To communicate with pgBackRest TLS server you need correct pgBackRest config, for example:

[demo]
pg1-path=/var/lib/postgresql/16/main

[global]
repo1-host=backup
repo1-host-ca-file=/etc/pgbackrest/cert/pgbackrest-test-ca.crt
repo1-host-cert-file=/etc/pgbackrest/cert/pgbackrest-test-client.crt
repo1-host-key-file=/etc/pgbackrest/cert/pgbackrest-test-client.key
repo1-host-type=tls
repo1-retention-diff=2
repo1-retention-full=2

And run:

docker run -d \
    --name pgbackrest_exporter \
    -e BACKREST_UID=1001 \
    -e BACKREST_GID=1001 \
    -p 9854:9854 \
    -v /etc/pgbackrest/pgbackrest.conf:/etc/pgbackrest/pgbackrest.conf \
    -v /etc/pgbackrest/cert:/etc/pgbackrest/cert \
    pgbackrest_exporter

Running as systemd service

• Register pgbackrest_exporter (already builded, if not - exec make build before) as a systemd service:

make prepare-service

Validate prepared file pgbackrest_exporter.service and run:

sudo make install-service

• View service logs:

journalctl -u pgbackrest_exporter.service

• Delete systemd service:

sudo make remove-service

---

Manual register systemd service:

cp pgbackrest_exporter.service.template pgbackrest_exporter.service

In file pgbackrest_exporter.service replace {PATH_TO_FILE} to full path to pgbackrest_exporter.

sudo cp pgbackrest_exporter.service /etc/systemd/system/pgbackrest_exporter.service
sudo systemctl daemon-reload
sudo systemctl enable pgbackrest_exporter.service
sudo systemctl restart pgbackrest_exporter.service
systemctl -l status pgbackrest_exporter.service

RPM/DEB packages

You can use the already prepared rpm/deb package to install the exporter. Only the pgbackrest_exporter binary and the service file are installed by package.

For example:

rpm -ql pgbackrest_exporter

/etc/systemd/system/pgbackrest_exporter.service
/usr/bin/pgbackrest_exporter

Running tests

Run the unit tests:

make test

Run the end-to-end tests:

make test-e2e