postgresql 物理备份 pg_rman
os: centos 7.4
postgresql: 9.6.9
pg_rman: REL9_6_STABLE
pg_rman 是一款优秀的postgresql 在线备份和恢复的工具，在github上可以找到该软件。
下面是pg_rman主页面的描述:
pg_rman is an online backup and restore tool for PostgreSQL.
The goal of the pg_rman project is to provide a method for online backup and PITR that is as easy as pg_dump. Also, it maintains a backup catalog per database cluster. Users can maintain old backups including archive logs with one command.
下载
https://github.com/ossc-db/pg_rman
由于本次的postgresql 版本为 9.6.9，所以需要下载相应的pg_rman REL9_6_STABLE : branch for PostgreSQL 9.6
# su - postgres
$ cd /tmp
$ git clone https://github.com/ossc-db/pg_rman.git
$ cd pg_rman/
$ git branch -a
* master
remotes/origin/HEAD -> origin/master
remotes/origin/REL9_2_STABLE
remotes/origin/REL9_3_STABLE
remotes/origin/REL9_4_STABLE
remotes/origin/REL9_5_STABLE
remotes/origin/REL9_6_STABLE
remotes/origin/REL_10_STABLE
remotes/origin/master
remotes/origin/pre-9.2
$ git checkout REL9_6_STABLE
Already on 'REL9_6_STABLE'
$ git status
$ On branch REL9_6_STABLE
nothing to commit, working directory clean
安装
前提是postgresl 9.6 已经安装好了。
# su - postgres
$ make
$ make installcheck
$ exit
# cd /tmp/pg_rman
# make install
/usr/bin/mkdir -p '/usr/pgsql-9.6/bin'
/usr/bin/install -c pg_rman '/usr/pgsql-9.6/bin'
# ls -l /usr/pgsql-9.6/bin | grep -i rman
-rwxr-xr-x. 1 root root 633680 Jun 11 23:08 pg_rman
pg_rman --help
$ which pg_rman
/usr/pgsql-9.6/bin/pg_rman
$ pg_rman --help
pg_rman manage backup/recovery of PostgreSQL database.
Usage:
pg_rman OPTION init
pg_rman OPTION backup
pg_rman OPTION restore
pg_rman OPTION show [DATE]
pg_rman OPTION show detail [DATE]
pg_rman OPTION validate [DATE]
pg_rman OPTION delete DATE
pg_rman OPTION purge
Common Options:
-D, --pgdata=PATH location of the database storage area
-A, --arclog-path=PATH location of archive WAL storage area
-S, --srvlog-path=PATH location of server log storage area
-B, --backup-path=PATH location of the backup storage area
-c, --check show what would have been done
-v, --verbose show what detail messages
-P, --progress show progress of processed files
Backup options:
-b, --backup-mode=MODE full, incremental, or archive
-s, --with-serverlog also backup server log files
-Z, --compress-data compress data backup with zlib
-C, --smooth-checkpoint do smooth checkpoint before backup
-F, --full-backup-on-error switch to full backup mode
if pg_rman cannot find validate full backup
on current timeline
NOTE: this option is only used in --backup-mode=incremental or archive.
--keep-data-generations=NUM keep NUM generations of full data backup
--keep-data-days=NUM keep enough data backup to recover to N days ago
--keep-arclog-files=NUM keep NUM of archived WAL
--keep-arclog-days=DAY keep archived WAL modified in DAY days
--keep-srvlog-files=NUM keep NUM of serverlogs
--keep-srvlog-days=DAY keep serverlog modified in DAY days
--standby-host=HOSTNAME standby host when taking backup from standby
--standby-port=PORT standby port when taking backup from standby
Restore options:
--recovery-target-time time stamp up to which recovery will proceed
--recovery-target-xid transaction ID up to which recovery will proceed
--recovery-target-inclusive whether we stop just after the recovery target
--recovery-target-timeline recovering into a particular timeline
--hard-copy copying archivelog not symbolic link
Catalog options:
-a, --show-all show deleted backup too
Delete options:
-f, --force forcibly delete backup older than given DATE
Connection options:
-d, --dbname=DBNAME database to connect
-h, --host=HOSTNAME database server host or socket directory
-p, --port=PORT database server port
-U, --username=USERNAME user name to connect as
-w, --no-password never prompt for password
-W, --password force password prompt
Generic options:
-q, --quiet don't show any INFO or DEBUG messages
--debug show DEBUG messages
--help show this help, then exit
--version output version information, then exit
Read the website for details. <http://github.com/ossc-db/pg_rman>
Report bugs to <http://github.com/ossc-db/pg_rman/issues>.
pg_rman init
pg_rman 需要一个备份目录
# mkdir -p /mnt/walbackup
# mkdir -p /mnt/pg_rman_backupset
# chown -R postgres:postgres /mnt
$ ls -l
total 0
drwxr-xr-x. 2 postgres postgres 6 Jun 11 23:24 pg_rman_backupset
drwxr-xr-x. 2 postgres postgres 6 Jun 11 23:05 walbackup
$ vi ~/.bash_profile
export BACKUP_PATH=/mnt/pg_rman_backupset
pg_rman init 初始化
$ pg_rman init
INFO: ARCLOG_PATH is set to '/mnt/walbackup'
INFO: SRVLOG_PATH is set to '/var/lib/pgsql/9.6/data/pg_log'
pg_rman backup
pg_rman 全量备份
$ pg_rman backup --backup-mode=full --with-serverlog --progress
INFO: copying database files
Processed 1166 of 1166 files, skipped 0
INFO: copying archived WAL files
Processed 3 of 3 files, skipped 0
INFO: copying server log files
Processed 1 of 1 files, skipped 0
INFO: backup complete
INFO: Please execute 'pg_rman validate' to verify the files are correctly copied.
pg_rman 校验备份集
pg_rman 的备份必须都是经过验证过的，否则不能进行恢复和增量备份
$ pg_rman validate
INFO: validate: "2018-06-11 23:30:47" backup, archive log files and server log files by CRC
INFO: backup "2018-06-11 23:30:47" is valid
pg_rman 列出备份集
$ pg_rman show
=====================================================================
StartTime EndTime Mode Size TLI Status
=====================================================================
2018-06-11 23:30:47 2018-06-11 23:30:49 FULL 58MB 1 OK
$ ls -l /mnt/pg_rman_backupset/
total 8
drwx------. 3 postgres postgres 20 Jun 11 23:30 20180611
drwx------. 4 postgres postgres 35 Jun 11 23:27 backup
-rw-r--r--. 1 postgres postgres 75 Jun 11 23:27 pg_rman.ini
-rw-r--r--. 1 postgres postgres 40 Jun 11 23:27 system_identifier
drwx------. 2 postgres postgres 6 Jun 11 23:27 timeline_history
pg_rman 增量备份
增量备份是基于文件系统的update time时间线
增量备份必须有个对应的全库备份
$ pg_rman backup --backup-mode incremental --progress --compress-data
INFO: copying database files
Processed 1435 of 1435 files, skipped 1135
INFO: copying archived WAL files
Processed 6 of 6 files, skipped 3
INFO: backup complete
INFO: Please execute 'pg_rman validate' to verify the files are correctly copied.
$ pg_rman validate
INFO: validate: "2018-06-11 23:40:57" backup and archive log files by CRC
INFO: backup "2018-06-11 23:40:57" is valid
$ pg_rman show
=====================================================================
StartTime EndTime Mode Size TLI Status
=====================================================================
2018-06-11 23:40:57 2018-06-11 23:40:59 INCR 907kB 1 OK
2018-06-11 23:30:47 2018-06-11 23:30:49 FULL 58MB 1 OK
pg_rman 删除备份集
若果提示不能删除，请执行查看输出信息。如果实在手贱，可以指定 -f 参数。
$ pg_rman delete '2018-06-11 23:30:47'
WARNING: cannot delete backup with start time "2018-06-11 23:30:47"
DETAIL: This is the latest full backup necessary for successful recovery.
$ pg_rman delete -f '2018-06-11 23:30:47'
INFO: delete the backup with start time: "2018-06-11 23:30:47"
$ pg_rman show
=====================================================================
StartTime EndTime Mode Size TLI Status
=====================================================================
2018-06-11 23:40:57 2018-06-11 23:40:59 INCR 907kB 1 OK
pg_rman restore
以下操作为模拟目录误删除。
# systemctl stop postgresql-9.6.service
# ps -ef|grep -i postmaster |grep -v grep
# su - postgres
$ cd $PGDATA/..
$ ls -l
total 8
drwx------. 2 postgres postgres 6 May 10 03:36 backups
drwx------. 20 postgres postgres 4096 Jun 12 02:51 data
-rw-------. 1 postgres postgres 878 Jun 11 21:47 initdb.log
$ mv ./data ./data.bak
$ ls -l
total 8
drwx------. 2 postgres postgres 6 May 10 03:36 backups
drwx------. 20 postgres postgres 4096 Jun 12 02:51 data.bak
-rw-------. 1 postgres postgres 878 Jun 11 21:47 initdb.log
$ mkdir data
$ chmod 700 ./data
$ ls -l
使用 pg_rman restore 还原
$ pg_rman restore
WARNING: pg_controldata file "/var/lib/pgsql/9.6/data/global/pg_control" does not exist
INFO: the recovery target timeline ID is not given
INFO: use timeline ID of latest full backup as recovery target: 1
INFO: calculating timeline branches to be used to recovery target point
INFO: searching latest full backup which can be used as restore start point
INFO: found the full backup can be used as base in recovery: "2018-06-12 02:36:40"
INFO: copying online WAL files and server log files
INFO: clearing restore destination
INFO: validate: "2018-06-12 02:36:40" backup, archive log files and server log files by SIZE
INFO: backup "2018-06-12 02:36:40" is valid
INFO: restoring database files from the full mode backup "2018-06-12 02:36:40"
INFO: searching incremental backup to be restored
INFO: searching backup which contained archived WAL files to be restored
INFO: backup "2018-06-12 02:36:40" is valid
INFO: restoring WAL files from backup "2018-06-12 02:36:40"
INFO: restoring online WAL files and server log files
INFO: generating recovery.conf
INFO: restore complete
HINT: Recovery will start automatically when the PostgreSQL server is started.
$ cd $PGDATA/
$ ls -l
total 60
-rw-r--r--. 1 postgres postgres 213 Jun 12 02:57 backup_label
drwx------. 7 postgres postgres 67 Jun 12 02:57 base
drwx------. 2 postgres postgres 4096 Jun 12 02:57 global
drwx------. 2 postgres postgres 18 Jun 12 02:57 pg_clog
drwx------. 2 postgres postgres 6 Jun 12 02:57 pg_commit_ts
drwx------. 2 postgres postgres 6 Jun 12 02:57 pg_dynshmem
-rw-------. 1 postgres postgres 4224 Jun 12 02:57 pg_hba.conf
-rw-------. 1 postgres postgres 1636 Jun 12 02:57 pg_ident.conf
drwx------. 2 postgres postgres 6 Jun 12 02:57 pg_log
drwx------. 4 postgres postgres 39 Jun 12 02:57 pg_logical
drwx------. 4 postgres postgres 36 Jun 12 02:57 pg_multixact
drwx------. 2 postgres postgres 18 Jun 12 02:57 pg_notify
drwx------. 2 postgres postgres 6 Jun 12 02:57 pg_replslot
drwx------. 2 postgres postgres 6 Jun 12 02:57 pg_serial
drwx------. 2 postgres postgres 6 Jun 12 02:57 pg_snapshots
drwx------. 2 postgres postgres 6 Jun 12 02:57 pg_stat
drwx------. 2 postgres postgres 6 Jun 12 02:57 pg_stat_tmp
drwx------. 2 postgres postgres 18 Jun 12 02:57 pg_subtrans
drwx------. 2 postgres postgres 6 Jun 12 02:57 pg_tblspc
drwx------. 2 postgres postgres 6 Jun 12 02:57 pg_twophase
-rw-------. 1 postgres postgres 4 Jun 12 02:57 PG_VERSION
drwx------. 3 postgres postgres 28 Jun 12 02:57 pg_xlog
-rw-------. 1 postgres postgres 88 Jun 12 02:57 postgresql.auto.conf
-rw-------. 1 postgres postgres 22304 Jun 12 02:57 postgresql.conf
-rw-------. 1 postgres postgres 60 Jun 12 02:57 postmaster.opts
-rw-r--r--. 1 postgres postgres 118 Jun 12 02:57 recovery.conf
$ cat recovery.conf
# recovery.conf generated by pg_rman 1.3.6
restore_command = 'cp /mnt/walbackup/%f %p'
recovery_target_timeline = '1'
启动postgresql
# systemctl start postgresql-9.6.service
# ps -ef|grep -i postgres
postgres 29509 1 0 02:59 ? 00:00:00 /usr/pgsql-9.6/bin/postmaster -D /var/lib/pgsql/9.6/data/
postgres 29512 29509 0 02:59 ? 00:00:00 postgres: logger process
postgres 29515 29509 0 02:59 ? 00:00:00 postgres: checkpointer process
postgres 29516 29509 0 02:59 ? 00:00:00 postgres: writer process
postgres 29518 29509 0 02:59 ? 00:00:00 postgres: stats collector process
postgres 29525 29509 0 02:59 ? 00:00:00 postgres: wal writer process
postgres 29526 29509 0 02:59 ? 00:00:00 postgres: autovacuum launcher process
postgres 29527 29509 0 02:59 ? 00:00:00 postgres: archiver process last was 00000002.history
postgres 29559 16928 0 03:00 pts/1 00:00:00 ps -ef
postgres 29560 16928 0 03:00 pts/1 00:00:00 grep --color=auto -i postgres
有时候restore后启动会碰到如下错误：
invalid primary checkpoint record
invalid secondary checkpoint record
could not locate a valid checkpoint record
此时只能重置xlog，并取消恢复模式
$ pg_resetxlog -f $PGDATA
$ mv $PGDATA/recovery.conf $PGDATA/recovery.done
这里有个问题需要煮一下，使用pg_rman备份时对wal的归档会是通过软链接来实现。建议添加 --hard-copy
参考：
https://github.com/ossc-db/pg_rman/tree/master
http://ossc-db.github.io/pg_rman/index.html
