NetBackup Accelerator backup fails STATUS 84
NetBackup Accelerator backup fails STATUS 84
Problem
The NetBackup Accelerator feature enables the ability for repeated Full backups but with the performance as good as an Incremental backup.
Additionally Accelerator backups with an incremental type schedule can be defined as Incremental backups which will run quicker and not create a full catalog reference in the NetBackup images database. The Accelerator feature relies on the Deduplication storage such as the Media Server Deduplication Pool (MSDP) to have the ability of storing only the data which has changed since the previous Accelerator backup.
There are numerous reasons why a backup will fail with STATUS 84 (media write error), however, this technical article addresses one specific problem when the job details state "get frag failed, backup id".
Error Message
Job Details:
29.jan.2021 02:48:17 - Info ndmpagent (pid=50203) <CLIENTNAME>: DUMP: Fri Jan 29 02:48:18 2021 : We have written 1880787110 KB.
29.jan.2021 02:53:17 - Info ndmpagent (pid=50203) <CLIENTNAME>: DUMP: Fri Jan 29 02:53:18 2021 : We have written 1914286917 KB.
29.jan.2021 02:58:18 - Info ndmpagent (pid=50203) <CLIENTNAME>: DUMP: Fri Jan 29 02:58:18 2021 : We have written 1949038217 KB.
29.jan.2021 03:01:12 - end writing; write time: 6:39:23
29.jan.2021 03:02:43 - Critical bptm (pid=50205) get frag failed, backup id (<CLIENTNAME>_1603821638)
29.jan.2021 03:02:43 - Critical bptm (pid=50205) image write failed: error 227: Not an OST error code
29.jan.2021 03:02:52 - Error bptm (pid=50205) cannot write image to disk, Invalid argument
29.jan.2021 03:02:52 - Info ndmpagent (pid=50203) Received ABORT request from bptm
29.jan.2021 03:02:52 - Error ndmpagent (pid=50203) NDMP backup failed, path = /<VOLUME_NAME>/<SUB_DIRECTORY>/
29.jan.2021 03:02:52 - Error ndmpagent (pid=50203) <CLIENTNAME>: DUMP: Error: Write to socket failed
29.jan.2021 03:02:53 - Error ndmpagent (pid=50203) <CLIENTNAME>: DUMP: Error: DUMP IS ABORTED
29.jan.2021 03:02:53 - Info ndmpagent (pid=50203) <CLIENTNAME>: DUMP: Deleting "/<VOLUME_NAME>/<SUB_DIRECTORY>/../snapshot_for_backup.2518" snapshot.
29.jan.2021 03:02:53 - Error ndmpagent (pid=50203) <CLIENTNAME>: DATA: Operation terminated (for /<VOLUME_NAME>/<SUB_DIRECTORY>/).
29.jan.2021 03:02:54 - Info bptm (pid=50205) EXITING with status 84 <----------
BPTM log - demonstrating the backup ID "<CLIENTNAME>_1603821638" could NOT be located in the MSDP catalog, which is what references the backup images held on the MSDP storage:
03:02:43.794 [50205] <2> 41124714:bptm:50205:svvpmediasrv52: [DEBUG] PDSTS: impl_open_image: entry: 0x9 R, img_basename=<CLIENTNAME>_1603821638_C1_F1
03:02:43.794 [50205] <2> 41124714:bptm:50205:svvpmediasrv52: [DEBUG] PDSTS: acquire_svh: svh:<0x13633d0>, nhandle:<22>
03:02:43.794 [50205] <2> 41124714:bptm:50205:svvpmediasrv52: [DEBUG] PDSTS: impl_image_handle: entry: 2 8 m=(R) (PureDiskVolume) (<CLIENTNAME>_1603821638_C1_F1)
03:02:43.794 [50205] <2> 41124714:bptm:50205:svvpmediasrv52: [DEBUG] PDSTS: acquire_svh: svh:<0x13633d0>, nhandle:<23>
03:02:43.794 [50205] <2> 41124714:bptm:50205:svvpmediasrv52: [DEBUG] PDSTS: impl_image_handle: copy=1, frag=1
03:02:43.794 [50205] <2> 41124714:bptm:50205:svvpmediasrv52: [DEBUG] PDSTS: impl_get_imh_image_prop: entry: imh(0x0x5ae0c550) imgdef(0x0x7ffec41c8620)
03:02:43.794 [50205] <2> 41124714:bptm:50205:svvpmediasrv52: [DEBUG] PDSTS: impl_get_imh_image_prop: imh->imh_infopath(/svvpmediasrv52#1/2/<CLIENTNAME>/<POLICY_NAME>/<CLIENTNAME>_1603821638_C1_F1.info)
03:02:43.794 [50205] <2> 41124714:bptm:50205:svvpmediasrv52: [DEBUG] PDSTS: impl_get_imh_image_prop: imh->imh_imgpath(/svvpmediasrv52#1/2/<CLIENTNAME>/<POLICY_NAME>/<CLIENTNAME>_1603821638_C1_F1.img)
03:02:43.794 [50205] <2> 41124714:bptm:50205:svvpmediasrv52: [DEBUG] PDSTS: impl_get_imh_image_prop: imh->imh_hdrpath(/svvpmediasrv52#1/2/<CLIENTNAME>/<POLICY_NAME>/<CLIENTNAME>_1603821638_C1_F1.hdr)
03:02:43.794 [50205] <2> 41124714:bptm:50205:svvpmediasrv52: [DEBUG] PDSTS: impl_get_imh_image_prop: imh->imh_mappath(/svvpmediasrv52#1/2/<CLIENTNAME>/<POLICY_NAME>/<CLIENTNAME>_1603821638_C1_F1.map)
03:02:43.795 [50205] <2> 41124714:bptm:50205:svvpmediasrv52: [DEBUG] PDSTS: impl_get_imh_image_prop: image <CLIENTNAME>_1603821638_C1_F1 not found
03:02:43.795 [50205] <2> 41124714:bptm:50205:svvpmediasrv52: [DEBUG] PDSTS: impl_get_imh_image_prop: exit (2060018:file not found)
Cause
Accelerator backups have a dependency on the all previous backups leading back to the last successful Full. If the option of performing Accelerator Incremental backups is used, these backups must be maintained (long enough retention period) going back to the last successful Full, because if any expire or are lost before the next Full is attempted, then a failure will occur.
If any *previous* Incremental backup going back to the last successful Full is missing from the catalog and/or storage, then the next Accelerator (Full or Incremental) backup will fail. The details for the previous backups are held in the track logs and the NetBackup images, which have the reference of the one previous backup image, therefore, if the chain is broken the failures will occur.
Solution
The solution is to perform a new baseline Full backup for the specific client/policy. This can be achieved by running an Accelerator Forced Rescan backup, however, to really start over, it is recommended to *remove* the existing track logs and then perform the Full Accelerator backup.
Track log locations:
Master server (NDMP Accelerator backups):
UNIX: /usr/openv/netbackup/db/track
Windows: install_path\NetBackup\db\track\
Media server(NDMP Accelerator backups):
UNIX: /usr/openv/netbackup/track
Windows: install_path\NetBackup\track\
Client (Accelerator backups which are NOT NDMP):
UNIX: /usr/openv/netbackup/track
Windows: install_path\Veritas\NetBackup\track\
Note: Under these directories, there will be a number of directories created that correspond to the master server name, storage server name, policy name, client name, and backup selections, therefore, only remove track logs related to the specific Accelerator backup needing attention.
