archiving stopped caused by backup task

Doc ID    SOLN270500
Version:    2.0
Status:    Published
Published date:    07 Oct 2016
Created Date:    19 Jun 2015
Author:   
Ming Jiang
 

Details

since June 12, customer claim their CMS historical report is empty.

Your maintenance backup has some problem, and it does not finish successfully on Jun 12 04:59:37 and stopped there.

Start backup of table dbitems acd 0 Fri Jun 12 04:59:09 2015
End date: 42165(06/11/2015)
status: Backing up Historical data: d_secs
(ACD = 4)
Start backup of table d_secs acd 5 Fri Jun 12 04:59:35 2015

End date: 42165(06/11/2015)
status: Backing up Historical data: d_secs
(ACD = 5)
Start backup of table d_secs acd 6 Fri Jun 12 04:59:37 2015
End date: 42165(06/11/2015)
 

Problem Clarification

archiving is not running successfully since Jun 12.


Archiver Completed: Fri Jun 12 00:49:50 2015 acd: 6

Waiting for backup to complete...
Warning Archiver waited for backup 3600 seconds, still waiting
Warning Archiver waited for backup 7200 seconds, still waiting
Warning Archiver waited for backup 10800 seconds, still waiting
Warning Archiver waited for backup 14400 seconds, still waiting
Warning Archiver waited for backup 18000 seconds, still waiting
Warning Archiver waited for backup 21600 seconds, still waiting
Warning Archiver waited for backup 25200 seconds, still waiting
Warning Archiver waited for backup 28800 seconds, still waiting
Warning Archiver waited for backup 32400 seconds, still waiting
Warning Archiver waited for backup 36000 seconds, still waiting
Warning Archiver waited for backup 39600 seconds, still waiting
Warning Archiver waited for backup 43200 seconds, still waiting

Warning Archiver waited for backup 111600 seconds, still waiting
Warning Archiver waited for backup 115200 seconds, still waiting
Warning Archiver waited for backup 118800 seconds, still waiting
Warning Archiver waited for backup 122400 seconds, still waiting
Warning Archiver waited for backup 126000 seconds, still waiting

Cause

The culprit is at the time of backup, someone manually unplugged the removable disk:

Backup stop at 04:59:36

Start backup of table dbitems acd 0 Fri Jun 12 04:59:09 2015
End date: 42165(06/11/2015)
status: Backing up Historical data: d_secs
(ACD = 4)
Start backup of table d_secs acd 5 Fri Jun 12 04:59:35 2015

End date: 42165(06/11/2015)
status: Backing up Historical data: d_secs
(ACD = 5)
Start backup of table d_secs acd 6 Fri Jun 12 04:59:37 2015
End date: 42165(06/11/2015)

System log shows since 04:59:36 Jun 12, the removable disk is gone:

Jun 12 04:59:36 1220BDYF04 scsi: [ID 107833 kern.warning] WARNING: /pci@400/pci@
2/pci@0/pci@f/pci@0/usb@0,2/hub@2/storage@2/disk@0,0 (sd1):
Jun 12 04:59:36 1220BDYF04 Command failed to complete...Device is gone
Jun 12 04:59:36 1220BDYF04 scsi: [ID 107833 kern.warning] WARNING: /pci@400/pci@
2/pci@0/pci@f/pci@0/usb@0,2/hub@2/storage@2/disk@0,0 (sd1):
Jun 12 04:59:36 1220BDYF04 Command failed to complete...Device is gone
Jun 12 04:59:36 1220BDYF04 scsi: [ID 107833 kern.warning] WARNING: /pci@400/pci@
2/pci@0/pci@f/pci@0/usb@0,2/hub@2/storage@2/disk@0,0 (sd1):
Jun 12 04:59:36 1220BDYF04 Command failed to complete...Device is gone
Jun 12 04:59:36 1220BDYF04 scsi: [ID 107833 kern.warning] WARNING: /pci@400/pci@
2/pci@0/pci@f/pci@0/usb@0,2/hub@2/storage@2/disk@0,0 (sd1):


Jun 12 04:59:36 1220BDYF04 Command failed to complete...Device is gone
Jun 12 04:59:36 1220BDYF04 scsi: [ID 107833 kern.warning] WARNING: /pci@400/pci@
2/pci@0/pci@f/pci@0/usb@0,2/hub@2/storage@2/disk@0,0 (sd1):
Jun 12 04:59:36 1220BDYF04 Command failed to complete...Device is gone
Jun 12 04:59:36 1220BDYF04 scsi: [ID 107833 kern.warning] WARNING: /pci@400/pci@
2/pci@0/pci@f/pci@0/usb@0,2/hub@2/storage@2/disk@0,0 (sd1):
Jun 12 04:59:36 1220BDYF04 Command failed to complete...Device is gone
Jun 12 04:59:36 1220BDYF04 scsi: [ID 107833 kern.warning] WARNING: /pci@400/pci@
2/pci@0/pci@f/pci@0/usb

Solution

You should never unplug the removable Disk when backup is running, this will cause unexpected troubles and effort.

kill the backup process and clean the 13005 job in bug post (13005 job is for backup)

since zpool can not be killed. the whole server needs to be hard rebooted to clean out the zpool

then manually do the archiving for the missing days.

Additional Relevant Phrases

archiver not able to run data summarizing is not successful

Avaya -- Proprietary. Use pursuant to the terms of your signed agreement or Avaya policy