CMS: Schedule backup can't be run

Doc ID    SOLN267904
Version:    7.0
Status:    Published
Published date:    16 Oct 2017
Created Date:    04 May 2015
Author:   
Yingnan Dong
 

Details

 CMS version: cms-r17-jb.a.x86_64; also seen on R18.0.0.1-fb.g

Customer stated that the scheduled fullback up cannot be run since 4/20, before it was working, however now even manual backup can not be run normally.

Check the timetable setting and it is ok, no changes made to timetable.

Check Backup/Restore Devices, it is NFS backup, check "mount" and "df -kh", NFS is correctly mounted. 

Check the backup logm found out since 4/20, full backup can not be run:

/cms/install/bin/compress_backup[257]: /NS_backup/CMS-bkup.incomplete-06-kt_cms000000000: cannot create [Permission denied]

cat /cms/maint/.cmp_pipe | gzip > /NS_backup/CMS-150428-01-LSAC-00-F-01-06-kt_cms000000000 failed: exit 1

state: 1

/cms/install/bin/compress_backup exit code: 1

/cms/install/bin/compress_backup failed: Tue Apr 28 15:35:59 HKT 2015

/cms/install/bin/compress_backup -s /NS_backup CMS-150428-01-LSAC-00-F-02 started: Tue Apr 28 15:35:59 HKT 2015

Available space in /NS_backup: 289398368KB

/cms/install/bin/compress_backup[257]: /NS_backup/CMS-bkup.incomplete-06-kt_cms000000000: cannot create [Permission denied]

cat /cms/maint/.cmp_pipe | gzip > /NS_backup/CMS-150428-01-LSAC-00-F-02-06-kt_cms000000000 failed: exit 1

cat: write error: Broken pipe

/cms/install/bin/compress_backup exit code: 1

/cms/install/bin/compress_backup failed: Tue Apr 28 15:35:59 HKT 2015

error:

status: Last backup cancelled 04/28/2015 15:36:37.

 

state: 0

Problem Clarification

 Obviously, the root cause is located in NFS permission, since CMS can not create backup files in NFS folder, which lead to the backup failed.

 

2) NFS Backup was failing because of a permissions issue on NFS

 

 

When attempting to mount the NFS remote location, the Secondary CMS backgrounds the mount process. This is never a good sign as it should mount immediately without any issues. Looking further into the logs, we can see that Stale File Handles are causing issues and prevent the NFS from being mounted. The mount processes are never automatically cleared and have to be stopped before restarting the NFS client services on the CMS.

 

Further research on this issue has lead us down a troubleshooting path on the CMS which ultimately lead to the reboot. Every possible causes of the Stale File Handle have been verified as far as the Secondary CMS system goes. This would suggest potential issues on the NFS server where it has potentially not realized that it’s no longer mounted to the Secondary CMS system which would result in the Stale File Handle we are seeing.

Cause

The NFS backup folder has been changed for the permission, now CMS don't have write permission to this backup folder.

Solution

  Suggest customer to check the NFS admin, see if any changes have been made so that the permission has been changed, give the write permission to NFS backup folder, so that the NFS backup can be run normally.

  Got the feedback from customer, they modify the permission in their Windows 2008 server for NFS backup, and the issue was fixed.


Avaya -- Proprietary. Use pursuant to the terms of your signed agreement or Avaya policy