System Manager: VMware based template gets file system corruption after first reboot.

Doc ID    SOLN270659
Version:    4.0
Status:    Published
Published date:    28 Aug 2020
Created Date:    22 Jun 2015
Author:   
brianking
 

Details

 This particular customer was running SMGR 6.3.0 and attempting to upgrade to a newer service pack.  That would fail with errors that file system has gone to read-only mode and next reboot would show detected file system corruption.  Further troubleshooting showed that the service pack had no relevence, even a reboot would show the file system corruption as it tried to come back up.

Had another customer running System Manager 6.3.11 and had a severe power outage. Other VMs on same hardware were fine but System Manager and its Geo-Redundant (GR) counterpart hosted on another piece of hardware both came back up in read-only.

Problem Clarification

System Manager 6.3 gets a corrupt filesystem running on VMware on reboot or possibly abrupt, ungraceful reboot (like a power outage). Attempts were made to run file system checks (fsck) but these would not run due to the severity of the corruption.

Cause

The VMware or server firmware was the root cause in one case.  The OVA was checked for consistency, but something specific to this Dell and VMware caused this one OVA to become corrupt almost immediately. If the hardware is up to date the VM could have suffered sever file system corruption.

Solution

Here is a list of the upgrades that were done on the Dell server to fix this issue.
Firmwares and Drivers:

BIOS 1.0.4 -> 1.2.10
PERC H730 Mini (RAID Cont) 25.2.1.0037 -> 25.2.2-0004 A03 13G OS Drivers Pack 14.08.01 -> 15.03.00 A00 Intel NIC Gigabit 4P i350-t -> 16.5.20 A00

OS:
VMware ESXi 5.5 Update 2 A00 -> VMware ESXi 5.5 Update 2 A02

Management:
iDRAC8 and Lifecycle Controller 2.02.01.01 -> 2.10.10.10

In the event the hardware is up to date but the Virtual Machine (VM) appears stuck in read-only then the VM should be restored from a snapshot or redeployed and restored from a backup. One could try a shutdown -F now as root which should force a file system check on reboot.
 


Avaya -- Proprietary. Use pursuant to the terms of your signed agreement or Avaya policy