AACC 7.0: IIS service is down after running "Qualys" or any other Network Security Scan
SOLN325828 | Version: 5.0 | Published: 05/30/2018 | Updated: 11/05/2020
Available To: Customer
Categories: Avaya Aura® Contact Center, Avaya Contact Center Select, Break/Fix, Configuration
Details

AACC 7.0 FPx SIP or AML
ACCS 7.0 FPx IPOP

Problem Clarification

The customer runs a periodic network security scan using an external 3rd party service by Secureworks. They have found that this causes the w3wp service to terminate, and CCMA is no longer accessible.

The Windows Application and System events do not show any indication of an issue in CCMA.

The faulting application is w3wp.exe, which is the iisworker process. The faulting module is msvcrt.dll, which is the Microsoft C standard library. CCMA does use these, but they are Microsoft components, and not part of CCMA itself

Error 5/5/2018 2:39:41 PM Application Error 1000 (100) "Faulting application name: w3wp.exe, version: 8.5.9600.16384, time stamp: 0x52157ba0
Faulting module name: msvcrt.dll, version: 7.0.9600.17415, time stamp: 0x54504b2e
Exception code: 0xc0000005
Fault offset: 0x0000f407
Faulting process id: 0x3aa4
Faulting application start time: 0x01d3e4a06c53a982
Faulting application path: C:\Windows\SysWOW64\inetsrv\w3wp.exe
Faulting module path: C:\Windows\SYSTEM32\msvcrt.dll
Report Id: ab6b97f9-5093-11e8-80f7-005056ab0ee9
Faulting package full name:
Faulting package-relative application ID: "
 

From this, there is no indication that the issue began with CCMA. Instead, it appears that the CCMA component failed two seconds later, after IIS is terminated:

Warning 5/5/2018 2:39:43 PM NGen 40592 Minor Event from Fault: NT Log Monitor[0] : Event from NT System Log[WAS] , Event: ID= 5011, Description: A process serving application pool 'CCMA_DefaultAppPool' suffered a fatal communication error with the Windows Process Activation Service. The process id was '15012'. The data field contains the error number. .
Error 5/5/2018 2:39:41 PM Application Error 1000 (100) "Faulting application name: w3wp.exe, version: 8.5.9600.16384, time stamp: 0x52157ba0
Faulting module name: msvcrt.dll, version: 7.0.9600.17415, time stamp: 0x54504b2e
Exception code: 0xc0000005
Fault offset: 0x0000f407
Faulting process id: 0x3aa4
Faulting application start time: 0x01d3e4a06c53a982
Faulting application path: C:\Windows\SysWOW64\inetsrv\w3wp.exe
Faulting module path: C:\Windows\SYSTEM32\msvcrt.dll
Report Id: abc5ee69-5093-11e8-80f7-005056ab0ee9
Faulting package full name:
Faulting package-relative application ID: "

 

System Event Log:

Classic   6/13/2018 2:58:24 PM   Microsoft-Windows-WAS             5002       None     Application pool 'CCMA_DefaultAppPool' is being automatically disabled due to a series of failures in the process(es) serving that application pool.
Classic   6/13/2018 2:58:24 PM   Microsoft-Windows-WAS             5011       None     A process serving application pool 'CCMA_DefaultAppPool' suffered a fatal communication error with the Windows Process Activation Service. The process id was '26152'. The data field contains the error number.
Classic   6/13/2018 2:58:09 PM   Service Control Manager              7036       None     The Windows Modules Installer service entered the running state.
Classic   6/13/2018 2:58:05 PM   Service Control Manager              7036       None     The Software Protection service entered the running state.
Classic   6/13/2018 2:57:58 PM   Microsoft-Windows-WAS             5011       None     A process serving application pool 'CCMA_DefaultAppPool' suffered a fatal communication error with the Windows Process Activation Service. The process id was '26272'. The data field contains the error number.
Classic   6/13/2018 2:57:57 PM   Microsoft-Windows-WAS             5011       None     A process serving application pool 'CCMA_DefaultAppPool' suffered a fatal communication error with the Windows Process Activation Service. The process id was '3264'. The data field contains the error number.
Classic   6/13/2018 2:57:53 PM   Microsoft-Windows-WAS             5011       None     A process serving application pool 'CCMA_DefaultAppPool' suffered a fatal communication error with the Windows Process Activation Service. The process id was '9824'. The data field contains the error number.
Classic   6/13/2018 2:57:51 PM   Service Control Manager              7036       None     The Windows Error Reporting Service service entered the running state.
Classic   6/13/2018 2:57:52 PM   Microsoft-Windows-WAS             5011       None     A process serving application pool 'CCMA_DefaultAppPool' suffered a fatal communication error with the Windows Process Activation Service. The process id was '26464'. The data field contains the error number.

Application Event Log:

Classic   6/13/2018 2:58:23 PM   Application Error              1000       (100)      "Faulting application name: w3wp.exe, version: 8.5.9600.16384, time stamp: 0x52157ba0
Faulting module name: msvcrt.dll, version: 7.0.9600.17415, time stamp: 0x54504b2e
Exception code: 0xc0000005
Fault offset: 0x0000f407
Faulting process id: 0x6628
Faulting application start time: 0x01d403487918163e
Faulting application path: C:\Windows\SysWOW64\inetsrv\w3wp.exe
Faulting module path: C:\Windows\SYSTEM32\msvcrt.dll
Report Id: bec70cbc-6f3b-11e8-8105-005056ab7451
Faulting package full name:
Faulting package-relative application ID: "
Classic   6/13/2018 2:58:23 PM   Application Error              1000       (100)      "Faulting application name: w3wp.exe, version: 8.5.9600.16384, time stamp: 0x52157ba0
Faulting module name: msvcrt.dll, version: 7.0.9600.17415, time stamp: 0x54504b2e
Exception code: 0xc0000005
Fault offset: 0x0000f407
Faulting process id: 0x6628
Faulting application start time: 0x01d403487918163e
Faulting application path: C:\Windows\SysWOW64\inetsrv\w3wp.exe
Faulting module path: C:\Windows\SYSTEM32\msvcrt.dll
Report Id: be976f2a-6f3b-11e8-8105-005056ab7451
Faulting package full name:
Faulting package-relative application ID: "

Solution