IQ service is totally down, OAM and reporting are not accessible

Doc ID    SOLN311740
Version:    1.0
Status:    Published
Published date:    29 Jun 2017
Author:   
mjiang
 

Details

unable to login to IQ, Both CCR and OAm page down. None of the user can login
2017-06-28 08:48:32,837 com.avaya.ccr.common.license.TimerGracePeriod$GraceTimerTask [Timer-9] 296 TimerGracePeriod.java ERROR com.avaya.ccr.common.license.TimerGracePeriod CommonLicensing.Licensing.GRACEPERIODDAYSLEFT The license grace period will end in 30 days.

java.lang.reflect.InvocationTargetException
com.avaya.coreservice.dss.DSSException: INTERNAL.VESP_EXCEPTION Resource not available

@2017-06-28 05:29:09,001 EST||ERROR|com.avaya.ccr.eventmgt.hdd.RDAPRFilterImpl||RDAPRFilterImpl||RHC.HDR.RDAPRFILTERIMPL11 RDAPRFilterImpl.filterData(). Exception : com.avaya.ccr.eventmgt.hdd.exceptions.RDAPRException: RRDC.RDAPR.RDAPRUnknownException RDAPR Unknown Exception||||||
com.avaya.ccr.eventmgt.hdd.exceptions.RDAPRException: RRDC.RDAPR.RDAPRUnknownException RDAPR Unknown Exception
com.avaya.coreservice.dss.DSSException: ENVIRONMENT.ROUTING_FAILED Routing failed for Service set: [8ae88ab23c6a1cd0013c6a1cd7e70039:10.104.10.50:31050] The reason could be, the service initialization is not completed or The container did not start the service.unable to login to IQ , Possible database down issue

**** Initiating flush of ParamValueCache

2017-06-29 11:18:02,826 org.apache.jsp.errorDisplay_jsp [http-0.0.0.0-18443-20]? ? ERROR com.avaya.ccr.ui.aop.ErrorFrameorkAdvice RUI.ErrorHandler.ERRORHANDLEDMSG The ErrorHandler intercepted and prcessed an error
com.avaya.coreservice.dss.DSSException: ENVIRONMENT.SERVER_TOO_BUSY inbound bufer fulltodayno change

Problem Clarification

found system is out of resources, multiple service has been locked out and stopped.

Cause

ERROR java.util.concurrent.TimeoutException
2017-06-28 08:51:19,692 [DSS:31050-226] 530 VespAdapter.java ERROR java.util.concurrent.TimeoutException
2017-06-28 08:51:19,693 [DSS:31050-196] 530 VespAdapter.java ERROR java.util.concurrent.TimeoutException

2017-06-28 07:37:37,296 [DSS:31050-64] 530 VespAdapter.java ERROR java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space

2017-06-28 08:34:51,116 [Timer-0] 608 ThreadPoolAsynchronousRunner.java WARN com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@361093e -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!

2017-06-28 08:48:25,889 [Timer-12] 624 ThreadPoolAsynchronousRunner.java WARN com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@701c96be -- APPARENT DEADLOCK!!! Complete Status:
Managed Threads: 3
Active Threads: 3
Active Tasks:

Solution

restart all servers including database hosts.


Avaya -- Proprietary. Use pursuant to the terms of your signed agreement or Avaya policy