The Application was consistenly crashing.
From the log file:
<[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "612" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@18c5d23[
POST /wfo/control/signin?_sysTime=1470942528027 HTTP/1.1
Accept: application/x-ms-application, image/jpeg, application/xaml+xml, image/gif, image/pjpeg, application/x-ms-xbap, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */*
Referer: http://xxxx:7001/wfo/control/signin?_sysTime=1470942528027
Accept-Language: en-US
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/7.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET4.0C; .NET4.0E; InfoPath.2)
Content-Type: application/x-www-form-urlencoded
Accept-Encoding: gzip, deflate
Content-Length: 174
Connection: Keep-Alive
Cache-Control: no-cache
Cookie: USER_ID=jkenned2; LANGUAGE_ID=en_US; browserCheckEnabled=true; SiteIDs=%7C%7C; JSESSIONID=yLWqXsNQHh1Mt1GKqT2v1vrlJP1ynwLdLypLM1mHBStCYnW6mMyh!-532485062
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-45 "[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" {
Observed that the latest versions of the weblogic and jrockit was not applied at the time the Hotfix was applied on site.
Weblogic was currently on an old install 10.3.4 and needed to be updated to 10.3.6 and its minor updated
For a similar issue, the latest version of the weblogic did not fix the issue. Verint observed that the Recalculate Adapter was timing out thereby causing the WFO web application time outs.
below errors are being caused by the recalc adapter timing out
(self-tuning)''] [common.logging.ErrorDumpAppender] <##[{user=}]##> - Message: ERROR_DUMP_HEADER
####[ProductionServer]-[ejb.core] [2016-03-07 15:04:47.715-0500] [ERROR] [[ACTIVE] ExecuteThread: ''33'' for queue: ''weblogic.kernel.Default
(self-tuning)''] [web.core.SAMLFilter] <##[{user=}]##> - SAMLFilter.doFilter() failed.
java.lang.NullPointerException
at com.witness.web.uif.system.session.SessionManager.getSession(SessionManager.java:49)
at com.witness.web.uif.system.RequestContext.initSession(RequestContext.java:321)
at com.witness.web.uif.system.RequestContext.(RequestContext.java:148)
at com.verint.web.core.OAuth2Filter.doFilter(OAuth2Filter.java:194)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
at com.verint.web.core.SAMLFilter.doFilter(SAMLFilter.java:57)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
at com.verint.web.core.HTTPCookieFilter.doFilter(HTTPCookieFilter.java:72)
