At the very beginning we found from Reporting.log, there is no space available for reporting process to record information onto the disk
2017-12-20 17:01:27,057 [main] 336 ReportingPropertyRetriever.java ERROR RPTConfig.Common.PROP_ERR Error getting properties
java.io.IOException: No space left on device
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(Unknown Source)
at sun.nio.cs.StreamEncoder$CharsetSE.writeBytes(Unknown Source)
at sun.nio.cs.StreamEncoder$CharsetSE.implFlushBuffer(Unknown Source)
at sun.nio.cs.StreamEncoder$CharsetSE.implFlush(Unknown Source)
at sun.nio.cs.StreamEncoder.flush(Unknown Source)
at java.io.OutputStreamWriter.flush(Unknown Source)
at java.io.BufferedWriter.flush(Unknown Source)
at com.avaya.ccr.rptconfig.ReportingPropertyRetriever.main(ReportingPropertyRetriever.java:186)
then we found the /opt partition is 100% full and there is a huge file : catalina.out which is 36G
[root@wintelIQ CCR]# ls -lh /opt/Avaya/CCR/RTD/tomcat/apache-tomcat-5.5.27/logs/catalina.out
-rw-rw-r-- 1 avayaiq avayaiq 36G Dec 20 15:24 /opt/Avaya/CCR/RTD/tomcat/apache-tomcat-5.5.27/logs/catalina.out
After the removal of this file and restart of Reporting Application Service/Reporting web server and reporting JBoss, the reporting application server still failed to startup.
Looked into the setupReportingApp.log, which is saying
UUID is -DCONTAINER_UUID=13d0944f2a631b45012a633c530f00ac
C8BISRVR_CONTENT_MANAGER=1
C8BISRVR_GATEWAY=1
Copying Oracle JDBC driver
Using /opt/Avaya/CCR/jre/bin/java
log4j:WARN No appenders could be found for logger (Audit.RTUsage.cms.CAM.CRP.jcam).
log4j:WARN Please initialize the log4j system properly.
20/12/2017,20:18:25,Err,CAM-CRP-1093 Unable to read the contents of the keystore '/opt/Avaya/CCR/reporting/crn/configuration/encryptkeypair/jEncKeystore'. Reason: java.io.IOException: stream does not represent a PKCS12 key store, com.cognos.crconfig.data.crypto.ConfiguringSession.configure(ConfiguringSession.java:35)
20/12/2017,20:18:25,Err,com.cognos.crconfig.data.DataManagerException: CAM-CRP-1093 Unable to read the contents of the keystore '/opt/Avaya/CCR/reporting/crn/configuration/encryptkeypair/jEncKeystore'. Reason: java.io.IOException: stream does not represent a PKCS12 key store, com.cognos.crconfig.data.DataManager.generateCryptoKeys(DataManager.java:2730)
When I tried to list the key store file, found that it was zeroed out (0byte) on Dec 19th , 09:10
[root@wintelIQ CCR]# ls -lh /opt/Avaya/CCR/reporting/crn/configuration/encryptkeypair/jEncKeystore
-rw-r--r-- 1 avayaiq avayaiq 0 Dec 19 09:10 /opt/Avaya/CCR/reporting/crn/configuration/encryptkeypair/jEncKeystore
I copied the keystore file from our lab and uploaded to the IQ server, then restarted all the 3 reporting services, they are up and running now.
