CCE 4.1.3 Email not delivering

[rellerbrock]

We have an issue that occurs mainly on Monday mornings. Our Call Center opens at 8:00 a.m. so it is very busy right at 8. Usually we only get reports of slowness but today users would get the call associated with an email workitem but the work item never was delivered to CCE.

Having the users log out of CCE and back in seemed to resolve the issue but not an acceptable answer. Here is an agent log from one of the users having the issue. Has anyone experienced this before? Any suggestions on how to avoid it?

[Mon Mar 14 08:04:42.543],Major,5896,ASDirectoryPlugin.ASContactDatabaseMan ager.GetContactEntryByMatchingField():
System.Data.SqlClient.SqlException: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
at System.Data.SqlClient.SqlInternalConnection.OnErro r(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndW arning(TdsParserStateObject stateObj)
at System.Data.SqlClient.TdsParserStateObject.ReadSni Error(TdsParserStateObject stateObj, UInt32 error)
at System.Data.SqlClient.TdsParserStateObject.ReadSni (DbAsyncResult asyncResult, TdsParserStateObject stateObj)
at System.Data.SqlClient.TdsParserStateObject.ReadNet workPacket()
at System.Data.SqlClient.TdsParserStateObject.ReadBuf fer()
at System.Data.SqlClient.TdsParserStateObject.ReadByt e()
at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
at System.Data.SqlClient.SqlDataReader.ConsumeMetaDat a()
at System.Data.SqlClient.SqlDataReader.get_MetaData()
at System.Data.SqlClient.SqlCommand.FinishExecuteRead er(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderT ds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
at System.Data.SqlClient.SqlCommand.RunExecuteReader( CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
at System.Data.SqlClient.SqlCommand.RunExecuteReader( CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(Com mandBehavior behavior, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(Com mandBehavior behavior)
at ASDirectoryPlugin.DBHelp.ExecuteReader(SqlConnecti on connection, SqlTransaction transaction, CommandType commandType, String commandText, SqlParameter[] commandParameters, SqlConnectionOwnership connectionOwnership)
at ASDirectoryPlugin.DBHelp.ExecuteReader(String connectionString, CommandType commandType, String commandText, SqlParameter[] commandParameters)
at ASDirectoryPlugin.DBHelp.ExecuteReader(String connectionString, CommandType commandType, String commandText)
at ASDirectoryPlugin.ASContactDatabaseManager.GetCont actEntryByMatchingField(String loggedInDomainUser, String stationDN, String stationDNFieldName, String agentID, String agentIDFieldName)
[Mon Mar 14 08:07:05.984],Major,1776,ASWorkItemCreation.Worker.RefreshOneMe diaStorePrograms():
gtcp://VSPCCE01.central-insurance.com:29097/EmailQueueManager.rem
Genuine channels operation exception: The remote host has lost all information about us. It is recommended to resubscribe to all remote host's events.
[Mon Mar 14 08:07:06.750],Status,1776,ASMediaController.MediaController.DoM ediaTypeRegistered(): media type has registered.
[Mon Mar 14 08:10:06.958],Major,1776,ASWorkItemCreation.Worker.RefreshOneMe diaStorePrograms():
gtcp://VSPCCE01.central-insurance.com:29097/EmailQueueManager.rem
Genuine channels operation exception: The response has not been received within the specified period of time.

[droart]

The logs that you provided don't line up with an email being delivered. You may need to turn logging up to 1 on the profiles that are having this issue.

But speaking to the issue without having the correct logs.. This could happen if the Media Controller Plug-in loses the connection to the Media Director. Another issue that I have seen is when the Media Proxy service locally on the host is stopped. But since a restart of the client fixes the issue i doubt it is that.

What version of CCE are you running?

[rellerbrock]

Are you talking changing the logging level at the desktop? As this didn't affect everyone it's difficult to foresee who will have the issue. Changing it on everyone's is difficult as we have 125 CCE users. I am not aware of a way to mass change the logging level for agents.

We are running CCE 4.1.3.

I have seen the issue with the Media Proxy before and would agree that is probably not the issue here.

We made a few changes since this was an 8 a.m. issue and the error referenced SQL. This last Monday was much better. Not sure if it matters but our SQL server only reboots monthly. The Monday we had the blank emails the SQL server had just restarted the night before. It will be restarting again this Sunday so will see if that could be causing the issues..

In case it would help others, here is what we have done.
- Ran stored procedure to create new AMSDataX database. The current one was over 3 GB. I found out Avaya likes this to stay under 2GB

- Adjusted the email queues to poll emails after hours. Our implementation vendor set these up to start polling at 8:00 a.m when the schedules for the queues start delivering email.

- Our SQL administrator moved the CCE databases to a separate drive on our SQL server.

.

[sfitzg]

The logs indicate that the client lost a .net remoting connection with the email mediastore queue and media director. The connection to Media Director was re-established but by the end of the snippet the connection to Email mediastore had not been. You would appear to be having network related issues and the Media Proxy log may help identify the problem.

As an aside, have the emails polled continuously. If you turn off polling, then re-enable it you could potentially have large numbers of emails to download and process which could be resource intensive. Your reporting is not an accurate representation of email processing either.

Simon

[rellerbrock]

Originally we didn't have emails poll continuously. I recently changed it so they would poll continuously. Unfortunately we had emails not get processed when the server restarted. It seemed like any emails that were received after the queue closed and before the server restarted never got processed. I have a case open with Avaya on this but have made much progress.

The original log entries I included were from the Media Proxy log. Is there more I can include?

It seems strange that this is a network issue when we don't experience any issues with any of the other applications our users use in the morning.

[sfitzg]

The log trace above shows an SQL timeout, which is probably a busy SQL server or an incorrect configuration for ASContact rather than a network issue. This will make the CCE application 'Busy' for a long period of time and may be causing other things to timeout.

Make sure when the agent first logs using their agent ID that they are in AUX work mode and not available. Then ensure the application is fully started before the agent goes available. See if that resolves your morning startup problem.

Failing that full information level CCE Desktop logs would be required to investigate further.

[rellerbrock]

When users log in, they are in an AUX state. They have to make themselves available. On busy mornings, the agent login can take awhile. Making themselves available goes quickly.

You mention an incorrect configuration for the ASContact database. Where can I verify that configuration? Is it possible that the "Preview Contact" is causing load on SQL? Can this be disabled as we really don't use the "Preview Contact" feature?

I will look at changing the logging to 1 on some user's profiles. Maybe I can catch the issue on one of them Monday.

[sfitzg]

1) Does the directory plugin work once CCE Desktop is running. If so then the configuration is correct and it is probably just a timeout. If not it is probably configured wrong.

2) yes Preview Contact can place a load on the system. Disabling the service as a test might help isolate your problem.

3) Consider disabling plugins in CCE Desktop that are unused, such as Session Notes for example. Do this carefully as there is some reliance from one plugin to another (and no it is not documented) so trial a test configuration first before applying to general users. Examples of problems for example is large content in session notes with spell checker enabled, so on startup the the application will run slow.

[rellerbrock]

The directory plugin works as our users do use that to populate the "to" field in emails for internal addresses they don't want to type.

I will look into disabling the Preview Contact services to see if it helps. I assume the Preview Contact Plug should also be disabled. Correct?

I have disabled the DMCC plugin before because it was causing errors in the logs. I will also look into disabling some more plugins. We don't use Session Notes so that will be one of the first.

[shadowpax]

hi,, does anyone of you encounter the issue in CCE which the desktop application suddenly hang for about 15 seconds and this will occur every 20 seconds approximately... ?

thanks

[sfitzg]

Generally this is because a connection cannot be made to the XML server. It attempts to reconnect to the XML server every 15 seconds. Review the log files to check what is happeing.

[k2cinswedenab]

Yes,

To fix it i had to disable DMCC plug-in, i believe it was the one causing it, but i have disabled a couple of more things that wasn't needed.

Regards
Andreas