Digital Stations Working Great But IP Phones Problems

[rogund]

I have an IP Office 500v2 with 8 IP1608 & 1616 handsets and two digital stations which has been working fine for months. Suddenly last week, the IP phones started experiencing voice quality problems, the keypad on the handsets no longer works if you connect to an outside line to a menu system. Lines cuts out instead of dialing. Echoes when you are talking to external calls.
In the meantime, the two digital stations are working great. I think this rules out line problems.
I had version 7.x firmware. I have since upgraded to version 8.x still no luck with the IP phones
Has anyone got any idea please?
TX

[pdgavin]

Digital Stations Working Great But IP Phones Problems

Are you using SSA to troubleshoot the issues? Do you have SSA setup to collect QoS information?

Pete

[rogund]

No, Not using SSA. I also don't think this is a QoS issues. I have eliminated the fact that it could be the PoE switch because if I powered one of the IP phones using a PoE injector and connect directly to the LAN port of the IP Office 500 box,the phone works i.e incoming and Outgoing calls are ok. However I still get the same issues where it echoes back, calls not completing sometimes, when you dial an external number requesting you to press 1 for this and 2 for that..pressing these numbers on the phone does not send the numbers to the other end...same issues whichever IP Phone I use..1608,1616 or even 9641G, Makes you think all the phones can't have corrupt firmware?

[pdgavin]

Digital Stations Working Great But IP Phones Problems

Sounds like a line issue.

Pete

[rogund]

Pete, that's was my initial suspicion. However the digital handsets are all working fine. If I call out BT to check line and can't find problem then I have to pay £180. BT already said can't find problem on the line

[richa83]

Have you tried rebooting the kit? ...this could be vcm related , also run monitor take some traces, if you have development tracing ticked on the system tab , you can look at RTP traffic, packet loss(true packet loss not like ssa) and obviously look at the traces to define what is potentially causing the issue.

If you have a power block try rebooting the switch first first and if no fix, a different switch as even if the phones are not powered by the poe they are still routing all the packets over it.

[pdgavin]

Digital Stations Working Great But IP Phones Problems

Rather than dismiss the idea that it may be QoS issues why not use the free and very easy to use tools that are built into the phones and the IP Office. It is very easy to setup and if you set it up in the System/Events tab to email you, you don't even have to do anything except sit back and wait for an email. If you can run it for a week and you are not notified that there was a problem then you can safely eliminate it as a possible cause.

"When you have eliminated the impossible, whatever remains, however improbable, must be the truth". Sherlock Holmes.

http://marketingtools.avaya.com/knowledgebase/businesspartner/ipoffice/mergedProjects/manager/configuration.htm

QoS Parameters: Release 5.0+.
These parameters are used if Enable RTCP Monitor on Port 5005 is selected (Systems | LAN1 | VoIP). They are used as alarm thresholds for the QoS data collected by the system for calls made by Avaya H.323 phones and for phones using VCM channels. If a monitored call exceeds any of the threshold an alarm is sent to the System Status application. Quality of Service alarms can also be sent from the system using Alarms.
•The alarm occurs at the end of a call. If a call is held or parked and then retrieved, an alarm can occur for each segment of the call that exceeded a threshold.
•Where a call is between two extensions on the system, it is possible that both extensions will generate an alarm for the call.
•An alarm will not be triggered for the QoS parameters recorded during the first 5 seconds of a call.
•Round Trip Delay (msec): Default = 350.
Less than 160ms is high quality. Less than 350ms is good quality. Any higher delay will be noticeable by those involved in the call. Note that, depending on the compression codec being used, some delay stems from the signal processing and cannot be removed: G.711 = 40ms, G.723a = 160ms, G.729 = 80ms.
•Jitter (msec): Default =20.
Jitter is a measure of the variance in the time for different voice packets in the same call to reach the destination. Excessive jitter will become audible as echo.
•Packet Loss (%): Default = 3.0.
Excessive packet loss will be audible as clipped words and may also cause call setup delays.
Round Trip Delay

Good Quality

High Quality

Round Trip Delay

< 350ms

< 160ms

Jitter

< 20ms

< 20ms

Packet Loss

< 3%

< 1%

http://marketingtools.avaya.com/knowledgebase/businesspartner/ipoffice/mergedProjects/manager/alarms.htm

SSA:

Quality of Service Alarms

IP Office 5.0+ supports Quality of Service (QoS) monitoring for IP Office extensions. This is enabled through the Enable RTCP Monitoring on Port 5005 (System | LAN1 | VoIP) setting within the IP Office configuration.
The current quality of service information for a call is displayed within SSA on the extension's Extension Status form. That information is displayed for Avaya H323 IP phones registered with the IP Office. It is also displayed for other extension when they are on a call involving an IP Office VCM channel.
The thresholds for quality of service alarms are set within the IP Office configuration (System | System Events | QoS Parameters). Separate thresholds are set for Round Trip Delay (default 350ms), Jitter (default 20ms) and Packet Loss (0.5%). At the end of a call where any one of the thresholds has been exceeded, the IP Office will output an QoS alarm containing details of the call and the maximum value of each of the QoS measures during the call.

For calls that are held or parked and then resumed, separate QoS alarms may be output for each segment of the call. If the call involves several extension, separate alarms may also be output for each extension.

http://marketingtools.avaya.com/knowledgebase/businesspartner/ipoffice/mergedProjects/ssa/ssa_quality_of_service_alarms.htm

Is it only a single phone or is it all IP Phones. If it is all IP phones what is the likelyhood that they all went bad at the exact same moment? If they were working fine and all of a sudden they started having problems it is unlikely to be software releated. I suppose it could be the IP Office but then why are the digital phones working okay? It may be the VCM or it may be the lan1 but I haven't heard of those going bad although I suppose it is possible. It is more likely that something changed on the network, like a duplicate IP address or a broadcast storm or a bad nic card causing chatter on the network. If you were going to escalate the case to Avaya Support they would want to have a network diagram and network assement and system monitor traces along with sniffer traces. SSA and IP Office system Event monitoring should be your first step. It is extreemly easy to setup and if it passes at least you can move on to other possible causes before you escalate.

Peter Gavin| Technical Support Engineer|ATAC - IOC|8744 Lucent Blvd. |Rm 446E276 |Avaya - ACA, ACS-M, ACS-I, ACE, ACSS – SME Communications

[reid12]

We have similar problems with "all" our IP Office deployments with IP Phones. We have run QOS tests with testing gear, and have found no issues with the network. Keep in mind with all our IP Office deployments they are local LAN networks, with 1 or 2 network switches at most.

As well, digitized voice and one way talk path issues, typically can be controlled by the network. But echo and volume issues, are typically a sign of issues with digital to analog conversions. When using digital stations, there is no conversion since going from analog (phone set) to analog (carrier signal). When going from an IP Phone to a carrier signal that is analog, then you will have a conversion, and hence a potential for echo or volume issues. Would seem logical then if using an IP Phone, may be better to use SIP trunking to the carrier since it will stay digital to digital.

Thus, with the current scenario where we have IP Phones and a carrier signal that is an analog circuit, we have still not found a resolution. Customers have all been complaining about echo problems. We have changed out analog cards within the IP Office and still no resolve.

Is anyone else having these issues? Has anyone else found a fix for this problem?

We tried adjusting the echo gain parameters, but has minimal impact.

I have heard that if using AT&T as the carrier, may have better results if having to keep with analog circuits. Anyone using AT&T for their PSTN circuits?

[asimbu]

Issue with echo when making to external call

I have similar issue. Call between IP phones are excellent but when making external call to mobile and landline over PSTN; echo can be heard during 2 - 3 second of call then disappeared. How do I check on this issue?

[furrerm]

You have not indicated what type of trunks you have.

And, as Pete has mentioned, do not dismiss QoS.

Are you using CCR? If so, Are the CCR agents using IP sets? Are those agents in alot of hunt groups?

Are Vlans set up for voice? Who set them up?

[pdgavin]

Echo is a very complicated issue to trouble shoot. There are several types of echo and even multiple causes of the same type. To make things even more complicated you may have several of these issues at the same time. You also did not state who is hearing the echo. For the most common cause of echo, in general if you hear echo the problem causing it is usually on the other end so are the people inside the office hearing it or are people calling in hearing it? Imagine yourself in a cavern and shout hello. Your voice bounces off the wall and comes back to you. If the wall wasn't there you would not hear the echo. Please note that although the problem may be on the other end you may be able to eliminate or reduce it on your end but in most cases you will have to eliminate the problem at the source, if the problem is caused by someone on an analog phone line calling into your system and their line is not properly setup there may not be a lot you can do on your end. A PRI has 4 wires and the send and receive talk path are each isolated on their own pair and you have echo cancelation circuitry built into the PRI lines that buffers several milliseconds of signal and looks to see if that signal repeats and if it does it drops that echo. Analog lines are only 2 wires so both the send and receive are on the same pair, because the send and receive are not isolated your voice can bounce off the analog set and get reflected back to you. If the analog lines are not properly balanced/provisioned then they can be the source of echo.

Delay can cause problems with the echo cancelation circuitry, this is because the echo cancellation buffers are only so big and if the delay is larger than the buffer the echo slips through. Delay is cumulative, multiple conversions from VoIP to TDM and back will cause more and more delay. One way to help with this is to change/lock down the codec in the IP Office that the phones use to G711 to reduce the latency inherent in compressing speech, think of ripping a CD to an MP3. It takes time to process the sound:

Latency: Less than 180ms for good quality. Less than 80ms for toll quality.
This is the measurement of packet transfer time in one direction. The range 80ms to 180ms is generally acceptable. Note that the different audio codecs used each impose a fixed delay caused by the codec conversion as follows:
G.711: 20ms.
G.722: 40ms.
G.729: 40ms.

You can also try the G.722 codec on the 9600 series phones this will improve the bandwidth and sound quality without significantly increasing the delay.

If you are using analog lines there is an impedance wizard you can run to try to correct a poorly engineered analog trunk if you are using analog trunks on your IP Office. It will not do anything if the problem is on the other end but if you have analog lines you should use this tool to make sure the impedance settings are at the optimum value.

If you are an authorized Avaya business partner there are a couple of tech tips written on echo, they are available on the IP Office Knowledgebase if you are an authorized BP. Tech tip 141 and 195. There are other documents on the support web site as well and you can probably find them outside Avaya’s web sites too.

Other things you can try to improve the overall voice quality on IP sets:


You can also turn off AGC on the phone for the headset, handset and speaker phone depending upon where you are having the problem. There are also other audio environment settings that can be changed by the setting file for an IP phone but there are several hundred values so you really should seek out help in setting them.



Please note that these are a few very simple explanations and suggestions of a very complicated issue. Even for experienced engineers troubleshooting echo is a very complicated issue. It requires that you define exactly what type of echo you are experiencing and where the source is and what can be done to improve the issue. I would recommend you create a trouble ticket and escalate it to Avaya Backbone support if you are having issues. If we had total control of all the components from handset to handset we wouldn’t have any of these issues When you use digital phones we eliminate the customers network equipment and the conversion from TDM to VoIP which simplifies things, unfortunately the lack of total control makes it more complicated for IP phones.


Pete G

[pdgavin]

More on echo


3.3 Echo
There are only two physical sources of echo in telephony: electrical echo (or network echo), and
acoustic echo. Electrical echo is caused by a reflection of the speech signal at 2-to-4-wire hybrid
circuitry. This circuitry is present in analog trunk cards, and it also exists deep within the PSTN
(at customer premises, for example). Acoustic echo is caused by the physical coupling (air path,
appliance-body path) between a loudspeaker and a microphone, for example, in a
speakerphone, a handset and a headset. Whether or not a talker actually perceives electrical or
acoustic echo depends on the loudness of his/her reflected voice signal and the roundtrip delay
that that reflection suffers. The loudness of the reflection at the point of reflection depends upon
the electrical impedance mismatch, for electrical echoes, and the acoustic gain of the
loudspeaker-to-microphone path, for acoustic echoes. The roundtrip delay is a function of the
path the reflected signal traverses, which in turn is a function of the call topology.
3.3.1 Electrical echo, also called network echo: reflection of a talker's speech signal at a
point of 2-to-4-wire conversion caused by an impedance mismatch at the point of
analog-to-digital conversion.
3.3.2 Acoustic echo: reflection of a talker's speech signal at an acoustic endpoint caused by
the acoustic coupling between the loudspeaker and microphone.
3.3.3 Constant echo: when talking, the perception of echo with every utterance. Such cases
occur when there is a physical electrical or acoustic echo path but no echo controller in
the call topology to control echo. Additionally, constant echo may result even though an
echo controller is known to be in the call path; this indicates a complete failure of the
echo controller, usually because the capabilities of the echo controller are exceeded
(e.g., the echo tail length exceeds the specifications of the echo controller).
3.3.4 Intermittent echo: when talking, the occasional perception of echo. Intermittent echo
often caused by the intermittent failure of an echo controller in the call path. The echo
suppressor within the echo controller may fail to engage (to apply echo attenuation)
when necessary, with the result that short bursts of echo become audible. In acoustic
echo control applications (speakerphone) in which people or objects close to the
speakerphone are moving, the change to the physical echo path often results in audible
intermittent acoustic echo to listeners at the other end of the call.
3.3.5 Residual echo: when talking, the perception of very low-level (quiet) echo. The echo
could be either constant or intermittent. Residual echo can be caused by PSTN
electrical echo that is not entirely removed by the echo controller in the call path.
3.3.6 Distorted or buzz-like echo: when talking the perception of a distorted echo or buzzlike
sound. This can be caused by a non-linear echo source. An example of this is
saturation distortion at an analog trunk interface. In this case, signals low in amplitude
are reflected cleanly, but signals high in amplitude are returned with significant distortion
making it difficult for an echo canceler to control echo. Such distorted echo can be
perceived constantly or intermittently, depending on the degree of distortion and the
echo canceler(s) involved.
3.3.7 Slapback or kickback acoustic echo: this is strictly a phenomenon of acoustic echo.
With speakerphones, slapback or kickback echo is the intermittent echo perceived at the
ends of one's utterances. This can occur with both older-model half-duplex
©2005 Avaya Inc. All Rights Reserved. Page 6
speakerphones and newer-model acoustic-echo canceling speakerphones. For
example, a talker speaking into a handset utters the phrase “Please send me the check”
and perceives echo primarily at the end of his/her sentence. This echo is described as
hearing just the sound “eck” or “k” of the word “check,” or as a slapping sound such as
that made by slapping one’s palm against a desktop. Commonly, slapback/kickback
echo is caused by acoustically reverberant rooms. Large offices and conference rooms
can have long reverberation times. In such rooms, the speakerphone senses at its
microphone a reverberated version of the word “check” (our prior example) several tens
or even hundreds of milliseconds after the far talker has finished saying the word
“check.” The speakerphone algorithm detects this reverberated speech at its
microphone, detects no speech at its receive-path driving the loudspeaker, and decides
to transition to transmit mode. The reverberated version of "check" is transmitted back to
the far talker, where it is perceived as echo.
3.3.8 Sidetone: in handsets and headsets, a portion of the microphone energy is fed back to
the earpiece so that the user of the handset/headset does has a psychoacoustic
experience that simulates the case in which the user's ear is not occluded by an object
(the handset earpiece). Without sidetone injection, the user experiences the
psychoacoustically bothersome condition that can be demonstrated to oneself by
pressing a finger into one ear while speaking. With one ear occluded, the sound of one’s
own voice is dominated by the path through the interior of the head (skull, etc.) instead
of around the head, an effect that most people find objectionable.
3.3.9 Hot sidetone: in a handset or headset, microphone-to-earpiece sidetone injection is not
normally noticed. Some digital phones, in particular, IP phones in which the internal
audio processing frame rate is 5 ms or greater, inject sidetone with an appreciable delay
(e.g., 5 ms) in the microphone-to-earpiece signal path. This delay causes the sidetone to
sound reverberant and/or louder than normal, or hot. Though hot sidetone is a type of
echo source – because some people may use the term “echo” to describe hot sidetone
– it is generated local to the telephone, not at some point within the telephone network.


Pete G

[pdgavin]

even more on echo

3.3.10 Short-path acoustic echo, short-path electrical echo: acoustic or electrical echo that
occurs in a very short roundtrip call topology. This type of echo is commonly described
as a hollow sound or sound of speaking in a barrel (see 2.1.3). In a digital-to-digital
phone call (think DCP-to-DCP), station-to-station, the roundtrip delay is usually very
small, less than 10 ms. Some digital speakerphones produce significant acoustic echo,
which is not canceled, suppressed, or otherwise controlled in this simple call topology. In
these cases, and depending on the volume setting of the far-party’s speakerphone and
near-party’s listening handset, the near party may perceive echo and refer to this as hot
sidetone. Again, this is truly acoustic echo from the speakerphone but is returned to the
talker with such a short roundtrip delay that it is perceived as hollowness or
reverberance rather than as classic echo. Because of the short roundtrip delay in this
case, it can be difficult to distinguish between hot sidetone (see definition) and short-path
echo.


Pete G

[asimbu]

You have not indicated what type of trunks you have.

And, as Pete has mentioned, do not dismiss QoS.

Are you using CCR? If so, Are the CCR agents using IP sets? Are those agents in alot of hunt groups?

Are Vlans set up for voice? Who set them up?

We using analogue trunk lines. By the way, our company based in Brunei and I'm not sure if the IP office needs to match with local telephony provider trunk lines?

[jtobon]

En la central IP Office release 7.0 (32) y en 8.0 (44) en adelante se corrige la falla que se te biene presentando con los ip phones 16XX.
cambia en lineas analogas/opciones analogicas/ Ganancia en ambos campos pasarla a -2.5 Db y se te corrige la falla, recuerda que requiere de reinicio.