G700 Traps

G700 Traps

G700 Media Gateways (serving either as standalone port networks, or as port networks within an IP Multi-Connect system) report alarms to the primary server (either an S8300 or S8700 Media Server) using SNMP traps. Like the primary server's own alarms, alarms from a G700 Media Gateway:

Reside in the primary server's alarm log.
Can be viewed using the SAT command display alarms
Can be viewed using the Web Interface Display Alarms option.
Can be viewed using the server's Linux command almdisplay -v

However, the format of these displayed alarms is slightly different. Using the G700 MO's Event�ID�#1 (the first entry in Table 5) as an example, a displayed G700 alarm has the following format:

n CMG 1 WRN 07/17/2002:13:45 121.1.1.2:cmgMultipleFanFault

Within the previous alarm-display string, the value:

"n" is a sequential alarm ID.
"CMG" identifies a G700 Media Gateway as the MO.
"1" is the event's ID (1st column of Table 5).

This table also contains each alarm's corresponding SNMP trap # in the 2nd column of Table 5. However, many of the MIB-defined traps have been excluded, either because:

A specific trap (such as, Trap #3) is the SNMP mechanism to clear an alarm logged by another specific trap (in this case, Trap #2).
The specific event indicated by a trap is not severe enough to justify an entry in the primary server's alarm log.
A trap is defined, but not implemented.
A trap # is reserved for future use.

"WRN" is the event's severity (3rd column of Table 5).
"07/17/2002:13:45" is the event's date and time stamp.
"121.1.1.2" is the IP address for Telnet access to the alarmed G700 Media Gateway Processor (MGP).
"cmgMultipleFanFault" is an alarm description (4th column of Table 5).

Although these alarms can be viewed from the primary server, they are normally resolved from within the G700 Media Gateway. The G700 generates the following traps. Follow the error resolution procedures in Table 5 to resolve errors indicated by these traps.

Table 5. G700 Alarms (SNMP Traps) to the Primary Media Server
Event ID	Trap #	Alarm Level	Description / Recommendation
1	2	WRN	cmgMultipleFanFault -- At least two G700 fans have been operating at less than 90% of their nominal speed for >= 5 minutes. (This may be an early warning of overheating.) Verify there are faults in the system. Use the Avaya G700 Media Gateway Processor (MGP) Command Line Interface (CLI) command show faults to display any faults on the G700. If there is a fan/temperature fault, check to see if the fans are working, and/or if there is sufficient space around the G700 for air circulation. Maintenance software monitors voltages applied to the Media Modules and other components of the G700, and compares these to the general power supply unit (PSU) status bit. If none of these voltages are out of tolerance, but the PSU status indicates failure,this generates the fan fault, which will be indicated in the show faults command output. Replace the entire G700. Fans and the PSU are not field replaceable.
2	4	MIN	cmgPsuBriefFanFault -- A G700's PSU fan has been operating at less than 90% of its nominal speed for between 5 and 15 minutes. (This may be an early warning of overheating.) Verify there are faults in the system. Use the Avaya G700 Media Gateway Processor (MGP) Command Line Interface (CLI) command show faults to display any faults on the G700. If there is a fan/temperature fault, check to see if the fans are working and/or if there is sufficient space around the G700 for air circulation. Maintenance software monitors voltages applied to the Media Modules and other components of the G700, and compares these to the general power supply unit (PSU) status bit. If none of these voltages are out of tolerance, but the PSU status indicates failure, this generates the fan fault, which will be indicated in the show faults command output. Replace the entire G700. Fans and the PSU are not field replaceable.
3	6	MIN	cmgPsuProlongedFanFault -- A G700's fan has been operating at less than 90% of its nominal speed for >= 15 minutes. (This may be an early warning of overheating.) Verify there are faults in the system. Use the Avaya G700 Media Gateway Processor (MGP) Command Line Interface (CLI) command show faults to display any faults on the G700. If there is a fan/temperature fault, check to see if the fans are working and/or if there is sufficient space around the G700 for air circulation. Maintenance software monitors voltages applied to the Media Modules and other components of the G700, and compares these to the general power supply unit (PSU) status bit. If none of these voltages are out of tolerance, but the PSU status indicates failure, this generates the fan fault, which will be indicated in the show faults command output. Replace the entire G700. Fans and the PSU are not field replaceable.
4	10	MIN	cmgCpuTempWarningFault -- A G700's CPU temperature sensor has exceeded its warning threshold. Verify there are faults in the system. Use the Avaya G700 Media Gateway Processor (MGP) Command Line Interface (CLI) command show faults to display any faults on the G700. If there is a temperature fault, turn off the G700 and allow it to cool. Reboot the G700. Check to see if the fans are working and/or if there is sufficient space around the G700 for air circulation. Use the MGP CLI show faults command to check for fan problems. Low voltage may be responsible for slower fans. Voltage may be reduced by a short in one of the Media Modules or a bad power supply. If there are no fan faults, use the MGP CLI command show voltages to display voltages applied to components on the motherboard and to the Media Modules. If the Media Module voltage is out of tolerance, systematically, remove each Media Module to determine if one of the Media Modules is responsible for reducing the voltage level. If one is found, replace the Media Module. If no Media Module is found to be bad, the power supply is suspect. Replace the G700.
5	12	MIN	cmgDspTempWarningFault -- The temperature sensor in a G700's DSP complex has exceeded its warning threshold. Verify there are faults in the system. Use the Avaya G700 Media Gateway Processor (MGP) Command Line Interface (CLI) command show faults to display any faults on the G700. If there is a temperature fault, turn off the G700 and allow it to cool. Reboot the G700. Check to see if the fans are working and/or if there is sufficient space around the G700 for air circulation. Low voltage may be responsible for slower fans. Voltage may be reduced by a short in one of the Media Modules or a bad power supply. If there are no fan faults, use the MGP CLI command show voltages to display voltages applied to components on the motherboard and to the Media Modules. If the Media Module voltage is out of tolerance, systematically, remove each Media Module to determine if one of the Media Modules is responsible for reducing the voltage level. If one is found, replace the Media Module. If no Media Module is found to be bad, the power supply is suspect. Replace the G700.
6	14	MAJ	cmgTempShutdownFault -- The temperature sensor in a G700's CPU has exceeded its shutdown threshold. The system is about to begin controlled shutdown. Verify there are faults in the system. Use the Avaya G700 Media Gateway Processor (MGP) Command Line Interface (CLI) command show faults to display any faults on the G700. If there is a temperature fault, turn off the G700 and allow it to cool. Reboot the G700. Check to see if the fans are working and/or if there is sufficient space around the G700 for air circulation. Issue MGP CLI command show voltages to determine voltages for Media Modules and other components of the G700. While the +12V fan supply is not directly monitored, it is included in the general power supply unit (PSU) status bit. Use the show faults command to determine if there is a fan problem. If none of the monitored voltages are out of tolerance, but the PSU status indicates failure, this would generate a fan fault. If fans are faulty, then replace the entire G700. Fans are not field replaceable. Voltage may also be reduced by a short in one of the Media Modules. Systematically, remove each Media Module to determine if one of the Media Modules is responsible for reducing the voltage levels.
7	16	MAJ	cmgMgpPowerFault -- The voltage reading at the server's +5.1V power source is out of tolerance. Check voltages. Issue the MGP CLI command show voltages to determine voltages for Media Modules and other components of the G700. Voltage may be reduced by a short in one of the Media Modules or a bad power supply. Systematically, remove each Media Module to determine if one of the Media Modules is responsible for reducing the voltage levels. Replace faulty Media Module. If the alarm clears in 10-20 seconds, it was a probable voltage spike. Do not replace the G700. Use a power monitor to monitor the power line. If a brown-out condition is suspected, use a power monitor to monitor the power line. If the fault persists and the show voltages command continues to show the level is out of tolerance, then replace the G700.
8	18	MAJ	cmgMediaModulePowerFault -- The voltage reading at the Media Modules' -48V power source is out of tolerance. Check voltages. Issue the MGP CLI command show voltages to determine voltages for Media Modules and other components of the G700. Voltage may be reduced by a short in one of the Media Modules or a bad power supply. Systematically, remove each Media Module to determine if one of the Media Modules is responsible for reducing the voltage levels. Replace faulty Media Module. If the alarm clears in 10-20 seconds, it was a probable voltage spike. Do not replace the G700. Use a power monitor to monitor the power line. If a brown-out condition is suspected, use a power monitor to monitor the power line. If the fault persists and the show voltages command continues to show the level is out of tolerance, then replace the G700.
9	20	MAJ	cmgVoipPowerFault -- The voltage reading at the VOIP complexes' +3.4V power source is out of tolerance. Check voltages. Issue the MGP CLI command show voltages to determine voltages for Media Modules and other components of the G700. Voltage may be reduced by a short in one of the Media Modules or a bad power supply. Systematically, remove each Media Module to determine if one of the Media Modules is responsible for reducing the voltage levels. Replace faulty Media Module. If the alarm clears in 10-20 seconds, it was a probable voltage spike. Do not replace the G700. Use a power monitor to monitor the power line. If a brown-out condition is suspected, use a power monitor to monitor the power line. If the fault persists and the show voltages command continues to show the level is out of tolerance, then replace the G700.
10	22	MAJ	cmgDspPowerFault -- The voltage reading at the DSP units' +1.58V power source is out of tolerance. Check voltages. Issue the MGP CLI command show voltages to determine voltages for Media Modules and other components of the G700. Voltage may be reduced by a short in one of the Media Modules or a bad power supply. Systematically, remove each Media Module to determine if one of the Media Modules is responsible for reducing the voltage levels. Replace faulty Media Module. If the alarm clears in 10-20 seconds, it was a probable voltage spike. Do not replace the G700. Use a power monitor to monitor the power line. If a brown-out condition is suspected, use a power monitor to monitor the power line. If the fault persists and the show voltages command continues to show the level is out of tolerance, then replace the G700.
11	24	MAJ	cmg8620PowerFault -- The voltage reading at the VoIP Processor's +2.5V power source is out of tolerance. Check voltages. Issue the MGP CLI command show voltages to determine voltages for Media Modules and other components of the G700. Voltage may be reduced by a short in one of the Media Modules or a bad power supply. Systematically, remove each Media Module to determine if one of the Media Modules is responsible for reducing the voltage levels. Replace faulty Media Module. If the MM760 Media Module is not present, the VoIP engine on the motherboard is possibly bad. Use the show faults command to check for VoIP faults. Replace the G700. If the alarm clears in 10-20 seconds, it was a probable voltage spike. Do not replace the G700. Use a power monitor to monitor the power line. If a brown-out condition is suspected, use a power monitor to monitor the power line. If the fault persists and the show voltages command continues to show the level is out of tolerance, then replace the G700.
14	30	MAJ	cmgSyncSignalFault -- Synchronization signal lost. Check that provisioned clock-sync source has a good signal by issuing a Media Gateway Processor (MGP) Command Line Interface (CLI) command show sync timing Procedure for setting synchronization timing sources on E1/T1 MM or MM710: Be sure that the E1/T1 MM has been added properly on the Avaya S8300 Media Server or Avaya S8700 Media Server, otherwise go to System Access Terminal (SAT) and issue a ADD DS1 command before going to the MGP CLI and issuing a set sync interface or set sync source command. Otherwise the MGP CLI will not allow these commands to be executed. Go to the MGP's CLI, and first specify the primary and secondary clock sources for synchronizing the E1/T1 span, using the set synch interface command. Note: You don't specify the internal clock source from the CLI - only the primary and secondary. The local clock is "built-in" and not provisionable. Issue a set sync source command to set to the specific MM710 E1/T1 Media Module to be used as the active clock reference. Verify whether or not these commands were executed by issuing a show sync timing command to ensure that the source is provisioned and active, or visually inspect the Yellow LED on the MM710 Media Module. Note: When the Yellow LED is on 2.7 seconds and off 0.3 seconds, this means the tone-clock synchronizer is in "active" mode, and an external synchronization source is being used as a synchronization reference. Setting the sync timing was successful. When the Yellow LED is on 0.3 seconds and off 2.7 seconds, this means the tone-clock synchronizer is in "active" mode and the internal (on-board) clock is being used as a synchronization reference. Setting the sync timing was not successful. If there is more than one MM710 Media Module, and they have been set up as primary and secondary, this behavior could be on the second and not the timing of the bus. For more details, please consult the maintenance documentation under LEDs and E1/T1 Media Module, or see http://support.avaya.com/elmodocs2/S8300/cd/index.htm
15	32	MAJ	cmgVoipHardwareFault -- A DSP complex serving the VoIP engines has failed. Check IP configuration. Reset or replace Media Module.
	50		cmgModuleRemove -- A Media Module has been removed.
16	52		cmgModuleInsertFault -- The insertion sequence for a Media Module has failed.
	53		cmgModuleInsertSuccess -- A Media Module has been inserted.
	54		cmgMgBusyout -- An administrator has busied-out a Media Module or a port.
	70		cmgFirmwareDownloadBegun -- A Media Gateway began downloading a software module.
	71		cmgFirmwareDownloadSuccess -- A Media Gateway successfully downloaded a software module.
	73		cmgRegistrationSuccess -- A Media Gateway successfully registered with a controller.
	74		cmgMgManualReset -- A Media Gateway is beginning a user-requested reset operation.
	75		cmgModuleManualReset -- A Media Module is beginning a user-requested reset operation.
	76		cmgVoipManualReset -- A VoIP engine is beginning a user-requested reset operation.
	77		cmgDsuManualReset -- An E1 or T1's DSU is beginning a user-requested reset operation.
	78		cmgConfigUploadBegun -- A Media Gateway began uploading a configuration file.
	79		cmgConfigUploadSuccess -- The Media Gateway successfully uploaded the configuration file.
17	90	MAJ	cmgMemoryFault -- The Media Gateway Processor has detected a low processor memory condition. If an attempt by a software module to allocate memory fails, or if available memory falls below four million bytes, this trap is generated. If available memory later rises above five million bytes, this trap is cleared. If available memory falls below a half a million bytes, the Media Gateway Processor is automatically reset. Check the Media Gateway Processor and insure that it has the latest version of firmware installed. If it does not, install the latest version of firmware and continue to monitor. If this trap occurs infrequently and is automatically cleared, the trap may be due to an unusual transient condition. Monitor future traps. If this trap occurs frequently and is automatically cleared, it is likely that the Media Gateway Processor software has the wrong limits set for its memory monitoring. These limits are hard coded in the software. Escalate. If this trap occurs and does not clear, the Media Gateway may be functionally impaired. Do not reset the Media Gateway. Escalate and have Tier 3 personnel capture the trap information and the event logs for analysis. If this trap occurs and the Media Gateway Processor automatically resets, then a severe processor memory shortage occured. Escalate and have Tier 3 personnel capture the trap information and the event logs for analysis.
18	92	MAJ	cmgDhcpRequestFault -- The Media Gateway cannot contact its DHCP server, or the DHCP server did not respond to a request. Either correct the DHCP problem, or correct the Media Module's configuration file.
19	94	MAJ	cmgFirmwareDownloadFault -- An attempt to download a software module has failed. Check the event log to find the specific error. Troubleshoot the specific error according to the information found. For example, if the string "File not found" appears in the log, then verify that the image file: a. Exists b. Has the correct name c. Resides in the correct directory
20	96	WRN	cmgProcessRestart -- a software module on the Media Gateway Processor failed. The Media Gateway Processor will attempt to restart the failing module. A successful restart of the module will clear this trap Check the Media Gateway Processor and insure that it has the latest version of firmware installed. If it does not, install the latest version of firmware and continue to monitor. If this trap occurs infrequently and is automatically cleared, the trap may be due to an unusual transient condition.Monitor future traps. If the trap occurs frequently and is automatically cleared, it may indicate an issue with a particular software module. Reset the Media Gateway at a time convenient with the customer. Escalate and have Tier 3 personnel capture the trap information and the event logs for analysis. If the trap occurs and does not clear, the Media Gateway may be functionally impaired. Reset the Media Gateway at a time convenient with the customer and consistent with the impairment. Escalate and have Tier 3 personnel capture the trap information and the event logs for analysis..
21	98	MAJ	cmgIccMissingFault -- An internal communications controller, expected in Slot 1, is missing. Check for presence of S8300. If present, check the G700 administration. If the administration is correct, suspect the S8300.
22	100	MAJ	cmgIccAutoReset -- The Media Gateway automatically reset the internal communications controller. If the problem persists, escalate.
23	102	MAJ	cmgPrimaryControllerFault -- The Media Gateway cannot contact the first controller in its controller list. Verify that the controller list is correct. From the MGP CLI, issue the command show mgc list. The IP address should match the S8700 Media Server CLAN or the S8300 Media Server IP addresses. If needed, correct this in 'configure' mode on the MGP's CLI by clearing the mgc list first with the clear mgc list command, and then issuing a set mgc list with the correct IP addresses. If so, verify that the primary controller is up. If so, shut down every LSP.
24	104	MAJ	cmgNoControllerFault -- The Media Gateway does not have any controllers in its controller list. Verify that the controller list is empty. From the MGP CLI, issue the command show mgc list to verify that there are no controllers listed. If none are listed, correct this by adding the correct IP address of the S8700/S8300. In 'configure' mode on the MGP's CLI, issue a set mgc list command with the correct IP address.
25	106	MAJ	cmgRegistrationFault -- The Media Gateway cannot register with any controllers in its controller list. Verify that the controller list is correct. From the MGP CLI, issue the command show mgc list. The IP address should match the S8700 Media Server CLAN or the S8300 Media Server IP addresses. If needed, correct this in 'configure' mode on the MGP's CLI by clearing the mgc list with the clear mgc list command, and then issuing a set mgc list with the correct IP addresses. If the IP address in the mgc list matches the S8700 Media Server CLAN or the S8300 Media Server IP addresses, there may be a network problem. Verify that the primary controller is up.
26	108	MIN	cmgH248LinkDown -- An H.248 link between the Media Gateway and its controller is down. Check the S8300 or S8700. If down, bring up. If not, check the G700 administration. Since the following command causes a brief service outage, it should only be executed at the customer's convenience. If the administration is correct, reboot the G700. If the problem persists, check network connectivity. Use ping or traceroute to the S8300/S8700 to check connectivity. If the problem persists, escalate.
29	114	WRN	cmgMgAutoReset -- The Media Gateway Processor automatically reset (rebooted). The processor automatically resets when a critical error occurs from which it cannot recover. The error may be software or hardware related. It may also automatically reset in response to a maintenance test running on the call controller. (A reset performed at the request of a SAT or CLI command entered by craft or Tier technical support generates a different trap indicating a manual reset.) Finally, it may automatically reset when it reregisters with a call controller after being out of touch for too long. This trap is is generated as the Media Gateway Processor comes back up after resetting. If the Media Gateway Processor resets and fails to come back up, this trap will not be generated. Check to see if a maintenance test that is supposed to reset the processor was run. Check that the reset was not due to the link with the call controlling going down. If the reset is due to a link failure with the call controller, follow call controller link failure troubleshooting procedures. Check the Media Gateway Processor and insure that it has the latest version of firmware installed. If it does not, install the latest version of firmware and continue to monitor. If this trap occurs infrequently, the trap may be due to an unusual transient condition. Monitor future traps. If this trap occurs and the Media Gateway Processor is frequently resetting, manually reset the media gateway. Escalate and have Tier 3 personnel capture the trap information and the event logs for analysis. If this trap occurs frequenty and the Media Gateway Processor is not resetting, the Media Gateway may be functionally impaired, and is not capable of resetting itself to restore service. If service is impaired, reset the Media Gateway manually. Escalate and have Tier 3 personnel capture the trap information and the event logs for analysis.
30	116	WRN	cmgModuleAutoReset -- A Media Module in the Media Gateway automatically reset (rebooted). A Media Module automatically resets when it fails a sanity test performed by the Media Gateway Processor. It may also automatically reset in response to a maintenance test running on the call controller. (A reset performed at the request of a SAT or CLI command entered by craft or Tier technical support generates a different trap indicating a manual reset.). Check to see if a maintenance test that is supposed to reset the Media Module was run. Check the Media Module and insure that it has the latest version of firmware installed. If it does not, install the latest version of firmware and continue to monitor. If this trap occurs infrequently, the trap may be due to an unusual transient condition. Monitor future traps. If this trap occurs and the Media Module does not return to service, or if this trap occurs frequently, attempt to reset the failing module from the SAT or CLI and see if this returns it to stable service. If manually resetting the Media Module does not return it to service, and if a spare Media Module of the same time is available, replace the failing Media Module with the spare and see if the spare Media Module goes into service. If so, follow procedures for dealing with the original, bad, Media Module. If the spare Media Module fails to go into service, it is of course possible that the spare Media Module is bad as well. But that aside, try manually resetting the Media Gateway Processor at a time convenient to the customer and see if this restores service. If so, the both the original and the spare Media Modules can probably be considered okay, and the problem is probably with the Media Gateway Processor itself. Escalate and have Tier 3 personnel capture the trap information and the event logs for analysis.
32	118	MIN	cmgModulePostFault -- A Media Module failed its power-on start-up test. Reset or replace the Media Module.
34	122	MAJ	cmgConfigUploadFault -- An attempt to upload a configuration file failed. Check the event log for an error message during the backup/restore process. Troubleshoot the specific error according to the information found. Retry the upload (backup) command; for example: copy mgp-config tftp <filename> <ipaddress> Since the following command causes a brief service outage, it should only be executed at the customer's convenience. If the problem persists, reboot the G700.
37	128	WRN	cmgVoipAutoReset -- A VOIP (Voice Over IP) module in the Media Gateway automatically reset (rebooted). A VOIP module automatically resets when it fails a sanity test performed by the Media Gateway Processor. It may also automatically reset in response to a maintenance test running on the call controller. (A reset performed at the request of a SAT or CLI command entered by craft or Tier technical support generates a different trap indicating a manual reset.) It automatically resets when its IP address is administered. Finally, it automatically resets when it fails a ping test performed by the Media Gateway Processor against the VOIP module's IP address. Check to see if a maintenance test that is supposed to reset the VOIP module was run. Check to see if the VOIP module had its IP address readministered. Check to see if the IP address administered on the VOIP module is correct. Check to see if the IP address of the Media Gateway itself can be pinged. Physical or logical connectivity issues (cabling or routing problems) in the data network can cause ping failures. Check the VOIP module and insure that it has the latest version of firmware installed. If it does not, install the latest version of firmware and continue to monitor. If this trap occurs infrequently, the trap may be due to an unusual transient condition. Monitor future traps. If this trap occurs and the VOIP module does not return to service, or if this trap occurs frequently, attempt to reset the failing module from the SAT or CLI and see if this returns it to stable service. If manually resetting the VOIP module does not return it to service, and if a spare VOIP module of the same type is available, replace the failing VOIP module with the spare and see if the spare VOIP module goes into service. If so, follow procedures for dealing with the original, bad, VOIP module. (Steps continued on next page)
37	128	WRN	If the spare VOIP module fails to go into service, it is of course possible that the spare VOIP module is bad, as well. There may be a power issue, also. Try manually resetting the Media Gateway Processor at a time convenient to the customer and see if this restores service. If so, both the original and the spare VOIP modules can probably be considered okay, and the problem is probably with the Media Gateway Processor itself. Capture the trap information. If possible, capture the event logs, using the show event-log CLI command, for analysis. Escalate. If none of this works, capture the trap information. If possible, capture the event logs, using the show event-log CLI command, for analysis. Escalate.
Continued on next page