Health Monitor Messages
This section summarizes Health Monitor messages and the conditions under which they are generated (for general information about the Health Monitor, see About the Health Monitor).
|
Note:
|
The exact text and availability of a Health Monitor message are dependent on the version of your xMatters deployment; the text of the messages displayed on your system may not be identical to the message examples described below. |
Component Misconfigured
Device Engines report the following message when a configuration change has been applied that prevents notification delivery (where possible, more detail about the source of the problem is included).
- The xMatters Health Monitor on node <node name> has detected that the <device engine name> has been misconfigured and can no longer function. Message: <detailed message>. Consult the node logs for further details.
Node Failure/Recovery
Nodes periodically update their database record with a timestamp. If a Health Monitor on a running node detects that another node has failed to update its timestamp, the first message below is sent. When a node has failed, the processing it was responsible for at the time of failure will be recovered by another running node (if there is one).
When a node that has previously failed (i.e., its heartbeats stopped) resumes its heartbeats, the second message below is sent.
Messages
- The xMatters Health Monitor on node <running node name> has detected that node <failed node name> has stopped sending heartbeats. Ensure that the node process is still running and can access the database.
- The xMatters Health Monitor on node <running node name> has detected that node <recovered node name> has recovered.
Multi-Node Communication
In some cases, node-to-node communication can fail (e.g., due to regular maintenance, network outages, faulty equipment, socket exhaustion, etc.), generating a Health Monitor message.
- The xMatters Health Monitor on node <node name> has detected that communication with an xMatters node has been lost: <node name>.
Node Network Monitor Parameters
The following Health Monitor Messages are triggered when the Node Network Monitor threshold parameters specified on the Global Configuration page are exceeded (for details, see Global Configuration):
- The xMatters Health Monitor on node <node name> has detected <communication defect> communication with AlarmPoint node <node name> over the last <number of seconds> seconds. The affected nodes will not exchange messages until bidirectional communication is established, which may result in apparent loss of service. Corrective action should be taken to allow bidirectional communication between the affected nodes. Consult the logs of the affected nodes for further details on communications errors.
- The xMatters Health Monitor on node <node name> has detected that communication between <node name1> and <node name2> has been interrupted <number of times> times for the last <number of minutes> minutes. If this consistently occurs on a specific node, ensure that the node process is active. If this occurs erratically and/or frequently, there may be networking issues affecting inter-node or node-to-database communication. Consult the logs of the affected nodes for further details.
- The xMatters Health Monitor on node <node name> has detected that communication between <node name1> and <node name2> is too slow. For the last <number of minutes> minutes, the average round trip latency was <number of milliseconds> ms.
This may be caused by a networking problem, or the load on the servers or the network may be reaching capacity.
The following messages are sent when application layer latency has recovered to within the specified parameters:
- The xMatters Health Monitor on node <node name> has detected that communication between <node name1> and <node name2> is recovered. Symmetric (i.e., bidirectional) communication has been restored.
- The xMatters Health Monitor on node <node name> has detected that communication between <node name1> and <node name2> is recovered. For the last <number of minutes> minutes, the communication interruption time has been below the configured threshold.
- The xMatters Health Monitor on node <node name> has detected that communication between <node name1> and <node name2> is recovered. For the last <number of minutes> minutes, the average round trip latency was <number of milliseconds> ms.
Node Licenses Exceeded
The following message is sent when a node could not be started without exceeding the licensed limit:
- Startup of <node name> has been aborted because the number of active nodes would have exceeded the licensed limit. Contact support @xmatters.com to obtain a license that allows more nodes.
Node Clock Off Sync
The system clocks for all components in your deployment must be synchronized; if xMatters detects that an internal clock for a node has deviated from the database clock, the Health Monitor sends the following message:
- The xMatters Health Monitor has detected that the internal clock for this node has deviated <number of milliseconds> ms from the database's internal clock. As a result, messages may be lost. Adjust the node or database clock to match. To avoid having to manually adjust the clock, consider, automatically synchronizing system clocks against an NTP server.
Unhandled Exception
The following message is sent when an unexpected error occurs.
Message
- The xMatters Health Monitor on node <node name> has detected an error: <error>.
Consult the node logs for further details.
Example
The xMatters Health Monitor on node SISC_APP_NODE1 (10.10.60.78) has detected an error: No row with the given identifier exists:[com.invoqsystems.foundation.interpreter.invoqscript.persistance.
ScriptObjectProxy#214410]; nested exception is org.hibernate.
ObjectNotFoundException: No row with the given identifier exists: [com.invoqsystems.foundation.interpreter.invoqscript.persistance.
ScriptObjectProxy#214410].
Consult node logs for further details.
Troubleshooting
To troubleshoot an unhandled exception alert, analyze the logs on the node that caused the exception:
- Search the log for the exception detailed in the Health Monitor email.
- Each log entry contains a thread number. Search the log for the entries corresponding to the thread number that generated the exception, and look for an Event ID or a target name (Recipient) in those log entries.
- In the following example, the thread numbers are in bold, the Event ID is italicized, and the recipient is underlined:
2008-09-10 12:14:51,465 [_httpConnector#1297127._ManagerProvider.receiver.295] DEBUG EV[282766] - NTFN[2134532] - BEGIN PROCESSING PERSON Target: U368222 Id: 2133915 State: PROCESSING Event: SUCCESSFUL_DELIVERY
2008-09-10 12:14:51,480 [_httpConnector#1297127._ManagerProvider.receiver.324] DEBUG - - LIVE NOTIFICATION ID: 2134136 Change state from SERVICEABLE to SERVICEABLE.
- If a recipient is found, log in to the xMatters web user interface and search for that recipient. You may find issues with the User’s Devices.
- If an Event ID is found, log in and look at reports for that Event ID.
- The Exhaustive Report, Event Summary Report and Notification Summary Report may contain explanations for the error in the log.
Expired Message Bus
Node components communicate with one another using messages. Most messages are eligible for processing by a number of components; this means that if a component fails or is stopped, another can process any outstanding messages. However, if a non-redundant component (e.g., a Device Engine that is the sole servicer of a protocol) is stopped, messages may go undelivered for an extended period. When this condition is detected the message below is sent.
- The xMatters Health Monitor on node <node name> has detected an error during message processing. This may indicate that a non-redundant system component (node or device engine) was stopped or has failed.
Additional details:
Company: <Company name>
Message Type: <Message>
Message ID: <Message>
Source Component: <component>
Source Node: <node name>
Target Component: <component>
Target Node: <node name> (<details>)
Time Posted: <time>
Time-to-Live: <number> ms.
Database Connectivity Failure/Recovery
Each xMatters node periodically checks its connectivity with the database. If a check fails, the first message below is sent, and checking continues. If another check is successful, the second message below is sent.
Messages
- The xMatters Health Monitor on node <node name> has detected that database communication has failed.
Check the database server status and whether the node can ping the database server's host. Consult the node logs for further details.
- The xMatters Health Monitor on node <node name> has detected that database communication has been restored.
User Service Provider Empty
Through configuration it is possible to create an empty User Service Provider by either removing all of its Protocol Providers or marking each of them as inactive. When this prevents notification, the Health Monitor reports the message below (subject to the configuration documented in the "Global Configuration settings" table in About the Health Monitor).
- The xMatters Health Monitor on node <node name> has detected that the <provider name> User Service Provider from Company <Company name> has no active protocol providers; as a result, notifications have not been delivered.
Configure at least one Protocol Provider and a corresponding Device Engine to send messages to Devices that use this User Service Provider.
User Service Provider Exhausted
When notification delivery fails for all Protocol Providers in a given User Service Provider, the Health Monitor sends the message below (subject to the configuration documented in the "Global Configuration settings" table in Global Configuration).
- The xMatters Health Monitor on node <node name> has detected that notification to <recipient target name> of company <company name> has failed because all Protocol Providers have been attempted and failed for User Service Provider <provider>.
If this occurs whenever a message is sent to this provider, the most likely cause is a configuration problem with the provider or its corresponding Device Engine. If this happens consistently only for this Device, the Device is likely invalid or misconfigured. If none of the latter conditions apply, this may be a transient problem with the provider or the network. Consult the node logs for further details.
Protocol Provider Failed
When delivery of a notification to a particular provider fails, the Health Monitor sends one of the messages below (subject to the configuration documented in the "Global Configuration settings" table in About the Health Monitor).
Messages
- The xMatters Health Monitor on node <node name> has detected that notification to <User> of Company <Company name> on <User Device name> has failed because all protocol providers have been attempted and failed for user service provider <Service Provider name>. Consult the node logs for further details.
- The xMatters Health Monitor on node <node name> has detected that the <protocol provider type> Protocol Provider of Company <Company name> has failed beyond the configured threshold. The most recent failure occurred on node <node name> with message <message text>. The service provider may be unavailable or the Protocol Provider may be incorrectly configured. Consult the node logs for further details.
- There is no Device Engine to process the notifications that target the Protocol Provider <Protocol Provider name>.
Configure at least one Device Engine that can handle the protocol.
- The xMatters Health Monitor on node <node name> has detected that <Protocol Provider Name> protocol provider has failed. Message <message text>.
Consult the node logs for further details.
- The xMatters Health Monitor on node <node name> has detected that the <SIP Service Provider name> SIP service provider has failed beyond the configured threshold. The most recent failure occurred on node <node name>. Ensure that the provider is operational and that the node is properly configured to use it. Consult the node logs for further details.
- The xMatters Health Monitor on node <node name> has detected that the <Phone Service Provider name> phone service provider of Company <Company name> has failed beyond the configured threshold. The most recent failure occurred on node <node name>. Ensure that the phone line is operational and that the node is properly configured to use it. kConsult the node logs for further details.
Device Engine Failed
The Health Monitor for each node monitors the Device Engines, and sends one of the following messages whenever it detects a failure:
- The xMatters Health Monitor on node <node name> has detected that the <Device Engine name> has failed for <#> consecutive registration attempts.
Message: <message>
Consult the node logs for further details.
- The xMatters Health Monitor on node <node name> has detected that the <Device Engine name> has failed to register with its associated server.
Check the configuration logs and network connectivity.
Consult the node logs for further details.
- The xMatters Health Monitor on node <node name> has detected that the <Device Engine name> has recovered.
SIP Device Engine Cannot Register with SIP Server
When an xMatters SIP Device Engine cannot register with its associated SIP server, the Health Monitor generates a message similar to the one below:
- The xMatters Health Monitor on node <node name> has detected that SIP Device Engine protocol provider has failed. Message Failed to register to SIP server <SIP Sever IP address and port>
Message Queue Threshold Exceeded
The Health Monitor for each node monitors the capacity of internal message queues. When it detects that a message queue is nearly full, the message below is sent:
- The xMatters Health Monitor on node <node name> has detected that its host node is nearing its maximum capacity. Incoming message acceptance may slow if the current load is maintained.
Node Startup Aborted
Each xMatters license includes a limit on the number of running nodes allowed. When this limitation is about to be exceeded by node startup, the Health Monitor of the node starting up will send the message below and node startup will be aborted.
- Startup of <node name> has been aborted because the number of active nodes would have exceeded the licensed limit.
Integration Agent/Service Failed/Recovered
The conditions under which these messages are sent are documented in the “Reported Status and Health Monitor Messages” section of the xMatters Integration Agent Guide.
|
Note:
|
The xMatters integration agent has its own Health Monitor and related notifications, which are summarized in the xMatters Integration Agent Guide. |
Messages
- At <timestamp>, No connection could be made between Integration Agent <URL> and any of the primary or secondary xMatters Web Servers. The Integration Agent will continue sending heartbeats, and a notification will be sent when the heartbeat recovers.
- At <timestamp>, Integration Agent <URL> successfully sent a fully accepted heartbeat to primary xMatters Web Server <URL>. The Integration Agent is fully functional. A notification will be sent if the heartbeat status changes.
- At <timestamp>, A connection was made between Integration Agent <URL> and primary xMatters Web Server <URL>, but the heartbeat generated the following error: The xMatters Server at <URL> rejected the heartbeat/registration with the following reason: AUTHENTICATION_ERROR. The Integration Agent will continue to send heartbeats to the primary servers, but Integration Agent functionality may be limited until the heartbeat is fully accepted. A notification will be sent when the heartbeat is fully accepted. Consult the Integration Agent log for further details.
- At <timestamp>, No connection could be made between Integration Agent <URL> and any of the primary xMatters Web Servers. The heartbeat was fully accepted by secondary xMatters Web Server <URL>. The Integration Agent is fully functional in failover mode. The Integration Agent will continue to send heartbeats to the secondary servers. A heartbeat to the primary servers will be reattempted every 600 seconds (0 indicates indefinitely), or if the secondary heartbeat fails. A notification will be sent when the primary heartbeat recovers.
- At <timestamp>, No connection could be made between Integration Agent <URL> and any of the primary xMatters Web Servers. A connection was made to secondary xMatters Web Server <URL>, but the heartbeat generated the following error: The xMatters Server at <URL> rejected the heartbeat/registration of the following Integration Services: Domain: ping, Name: ping, Reason: UNKNOWN_SERVICE . The Integration Agent will continue to send heartbeats to the secondary servers, but Integration Agent functionality may be limited until the heartbeat is fully accepted. A heartbeat to the primary servers will be reattempted every 600 seconds (0 indicates indefinitely), or if the secondary heartbeat fails. A notification will be sent when the primary heartbeat recovers or the secondary heartbeat is fully accepted. Consult the Integration Agent log for further details.
Invalid Event Received
When an Event Domain receives an invalid event, the Health Monitor sends the following message:
- The xMatters Health Monitor on node <node name> has detected that an invalid event to Company <Company name> has been received on the <domain name> Event Domain with an incident ID of <ID>. This is a summary report; more invalid events may be rejected.
General Troubleshooting
When troubleshooting Health Monitor error messages, you may find the following reports useful:
- Submitted Notifications Report: this report displays all notifications sent by the system across multiple events during a specific time period. For more information, see Submitted Notifications Report.
- Live Notifications Report: this report displays all notifications currently in progress. For more information, see Live Notifications Report.
Note that two-way pager notifications sent via SCTP and SNPP appear in the Live Notifications Report for one hour after the notification is sent. This is due to the mechanism used by these protocols to query for delivery updates and replies.
- Application Audit Report: this report lists all changes made to any type of object within the system, such as Users, Groups, Devices, Schedules, etc. For more information, see Application Audit Report.