This section summarizes Health Monitor messages and the conditions under which they are generated (for general information about the Health Monitor, see About the Health Monitor).
| Note: | The xMatters integration agent has its own Health Monitor and related notifications, which are summarized in the xMatters Integration Agent Guide. |
Device Engines report the following message when a configuration change has been applied that prevents notification delivery (where possible, more detail about the source of the problem is included).
The xMatters Health Monitor on node <node name> has detected that the <device engine name> has been misconfigured and can no longer function. Message: <detailed message>.
Nodes periodically update their database record with a timestamp. If a Health Monitor on a running node detects that another node has failed to update its timestamp, the first message below is sent. When a node has failed, the processing it was responsible for at the time of failure will be recovered by another running node (if there is one). When this occurs the second message below is sent.
In some cases, node-to-node communication can fail (e.g., due to regular maintenance, network outages, faulty equipment, socket exhaustion, etc.), generating a Health Monitor message.
The following Health Monitor Messages are triggered when the Node Network Monitor threshold parameters specified on the Global Configuration page are exceeded (for details, see Global Configuration):
The following messages are sent when application layer latency has recovered to within the specified parameters:
The following message is sent when an unexpected error occurs.
The xMatters Health Monitor on node SISC_APP_NODE1 (10.10.60.78) has detected that the application has encountered an error: No row with the given identifier exists:[com.invoqsystems.foundation.interpreter.invoqscript.persistance.
ScriptObjectProxy#214410]; nested exception is org.hibernate.
ObjectNotFoundException: No row with the given identifier exists: [com.invoqsystems.foundation.interpreter.invoqscript.persistance.
ScriptObjectProxy#214410]. Consult node logs for further details.TroubleshootingTo troubleshoot an unhandled exception alert, analyze the logs on the node that caused the exception:
2008-09-10 12:14:51,465 [_httpConnector#1297127._ManagerProvider.receiver.295] DEBUG EV[282766] - NTFN[2134532] - BEGIN PROCESSING PERSON Target: U368222 Id: 2133915 State: PROCESSING Event: SUCCESSFUL_DELIVERY
2008-09-10 12:14:51,480 [_httpConnector#1297127._ManagerProvider.receiver.324] DEBUG - - LIVE NOTIFICATION ID: 2134136 Change state from SERVICEABLE to SERVICEABLE.
Node components communicate with one another using messages. Most messages are eligible for processing by a number of components; this means that if a component fails or is stopped, another can process any outstanding messages. However, if a non-redundant component (e.g., a Device Engine that is the sole servicer of a protocol) is stopped, messages may go undelivered for an extended period. When this condition is detected the message below is sent.
Each xMatters node periodically checks its connectivity with the database. If a check fails, the first message below is sent, and checking continues. If another check is successful, the second message below is sent.
Through configuration it is possible to create an empty User Service Provider by either removing all of its Protocol Providers or marking each of them as inactive. When this prevents notification, the Health Monitor reports the message below (subject to the configuration documented in the "Global Configuration settings" table in About the Health Monitor).
When notification delivery fails for all Protocol Providers in a given User Service Provider, the Health Monitor sends the message below (subject to the configuration documented in the "Global Configuration settings" table in Global Configuration).
When delivery of a notification to a particular provider fails, the Health Monitor sends one of the messages below (subject to the configuration documented in the "Global Configuration settings" table in About the Health Monitor).
When an xMatters SIP Device Engine cannot register with its associated SIP server, the Health Monitor generates a message similar to the one below.
The Health Monitor for each node monitors the capacity of internal message queues. When it detects that a message queue is nearly full, the message below is sent.
Each xMatters license includes a limit on the number of running nodes allowed. When this limitation is about to be exceeded by node startup, the Health Monitor of the node starting up will send the message below and node startup will be aborted.
The conditions under which these messages are sent are documented in the “Reported Status and Health Monitor Messages” section of the xMatters Integration Agent Guide.
At <timestamp>, No connection could be made between Integration Agent <URL> and any of the primary or secondary xMatters Web Servers. The Integration Agent will continue sending heartbeats, and a notification will be sent when the heartbeat recovers.
At <timestamp>, Integration Agent <URL> successfully sent a fully accepted heartbeat to primary xMatters Web Server <URL>. The Integration Agent is fully functional. A notification will be sent if the heartbeat status changes.
At <timestamp>, A connection was made between Integration Agent <URL> and primary xMatters Web Server <URL>, but the heartbeat generated the following error: The xMatters Server at <URL> rejected the heartbeat/registration with the following reason: AUTHENTICATION_ERROR. The Integration Agent will continue to send heartbeats to the primary servers, but Integration Agent functionality may be limited until the heartbeat is fully accepted. A notification will be sent when the heartbeat is fully accepted. Consult the Integration Agent log for further details.
At <timestamp>, No connection could be made between Integration Agent <URL> and any of the primary xMatters Web Servers. The heartbeat was fully accepted by secondary xMatters Web Server <URL>. The Integration Agent is fully functional in failover mode. The Integration Agent will continue to send heartbeats to the secondary servers. A heartbeat to the primary servers will be reattempted every 600 seconds (0 indicates indefinitely), or if the secondary heartbeat fails. A notification will be sent when the primary heartbeat recovers.
At <timestamp>, No connection could be made between Integration Agent <URL> and any of the primary xMatters Web Servers. A connection was made to secondary xMatters Web Server <URL>, but the heartbeat generated the following error: The xMatters Server at <URL> rejected the heartbeat/registration of the following Integration Services: Domain: ping, Name: ping, Reason: UNKNOWN_SERVICE . The Integration Agent will continue to send heartbeats to the secondary servers, but Integration Agent functionality may be limited until the heartbeat is fully accepted. A heartbeat to the primary servers will be reattempted every 600 seconds (0 indicates indefinitely), or if the secondary heartbeat fails. A notification will be sent when the primary heartbeat recovers or the secondary heartbeat is fully accepted. Consult the Integration Agent log for further details.
When troubleshooting Health Monitor error messages, you may find the following reports useful:
Note that two-way pager notifications sent via SCTP and SNPP appear in the Live Notifications Report for one hour after the notification is sent. This is due to the mechanism used by these protocols to query for delivery updates and replies.