Health Monitor Messages

This section summarizes Health Monitor messages and the conditions under which they are generated (for general information about the Health Monitor, see About the Health Monitor).

Note: The xMatters integration agent has its own Health Monitor and related notifications, which are summarized in the xMatters Integration Agent Guide.

Component Misconfigured

Device Engines report the following message when a configuration change has been applied that prevents notification delivery (where possible, more detail about the source of the problem is included).

Message

The xMatters Health Monitor on node <node name> has detected that the <device engine name> has been misconfigured and can no longer function. Message: <detailed message>.

Node Failure/Recovery

Nodes periodically update their database record with a timestamp. If a Health Monitor on a running node detects that another node has failed to update its timestamp, the first message below is sent. When a node has failed, the processing it was responsible for at the time of failure will be recovered by another running node (if there is one).

When a node that has previously failed (i.e., its heartbeats stopped) resumes its heartbeats, the second message below is sent.

Messages

Multi-Node Communication

In some cases, node-to-node communication can fail (e.g., due to regular maintenance, network outages, faulty equipment, socket exhaustion, etc.), generating a Health Monitor message.

Message

Node Network Monitor Parameters

The following Health Monitor Messages are triggered when the Node Network Monitor threshold parameters specified on the Global Configuration page are exceeded (for details, see Global Configuration):

The following messages are sent when application layer latency has recovered to within the specified parameters:

Unhandled Exception

The following message is sent when an unexpected error occurs.

Message
Example
The xMatters Health Monitor on node SISC_APP_NODE1 (10.10.60.78) has detected that the application has encountered an error: No row with the given identifier exists:[com.invoqsystems.foundation.interpreter.invoqscript.persistance.
ScriptObjectProxy#214410]; nested exception is org.hibernate.
ObjectNotFoundException: No row with the given identifier exists: [com.invoqsystems.foundation.interpreter.invoqscript.persistance.
ScriptObjectProxy#214410].  Consult node logs for further details.Troubleshooting
 

To troubleshoot an unhandled exception alert, analyze the logs on the node that caused the exception:

  1. Search the log for the exception detailed in the Health Monitor email.
  2. Each log entry contains a thread number. Search the log for the entries corresponding to the thread number that generated the exception, and look for an Event ID or a target name (Recipient) in those log entries.
    1. In the following example, the thread numbers are in bold, the Event ID is italicized, and the recipient is underlined:
2008-09-10 12:14:51,465 [_httpConnector#1297127._ManagerProvider.receiver.295] DEBUG EV[282766] - NTFN[2134532] - BEGIN PROCESSING PERSON Target: U368222 Id: 2133915 State: PROCESSING  Event: SUCCESSFUL_DELIVERY 
2008-09-10 12:14:51,480 [_httpConnector#1297127._ManagerProvider.receiver.324] DEBUG  -  - LIVE NOTIFICATION ID: 2134136 Change state from SERVICEABLE to SERVICEABLE.                 
  1. If a recipient is found, log in to the xMatters web user interface and search for that recipient. You may find issues with the User’s Devices.
  2. If an Event ID is found, log in and look at reports for that Event ID.
    1. The Exhaustive Report, Event Summary Report and Notification Summary Report may contain explanations for the error in the log.

Expired Message Bus

Node components communicate with one another using messages. Most messages are eligible for processing by a number of components; this means that if a component fails or is stopped, another can process any outstanding messages. However, if a non-redundant component (e.g., a Device Engine that is the sole servicer of a protocol) is stopped, messages may go undelivered for an extended period. When this condition is detected the message below is sent.

Message

Database Connectivity Failure/Recovery

Each xMatters node periodically checks its connectivity with the database. If a check fails, the first message below is sent, and checking continues. If another check is successful, the second message below is sent.

Messages

User Service Provider Empty

Through configuration it is possible to create an empty User Service Provider by either removing all of its Protocol Providers or marking each of them as inactive. When this prevents notification, the Health Monitor reports the message below (subject to the configuration documented in the "Global Configuration settings" table in About the Health Monitor).

Message

User Service Provider Exhausted

When notification delivery fails for all Protocol Providers in a given User Service Provider, the Health Monitor sends the message below (subject to the configuration documented in the "Global Configuration settings" table in Global Configuration).

Message

Protocol Provider Failed

When delivery of a notification to a particular provider fails, the Health Monitor sends one of the messages below (subject to the configuration documented in the "Global Configuration settings" table in About the Health Monitor).

Messages

SIP Device Engine Cannot Register with SIP Server

When an xMatters SIP Device Engine cannot register with its associated SIP server, the Health Monitor generates a message similar to the one below.

Message

Message Queue Threshold Exceeded

The Health Monitor for each node monitors the capacity of internal message queues. When it detects that a message queue is nearly full, the message below is sent.

Message

Node Startup Aborted

Each xMatters license includes a limit on the number of running nodes allowed. When this limitation is about to be exceeded by node startup, the Health Monitor of the node starting up will send the message below and node startup will be aborted.

Message

Integration Agent/Service Failed/Recovered

The conditions under which these messages are sent are documented in the “Reported Status and Health Monitor Messages” section of the xMatters Integration Agent Guide.

Messages

General Troubleshooting

When troubleshooting Health Monitor error messages, you may find the following reports useful:

Note that two-way pager notifications sent via SCTP and SNPP appear in the Live Notifications Report for one hour after the notification is sent. This is due to the mechanism used by these protocols to query for delivery updates and replies.