Health Monitor Messages
This section summarizes Health Monitor messages and the conditions under which they are generated (for general information about the Health Monitor, see About the Health Monitor).
Component Misconfigured
Device Engines report the following message when a configuration change has been applied that prevents notification delivery (where possible, more detail about the source of the problem is included).
Message
The AlarmPoint Health Monitor on node <node name> has detected that the <device engine name> has been misconfigured and can no longer function. Message: <detailed message>.
Node Failure/Recovery
Nodes periodically update their database record with a timestamp. If a Health Monitor on a running node detects that another node has failed to update its timestamp, the first message below is sent. When a node has failed, the processing it was responsible for at the time of failure will be recovered by another running node (if there is one). When this occurs the second message below is sent.
Messages
- The AlarmPoint Health Monitor on node <running node name> has detected that node <failed node name> has failed.
- The AlarmPoint Health Monitor on node <running node name> has detected that node <recovered node name> has been recovered.
Multi-Node Communication
In some cases, node-to-node communication can fail (e.g., due to regular maintenance, network outages, faulty equipment, socket exhaustion, etc.), generating a Health Monitor message.
Message
- The AlarmPoint Health Monitor on node <node name> has detected that communication with an AlarmPoint node has been lost: <node name>.
Unhandled Exception
The following message is sent when an unexpected error occurs.
Message
- The AlarmPoint Health Monitor on node <node name> has detected that the application has encountered an error: <error>. Consult node logs for further details.
Example
The AlarmPoint Health Monitor on node SISC_APP_NODE1 (10.10.60.78) has detected that the application has encountered an error: No row with the given identifier exists:[com.invoqsystems.foundation.interpreter.invoqscript.persistance.
ScriptObjectProxy#214410]; nested exception is org.hibernate.
ObjectNotFoundException: No row with the given identifier exists: [com.invoqsystems.foundation.interpreter.invoqscript.persistance.
ScriptObjectProxy#214410]. Consult node logs for further details.Troubleshooting
To troubleshoot an unhandled exception alert, analyze the logs on the node that caused the exception:
- Search the log for the exception detailed in the Health Monitor email.
- Each log entry contains a thread number. Search the log for the entries corresponding to the thread number that generated the exception, and look for an Event ID or a target name (Recipient) in those log entries.
- In the following example, the thread numbers are in bold, the Event ID is italicized, and the recipient is underlined:
2008-09-10 12:14:51,465 [_httpConnector#1297127._muleManagerProvider.receiver.295] DEBUG EV[282766] - NTFN[2134532] - BEGIN PROCESSING PERSON Target: U368222 Id: 2133915 State: PROCESSING Event: SUCCESSFUL_DELIVERY
2008-09-10 12:14:51,480 [_httpConnector#1297127._muleManagerProvider.receiver.324] DEBUG - - LIVE NOTIFICATION ID: 2134136 Change state from SERVICEABLE to SERVICEABLE.
- If a recipient is found, log in to the AlarmPoint Web User Interface and search for that recipient. You may find issues with the User’s Devices.
- If an Event ID is found, log in and look at reports for that Event ID.
- The Exhaustive Report, Event Summary Report and Notification Summary Report may contain explanations for the error in the log.
Expired Message Bus
Node components communicate with one another using messages. Most messages are eligible for processing by a number of components; this means that if a component fails or is stopped, another can process any outstanding messages. However, if a non-redundant component (e.g., a Device Engine that is the sole servicer of a protocol) is stopped, messages may go undelivered for an extended period. When this condition is detected the message below is sent.
Message
- The AlarmPoint Health Monitor on node <node name> has detected an error during message processing. This may indicate that a non-redundant system component (node or device engine) was stopped or has failed.
Database Connectivity Failure/Recovery
Each AlarmPoint node periodically checks its connectivity with the database. If a check fails, the first message below is sent, and checking continues. If another check is successful, the second message below is sent.
Messages
- The AlarmPoint Health Monitor on node <node name> has detected that database communication has failed.
- The AlarmPoint Health Monitor on node <node name> has detected that database communication has been restored.
User Service Provider Empty
Through configuration it is possible to create an empty User Service Provider by either removing all of its Protocol Providers or marking each of them as inactive. When this prevents notification, the Health Monitor reports the message below (subject to the configuration documented in the "Global Configuration settings" table in About the Health Monitor).
Message
- The AlarmPoint Health Monitor on node <node name> has detected that the <provider name> user service provider of company <Company name> has no active protocol providers and therefore a notification has not been delivered.
User Service Provider Exhausted
When notification delivery fails for all Protocol Providers in a given User Service Provider, the Health Monitor sends the message below (subject to the configuration documented in the "Global Configuration settings" table in Global Configuration).
Message
- The AlarmPoint Health Monitor on node <node name> has detected that notification to <recipient target name> of company <company name> has failed because all protocol providers have been attempted and failed for user service provider <provider>.
Protocol Provider Failed
When delivery of a notification to a particular provider fails, the Health Monitor sends one of the messages below (subject to the configuration documented in the "Global Configuration settings" table in About the Health Monitor).
Messages
- The AlarmPoint Health Monitor on node <node name> has detected that notification to <User> of company <Company name> on <User Device name> has failed because all protocol providers have been attempted and failed for user service provider <Service Provider name>.
- The AlarmPoint Health Monitor on node <node name> has detected that the <protocol provider type> protocol provider has failed beyond the configured threshold. The most recent failure occurred on node <node name> with message <message text>. The service provider may be unavailable or the protocol provider may be incorrectly configured.
- The AlarmPoint Health Monitor on node <node name> has detected that <Protocol Provider Name> protocol provider has failed. Message <message text>.
- The AlarmPoint Health Monitor on node <node name> has detected that the <SIP Service Provider name> SIP service provider has failed beyond the configured threshold. The most recent failure occurred on node <node name>.
Message Queue Threshold Exceeded
The Health Monitor for each node monitors the capacity of internal message queues. When it detects that a message queue is nearly full, the message below is sent.
Message
- The AlarmPoint Health Monitor on node <node name> has detected that its host node is nearing its maximum capacity. Incoming message acceptance may slow if the current load is maintained.
Node Startup Aborted
Each AlarmPoint license includes a limit on the number of running nodes allowed. When this limitation is about to be exceeded by node startup, the Health Monitor of the node starting up will send the message below and node startup will be aborted.
Message
- Startup of <node name> has been aborted because the number of active nodes would have exceeded the licensed limit.
Integration Agent/Service Failed/Recovered
The conditions under which these messages are sent are documented in the “Reported Status and Health Monitor Messages” section of the AlarmPoint Mobile Gateway Guide.
Messages
- The <service> integration service running under event domain <domain> has failed.
- The <service> integration service running under event domain <domain> has recovered.
- Integration agent <agent> has failed.
- Integration agent <agent> has recovered.
General Troubleshooting
When troubleshooting Health Monitor error messages, you may find the following reports useful:
- Submitted Notifications Report: this report displays all notifications sent by the system across multiple events during a specific time period. For more information, see Submitted Notifications Report.
- Live Notifications Report: this report displays all notifications currently in progress. For more information, see Live Notifications Report.
Note that two-way pager notifications sent via SCTP and SNPP appear in the Live Notifications Report for one hour after the notification is sent. This is due to the mechanism used by these protocols to query for delivery updates and replies.
- Application Audit Report: this report lists all changes made to any type of object within the system, such as Users, Groups, Devices, Schedules, etc. For more information, see Application Audit Report.