OpenNMS Integration package will not be served for a while. Hence, "Scenario 2: Rich Alert Integration" will be unavailable during this time.
OpenNMS is a network management platform providing automated discovery, service monitoring, event management, and notifications. OpsGenie enables OpenNMS users to receive OpenNMS events (referred as alerts in OpsGenie) through email, SMS, phone call, and iPhone/Android push notifications, and work with OpenNMS events from mobile phones. OpsGenie alerts can contain not only the event but also supporting information such as managed object details, performance charts, etc. enabling users to make better decisions when they are notified.
OpsGenie provides various integration tools and APIs to integrate monitoring tools with OpsGenie, so that users can be notified when something important occurs in their environment. You can use the following integration scenarios and packages provided by OpsGenie as a starting point to integrate OpenNMS with OpsGenie alerting system.
Scenario 1: Email Integration
OpenNMS has the capability to send emails to notify users for specified events. Using OpsGenie email integration capabilities, OpsGenie can notify users via SMS, iPhone or Android push notifications, or phone calls. Instructions below describe how to configure OpenNMS and OpsGenie to send push notifications to iPhone/Android apps using OpsGenie email integration for OpenNMS authentication failed events as an example.
Step1: Setup OpsGenie
As a first step, you need an OpsGenie account if you don't have one already. You can sign up for a free account. Once the account is created, you can invite other users as well. Users can set their notification preferences (how and when they will be notified, quiet hours, time zones, etc) in their profile pages. You can find more information on profile setup in OpsGenie Getting Started documentation.
In the example above, the user has configured to receive notifications via email and iPhone push immediately. If the user does not see the alert within 5 minutes, OpsGenie will send a notification via SMS, and another one via a phone call, if the user still has not seen the notification within 10 minutes.
You will need to know the usernames (email of the user) of the users that will be notified. Alternatively, you can create a group in OpsGenie (for example web_operations), make users member of the group and specify the group as the recipient when creating the alert.
Next, we need to setup OpsGenie email rules which will create alerts from emails sent from OpenNMS. When your account is created OpsGenie provides a default email address and a default email rule for you. You can modify your email address and rules in the Settings page. We will use the default rule, which creates alerts for every incoming mail, in this example. You can find more information on email integration here.
Step2: OpenNMS Configuration
The first thing you should do is to create a user account for OpsGenie integration in OpenNMS web UI. You can do so by clicking the "Admin" link in the menu bar, going to "Configure Users, Groups and On-Call Roles", and then clicking "Configure Users". Then, click the "Add New User" link, and type "OpsGenie" in the username field and provide a password in the form, then click "OK".
Next, add an OpsGenie Email Integration, copy the email address from the OpsGenie Email Integration and paste it to Email field in OpenNMS user form and then click "Finish".
To receive a notification from OpenNMS, your user or a group that your user is in needs to be a part of a destination path. Go to the Admin page, click "Configure Notifications" in the "Operations" section, then "Configure Destination Paths," then click the "New Path" button. Give it a name, like "OpsGenieEmailPath", then click "Edit". Add "OpsGenie" user by selecting it, then click "Next >>>". On the next page, leave the defaults ("javaEmail" and "on") and click "Next >>>" again. Finally, click "Finish" and you should see your new destination path in the "Existing Paths" section.
Next, we need to associate OpenNMS events with our new destination path. Go to the Admin page, click "Configure Notifications" in the "Operations" section, then "Configure Event Notifications". Click the "Add New Event Notification" button at the top of the Event Notifications page and select "OpenNMS-defined internal event: an authentication failure has occurred in WebUI" event for the time being to test the setup easily. You can change this configuration later.
Then click "Next >>>". Click "Skip results validation >>>" in the next page.
Finally, you can configure the information used to send the notification. Fill in the following values:
- Name: Authentication Failed
- Choose A Path: OpsGenieEmailPath
- Text Message: The OpenNMS Web UI had a failed login attempt, by user '%parm[user]%', from IP address %parm[ip]%
- Short Message: Authentication failed by user '%parm[user]%'
- Email Subject: Authentication failed by user '%parm[user]%'
...and then click "Finish".
You should now see your notification event in the list, but it will be disabled. Set the radio button to "On" to enable it. In the main OpenNMS Admin page, there is a radio selector labeled, "Notification Status." Change it to "On" and click update. And configure your mail server in $OPENNMS_HOME/etc/javamail-configuration.properties file and restart OpenNMS server.
You are now ready to create an OpenNMS event and receive an OpsGenie push notification. Logout from OpenNMS web UI and try to login with an invalid account. You should receive a push notification using your smartphone, if you don't, take a look at "$OPENNMS_HOME/logs/daemon/notifd.log" and see if you get any error messages. If no problem exists look at OpsGenie alerts page and check whether your new alert exist in the alert list.
You can find more information on OpenNMS notifications in the following pages;
Scenario 2: Rich Alert Integration
OpenNMS notification system allows executing arbitrary external programs, which opens endless possibilities to integrate with OpsGenie alerting system through OpsGenie client tools like lamp, marid, etc. OpsGenie provides a downloadable OpenNMS integration package which shows rich alerting capabilities of our system. Integration files include example groovy script that forwards node events, script also retrieves most recent node outages to the alert from OpenNMS and attaches to the alert to make this information available to the recipients.
- Download OpsGenie OpenNMS Integration files that include:
- Lamp - OpsGenie command line utility to create alerts
- OpenNMS Integration scripts
- Unzip OpenNMS Integration
- OpenNMS Integration Files contains a customized version of lamp , you need to install this version.
- Install lamp by referring to Lamp Documentation - Installation steps. Do not download lamp again.
- Make sure your OpenNMS shell scripts have correct LAMP_HOME, your lamp installation directory
- Create an OpsGenie API Integration, copy the integration apiKey, put into the opsgenie-integration/conf/opsgenie-integration.conf file.
- Verify Lamp is working : Create a test alert, using the lamp utility from the command line , make sure your alert is visible on OpsGenie dashboard . Please refer to Lamp Documentation for more information.
- Groovy script configuration: Instead of implementing all logic in shell script, the integration package provides a script (ogCreateAlert.groovy) written in groovy - a powerful java based scripting language- which can be executed by lamp. Briefly, the script creates an OpsGenie alert, retrieve node outages through OpenNMS rest api, creates an html file showing the outages and attaches it to the alert.
/********************CONFIGURATIONS****************************/ // Recipients should be specified here for automatic tools. // Recipients can be users or groups created in OpsGenie RECIPIENTS="all" SOURCE="OpenNMS" //OpenNMS credentials are needed for extra information to be fetched through REST API OPENNMS_USER = "admin" OPENNMS_PASSWORD = "admin" OPENNMS_HOST="localhost" OPENNMS_PORT=8980
- RECIPIENTS: Users or group of users who will be notified by OpsGenie. Defaults to "all".
- SOURCE: Alert source
- OPENNMS_USER and OPENNMS_PASSWORD: Credentials to authenticate OpenNMS rest API.
- OPENNMS_HOST and OPENNMS_PORT:OpenNMS web server host and port which will be used for rest API calls.
Prepare opennmsShellScripts/opsgenie shell script :
- Copy the shell script anywhere in the same system where OpenNMS installed. This path will be pointed when OpenNMS notification commands are configured.
- Configure LAMP_HOME to point your lamp installation directory.
- Make the script executable in linux (chmod +x)
Prepare OpsGenie Groups & Users
- Create groups or users that you want to notify from Smarts
- Make sure groups have at least one user
- Add a custom notification command by adding the following section to the $OPENNMS_HOME/etc/notificationCommands.xml configuration file. Check OpenNMS Notification Configuration document for details.
<command binary="true"> <name>OpsGenie</name> <execute>/usr/bin/opsgenie</execute> <comment>OpsGenie alert creation</comment> <argument streamed="false"> <switch>-subject</switch> </argument> <argument streamed="false"> <switch>-tm</switch> </argument> <argument streamed="false"> <switch>-nodeid</switch> </argument> </command>
- Modify the <execute> tag so that it points the shell script provided with OpsGenie integration package.
- Restart OpenNMS server.
- Add an OpenNMS user and give it a name like "OpsGenie".
- The user added should be a part of a destination path so that the shell script provided can be executed with OpenNMS.Go to the Admin page, click "Configure Notifications" in the "Operations" section, then "Configure Destination Paths," then click the "New Path" button. Give it a name, like "OpsGenieScriptPath", then click "Edit". Add the user by selecting it, then click "Next >>>". On the next page, check "OpsGenie" custom command and "on" and click "Next >>>" again. Finally, click "Finish" and you should see your new destination path in the "Existing Paths" section.
OpenNMS events should be associated with the new destination path. Go to the Admin page, click "Configure Notifications" in the "Operations" section, then "Configure Event Notifications". Click the "Add New Event Notification" button at the top of the Event Notifications page and select "OpenNMS-defined node event: nodeLostService" event.Then click "Next >>>". Click "Skip results validation >>>" in the next page. Configure the information used to send the notification. Fill in the following values:
- Name: Node Lost Service
- Choose A Path: OpsGenieShellPath
- Text Message: The %service% service poll on interface %interfaceresolve% (%interface%) on node %nodelabel% failed at %time%.
- Short Message: %service% down on %interfaceresolve% (%interface%) on node %nodelabel%.
- Email Subject: %service% down on %interfaceresolve% (%interface%) on node %nodelabel%.
- Then click "Finish".
- The notification event in the list will be disabled. Set the radio button to "On" to enable it.
- In the main OpenNMS Admin page, there is a radio selector labeled, "Notification Status." Change it to "On" and click update.
To test the integration package, I make OpenNMS server to scan my local PC which runs an HTTP server on port 8080, with the following command:
perl $OPENNMS_HOME/bin/send-event.pl --interface 192.168.1.102 uei.opennms.org/internal/discovery/newSuspect
After the scan completed OpenNMS creates the node, its interfaces and services including my HTTP server.
When I shut down my HTTP server, OpenNMS detects it in the next service poll and creates a "nodeLostService" event for it.
The event triggers the notification system which executes the shell script installed. As a result a new alert is created in OpsGenie system and most recent outages of my PC can be displayed in OpsGenie console.