Zenoss is an open-source application, server, and network management platform. OpsGenie is an alert and notification management solution that is highly complementary to Zenoss.
OpsGenie Zenoss plugin supports bi-directional integration with Zenoss. Integration leverages OpsGenie's zenoss-specific executable and marid utility to automatically create alerts and synchronize alert status between Zenoss and OpsGenie.
OpsGenie Zenoss Plugin Integration
OpsGenie Zenoss integration plugin utilizes full capabilities of OpsGenie and provides bi-directional integration with Zenoss. Integration leverages OpsGenie's Zenoss-specific executable and marid utility to automatically create alerts and synchronizes alert status between Zenoss and OpsGenie.
The steps below describe how to integrate OpsGenie and Zenoss using OpsGenie Zenoss integration plugin. Note that you may need to slightly alter these instructions depending on your exact Linux distribution and your Zenoss configuration.
The plugin should be installed on the same server Zenoss is running.
Packages provided support the following systems:
- Red Hat based linux distributions
- Debian based linux distributions
Download OpsGenie Zenoss Plugin
- Download OpsGenie Zenoss (Linux RPM)
- Run the following command :
rpm -i opsgenie-zenoss-<your_version>.rpm
rpm -U --force opsgenie-integration-<your_version>.rpm
We suggest that you backup your configuration files before update!
- Download OpsGenie Zenoss (Linux DEB)
- Run the following command :
dpkg -i opsgenie-zenoss-<your_version>.deb
Add Zenoss integration in OpsGenie
To add Zenoss integration in OpsGenie, go to OpsGenie Zenoss Integration page
Click on "Save Integration" button to save the integration. An "API Key" is generated for the integration. This key will be used by Zenoss to authenticate with OpsGenie and specify the integration that should be used to process Zenoss alerts.
OpsGenie Plugin Configuration in Zenoss
The plugin uses a golang-executable file (included in the plugin as zenoss2opsgenie) to create, acknowledge and close alerts in OpsGenie. Zenoss should be configured to execute this file on events to create, acknowledge and close alerts in OpsGenie.
Setting the apiKey is required. Other configuration parameters are set to defaults that work with most Zenoss implementations but may need to be modified as well.
|apiKey||Copy the API key from the Zenoss integration you've created above. zenoss2opsgenie uses this key to authenticate to OpsGenie. API key is also used to identify the right integration configuration that should be used to process alerts.|
|zenoss.command_url||URL to get detailed event data from Zenoss in zenoss2opsgenie.|
|zenoss.user||Credentials to authenticate Zenoss web server|
|recipients||Recipients field is used to specify who should be notified for the Zenoss alerts. This field is used to set the default recipients field value. It can be modified to route different alerts to different people in OpsGenie Zenoss integration, Advanced Settings page. Recipients can be set to users, groups, escalations or schedules who will be notified by OpsGenie. If you did not set recipients in the integration, this field is required.|
|teams||Teams field is used to specify which teams should be notified for the Zenoss alerts. This field is used to set the default teams field value. It can be modified to route different alerts to different teams in OpsGenie Zenoss integration, Advanced Settings page.|
|tags||Tags field is used to specify the tags of the alert that created in Opsgenie.|
|viaMaridUrl||viaMaridUrl field is used to send alerts to OpsGenie through Marid. You should enter host and port values of your working Marid.
|logPath||Specifies the full path of the log file. (Default value is /var/log/opsgenie/zenoss2opsgenie.log)|
There are three ways to configure golang-executable file:
- Configuring from conf file: You can configure from /etc/opsgenie/conf/opsgenie-integration.conf file. Configuring from conf file will overwrite the configurations made in the script.
- Configuring by using Golang Flags: You can configure by entering flags to command of the notification you created in Zenoss, which is described in "Configure Triggers in Zenoss" section. Use -apiKey flag for your apiKey.
- Configuring from script: You can configure apiKey zenoss2opsgenie.go script. If you use this option, you need to build the script again and put the new executable to /usr/bin directory. You can find information about the location of the zenoss2opsgenie.go and how to build a go script in the "Source" section.
You can configure the golang-executable to use a proxy for sending HTTP requests by defining the environment variable HTTP_PROXY=http://host:port
Configure Triggers in Zenoss
Before creating a notification you should:
Select Events > Triggers from the Navigation menu
Create a trigger named opsgenie
After creating the trigger, follow the steps below:
Select Events > Triggers from the Navigation menu
Select Notifications in the left panel
Create a notification
Choose the notification you created and click edit button
In "Notification" tab, enable the notification, set "Send Clear" as checked and add the trigger named "opsgenie" from the trigger list and click "Add"
In "Content" tab, put the following into "Command" and "Clear Command" fields. If you add optional
-eventState=closeto your Clear Command, zenoss2opsgenie executable will not try to get event details from Zenoss and will directly close the event's alert in OpsGenie.
In "Subscribers" tab, choose the subscribers and click "SUBMIT".
Configure OpsGenie to Zenoss Integration (Optional)
The plugin uses Marid utility (included in the plugin) to update the state of alerts in Zenoss when they get updated in OpsGenie. For example, when users acknowledge/close an alert from their mobile devices using the OpsGenie app, alert gets acknowledged/closed in Zenoss. Marid subscribes to alert actions in OpsGenie and reflects these actions on Zenoss using Zenoss JSON API.
- To start Marid, run following command:
- To stop Marid, run following command:
Marid is a java application; therefore requires the Java Runtime version 1.6+ Both the Open JDK and Oracle JVMs can be used.
#JAVA_HOME=<path/to/JDK or JRE/install>
To be able to execute actions in Zenoss, Marid gets the configuration parameters from /etc/opsgenie/conf/opsgenie-integration.conf file.
|zenoss.command_url||URL to update Zenoss events when alerts get acknowledged, closed, etc.|
|zenoss.user||Credentials to authenticate on Zenoss web server.|
If you see "JAVA_HOME not defined" error in /var/log/opsgenie/zenoss2opsgenie.log, you should define it in /etc/opsgenie/profile shell script.
FAQ and Troubleshooting
If you're having trouble getting the integration to work, please check if your problem is mentioned below, and follow our advice:
1- Zenoss alerts are not getting created in OpsGenie:
Run the following test command from the shell. Check if the test alert is created in OpsGenie:
- If you're getting a "Trace/breakpoint trap" error: It means your zenoss2opsgenie plugin isn't compatible with your server distribution. Follow the "Source and Recompiling zenoss2opsgenie" section below and rebuild your zenoss2opsgenie.go according to your specific server environment.
- If the alert is created in OpsGenie: It means the integration is installed correctly. The problem might be that Zenoss is not notifying the OpsGenie contact for alerts. Check your Zenoss alert notifications log.
- If not: Check the logs at /var/log/opsgenie/zenoss2opsgenie.log. Look for the following errors in the log file:
- If you're seeing "RestException[Could not authenticate.]" in the logs, it means OpsGenie couldn't identify your api key. Check if you've set the API key correctly, as explained in "OpsGenie Plugin Configuration in Zenoss" above.
- If you can't make sense of the problem, set the plugin's log level to debug, try again and send the logs to us at firstname.lastname@example.org.
- If there is no /var/log/opsgenie/zenoss2opsgenie.log file, or there are no logs in it, check the following:
- First, ensure that the zenoss user has permission to write to /var/log/opsgenie directory.
- Now check your Zenoss server logs at /opt/zenoss/log/zeneventd.log. See if there are error logs regarding zenoss2opsgenie, and contact us with them.
Setting zenoss2opsgenie plugin's log level to DEBUG:
Change the line zenoss2opsgenie.logger=warning to zenoss2opsgenie.logger=debug in /etc/opsgenie/conf/opsgenie-integration.conf file.
2- The Zenoss alert is not acknowledged when you ack the alert at OpsGenie:
- First, check your alert logs. If you don't see the "Posted [Acknowledge] action to Zenoss.." log, it means OpsGenie didn't send the Acknowledge action to Zenoss. Check the logs at OpsGenie Logs page, by searching with the id of the alert. The alert id can be retrieved from the alert logs.
- If you're seeing "Executed [Acknowledge] action via Marid with errors." log, it means the zenossActionExecutor.groovy script in your Marid has encountered an error. Check the logs at /var/log/opsgenie/marid/script.log for error logs.
- If you can't make sense of the problem, set the Marid's script log level to debug, try again and send the /var/log/opsgenie/marid/script.log file to us at email@example.com.
Setting Marid's script log level to DEBUG:
Change the line log4j.logger.script=WARN, script to log4j.logger.script=DEBUG, script in /etc/opsgenie/marid/log.properties file. Then, restart Marid service.
3- Marid is causing memory leak, or using up too much RAM:
Change the line log4j.rootLogger=WARN, marid to log4j.rootLogger=DEBUG, marid in /etc/opsgenie/marid/log.properties file. Then, restart Marid service and send the /var/log/opsgenie/marid/Marid.log file to us at firstname.lastname@example.org so we can analyze further.
Source and Recompiling zenoss2opsgenie
The source for the executable zenoss2opsgenie is located under /usr/bin/ and zenoss2opsgenie.go is located under /etc/opsgenie/ and is also available at GitHub OpsGenie Integration repository. If you wish to change the behavior of the executable, you can edit zenoss2opsgenie.go and build it using:
go build zenoss2opsgenie.go
For installing go, refer to http://golang.org/doc/install. Note that the executable in the plugin is built for linux/386 systems.