OpsGenie Icinga2 Integration
Icinga is an open source IT infrastructure monitoring tool that offers monitoring and alerting for systems, network devices, applications, and services. Icinga is highly configurable, easily extensible, hence very pervasive. OpsGenie is an alert and notification management solution that is highly complementary to Icinga.
What does OpsGenie offer to Icinga2 users?
OpsGenie Icinga2 integration plugin utilizes full capabilities of OpsGenie and provides two-way integration with Icinga2. Using the integraton:
- Icinga2 sends alerts to OpsGenie with scaringly detailed information. OpsGenie acts as a dispatcher for Icinga2 alerts, determining the right people to notify based on on-call schedules, using email, phone calls, text messages (SMS) and iPhone & Android push notifications, and escalating alerts until the alert is acknowledged or closed.
- OpsGenie automatically connects to Icinga2, gets performance data from Graphite for the host/service and attaches it to the alert.
- OpsGenie posts alert updates to Icinga2, e.g. acknowledging the alert will automatically ack the alert in Icinga2, alert comments are reflected to Icinga2 etc.
The steps below describe how to integrate OpsGenie and Icinga2 using OpsGenie Icinga2 integration plugin. Note that you may need to slightly alter these instructions depending on your exact Linux distribution and your Icinga2 configuration.
The plugin should be installed on the same server Icinga2 is running.
Packages provided support the following systems:
- Red Hat based linux distributions
- Debian based linux distributions
Download OpsGenie Icinga2 Plugin
- Download OpsGenie Icinga2 (Linux RPM)
- Run the following command :
rpm -i opsgenie-icinga2-<your_version>.rpm
- Remove icinga.alert_histogram_image_url, icinga.trends_image_url, icinga.command_url properties.
- Add icinga.api_url (Icinga2 API endpoint of your Icinga server) and icinga.graphite_url properties.
- Download OpsGenie Icinga2 (Linux DEB)
- Run the following command :
dpkg -i opsgenie-icinga2-<your_version>.deb
Add Icinga2 integration in OpsGenie
To add Icinga2 integration in OpsGenie, go to OpsGenie Icinga2 Integration page
Click on "Save Integration" button to save the integration. An "API Key" is generated for the integration. This key will be used by Icinga2 to authenticate with OpsGenie and specify the integration that should be used to process Icinga2 alerts.
OpsGenie Plugin Configuration in Icinga2
The plugin uses a golang-executable file (included in the plugin as icinga2opsgenie) to create, acknowledge and close alerts in OpsGenie. Icinga2 should be configured to execute this file on events to create, acknowledge and close alerts in OpsGenie.
Setting the apiKey is required. Other configuration parameters are set to defaults that work with most Icinga2 implementations but may need to be modified as well.
|apiKey||Copy the API key from the Icinga2 integration you've created above. icinga2opsgenie uses this key to authenticate to OpsGenie. API key is also used to identify the right integration configuration that should be used to process alerts.|
|recipients||Recipients field is used to specify who should be notified for the Icinga2 alerts. This field is used to set the default recipients field value. It can be modified to route different alerts to different people in OpsGenie Icinga2 integration, Advanced Settings page. Recipients can be set to users, groups, escalations or schedules who will be notified by OpsGenie. If you did not set recipients in the integration, this field is required.|
|teams||Teams field is used to specify which teams should be notified for the Icinga2 alerts. This field is used to set the default teams field value. It can be modified to route different alerts to different teams in OpsGenie Icinga2 integration, Advanced Settings page.|
|tags||Tags field is used to specify the tags of the alert that created in Opsgenie.|
|icinga_server||icinga_server field is used to identify the Icinga2 server in OpsGenie, and only required when there are multiple Icinga2 servers. This field is used by OpsGenie when sending actions executed by users (acknowledge, close, etc.) back to your Icinga2 servers via Marid|
|viaMaridUrl||viaMaridUrl field is used to send alerts to OpsGenie through Marid. You should enter host and port values of your working Marid.
|logPath||Specifies the full path of the log file. (Default value is /var/log/opsgenie/icinga2opsgenie.log)|
There are three ways to configure golang-executable file:
- Configuring from conf file: You can configure from /etc/opsgenie/conf/opsgenie-integration.conf file. Configuring from conf file will overwrite the configurations made in the script.
- Configuring by using Golang Flags: You can configure by entering flags to the command definition in /etc/icinga2/conf.d/opsgenie.conf file. Use -apiKey flag for your apiKey and -is flag for your icinga_server name. If you don't have multiple Icinga2 servers, you don't have to define the Icinga2 server. Using flags will overwrite all the other configuration methods mentioned above.
- Configuring from script: You can configure apiKey and icinga_server from icinga2opsgenie.go script. If you use this option, you need to build the script again and put the new executable to /usr/bin directory. You can find information about the location of the icinga2opsgenie.go and how to build a go script in the "Source" section.
You can configure the golang-executable to use a proxy for sending HTTP requests by defining the environment variable HTTP_PROXY=http://host:port
Configure OpsGenie to Icinga Integration (Optional)
The plugin uses Marid utility (included in the plugin) to enrich alerts when they get created and to update the state of the them in Icinga2 when they get updated in OpsGenie. For example, when an alert is created in OpsGenie, Marid gets the details(histogram, trends etc.) from Icinga2 and attaches them to the alert. Also when users acknowledge an alert from their mobile devices using the OpsGenie app, alert gets acknowledged in Icinga2, and when users add comments to alerts in OpsGenie, comments get posted to Icinga2 as well. Marid subscribes to alert actions in OpsGenie and reflects these actions on Icinga2 using Icinga2 API.
- To start Marid, run the following command:
- To stop Marid, run the following command:
Marid is a java application; therefore requires the Java Runtime version 1.6+ Both the Open JDK and Oracle JVMs can be used.
#JAVA_HOME=<path/to/JDK or JRE/install>
To be able to execute actions in Icinga2, Marid gets the configuration parameters from /etc/opsgenie/conf/opsgenie-integration.conf file.
|icinga.api_url||Marid uses this URL to post alert updates to Icinga2, like alert acknowledgement, comments, etc. Replace the "https://localhost:5665" with your Icinga2 server's API endpoint.|
|icinga.graphite_url||(Optional) Marid retrieves performance data graphs from Icinga2 using this URL. Replace the "http://localhost:5003" with your Icinga2 server's Graphite endpoint.|
|icinga.user||Credentials to authenticate to Icinga2 API.|
|icinga.http.timeout||Timeout duration in msecs to connect to Icinga2 API.|
|icinga.expire_acknowledgement_after||Removes acknowledgement after given value (in minutes.) Disabled by default.|
Multiple Icinga2 Server SupportMarid can be configured to forward OpsGenie alert actions to multiple Icinga2 servers, in order to do so :
- On each Icinga2 Server, modify /etc/opsgenie/conf/opsgenie-integration.conf icinga_server config property value to a unique name.
- On Marid server, modify /etc/opsgenie/conf/opsgenie-integration.conf, Add API endpoint, Graphite endpoint, user, password configuration for each Icinga2 server.
- opsgenie-integration.conf has a commented sample configuration for multiple Icinga servers :
#icinga.server1.api_url=http://icingaHost:port #icinga.server1.graphite_url=http://icingaHost:port #icinga.server1.user=icingaadmin #icinga.server1.password=icingaadmin #icinga.server1.http.timeout=30000
If you see "JAVA_HOME not defined" error in /var/log/opsgenie/icinga2opsgenie.log, you should define it in /etc/opsgenie/profile shell script.
If you use Marid, you will see rich alerts populated with host or service current status information in OpsGenie for every notification created in Icinga2.
FAQ and Troubleshooting
If you're having trouble getting the integration to work, please check if your problem is mentioned below, and follow our advice:
1- Icinga 2 alerts are not getting created in OpsGenie:
Run the following test command from the shell. Check if the test alert is created in OpsGenie:
/usr/bin/icinga2opsgenie -entityType=host -t=PROBLEM -hs=DOWN -hn=test_host
- If you're getting a "Trace/breakpoint trap" error: It means your icinga2opsgenie plugin isn't compatible with your server distribution. Follow the "Source and Recompiling icinga2opsgenie" section below and rebuild your icinga2opsgenie.go according to your specific server environment.
- If the alert is created in OpsGenie: It means the integration is installed correctly. The problem might be that Icinga2 is not notifying the OpsGenie contact for alerts. Check your Icinga2 alert notifications log.
- If not: Check the logs at /var/log/opsgenie/icinga2opsgenie.log. Look for the following errors in the log file:
- If you're seeing "RestException[Could not authenticate.]" in the logs, it means OpsGenie couldn't identify your api key. Check if you've set the API key correctly, as explained in "OpsGenie Plugin Configuration in Icinga" above.
- If you're seeing "Could not execute this action with apiKey of [Icinga2] integration" in the logs, you might have downloaded the wrong integration package. Make sure that you download the Icinga2 integration package, not Icinga or any other.
- If you can't make sense of the problem, set the plugin's log level to debug, try again and send the logs to us at email@example.com.
- If there is no /var/log/opsgenie/icinga2opsgenie.log file, or there are no logs in it, check the following:
- First, ensure that the icinga user has permission to write to /var/log/opsgenie directory.
- Now check your Icinga2 server logs at /var/log/icinga2/icinga2.log. See if there are error logs regarding icinga2opsgenie, and contact us with them.
Setting icinga2opsgenie plugin's log level to DEBUG:
Change the line icinga2opsgenie.logger=warning to icinga2opsgenie.logger=debug in /etc/opsgenie/conf/opsgenie-integration.conf file.
2- The Icinga2 alert is not acknowledged when you ack the alert at OpsGenie:
- First, check your alert logs. If you don't see the "Posted [Acknowledge] action to Icinga2.." log, it means OpsGenie didn't send the Acknowledge action to Icinga2. Check the logs at OpsGenie Logs page, by searching with the id of the alert. The alert id can be retrieved from the alert logs.
- If you're seeing "Executed [Acknowledge] action via Marid with errors." log, it means the icingaActionExecutor.groovy script in your Marid has encountered an error. Check the logs at /var/log/opsgenie/marid/script.log for error logs.
- If you can't make sense of the problem, set the Marid's script log level to debug, try again and send the /var/log/opsgenie/marid/script.log file to us at firstname.lastname@example.org.
Setting Marid's script log level to DEBUG:
Change the line log4j.logger.script=WARN, script to log4j.logger.script=DEBUG, script in /etc/opsgenie/marid/log.properties file. Then, restart Marid service.
3- Marid is causing memory leak, or using up too much RAM:
Change the line log4j.rootLogger=WARN, marid to log4j.rootLogger=DEBUG, marid in /etc/opsgenie/marid/log.properties file. Then, restart Marid service and send the /var/log/opsgenie/marid/Marid.log file to us at email@example.com so we can analyze further.
Source and Recompiling icinga2opsgenie
The source for the executable icinga2opsgenie is located under /usr/bin/ and icinga2opsgenie.go is located under /etc/opsgenie/ and is also available at GitHub OpsGenie Integration repository. If you wish to change the behavior of the executable, you can edit icinga2opsgenie.go and build it using:
go build icinga2opsgenie.go
For installing go, refer to http://golang.org/doc/install. Note that the executable in the plugin is built for linux/386 systems.