Splunk Integration

Splunk makes it simple to collect, analyze and act upon the untapped value of the big data generated by your technology infrastructure, security systems and business applications. Splunk has an alerting functionality. Splunk alerts can be used to monitor for and respond to specific events. Alerts use a saved search to look for events in real time or on a schedule. Alerts trigger when search results meet specific conditions. Alert actions can be used to respond when alerts trigger.

What does OpsGenie offers to Splunk users?

Splunk's Searching and Reporting app lets you search your data, create data models and pivots, save your searches and pivots as reports, configure alerts, and create dashboards. Through OpsGenie Alerts app, you can forward Splunk alerts to OpsGenie. With Splunk Integration, OpsGenie acts as a dispatcher for these alerts, determining the right people to notify based on on-call schedules, notifying them using email, text messages (SMS), phone calls and iPhone & Android push notifications, and escalating alerts until the alert is acknowledged or closed.


Functionality of the integration

  • OpsGenie has a Splunk specific alert app to send Splunk alerts to Opsgenie. OpsGenie also has a specific API for Splunk Integration, Splunk sends alerts through OpsGenie Alerts app to OpsGenie and OpsGenie handles automatic creation of alerts.

Add Splunk Integration in OpsGenie

  1. Please create an OpsGenie account if you haven't done already
  2. Go to OpsGenie Splunk Integration page,
  3. Specify who should be notified for Splunk alerts using the "Teams" and "Recipients" fields. Auto-complete suggestions will be provided as you type. 
  4. Copy the Api URL by clicking on the copy button or selecting.
  5. Click on "Save Integration".

Configuration in Splunk

  1. In Splunk install OpsGenie Alerts from Splunkbase.
  2. Run a search in Splunk to create an Alert.
  3. Click Save As and select Alert from the dropdown list.
  4. Populate alert title and specify conditions etc.
  5. Add a trigger action, select OpsGenie trigger action from the dropdown list.
  6. Paste the API URL into URL field. Only pasting API key will also work.
  7. Click Save.

Sample payload sent from Splunk and Draggable Fields in OpsGenie

Result field of the below content can differ according to the fields of the lines that matches to the search. That's why we provided some common fields of result object in available fields. Raw, Index, Serial, Source Type etc. can be example of the common fields.

We also added Result Object to the available fields, to make it possible to extract custom fields from result object.

Let's say you wanted to put the date_month field of result object to the alert: {{result.date_month}} solves the problem.

Create Alert payload:

{
    "session_key": "r41vK7psTN9iIp1HQXqgNxTHPz2AW_Ee3ELbdYM4FBqiBbI7L6f82o6f6IENt6Q_Xdq2V4jBSkjkyIfXIm56xbbcFcpWlcJNB0ZUZaezsImsTQ2lGWH26yiZ8l854Or8SPETrWuVgTKVeC",
    "search_name": "fail",
    "results_link": "http://Tuba-MacBook-Pro.local:8000/app/search/search?q=%7Cloadjob%20rt_scheduler__admin__search__fail_at_1464802733_32.0%20%7C%20head%201%20%7C%20tail%201&earliest=0&latest=now",
    "app": "search",
    "sid": "rt_scheduler__admin__search__fail_at_1464802733_32.0",
    "configuration": {
        "api_url": "http://4kmm916oxm9m.runscope.net"
    },
    "server_host": "Tuba-MacBook-Pro.local",
    "owner": "admin",
    "results_file": "/Applications/Splunk/var/run/splunk/dispatch/rt_scheduler__admin__search__fail_at_1464802733_32.0/per_result_alert/tmp_0.csv.gz",
    "server_uri": "https://127.0.0.1:8089",
    "result": {
        "date_month": "may",
        "index": "main",
        "_indextime": "1464802756",
        "date_minute": "15",
        "date_hour": "0",
        "splunk_server": "Tuba-MacBook-Pro.local",
        "date_mday": "11",
        "sourcetype": "secure",
        "source": "tutorialdata copy 2.zip:./www1/secure.log",
        "date_second": "2",
        "_serial": "0",
        "_sourcetype": "secure",
        "date_year": "2016",
        "eventtype": "",
        "_kv": "1",
        "timeendpos": "25",
        "timestartpos": "4",
        "linecount": "1",
        "date_zone": "local",
        "date_wday": "wednesday",
        "punct": "____::__[]:________...___",
        "_raw": "Thu May 11 2016 00:15:02 www1 sshd[4747]: Failed password for invalid user jabber from 118.142.68.222 port 3187 ssh2",
        "_eventtype_color": "",
        "_confstr": "source::tutorialdata copy 2.zip:./www1/secure.log|host::Tuba-MacBook-Pro.local|secure",
        "_time": "1462914902",
        "host": "Tuba-MacBook-Pro.local"
    }
}

Sample alert