Get started with Incident Intelligence

As part of Applied Intelligence, Incident Intelligence helps you correlate your incidents and reduce noise in your environment. It gives you an overview of all your incidents, their sources, and related events.

How it works

After you set up Incident Intelligence, our system will begin finding issues from your data sources.

In the issue feed, you can find an overview of all your issues, along with helpful information about them. You can also click any individual issue for more detail, including its analysis summary, event log, and details about correlated issues.

This image shows the AI issue feed in New Relic One.
This screenshot shows an example issue feed, which describes your issues' statuses, correlations, and more.

What's the difference between an issue, incident, and event? In short, these terms are like building blocks. Events are raw data from your sources. Incidents are made up of one or more events. Issues are composed of one or more incidents.

In more detail:

  • Events indicate a state change or trigger defined by your monitoring systems. An event contains information about the affected entity, and they are almost always triggered automatically by the system.

  • Incidents are groups of events that describe the "symptoms" of your system over time. These symptoms are detected by your monitoring tools, which evaluate your data streams and events.

  • Issues are groups of incidents that describe the underlying problem of your symptoms. When a new incident is created, Incident Intelligence opens an issue and evaluates other open issues for correlations.

Set up Incident Intelligence

To enable Incident Intelligence, follow these four steps. Afterwards, issues should start to appear in your issue feed.

1. Configure your environment (one-time)

To set up an environment in Incident Intelligence, you need an administrator to select a New Relic account for it. This account should be the one your team is using.

  • Who sets the environment? Only administrators, and only for accounts where they have admin privileges.​
  • Can administrators set more than one environment? They can set one environment per master account and its sub-accounts.​ More than one can be set if an administrator has privileges for more than one master account.
  • Need to change the environment's associated account? Reach out to your account executive or our support team for help.

Incident Intelligence is a cross-account product. This means you can send in data from any New Relic account or external source to correlate events.

2. Configure sources

After setting up your environment, determine your incident sources. These are your data inputs.

You can get data from any of the following sources:

By integrating Incident Intelligence with your alerts violations, you can get context and correlations from what you're monitoring.

To get data from alerts:

  1. From one.newrelic.com, click Alerts & AI, then click Go to Incident Intelligence.
  2. Click Sources and then choose New Relic Alerts.
  3. Select the policies you want to connect to Applied Intelligence, and click Connect.

You can add additional alerts policies or remove policies you've already connected in Sources > New Relic Alerts.

Adding alerts as a source will not affect your current configuration or notifications.

You can integrate Incident Intelligence directly with your PagerDuty services to ingest, process, and enhance all of your PagerDuty incidents.

To get data from PagerDuty:

  1. Make sure your PagerDuty API key has write access.
  2. From one.newrelic.com, click Alerts & AI, then click Go to Incident Intelligence.
  3. Click Sources and then choose PagerDuty.
  4. Enter your PagerDuty API key.
    • The key should be either a personal or general access key with write access. If it's created by a user, the user should be an admin.
  5. Select the PagerDuty services you want to connect to Applied Intelligence, and click Connect.

You can add additional services or remove services you've already connected in Sources > PagerDuty.

Connecting PagerDuty services to Applied Intelligence will not affect your current services or notifications.

By integrating Incident Intelligence with your Splunk log monitoring, you can:

  • Use your environment's log data for searches and key term reports.
  • Correlate alerts and search reports with your other metrics and incidents.

Applied Intelligence supports Splunk Light, Splunk Cloud, and Splunk Enterprise version 6.3 and higher.

To get data from Splunk:

  1. In your Splunk console, start a search for the relevant events.
  2. Save your search as an alert, configure your alert conditions, and then choose the webhook as the delivery method.
  3. Go to one.newrelic.com, click Alerts & AI, in the left nav under Incident Intelligence click Sources, then click Splunk.
  4. Copy the collector URL, and paste it into the webhook endpoint in the Splunk console.
  5. Optional: Use Splunk tokens to enrich alert data with Splunk metadata.

To enrich alerts data with your Splunk metadata, use Splunk tokens. This helps you leverage your search data, which includes metadata and values from the first row of search results.

If you want to... Do this...
Access search data

Use the format $<fieldname>$. For example, use $app$ for the app context for the search.

Access field values

To access field values from the first result row that a search returns, use the format $result.<fieldname>$. For example, use $result.host$ for the host value and $result.sourcetype$ for the source type.

Use variables

You can leverage any of the Selected fields in the Splunk search and add any unique fields to the Selected fields to make the data available as a variable.

The following fields will automatically provide hints to the correlation engine:

  • app: parsed as APPLICATION_NAME
  • application:parsed as APPLICATION_NAME
  • application_name: parsed as APPLICATION_NAME
  • cluster: parsed as CLUSTER_NAME
  • computer: parsed as HOST_NAME
  • Dc: parsed as DATACENTER_NAME
  • datacenter: parsed as DATACENTER_NAME
  • host: parsed as HOST_NAME
  • host_name: parsed as HOST_NAME
  • hostname: parsed as HOST_NAME
  • transaction: parsed as EVENT_ID
  • Transaction_id: parsed as EVENT_ID
  • user: parsed as USER_NAME

By integrating Incident Intelligence with Prometheus Alertmanager, you can receive and correlate your Prometheus alerts with events from other sources.

To integrate Prometheus Alertmanager:

  1. Set up your Alertmanager configuration file by running:

    ./alertmanager -config.file=simple.yml
  2. Go to one.newrelic.com, click Alerts & AI, in the left nav under Incident Intelligence click Sources, then click Prometheus Alertmanager.
  3. Copy the Prometheus Alertmanager URL, and paste it into the <webhook_config>/url section of your Alertmanager config file.

  4. Reload the Prometheus Alertmanager configuration with one of the two methods:
    • Send a SIGHUP to the process.
    • Send an HTTP POST request to the /-/reload endpoint.

You can integrate Incident Intelligence with Grafana's notifications for insight into events across your applications and environment. Grafana's webhook notification is a simple way to send information over HTTP to a custom endpoint.

To integrate Grafana as a new webhook:

  1. Log into your Grafana portal using Admin permissions, and choose Alerting.
  2. On the Grafana Notification Channels page, click New Channel > Webhook.
  3. Go to one.newrelic.com, click Alerts & AI, in the left nav under Incident Intelligence click Sources, then click Grafana.
  4. Copy the URL, and paste it into your new Grafana webhook.

You can integrate Incident Intelligence with Amazon CloudWatch to provide incident management for all of your AWS services.

To integrate Amazon CloudWatch:

  1. Go to one.newrelic.com, click Alerts & AI, in the left nav under Incident Intelligence click Sources, then click ​​​​​Amazon Web Services.
  2. Copy the URL.
  3. Create a new Amazon SNS topic.
  4. Set CloudWatch to forward all Alarms state changes to that topic:

    • In the Amazon CloudWatch UI, click Events > Event Pattern.
    • Select Service Name > CloudWatch.
    • Select Event Type > CloudWatch Alarm State Change.
    • Select Targets > SNS Topic, and select your new Amazon SNS topic.
  5. Create a new subscription:

    • In the Amazon AWS UI, click Create a Subscription.
    • Select your new Amazon SNS topic.
    • Select Protocol > choose HTTPS.
    • In Endpoint, paste the URL you previously copied from the Applied Intelligence Sources.

Incident Intelligence supports a dedicated REST API interface that lets you integrate with additional systems. The interface allows instrumentation of your code or other monitoring solutions to report any kind of metric or event.

  • A metric can be a raw data point such as CPU, memory, disk utilization, or business KPI.
  • An event can be a monitoring alert, deployment event, incident, exceptions or any other change in state that you want to describe.

You can also send any type of data to Incident Intelligence straight from your own systems or applications. The REST API supports secure token-based authentication and accepts JSON content as input.

For more information on authentication and the full API reference, see REST API for New Relic Applied Intelligence.

3. Configure destinations

Now that you've set up your sources, you can configure your destinations. These are the data outputs where you view your incidents.

You can set destinations using any of the following methods:

Send data to PagerDuty

Recommended: Create a new PagerDuty service to use as a destination. Because PagerDuty services can also be used as sources, this can help you distinguish your data input from your output.

To create a PagerDuty destination:

  1. Go to one.newrelic.com, click Alerts & AI, in the left nav under Incident Intelligence click Destinations, then click PagerDuty.
  2. Enter your PagerDuty API key.
    • The key should be either a personal or general access key with write access. If it's created by a user, the user should be an admin. If you've configured a PagerDuty source with an API key, you can use the same key.
  3. Select the PagerDuty services you want to connect to Applied Intelligence, and click Connect.

When you're ready, you can add policies for one or more PagerDuty destinations. You can also transfer the policies over from your existing services or leave them as sources if needed.

From the Destinations > PagerDuty page, you can also:

  • Review the permissions for your services. Click Authorize when you're done.
  • Add or delete existing services from the PagerDuty destination.
  • Edit permissions for any service.

To configure your PagerDuty destinations, use the following settings:

Configuration setting Description
Trigger new incidents Required. Trigger correlated parent incidents so you can identify issues faster.
Edit incident titles Required. Alter your incident titles to help you orient and understand issues.
Add new integrations Required. Add integrations to enable incident creation for selected services.
Add webhook extensions Add webhook extensions to sync user actions in PagerDuty to New Relic. This lets you update the correlated issue state.
Auto-resolve correlated incidents When enabled, this will resolve and automatically close correlated parent/child incidents.
Select a user to take actions in PagerDuty

You need to select a user before you can enable deep integration with PagerDuty. Once you do, the user can:

  • Add notes to incidents (required): Incident notes are used to enrich incidents with context.

  • Acknowledge triggered incidents: When enabled, Applied Intelligence will acknowledge and correlate newly triggered incidents in PagerDuty before you're notified.

  • Use the original escalation policy: When enabled, the escalation policy of the source service will be applied to each incident.

Send data via webhook

Incident Intelligence will send the event body in JSON format by HTTPS POST. The system expects the endpoint to return a successful HTTP code (2xx).

To configure Incident Intelligence to send data via webhook:

  1. Go to one.newrelic.com, click Alerts & AI, in the left nav under Incident Intelligence click Destinations, then click Webhook.
  2. Required: Configure the unique webhook key, used in Applied Intelligence to refer to this webhook configuration and its specific settings.
  3. Required: Configure the destination endpoint where the webhook payload will be sent.
  4. Optional steps:
    • Configure custom headers, which are key:value pairs of headers to be sent with the request. Example: "Authentication" "Bearer" <bearer token>
    • Configure a custom payload template that can be used to map New Relic fields to match the destination tool's expected name and format.
    • Configure priority mapping (critical, high, medium, or low), used to map New Relic's priorities to the priorities expected at the destination.

For examples of destination templates, webhook formats, and JSON schema, see the Incident Intelligence destination examples.

4. Configure pathways

To control when and where you want to receive notifications from your incidents, you can configure pathways.

To add a pathway:

  1. Go to one.newrelic.com, click Alerts & AI, in the left nav under Incident Intelligence click Pathways, then click Add a pathway.
  2. In the query builder box, select an attribute, such as application/name.
    • This can be from the list of all attributes available in PagerDuty incidents and New Relic alerts violations, or you can add your own attributes.
  3. Select a logical operator. For example, contains.
  4. Enter a specific value to complete the logical expression.

    • To include all issues created by your sources, select Send everything. (Use this if you only use one PagerDuty service to manage all incidents.)
    • To build more complex logic, use the AND/OR operators.
  5. Select one or more of your destinations.

To edit or remove existing pathways, mouse over the pathway's name on the Pathways page.

Use decisions

To further reduce noise or get improved incident correlation, you can change or customize your decisions. Decisions determine how Incident Intelligence groups incidents together.

To get started, see Decisions.

Use suggested responders

If you’re using PagerDuty as your incident notification tool, Incident Intelligence suggests relevant team members that can help resolve your issues.

Incident Intelligence learns from your PagerDuty alert data to provide suggestions for each new incident. Once you receive a suggestion, you can contact the responder or search for relevant documentation that person may have authored.

To get started, enable PagerDuty as a source for Incident Intelligence. Afterwards, you can view the suggestions in two places:

  • The issue feed, where you can also provide feedback​ on the suggestions.

  • Directly within PagerDuty (both UI and API.) If you’re also using PagerDuty as a destination, the suggestions will appear in your issue notifications payload.

This feature doesn't account for on-call availability at the time of incident.

In order to train the model, we use the information PagerDuty provides about individuals. We ingest incident information only, not users’ contact details.

EU and US datacenter and Incident Intelligence data

New Relic's Incident Intelligence service is performed solely in the United States. By using New Relic Incident Intelligence, you agree that New Relic may move your data to, and process your data in, the US region. This applies whether you store your data in New Relic's US region data center or in our EU region data center.

If you elect to use the Suggested Responder feature and manage EU-based individuals, you may need to confirm that an appropriate data processing agreement is in place.

For more help

If you need more help, check out these support and learning resources: