Proactive Detection with Applied Intelligence

With Applied Intelligence's Proactive Detection, you can be notified of unusual app behavior and access an automatic analysis of this unusual behavior. You can have notifications for anomalies delivered in Slack, or you can set up a webhook to deliver messages when you need them. These events are automatically available in NRDB for dashboarding, alerting, and integration with Incident Intelligence.

Why it matters

With Proactive Detection, Applied Intelligence delivers insights about anomalies in your production system, along with an automatic analysis of the anomaly. When an anomaly is detected, you can view it in an overview of anomalies, or we'll send notifications directly to your Slack channel or a webhook.

How it works

Proactive Detection uses the following methods to detect anomalies in your app data:

  1. Proactive Detection monitors metric data reported by an APM agent, building a model of your typical application dynamics, and focuses on key golden signals: throughput, response time, and errors.
  2. If one of these golden signals shows anomalous behavior, the system flags it and tracks recovery to normal behavior.
  3. The system adapts to changes in your data, and continuously updates models based on new data.

Receiving notifications: We send notifications when we detect anomalous changes in throughput or response time. The notifications are sent to selected Slack channels, or sent via webhook. When the anomaly goes back to normal, a recovery message is sent. If you don't want to receive notifications, you can choose to just save the events to NRDB.

Anomaly analysis: For each anomaly, we provide a link in Slack to an analyze anomaly page. This page generates automatic insights into the anomaly. The page is also available from the anomaly overview page, which lists all recent anomalies. This page uses your existing APM and Proactive Detection data to provide explanations as to the cause of the anomaly.

Activity stream: Inside the activity stream on the New Relic One homepage and the APM Summary page you will see relevant anomalies for applications that you've configured with Proactive Detection. Clicking on any of the anomaly events in the activity stream will bring up the analysis page for that anomaly.

Applications will not always generate anomalies, so it can be normal to not receive any detections.

Requirements

To use Proactive Detection, ensure you have:

  • An APM agent installed on applications to monitor
  • For Slack configuration: the Applied Intelligence Slack application installed into your Slack workspace by an IT administrator.

For more details, see Data limits.

Set up Proactive Detection

You can configure these features in the Proactive Detection UI:

Set up for Slack

To use Proactive Detection with Slack:

  1. Go to one.newrelic.com, in the top nav click Alerts & AI, then click Proactive Detection to go to Configuration.
  2. Click Slack, then click plus Add configuration.
  3. Input the following information into the form:

    • Choose a name for your configuration that helps you easily identify it from others in your account.
    • Select an account.
    • Select up to 1,000 applications. Note that certain applications with low throughput might not be good candidates for Proactive Detection, as they can be more sensitive to smaller amounts of data fluctuation.
    • Choose which Slack channels receive notifications, including existing or new ones. This prompts the workflow to add the Applied Intelligence Slack application to your selected channel.

      If you experience an error when assigning Slack channels, make sure that the New Relic AI Slack application has been added to your Slack workspace.

  4. Enable the configuration.

You can modify the applications for each configuration at any time by selecting Edit configuration in the configuration table.

Set up for webhooks

To use Proactive Detection with webhooks:

  1. Go to one.newrelic.com, in the top nav click Alerts & AI, then click Proactive Detection to go to Configuration.
  2. Click Webhook, then click plus Add configuration.
  3. Input the following information into the form:

    • Choose a name for your configuration that helps you easily identify it from others in your account.
    • Select an account.
    • Select up to 1,000 applications. Note that certain applications with low throughput might not be good candidates for Proactive Detection, as they can be more sensitive to smaller amounts of data fluctuation.
    • Provide the webhook URL.
    • Provide optional custom headers.
    • Choose to edit the custom payload, or enable using the default payload.
  4. Enable the configuration.

You can modify the applications for each configuration at anytime by selecting Edit configuration in the configuration table.

Set up without notifications

To use Proactive Detection when you don't need to receive notifications:

  1. Go to one.newrelic.com, in the top nav click Alerts & AI, then click Proactive Detection to go to Configuration.
  2. Click No notifications, then click plus Add configuration.
  3. Input the following information into the form:

    • Choose a name for your configuration that helps you easily identify it from others in your account.
    • Select an account.
    • Select up to 1,000 applications. Note that certain applications with low throughput might not be good candidates for Proactive Detection, as they can be more sensitive to smaller amounts of data fluctuation.
  4. Enable the configuration.

You can modify the applications for each configuration at anytime by selecting Edit configuration in the configuration table.

Mute notifications (Slack only)

In Slack, detections coming from specific applications can be muted temporarily or permanently. The entire channel can also be muted temporarily. This is useful in the case of an incident or when the channel should otherwise not be interrupted.

To mute in Slack, select Mute this app’s warnings or Mute all warnings, then select the duration. We will resume sending notifications for any detections once the muting duration has completed.

Muting an application permanently removes it from the configuration. To add it back in, go to one.newrelic.com, in the top nav click Alerts & AI, then click Proactive Detection, and select the configuration to edit.

Muting Proactive Detection notifications does not affect alerts.

Use Proactive Detection Slack messages

Each anomaly message has several key pieces of information you can use to learn more about and start troubleshooting the potential issue:

  • The application name and a link to more information about it in New Relic One.
  • The metric experiencing an anomaly and a link to its details in New Relic One.
  • A graph of the metric over time to provide a visual understanding of the anomaly’s behavior and degree.
  • An Analyze button that navigates to an analysis page in Applied Intelligence that identifies key attributes that are unique to the anomaly, anomalies found upstream or downstream, and any other relevant signals.

Once an anomaly has returned to normal, we send a recovery notification with the option to provide feedback. Your feedback provides our development team with input to help us improve detection quality. If we helped you, you can select Yes or No.

View Anomaly overview UI

In addition to notifications for anomalies that give you information via Slack or webhook, Proactive Detection also includes a UI view with more information about the anomalies in your environment. This provides a list of all the recent anomalies from every configuration in the selected account. You can also select each anomaly to view a detailed analysis.

Use anomaly events with NRDB

Once you configure Proactive Detection, anomaly events will be sent to New Relic’s database (NRDB). You can query NRDB for Proactive Detection anomaly events and use them in conjunction with other Applied Intelligence tools.

Query for Proactive Detection events from NRDB

You can use NRQL to query NRDB for any Proactive Detection events recorded over the previous 7 days:

FROM ProactiveDetection SELECT *

Attribute Description

anomaly.deliveredAt

Number

The time at which the anomaly event was initially recorded, in epoch milliseconds.

Example: 1584366819000

anomaly.description

String

A brief description of the anomaly event.

Example: Non-web throughput has returned to normal

anomaly.endedAt

String

The time at which the anomalous period ended, in epoch milliseconds.

Example: 1584366819000

anomaly.startedAt

Number

The time at which the anomalous period began, in epoch milliseconds.

Example: 1584366819000

anomaly.type

The type of data that was analyzed, including error_rate, response_time.web, response_time.non_web, throughput.web, or throughput.non_web.

anomaly.status

The state of the anomaly as the NRDB event is inserted, including open or close.

anomaly.uuid

A unique identifier for the NRDB anomaly event.

Example: 4ee3b908-d323-4677-94c8-339762c931bc

anomaly.id

ID of the anomaly associated with the event.

Example: 56341b5608bf68e1

entity.accountId

Number

The ID of the account for the entity.

entity.domain

Enum

The domain for the entity.

Example: APM

entity.domainId

String

The id used to uniquely identify the entity within the domain.

entity.guid

String

The guid used to uniquely identify the entity across all products.

entity.name

String

The name of the entity.

Example: Laura's coffee service

Send Proactive Detection events to Incident Intelligence

You can send Proactive Detection events to Incident Intelligence to be processed and correlated with other activity in your system:

  1. Create an alert condition for your NRQL query that pulls proactive detection events from NRDB.
  2. Configure a new incident intelligence source for your condition.
  3. (Optional) Create Decision logic to correlate future anomalies with related events.

Webhook payload and examples

Proactive Detection sends the event body in JSON format via HTTPS POST. The system expects the endpoint to return a successful HTTP code (2xx). If you use webhooks to configure Proactive Detection, use these examples of the webhook body format and JSON schema.

Attribute Description

category

Enum

The category of data that was analyzed.

Categories include web throughput, non-web throughput, web transactions, non-web transactions, and error class.

data

List

The time series data leading up to the detection.

data[].timestamp

Number

The timestamp of the data point in epoch milliseconds.
Example: 1584366819000

data[].unit

String

The unit describing the value of the data point.

Data units include count, milliseconds, and error_rate.

data[].value

Number

The value of the data point.

Example: 1.52

detectionType

Enum

The type of data that was analyzed.

Types include latency, throughput, and error_rate.

entity

Object

The entity that reported the unusual data.

entity.accountId

Number

The ID for the entity's account.

entity.domain

Enum

The domain for the entity.

Example: APM

entity.domainId

String

The id used to uniquely identify the entity within the domain.

entity.guid

String

The guid used to uniquely identify the entity across all products.

entity.name

String

The name of the entity.

Example: Laura’s coffee service

entity.link

String

A link to view the entity. Example:

https://rpm.newrelic.com/accounts/YOUR_ACCOUNT_ID/applications/987654321”

severity

Enum

A description of how unusual of a change occurred, including NORMAL, WARNING, or CRITICAL.

version

String

Version used to describe the data being provided.

Example: v1

JSON schema example

Applied Intelligence will send the event body in JSON format via HTTPS POST. The system expects the endpoint to return a successful HTTP code (2xx).

Template:

{
  "version": "{{version}}", 
  "entity": {
    "type": "{{entity.type}}",
    "name": "{{entity.name}}",
    "link": "{{entity.link}}",
    "entityGuid": "{{entity.entityGuid}}",
    "domainId": "{{entity.domainId}}",
    "domain": "{{entity.domain}}",
    "accountId": {{entity.accountId}}
  },
  "detectionType": "{{detectionType}}",
  "category": "{{category}}",
  "data": [{{#each data}}
    {
      "value": {{value}},
      "unit": "{{unit}}",
      "timestamp": {{timestamp}}
    }
    {{#unless @last}},{{/unless}}
  {{/each}}]
}
    

Sample payload:

{
  "version": "v1", 
  "entity": {
    "type": "APPLICATION",
    "name": "My Application",
    "link": "https://rpm.newrelic.com/accounts/ACCOUNT_ID/applications/123",
    "entityGuid": "foo",
    "domainId": "123",
    "domain": "APM",
    "accountId": YOUR_ACCOUNT_ID
  },
  "detectionType": "metric",
  "category": "web throughput",
  "severity": "CRITICAL",
  "data": [
    {
      "value": 100,
      "unit": "count",
      "timestamp": 1584047560917
    }
    ,
  
    {
      "value": 99,
      "unit": "count",
      "timestamp": 1584047620917
    }
    ,
  
    {
      "value": 0,
      "unit": "count",
      "timestamp": 1584047680917
    }
  ]
}

Data limits

In addition to requirements, data limits include:

  • APM app transactions per month: up to 100 million included free
  • Monitored APM applications: limited to 1,000 per configuration
  • Slack configurations: limited to 200 per account
  • Webhook configurations: limited to 200 per account
  • Configurations without notifications: limited to 200 per account

For more help

If you need more help, check out these support and learning resources: