Automatic anomaly detection is the easiest way for you and your team to detect and analyze unusual behavior in your system. It does this using the following methods:
Anomaly detection monitors metric data reported by an agent, building a model of your typical application dynamics, and focuses on three key golden signals: throughput, response time, and errors.
If one of these golden signals shows anomalous behavior, the system flags the behavior and tracks recovery to normal behavior.
The system adapts to changes in your data, and continuously updates models based on new data.
Automatically on: By default, anomaly detection monitors all your applications, with no action required by you. When an anomaly is detected, it's automatically surfaced in various activity streams, the applied intelligence anomalies feed and is available for querying via NRQL.
Correlation: Anomalies are connected to our correlation logic, otherwise known as decisions. If our correlation logic determines that issues are connected, then a single issue will be created from multiple incidents. You can see this correlated issue in the issues feed. Anomalies provide additional context about what changed around the time an issue occurred so your team can easily detect patterns and diagnose problems.
Receiving notifications: When New Relic detects anomalous changes in throughput, error rate, or response time, your team has the option to receive notifications about this behavior. We recommend sending notifications to selected Slack channels or email channels. When the anomalous behavior returns to normal then a recovery message is sent. If you don't want to receive notifications you can still access to the data using a NRQL query.
Anomaly analysis: For each anomaly, we provide a link in Slack to an analyze anomaly page. This page generates automatic insights into the anomaly. The page is also available from the anomalies tab, which lists recent anomalies. This page uses your existing and anomaly detection data to provide explanations as to the cause of the anomaly.
Activity stream: Inside various activity streams such as the New Relic homepage, summary page, entity list views, and New Relic Lookout, you'll see relevant anomalies from your APM-monitored applications. Clicking on any of the anomaly events in the activity stream brings up the analysis page for that anomaly.
Applications don't always generate anomalies, so it can be normal to not receive any detections.
To see configure an anomaly detection, go to one.newrelic.com > All capabilities > Alerts & AI and click Anomaly detection in the left pane.
Choose a name for your configuration that helps you easily identify it from others in your account.
Select the account you want to use for this configuration.
Select the applications and services you want to include.
Select the metrics you want to be notified on.
Select how you want to be notified. These are the options:
Slack. You need to select a Slack channel and the time zone the charts will use.
Webhook. You need to add the URL.
No notifications. If you prefer not to receive notifications, select this option.
Click Save configuration.
How to see the details of an anomaly
You can explore the details of any anomaly in your system's performance to better understand what errors you're receiving and why.
To see the details of a detected anomaly, go to one.newrelic.com > All capabilities > Alerts & AI. Click the Anomalies tab to see the dashboard and then click the anomaly you want to see.
The details of an anomaly gives you this information:
Name of the anomaly. You can click on it and it will take you to APM & Services capability to see a summary of the selected anomaly.
Description of the signal.
Graphical data by error rate or error count. You can change the frequency.
Analysis. See the found attributes, anomalies, and errors to investigate.
If you expand the page, you also can see specific data about the anomaly: metadata, entity, recent activity, and related dashboards.
You can also see all the anomalies related to a configuration:
Search the configuration you want to see its anomalies.
Click the icon of the configuration and select View anomalies.
To see the anomalies of a specific configuration, go to one.newrelic.com > All capabilities > Alerts & AI and click Anomaly detection in the left pane.
Anomaly set-up
Once you choose to monitor anomalous behavior in your system using either our custom or automatic anomaly detection, you will need to make sure that your team is notified of any unusual behavior and that you can query and understand your data. It doesn't matter if you choose custom or automatic anomaly detection, the set-up is the same.
To use anomaly detection of New Relic, ensure that you have:
Anomaly detection is enabled automatically at no additional cost. To receive notifications or to have a configuration (group of apps) that you can add as a source for incident intelligence, you'll need to create an anomaly detection configuration.
You can create a configuration in the anomaly detection UI:
Choose a name for your configuration that helps you easily identify it from others in your account.
Select an account.
Select up to 1,000 applications. Note that certain applications with low throughput might not be good candidates for anomaly detection, as they can be more sensitive to smaller amounts of data fluctuation.
Select the metrics you want to be notified on.
Select how you want to be notified. These are the options: Slack, Webhook, and no notifications.
Click Save configuration.
Use anomaly detection with Slack:
Select Slack.
Choose which Slack channel receives notifications. You can select any existing public or private channel. This prompts the workflow to add the applied intelligence Slack application to your selected channel. Or create a new channel for anomaly detection, create the channel in Slack first, then select that channel.
Tip
If you experience an error when assigning Slack channels, make sure that the New Relic applied intelligence Slack application has been added to your Slack workspace.
Save the configuration.
You can modify the applications for each configuration at any time by selecting the configuration in the configuration table.
Use anomaly detection with webhooks:
Select Webhook.
Input the following information into the form:
Provide the webhook URL.
Provide optional custom headers.
Choose to edit the custom payload, or enable using the default payload.
Save the configuration.
You can modify the applications for each configuration at any time by selecting the configuration in the configuration table.
Choose a name for your workflow that helps you easily identify it from others in your account.
Click the Advanced button to build an advanced filter for all attributes.
Click the Select or enter attribute selector and add origins. Then, select contains as the operator and anomalies as the value.
Select entitiesData.typescontainsApplication.
Select tag.entity.guid to select an entity of your choice.
Select the signalTypecontains and the signal you would like to be notified on.
Click Additional settings if you want to enrich your data.
Select a destination channel of your choosing.
Click Test workflow if you want to check if you workflow works.
Click Activate workflow.
To reduce noise, we recommend sending anomalies to a low priority notification channel such as Slack or email.
In Slack, you can temporarily or permanently mute detections coming from specific applications. You can also temporarily mute the entire channel. This is useful in the case of an incident or when the channel should otherwise not be interrupted.
To mute in Slack, select Mute this app's warnings or Mute all warnings, then select the duration. Notifications will resume once the muting duration has completed.
Muting an application permanently removes it from the configuration. To add it back in, go to one.newrelic.com > All capabilities > Alerts & AI > Anomaly detection, and select the configuration to edit.
Each anomaly message has several key pieces of information you can use to learn more about and start troubleshooting the potential issue:
The application name and a link to more information about it in the New Relic UI.
The metric experiencing an anomaly and a link to its details in the New Relic UI.
A graph of the metric over time to provide a visual understanding of the anomaly's behavior and degree.
An Analyze button that navigates to an analysis page in applied intelligence that identifies key attributes that are unique to the anomaly, anomalies found upstream or downstream, and any other relevant signals.
Once an anomaly has returned to normal, we send a recovery notification with the option to provide feedback. Your feedback provides our development team with input to help us improve detection quality. When feedback is provided on throughput anomalies, an evaluation is run each hour based on your feedback to find a more suitable model. If we helped you, you can select Yes or No.
In addition to notifications for anomalies that give you information via Slack or webhook, you can view more information about the anomalies in your environment via the anomalies Feed on the alerts and applied intelligence overview page. That tab provides a list of all the recent anomalies from every configuration in the selected account, and you can select an anomaly for a detailed analysis.
In addition to viewing anomalies in the anomalies feed, anomalies are correlated to other incidents and grouped into issues. Anomalies provide context to what has changed around a critical issue. Out of the box anomalies will be correlated with incidents of the same entity type. Correlates unique to your use case can be added using decisions.
Anomalies are displayed in various New Relic activity streams and in the applied intelligence anomalies feed. You can customize what is displayed using the anomaly visibility settings (for example, hiding throughput anomalies on an activity stream but keeping them in the anomalies feed).
To find these settings: from Alerts & AI, under Anomaly detection, click Settings.
Notes on using these settings:
These settings are applied at the user level. Changes you make won't affect others users in your organization.
Regardless of these settings, the anomalies are still reported and available for NRQL querying.
Details on these UI sections:
AI overview and anomalies tab: Use the AI overview and anomalies tab setting to hide anomalies from the AI overview and anomalies tab setting. Please note you also can use filters specific to these views as well.
Global activity stream: Use the global activity stream section to customize what anomalies are shown in the various New Relic activity streams, including the New Relic homepage, summary, and Lookout.
Anomaly types: Use the check boxes here to hide specific types of anomalies. For example, uncheck Web throughput and Non-web throughput anomalies to hide these types of anomalies from both the activity streams and the AI overview and anomalies tab. Note they are still reported and available for querying.
You can use NRQL to query and chart your anomaly detection data using the NrAiAnomaly event. For example:
FROM NrAiAnomaly SELECT *
Important
This data has previously been attached to the ProactiveDetection event. That event was deprecated on April 7, 2021. If you use ProactiveDetection in your custom charts, you should convert those queries to using NrAiAnomaly.
Here are important attributes attached to this event:
Attribute
Description
timestamp timestamp
The time at which the event was written.
anomalyId number
If this incident originated from a New Relic anomaly it will have an anomaly ID.
closeTime timestamp
The time when the anomaly ended. Example: 1615304100000.
configurationType string
The type of configuration monitoring the event. If at least one configuration is monitoring the entity, this is set to configuration. Otherwise, it's set to automatic.
entity.accountId number
The New Relic account ID to which the entity belongs.
entity.domain enum
The domain for the entity. Example: APM.
entity.domainId string
The id used to uniquely identify the entity within the domain.
entity.guid string
The GUID of the entity. This is used to identify and retrieve data about the entity via NerdGraph. Identical to entityGuid.
entity.name string
The name of the entity whose data was determined to be anomalous. Identical to entityName. Example: Laura's coffee service.
entity.type string
The type of entity (currently only APPLICATION but will change with future functionality).
entityGuid string
The GUID of the entity. This is used to identify and retrieve data about the entity via NerdGraph. Identical to entity.guid.
entityName string
The name of the entity whose data was determined to be anomalous. Identical to entity.name.
evaluationType string
This is always anomaly.
event string
Indicates whether it's the beginning (open) or end (close) of the anomalous data.
openTime timestamp
The time when the anomaly opened. Example: 1615303740000.
signalType string
The type of data that was analyzed. For example, error_rate or response_time.non_web.
title string
Description of the anomaly. Example: Error rate was much higher than normal.
By integrating incident intelligence with your anomaly detection, you can get context and correlations. To learn about doing this in incident intelligence, see Configure sources.
You can also select Connect to incident intelligence from inside of a configuration.
Automatic anomaly detection sends the event body in JSON format via HTTPS POST. The system expects the endpoint to return a successful HTTP code (2xx). If you use webhooks to configure automatic anomaly detection, use these examples of the webhook body format and JSON schema.
Attribute
Description
category enum
The category of data that was analyzed.
Categories include web throughput, non-web throughput, web transactions, non-web transactions, and error class.