Enhance notifications with Incident Workflows

BETA

New Relic Incident Workflows iis a flexible notification system that works with Alerts and Applied Intelligence. Incident Workflows uses a friendly user interface to build rich notification message templates and customize how, when, and where they’re sent.

Incident Workflows is currently in BETA. If you haven't signed up yet, go to one.newrelic.com > Alerts & AI and click Incident Workflows Request access.

Why it matters

Incident Workflows sends alert events to many destinations in any format, enriched with additional context. The destinations include various notification platforms, like webhooks and Jira. Before being sent to a destination, a notification is enriched with extra information, including from the New Relic database (NRDB). Not only that, a workflow creates a two-way connection between your notification platforms and New Relic. All of your Alerts policies can be grouped into a single incident workflow.

How it works

Workflows are a collection of triggers and actions for alert incidents. A workflow is a set of instructions that defines enriched notification templates, chooses triggers, and connects to multiple notification platforms.

Triggers are Alert policies that start workflow processes.

Actions are operations that execute when workflows trigger. Actions include enriching a notification with extra information and configuring your notification platform to define the notification message format.

When notifications are sent

A workflow uses the same incidents you defined by your Alerts policies.

In New Relic Alerts, incident preferences define how incidents and notifications are created when conditions in the alert policy are violated.

Create or edit a workflow

Use the workflows dashboard to create, edit, enable, or disable your workflows. Here’s a basic overview:

  1. Add and name the workflow.
  2. Select triggers.
  3. Enrich your notifications.
  4. Create notification actions.
  5. Test your workflow.
  6. Activate your workflow.

From one.newrelic.com, click Alerts & AI. In the left pane, click Incident Workflows to open the workflows dashboard.

Step 1. Add and name a workflow

When you add a workflow, you’ll give it a name. Write a unique, descriptive name that will help you remember its purpose later.

Step 2. Select your triggers

An alerts condition defines a violation. Alerts violations are workflow triggers. Each workflow has a single account and uses that account’s alert conditions for its triggers.

Each workflow must have at least one trigger. If you select more than one, only one of them needs to be true to trigger an action.

Step 3. Add NRQL query data to enrich your notifications

After you select triggers, enrich your notification data with NRQL queries. You'll use Query Builder to write a NRQL query. The results of the query are a string and are used as part of the notification message.

Use Click here to query for errors to create a NRQL query based on your errors. You can also write a query into your workflow, even if it returns an empty result.

Each workflow can have seven NRQL queries.

Screen capture showing where to add a query to enrich a notification.
An example of a NRQL query you might use to enrich your workflow.
You can query with violation specific variables - for example {{entity.id}}

NRQL query enrichment examples

Here are some examples of how to set up enrichment queries.

Check Synthetic health

If you have a variety of Synthetic monitors measuring your website, you can define an alert trigger for the number of unsuccessful health checks for your monitors.

This query lets your site reliability team track your monitors’ unsuccessful health checks without additional effort. This helps your team respond more quickly.

SELECT count(*) FROM SyntheticCheck WHERE result != 'SUCCESS' since since 10 minutes FACET monitorName
Know when violation-related traffic for an application drops

This query tracks the number of transactions on the entity with a condition violation, using the custom variable entity.name:

SELECT count(*) FROM Transaction WHERE appName = '{{entity.name}}' since 10 minutes ago

This query tracks the latest HTTP status code response, filtered by the entity that violated the condition’s threshold:

From Transaction select latest(httpResponseCode), average(duration) where appName = '{{entity.name}}'

This query gets the number and ingest time by product running within a Kubernetes pod so you can find a potential remedy:

SELECT uniqueCount(displayName), sum(nr.ingestTimeMs) from K8sServiceSample where entityName = '{{entity.name}}' since 1 hour ago

Add custom variables to your query

Custom variables are properties you can add to violations when you configure a workflow action as part of the configuration of a workflow action. You can get information about the alert, such as condition, violation, and entity, using double curly brackets: {{variable_name}}.

To get information about the entity that violated a condition, you can use custom variables as part of the WHERE statement of the query. For example, here's a query using an entity.name variable to get the state of an Amazon EC2 instance:

SELECT latest(ec2State) FROM ComputeSample where provider = 'Ec2Instance' where entityName = '{{entity.name}}'

This query has a single value result (stopped) because it only selects a single field. The entity.name is the entity identifier. You can also use entity.id and any other entity properties like this.

Step 4. Create notifications

You can use monitoring platforms ServiceNow, Atlassian Jira Cloud, and AWS EventBridge with Workflows. You can also configure a webhook to other destinations.

Every workflow requires at least one notification action. To create an action:

  1. Configure the notifier
  2. Build the message
  3. Test the notifier

Configure the notifier

To configure your notifier, choose a notification platform, and then configure its destination fields.

  1. Click + Add an action, and then under Notify your channels, click Add.
  2. Choose your notification platform.
  3. Complete the destination fields. Complete the destination fields based on your notification platform.
  4. To close an issue with Jira or ServiceNow, check Allow two-way integration. For example configurations, see Jira and ServiceNow

Configure your notification message

The notification message will be delivered to your notification platform. A notification message includes one or more key-value fields.

When using Jira and ServiceNow, you must map a workflow field name value to its corresponding field in your notification platform.

A field’s value can be any combination of the following:

  • Free text
  • Output enriched with NRQL query results
  • Output enriched with custom variables

In each notifier, the payload is populated with your query results and some custom variables. You can remove those and add other custom variables with #.

In a Value field, type # to see a list of available outputs and custom variables.

You can also write the custom variables manually:

  • {{enricher_name.result}} is replaced with its result.
  • {{variable_name}} is replaced with the variable’s value.

Choose the variable name from a list of common fields or create a custom string.

Screenshot showing the fields for building a notification message.
ServiceNow notification message example

Optional: Test the notifier

Send a notification message to verify that you have a good connection to your platform.

A workflow test creates a ticket with placeholders for the final values. You get live results from the NRQL queries used to enrich the notifications, but N/A when referring to violations-related data. This is expected behavior.

Updated variables are rendered on your notification platform when a real violation triggers the workflow. These values are part of the violation event. If you use variables as part of the notification message, the variables value uses the variable name (for example, entity.name) as a placeholder.

If the notifier test fails, check the following:

  • Make sure to use the proper domain, username, and password/token for your platform.
  • Confirm your destination is live/awake.
  • Make sure to use different names for your notifiers.

Notifier configuration

Here are some notifier configuration instructions for the different channels

Atlassian Jira notifier

With Jira as a notifier you can use incident violation data to create a new Jira ticket.

With two-way integration, changes to a Jira ticket will update incidents in New Relic.

Atlassian Jira notifiers need:

  • Notifier name
  • Jira endpoint
  • Jira API key
  • A user with write permission
  • Optional: Assignee and watchers for the ticket
  • A notification message where every field name of the message has a corresponding field name in the project schema (text field or multi-line text field)
Screenshot showing an Atlassian Jira setup.
An example Jira configuration with two-way notification enabled.

In order to configure two-way integrations with the Jira UI, create a new webhook:

  1. In Jira under Settings > System > Webhooks (category “Advanced”) > Create New, check to send updates back to New Relic.
  2. In the URL field, use: https://workflows.newrelic.com/proxy/NEW_RELIC_ACCOUNT_ID/incident/jira/
  3. In the Issue related events section, under issue, choose the update option.

Replace NEW_RELIC_ACCOUNT_ID with your New Relic account. For example, if the New Relic account is 1000, enter this: https://workflows.newrelic.com/proxy/1000/incident/jira/

AWS EventBridge notifier

Use workflows to hook New Relic events into AWS services with EventBridge, Amazon's serverless event bus. Then use EventBridge to deliver notification messages to route data to targets like AWS Lambda, SNS queues, CloudWatch logs, etc.

Configure an EventBridge integration:

  1. Complete the AWS account ID, the AWS region, the name of the event bus, and the notification message (JSON).
  2. Send a test notification, which creates a partner event source in your EventBridge service page.
  3. On Partner event sources, select Associate with event bus.
A screenshot showing how to associate the event source with an event bus.
The Associate with event bus button.
  1. Set an event rule to forward the events to whatever AWS service you need:
    • Select the custom or partner event bus option, and then select the name of the event bus.
    • Choose an AWS service to run when triggered by a workflow.

ServiceNow notifier

Using ServiceNow as a notifier would enable you to push valuable violation data, into a new ServiceNow Incidents tickets.

With 2-way integration - you could also make sure that state-updates are mirrored back to New Relic. More on that below.

ServiceNow notifier needs:

  • Notifier name
  • ServiceNow endpoint
  • A user with write permission
  • A workflow notification where every field name has a matching field name in the ServiceNow incident table
Screenshot showing the  fields for an enriched ServiceNow notifier.
An example of a ServiceNow notification message configuration.

Here's what that configuration would look like in ServiceNow:

Screenshot showing a ServiceNow test.
When ServiceNow receives an enriched notification from a workflow, it will populate an incident with that information.

Here are some notes about this example:

  • We used ServiceNow’s default field incident configuration, you or your company may have configured additional fields and values.
  • Default configurations:
    • Severity, Impact, Urgency are numeric values 1/2/3 that correspond with High/Medium/Low.
    • State defines values, such as new, in progress, closed, and others.
    • The Category and Subcategory fields are text fields with a limited value set. See ServiceNow for more detail.
    • The Category and Subcategory fields are text fields with a limited value set. See ServiceNow for more detail.
  • Description, Short-Description, and Work_notes are free text fields and can be enriched as part of the notification message.

ServiceNow two-way integration

You can configure a two-way integration with ServiceNow incidents so that when state updates for the incident (resolved or closed), would trigger an update in the corresponding New Relic Alerts incident state.

To enable auto-sync between the ServiceNow incident state and a workflow incident:

  1. Check Allow two-way integration when you create the Notifier.
Screenshot showing how to enable a two-way integration.
When you enable two-way integration, changes to ServiceNow incidents will update an alert's incident state.
  1. Open and download this XML file (which includes the business rule for Incident Workflows).
  2. In the ServiceNow sidebar menu, go to System Definition > Business Rule.
  3. Click the menu icon in one of the column headers, select Import XML, and upload the XML file you downloaded.

After you enable the auto-sync process, an incident state in ServiceNow changes to Resolved or Closed, and the corresponding workflow incident changes to Closed. For any state other than New, the workflow incident state changes to Acknowledged.

Webhook notifier

Use the webhook notifier to send notification messages to any endpoint. The webhook configuration requires a channel name, the endpoint of the target application, authorization, custom headers (if needed), and the notification message.

Webhook only supports JSON payloads.

Screenshot showing the fields to set up a webhook notifier.
This is an example of a webhook configuration you might use.

Step 5. Optional: Test your workflow

When you test the workflow, it runs the NRQL queries to enrich your notifications and sends test notification messages to your notification platforms.

If your workflow test fails, check the following:

  • Unique name for your workflow and notifiers
  • Valid queries (without error messages in the NRQL query)
  • Notifiers passed their test

Step 6. Activate your workflow

Make sure your workflow has at least one trigger and one notification channel, then save and activate the workflow.

If you need more help, check out these support and learning resources: