• English日本語한국어
  • Log inStart now

Log obfuscation: Hash or mask sensitive data in your logs

With log obfuscation rules, you can prevent certain types of information from being saved in New Relic.

Requirements

Our log obfuscation feature is available as part of our Data Plus option.

What is log obfuscation?

Our service automatically masks number patterns that we identify as likely being sensitive items, such as credit card or Social Security numbers.

If you need additional obfuscation, one option is to adjust the configuration of the log forwarder you use (for example, our infrastructure agent). But an easier option is to use our log obfuscation feature, available with Data Plus. This feature lets you set up log obfuscation rules directly from the log management UI, or via our NerdGraph API, without lengthy manual configuration. You'll define regular expressions matching your sensitive information, and then create rules to obfuscate that data. You can choose either to have sensitive information masked or hashed.

Definitions

Here are some important terms:

  • Obfuscation rules define what logs to apply obfuscation actions to.
  • Obfuscation rule actions define what attributes to look at, what text to obfuscate, and how to obfuscate (either by masking or hashing).
  • Obfuscation expressions are named regular expressions identifying what text to obfuscate.
  • Masking completely removes information, replacing it with X characters. You cannot search for specific values once this is done.
  • Hashing hides information. You can use the hashing tool to get the hash of a sensitive value, and then search for logs containing that hash.

How obfuscation works

The JSON objects displayed in the following example are simplifications of the payloads used by our NerdGraph API. This will help you better correlate the different API operations with their UI equivalent counterparts.

Example: Log record before obfuscation

Imagine you have the following log record:

{
"message": "The credit card number 4321-5678-9876-2345 belongs to user user@email.com (born on 01/02/2003) with SSN 123-12-1234",
"creditCardNumber": "4321-5678-9876-2345",
"ssn": "123-12-1234",
"department": "sales",
"serviceName": "loginService"
}

This log record contains several sensitive data. Ideally, you would like your log to end up looking like this:

{
"message": "The credit card number 9aa9bc1528859aee1b1df75795f1ebd54beb2f0d26c8a1d4580a71a07189cdd5 belongs to user user@email.com (born on XXXXXXXXXX) with SSN 30e6897f76dc102e32ee1d781c43417d259e586eac15c963d75ab8b5187769da",
"creditCardNumber": "9aa9bc1528859aee1b1df75795f1ebd54beb2f0d26c8a1d4580a71a07189cdd5",
"ssn": "30e6897f76dc102e32ee1d781c43417d259e586eac15c963d75ab8b5187769da",
"department": "sales",
"serviceName": "loginService"
}

Example: Basic process

Here is the basic process you would use to obfuscate the sensitive data in this example.

Checklist: Steps to obfuscate logs

To obfuscate your logs:

  1. Study the shape of your logs by identifying patterns of sensitive data that appear in them. For example:

    • Do all your logs contain sensitive information? Or can you be more specific (only the logs from service A or region B)?
    • What sensitive information do they contain: credit card numbers, driver's license numbers, national IDs, biometrics, other values?
  2. Create obfuscation expressions to identify how to extract sensitive data.

  3. Define obfuscation rules for each set of logs:

    • Define how you will capture them using NRQL.
    • Define which obfuscation actions need to be applied to each of them. Ask yourself: Will I need to query my logs using this sensitive information later (consider using HASH), or do I need to remove this information entirely from my logs (consider using MASK)?

Tip

The Logs obfuscation UI includes a Hashing tool so that you can find a hash from a known value and copy it for use with other expressions and rules.

CPU limits

Obfuscation has per-minute CPU limits. If an account hits these resource limits, logs won't be obfuscated as expected. To check your CPU limits, go to your system Limits page in the Data management UI.

If you exceed the obfuscation per-minute CPU limits and logs cannot be obfuscated or hashed, the attribute the obfuscation rule was applied to will be dropped and replaced with text indicating why the attribute was dropped. For example, if the obfuscation rule is applied to the message field, and the CPU per-minute limit is reached, the resulting log will look like this:

{
...
"message": "<OBFUSCATION> The account is over its obfuscation per-minute limit, attribute dropped",
...
}

This is to prevent PII or other sensitive data from being ingested inadvertently.

In addition, the following `NrIntegrationError' will be logged to the account:

{
"category": "RateLimit",
"level": "error",
"limitName": "Log API obfuscation per account per minute",
"message": "You’ve exceeded our limit of per-account obfuscation time per-minute for the Log ingestion pipeline. Please reduce your usage or contact New Relic support.",
"name": "ObfuscationTimeLimitReached",
"newRelicFeature": "Logs",
"rateLimitType": "ObfuscationTimePerMinute",
"timestamp": 1678819264283
}

To evaluate how well your obfuscation rules are working and see which ones are being skipped, go to Logs Obfuscation > Health. Obfuscation rules are CPU intensive, so these charts can help you decide which rules are most impacted by any resource limitations.

Obfuscation expressions

You can create, read, update, or delete obfuscation expressions by using the New Relic UI or by using NerdGraph, our GraphQL Explorer.

one.newrelic.com > All capabilities > Logs > Obfuscation: First create one or more obfuscation expressions, then create your obfuscation rules.

Sample expressions

Below, we've provided some sample regex expressions to obfuscate some of the most common sensitive data types. Obfuscation expressions must be created for each New Relic account where those expressions will be in use.

Tip

The following examples are regular expressions you could use in the UI. To use these in GraphQL, you'd need to escape them as shown in this example.

Obfuscation rules

You can create, read, update, or delete obfuscation rules by using the New Relic UI or by using NerdGraph, our GraphQL Explorer.

Copyright © 2024 New Relic Inc.

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.