Logs parsing: Built-in rules and custom parsing

New Relic parses log data according to rulesets. Learn how logs parsing works, how to use built-in rulesets, and how to create custom rules to filter your logs.

Why it matters

Parsing is the process of taking an unstructured text log and organizing it into attribute/value pairs. Parsing is valuable because it allows for aggregations and alerts to be applied to a subset of the log data.

A good example is an NGINX access log. By default, that type of log is an unstructured collection of text, useful for full-text searching but not much else. Here's an example of a typical line from an access log:

93.180.71.3 - - [10/May/1997:08:05:32 +0000] "GET /downloads/product_1 HTTP/1.1" 304 0 "-" "Debian APT-HTTP/1.3 (0.8.16~exp12ubuntu10.21)"

After parsing, the log can be organized into attributes, like response code and request URL:

{
   "remote_addr":"93.180.71.3",
   "time":"1586514731",
   "method":"GET",
   "path":"/downloads/product_1",
   "version":"HTTP/1.1",
   "response":"304",
   "bytesSent": 0,
   "user_agent": "Debian APT-HTTP/1.3 (0.8.16~exp12ubuntu10.21)"
}

Parsing makes it possible to create custom queries that facet on those values, letting you understand the distribution of response codes per-request URL and quickly find problematic pages.

How we parse logs

Here's an overview of how New Relic implements parsing of logs:

  • All parsing takes place against the message field; this is the only field that is parsed.
  • Parsing is applied based on an attribute (key-value pair) included in the unparsed log event. It must be available before parsing takes place, to help match the rule to the right logs. Attributes can either be a part of the original log or added when the log is shipped.
  • We highly recommend that you use logtype as the attribute name.
  • Parsing rules are written in Grok, a collection of patterns that abstract away complicated regular expressions. You must know Grok in order to create a parsing rule.
  • If the content of the message field is valid JSON, it's automatically parsed into attribute/value pairs.

How does our log parsing work?

New Relic provides a log ingestion pipeline that can parse data. Parsing is done by matching a log event to a rule that describes how the log should be parsed. There are two ways log events can be parsed: using a built-in rule or by defining a custom rule.

Rules are a combination of matching logic and parsing logic. Matching is done by defining a query match on an attribute of the logs. Rules are not applied retroactively: logs collected prior to the creation of a rule are not parsed by that rule.

The simplest way to organize your logs and how they are parsed is to include the logtype field in your log event, which tells New Relic what built-in ruleset must be applied to the logs.

Once a parsing rule is active, data parsed by the rule is permanently changed. This cannot be reverted.

Built-in parsing rulesets

Common log formats have well-established parsing rules already created for them. To get the benefit of built-in parsing rules, add the logtype attribute when shipping the logs. Set the value to one of those listed in the table below, and the rules for that type of log will be applied automatically.

The following logtype attribute values map to a standard parsing rulesets. See Built-in log parsing rulesets to learn what fields are parsed for each ruleset.

logtype Example matching query Description
alb logtype:alb AWS Application Load Balancer
apache logtype:apache Apache Access
cloudfront-web logtype:cloudfront-web Cloudfront Web
elb logtype:elb Amazon Elastic Load Balancer
iis_w3c logtype:iis_w3c IIS server logs - W3C format
monit logtype:monit Monit logs
mysql-error logtype:mysql-error MySQL Error
nginx logtype:nginx NGINX access logs
nginx-error logtype:nginx-error NGINX error logs
route-53 logtype:route-53 Amazon Route 53 logs

How to add the logtype attribute

When aggregating logs, it's important to provide metadata that makes it easy to organize, search, and parse those logs. One simple way of doing this is to add the attribute logtype to the log messages when they are shipped. Built-in parsing rules are applied by default to certain logtype values.

Below are some examples of how to add logtype to logs sent by some of our supported shipping methods.

Add logtype as an attribute. You must set the logtype for each named source.

logs:
  - name: file-simple
    path: /path/to/file
    attributes:
      logtype: fileRaw  
  - name: nginx-example
    path: /var/log/nginx.log
    attributes:
      logtype: nginx

Add a filter block to the .conf file, which uses a record_transformer to add a new field. In this example we use a logtype of nginx to trigger the build-in NGINX parsing rule. Check out other Fluentd examples here.

<filter containers>
  @type record_transformer
  enable_ruby true
  <record>
    #Add logtype to trigger a built-in parsing rule for nginx access logs
    logtype nginx
    #Set timestamp from the value contained in the field "time"
    timestamp record["time"]
    #Add hostname and tag fields to all records
    hostname "#{Socket.gethostname}"
    tag ${tag}
  </record>
</filter>

Add a filter block to the .conf file that uses a record_modifier to add a new field. In this example we use a logtype of nginx to trigger the build-in NGINX parsing rule. Check out other Fluent Bit examples here.

[FILTER]
    Name record_modifier
    Match *
    Record logtype nginx
    Record hostname ${HOSTNAME}
    Record service_name Sample-App-Name

Add a filter block to the Logstash configuration which uses an add_field mutate filter to add a new field. In this example we use a logtype of nginx to trigger the build-in NGINX parsing rule. Check out other Logstash examples here.

filter {
  mutate {
    add_field => {
      "logtype" => "nginx"
      "service_name" => "myservicename"
      "hostname" => "%{host}"
    }
  }
}

You can add attributes to the JSON request sent to New Relic. In this example we add a logtype attribute of value nginx to trigger the build-in NGINX parsing rule. ​Learn more about using the Logs API here.

POST /log/v1 HTTP/1.1
Host: log-api.newrelic.com
Content-Type: application/json
X-License-Key: YOUR_LICENSE_KEY
Accept: */*
Content-Length: 133
{
  "timestamp": TIMESTAMP_IN_UNIX_EPOCH,
  "message": "User 'xyz' logged in",
  "logtype": "accesslogs",
  "service": "login-service",
  "hostname": "login.example.com"
}

Create custom parsing rules

Many logs are formatted or structured in a unique way. In order to parse them, custom logic must be built and applied. New Relic's log management lets you create and manage custom parsing rules.

  1. Choose Manage parsing from the action menu dropdown.
    manage-parsing.png
  2. Click Create parsing rule.
  3. Choose an attribute and value to match on.
  4. Write your Grok pattern and test the rule.
  5. Enable and save the custom parsing rule.

To learn about Grok and custom parsing rules, read our blog post, How to parse logs with Grok patterns.

For more help

If you need more help, check out these support and learning resources: