Log obfuscation: Hash or mask sensitive data in your logs

You decide you want to apply the following obfuscation actions to all the logs coming from that service:

• HASH the credit card number present in the message and creditCardNumber attributes.
• MASK the birth date present in the message attribute.
• HASH the Social Security number present in the message and ssn attributes.

The first thing you need to do is to define some obfuscation expressions that allow you to capture this sensitive information.

{
  "name": "Credit Card Number",
  "regex": "((?:(?:4\d{3})|(?:5[1-5]\d{2})|6(?:011|5[0-9]{2}))(?:-?|\040?)(?:\d{4}(?:-?|\040?)){3}|(?:3[4,7]\d{2})(?:-?|\040?)\d{6}(?:-?|\040?)\d{5})"
}

{
  "name": "Social Security Number",
  "regex": "(\d{3}-\d{2}-\d{4})"
}

{
  "name": "Born date - loginService specific",
  "regex": "born on (.*)\)"
}

Each obfuscation expression defines how to capture some sensitive information out of a string (using a regex) and associates it with some friendly name so that you can easily identify it later.

Obfuscation expressions can be reusable: they are totally agnostic to how the log attribute containing the sensitive data is named. For instance, the Social Security expression defined above could be applied to a log attribute named ssn, socialSecurityNumber, or socSecNum.

But still, you can create non-reusable obfuscation expressions (like the Born date (loginService specific) one) that are tightly coupled to the log attribute's format. For example, you could use whatever comes after born on and before.

Now that we have defined how to capture our sensitive data, we need to specify which logs need to be obfuscated (the ones of the Login Service) and how (with the obfuscation actions we defined). To achieve this, we define an obfuscation rule.

{
  "name": "Obfuscate Login Service Logs",
  "filter": "serviceName = 'loginService' AND department = 'sales'",
  "actions": [
    {
      "attributes": ["message", "creditCardNumber"],
      "expression": { "name": "Credit Card Number" },
      "method": "HASH_SHA256"
    },
    {
      "attributes": ["message"],
      "expression": { "name": "Born date - loginService specific" },
      "method": "MASK"
    },
    {
      "attributes": ["message", "ssn"],
      "expression": { "name": "Social Security Number" },
      "method": "HASH_SHA256"
    }
  ]
}

This rule contains three main components:

As a final example, imagine we also needed to obfuscate logs coming from another service named "Checkout Service" that have an attribute serviceName = checkoutService as well as a ccn attribute that contains credit card information:

{
  "message": "Order completed",
  "ccn": "4321-5678-9876-2345",
  "department": "sales",
  "serviceName": "checkoutService"
}

To obfuscate the logs from this service, we would only have to define another obfuscation rule targeting these specific logs, and we would simply reuse the previously created Credit card number obfuscation expression:

{
  "name": "Obfuscate Checkout Service Logs",
  "filter": "serviceName = 'checkoutService' AND department = 'sales'",
  "actions": [
    {
      "attributes": ["ccn"],
      "expression": { "name": "Credit Card Number" },
      "method": "HASH_SHA256"
    }
  ]
}

Use either of these options to create an obfuscation expression:

Using the Logs UI:

1. Go to one.newrelic.com > Logs and from the left nav, select Obfuscation.

2. Click Create regex.

3. Enter a name for your new obfuscation rule, as well as a regular expression matching the sensitive data you want to capture. Use RE2 syntax.

   The following example shows a basic obfuscation expression that will match credit card numbers:

   

Using NerdGraph:

Use the logConfigurationsCreateObfuscationExpression mutator under logConfigurations. Refer to the populated example in GraphiQL as well as the related documentation in api.newrelic.com/graphiql.

Use either of these options to query an obfuscation expression:

Using the Logs UI:

1. Go to one.newrelic.com > Logs and from the left nav, select Obfuscation.
2. Select the Expressions tab to view all the available obfuscation expressions and their definitions.

Using NerdGraph:

Use the obfuscationExpressions fetcher under actor.account.logConfigurations. Refer to the populated example in GraphiQL as well as the related documentation in api.newrelic.com/graphiql.

Use either of these options to update an obfuscation expression:

Using the Logs UI:

1. Go to one.newrelic.com > Logs and from the left nav, select Obfuscation.

2. Select the Expressions tab.

3. Click the three dots menu ... at the right side of the obfuscation expression you want to edit, and click Edit.

4. Modify fields as needed, and click Update.

Using NerdGraph:

Use the logConfigurationsUpdateObfuscationExpression mutator under logConfigurations. Refer to the populated example in GraphiQL as well as the related documentation in api.newrelic.com/graphiql.

You do not need to specify all the fields of an obfuscation expression when updating it, only the id (mandatory) and the fields that you wish to modify.

Use either of these options to delete an obfuscation expression:

Using the Logs UI:

1. Go to one.newrelic.com > Logs and from the left nav, select Obfuscation.

2. Select the Expressions tab.

3. Click the three dots menu ... at the right side of the obfuscation expression you want to delete, and click Delete.

4. Confirm you want to delete the expression by clicking Delete.

Using NerdGraph:

Use the logConfigurationsDeleteObfuscationExpression mutator under logConfigurations. Refer to the populated example in GraphiQL as well as the related documentation in api.newrelic.com/graphiql.

The mutation will return a snapshot of the expression before being deleted.

Expression:

(\d{3}-\d{2}-\d{4})

Expression:

((?:(?:4\d{3})|(?:5[1-5]\d{2})|6(?:011|5[0-9]{2}))(?:-?|\040?)(?:\d{4}(?:-?|\040?)){3}|(?:3[4,7]\d{2})(?:-?|\040?)\d{6}(?:-?|\040?)\d{5})

Expression:

((?:\d{2})?\d\d(?:\\)?(?:\/)?\d\d(?:\\)?(?:\/)?\d{2}(?:\d{2})?)

Expression:

([a-zA-Z0-9!#$'*+?^_`{|}~.-]+(?:@|%40)(?:[a-zA-Z0-9-]+\.)+[a-zA-Z0-9-]+)

Use either of these options to create an obfuscation rule:

Using the Logs UI:

1. Go to one.newrelic.com > Logs and from the left nav, select Obfuscation.

2. Click Create obfuscation rule.

3. Enter a name for your new obfuscation rule, as well as a matching criteria (in NRQL format) to capture the target set of logs you want to obfuscate.

4. Add a new actions (the first one is added automatically) to specify the obfuscation expression (regex) to capture each set of attributes, as well as whether to MASK or HASH them.

   • Multiple attributes can be specified comma-separated.
   • MASK will replace all matching characters with Xes. If you use MASK, you will not be able to query for a particular obfuscated value later.
   • HASH will replace sensitive data with the SHA-256 hash value. If you use HASH, you will be able to query them using our hashing tool, provided you know its cleartext value.

   Your rule should look something like this:

   

5. Click Create rule to create and activate your obfuscation rule.

Using NerdGraph:

Use the logConfigurationsCreateObfuscationRule mutator under logConfigurations. Refer to the populated example in GraphiQL as well as the related documentation in api.newrelic.com/graphiql.

Use either of these options to query an obfuscation rule:

Using the Logs UI:

1. Go to one.newrelic.com > Logs and from the left nav, select Obfuscation.

2. Select the Rules tab (default) to view all the available obfuscation rules and their definitions.

Using NerdGraph:

Use the obfuscationRules fetcher under actor.account.logConfigurations. Refer to the populated example in GraphiQL as well as the related documentation in api.newrelic.com/graphiql

Use either of these options to update an obfuscation rule:

Using the Logs UI:

1. Go to one.newrelic.com > Logs and from the left nav, select Obfuscation.

2. Select the Rules tab (default).

3. Click the three dots menu ... at the right side of the obfuscation rule you want to edit, and click Edit.

4. Modify the fields as needed, and click Update rule.

Using NerdGraph:

Use the logConfigurationsUpdateObfuscationRule mutator under logConfigurations. Refer to the populated example in GraphiQL as well as the related documentation in api.newrelic.com/graphiql.

You do not need to specify all the fields of an obfuscation rule when updating it, only the id (mandatory) and the fields that you wish to modify. Here is an example to only update the name.

Use either of these options to delete an obfuscation rule:

Using the Logs UI:

1. Go to one.newrelic.com > Logs and from the left nav, select Obfuscation.

2. Select the Rules tab (default).

3. Click the three dots menu ... at the right side of the obfuscation rule you want to delete, and click Delete.

4. Confirm you want to delete the expression by clicking Delete.

Using NerdGraph:

Use the logConfigurationsDeleteObfuscationRule mutator under logConfigurations. Refer to the populated example in GraphiQL as well as the related documentation in api.newrelic.com/graphiql.

The mutation will return a snapshot of the rule before being deleted.