Python: Configure logs in context

This is the simplest approach, and it's a great choice for developers who may not have the access or interest in setting up a log forwarder, or for accounts that want to see the power of logs and other linking metadata in context of their apps, without a lot of overhead.

Using this option, your logs will include span.id, trace.id, hostname, entity.guid, and entity.name. These attributes automatically link your logs to spans, traces, and other telemetry, making it easier to troubleshoot.

All you need to do is install an agent version with log forwarding capabilities (Python agent 7.12.0.176 or higher), and update your configuration to enable forwarding.

newrelic.ini:

application_logging.enabled=true
application_logging.forwarding.enabled=true

Environment variable:

NEW_RELIC_APPLICATION_LOGGING_ENABLED=true
NEW_RELIC_APPLICATION_LOGGING_FORWARDING_ENABLED=true

This option may be on by default in a future agent version.

Optional adjustments:

Once this is enabled, you also have control over the maximum number of logs sent to New Relic every minute. The default value is 10,000. If more than 10,000 logs are received in a 60-second window, your logs will begin to be sampled.

Set it to a higher number to receive more logs. Set it to a lower number to receive fewer logs. Any integer up to 100,000 is valid.

newrelic.ini:

event_harvest_config.harvest_limits.log_event_data=10000

Environment variable:

NEW_RELIC_APPLICATION_LOGGING_FORWARDING_MAX_SAMPLES_STORED=10000

If you have an existing log forwarding solution and are updating your agent to use automatic logs in context, be sure to disable your manual log forwarder. Otherwise, your app will be sending double log lines. Depending on your account, this could result in double billing. For more information, follow the procedures to disable your specific log forwarder.

Already have a log forwarder you like? We've got you covered! Language agents can decorate your logs with the linking metadata needed to provide access to automatic logs-in-context features.

This option should not be used with in-agent forwarding. Using an external log forwarder to send logs to New Relic while in-agent forwarding is enabled will cause your logs to be sent up twice to New Relic. Depending on your account, this may result in double billing.

This option should also not be used with the manual log decorating formatter. If you have references to the manual formatter in your codebase, please remove them before enabling this option.

1. If you want to use this option, make sure you have the in-agent forwarding configuration option disabled.

   newrelic.ini:

   application_logging.forwarding.enabled=true

   Copy

2. Enable log decorating in your configuration, then relaunch the agent to start decorating your logs.

   newrelic.ini:

   application_logging.enabled=true
   application_logging.local_decorating.enabled=true

   Copy

   Environment variable:

   NEW_RELIC_APPLICATION_LOGGING_ENABLED=true
   NEW_RELIC_APPLICATION_LOGGING_LOCAL_DECORATING_ENABLED=true

   Copy

   Our decorator adds five attributes to every log message: entity.guid, entity.name, hostname, trace.id, and span.id. Example:

   This is my log message. NR-LINKING|{entity.guid}|{hostname}|{trace.id}|{span.id}|{entity.name}|

   Copy

   Some attributes may be empty if the log occurred outside a transaction or if they are not applicable to your application's context.

   We recommend this option over the manual decorating formatter, NewRelicContextFormatter.

You can also use our manual installation option to enable logs in context for APM apps monitored by Python.

1. Make sure you have already set up logging in New Relic. This includes configuring a supported log forwarder that collects your application logs and extends the metadata that is forwarded to New Relic.

2. Install or update to the latest Python agent version, and enable distributed tracing. Use Python agent version 5.4.0 or higher for logs in context.

3. Configure logs in context for your log handler. Before language agents had the ability to forward and decorate logs, you could enable logs in context by instantiating a log formatter and adding it to your log handler. This example uses a StreamHandler which by default writes logs to sys.stderr, but any handler can be used. For more information about configuring log handlers, see the Python.org documentation.

   # Import the logging module and the New Relic log formatter
   import logging
   from newrelic.agent import NewRelicContextFormatter
   
   # Instantiate a new log handler
   handler = logging.StreamHandler()
   
   # Instantiate the log formatter and add it to the log handler
   formatter = NewRelicContextFormatter()
   handler.setFormatter(formatter)
   
   # Get the root logger and add the handler to it
   root_logger = logging.getLogger()
   root_logger.addHandler(handler)

   Copy

4. To verify that you have configured the log appender correctly, run your application, then check your logs data in New Relic using the query operator has:span.id has:trace.id.

If everything is configured correctly and your data is being forwarded to New Relic with the enriched metadata, your logs should now be emitted as JSON and contain trace.id and span.id fields. If you don't see log data in the UI, follow the troubleshooting procedures.