• Log in

Amazon SageMaker MLOps integration

By integrating Amazon SageMaker's integration with New Relic, you'll be able to instrument, analyze, troubleshoot, and optimize your machine-learning performance across your entire system. Rigorously observe your capabilities to react quickly to changes in the model's input or output and the relationship between the two.

Take the next steps to monitor your Amazon SageMaker metrics and objects (that are sent to AWS CloudWatch) and view them as entities and dashboards in New Relic.

For a look at our SageMaker integration in action, watch this short YouTube video (2:57 minutes).

Stream AWS CloudWatch Metrics to New Relic

Start benefiting from New Relic MLOps entities in a single simple step (and just a few minutes!):

Important

Each metric sent to CloudWatch is automatically sent to New Relic's metric table in NRDB, according to the namespace filter. You can always query them using NRQL:

FROM Metric SELECT * WHERE aws.Namespace='/aws/sagemaker/Endpoints' LIMIT MAX SINCE 1 WEEK AGO

Manual option

Follow our docs to set up CloudWatch Metric Streams.

Automated option

You may automate the setup with the Terraform code:

module "example_usage" {
  source = "modules/nr-cloudwatch-metric-stream"

  name_suffix = "suffix" # optional
  aws_account_id = "your-aws-account-id"

  newrelic_collector_endpoint = "newrelic-endpoint-url"
  newrelic_trusted_account_id = "12345678"
  newrelic_license_key        = "[your-newrelic-license-key](/docs/apis/intro-apis/new-relic-api-keys/#ingest-license-key)"
}

When calling the module, please write the correct newrelic_collector_endpoint:

  • HTTP endpoint URL - US datacenter: https://aws-api.newrelic.com/cloudwatch-metrics/v1
  • HTTP endpoint URL - EU datacenter: https://aws-api.eu01.nr-data.net/cloudwatch-metrics/v1

When you set the metric stream you can choose to stream the metric from all the namespaces, or you can specify namespaces.

Important

You can view each entity's metrics in a dashboard that's created automatically when the metrics arrive at the New Relic.

Monitor your data and model in Amazon SageMaker, and send the metrics to CloudWatch

SageMaker automatically monitors your endpoints’ performance, and sends statistic metrics to CloudWatch. For more information, see Endpoint CloudWatch Metrics.

To obtain more benefits from the Amazon SageMaker MLOps integration, use the Amazon SageMaker Model Monitor tools. You'll have to define scheduled monitoring jobs to monitor the quality of your machine learning models in production and send metrics to CloudWatch.

The Amazon SageMaker Model Monitor provides the following types of monitoring:

Advanced options

You can aso publish metric data points to Amazon CloudWatch and define the namespaces and one of the above using the put_metric_data function.

If you use your own algorithm for hyperparameter tuning, make sure that it sends at least one metric by writing evaluation data to stderr or stdout. Read more on how to define metrics in automatic model tuning. See also the example notebook Develop, Train, Optimize and Deploy Scikit-Learn Random Forest.

Explore your MLOps entities and dashboards

We generate aws-entities (under the MLOps entity domain) for the detailed namespaces. For these entities, you can get out-of-the-box dashboards and views. You can also create your own dashboard to view metrics that are not being displayed as part of the entities' views.

New Relic entity

Namespace

Machine learning endpoint

/aws/sagemaker/Endpoints, AWS/SageMaker

Machine learning model data

aws/sagemaker/Endpoints/data-metrics

Machine learning model

aws/sagemaker/Endpoints/model-metrics, aws/sagemaker/Endpoints/explainability-metrics

Go to one.newrelic.com and select the Explorer to view:

  • Your machine-learning entities
  • The dashboard for the metrics of the endpoint from one of the Amazon SageMaker entities
  • The dashboard for the model data entity
Copyright © 2022 New Relic Inc.

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.