JMX monitoring integration

Our JMX integration allows users to monitor any application that exposes metrics with JMX. The integration includes a default collection file that automatically collects key metrics from the JVM. You can also customize your metric collection with YAML files to collect any subset of metrics.

Read on to install the integration, and to see what data we collect.

Compatibility and requirements

Our integration is compatible with Java 8 or higher. If you need to use a different Java version than the one configured in PATH, follow New Relic's configuration documentation on GitHub.

Before installing the integration, make sure that you meet the following requirements:

• A New Relic account. Don't have one? Sign up for free! No credit card required.
• If JMX is not running on Kubernetes or Amazon ECS, you must install the infrastructure agent on a host that's running JMX. Otherwise:
  • If running on Kubernetes, see these requirements.
  • If running on ECS, see these requirements.
• This integration does not support the IIOP protocol.

Install and activate

To install the JMX integration, follow the instructions for your environment:

Additional notes:

• Advanced: Integrations are also available in tarball format to allow for install outside of a package manager.
• When the Infrastructure agent executes the nri-jmx binary, it sets the path to PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin. The java binary must be in one of those paths.
• nrjmx and jmx need java binaries to be in the system PATH, which is NOT the PATH for the root user, and isn't affected if you change the PATH environment variable in /root/.bashrc. Read more on how to change the PATH system variable.
• On-host integrations do not automatically update. For best results, regularly update the integration package and the infrastructure agent.

Configure the integration

An integration's YAML-format configuration is where you can place required login credentials and configure how data is collected. Which options you change depend on your setup and preference.

There are several ways to configure the integration, depending on how it was installed:

• If enabled via Kubernetes: see Monitor services running on Kubernetes.
• If enabled via Amazon ECS: see Monitor services running on ECS.
• If installed on-host: edit the config in the integration's YAML config file, jmx-config.yml.

The JMX integration defines and collects integration data using two types of YAML files: the integration configuration options and the metrics collection options.

Configuration options are below. For examples, see Example config.

Integration configuration files

The configuration file has common settings applicable to all integrations, such as interval, timeout, or inventory_source. To read all about these common settings refer to our configuration format document.

Specific settings related to JMX are defined using the env section of the configuration file. These settings control the connection to your JMX instance as well as other security settings and features. The list of valid settings is described in the next section of this document.

Config options are below. For an example configuration, see the example config file.

JMX Instance Settings

The JMX integration collects both metrics (M) and inventory (I) information. Check the Applies To column below to find which settings can be used for each specific collection.

For an example, see the host connection file example.

COLLECTION_CONFIG='{"collect":[{"domain":"java.lang","event_type":"JVMSample","beans":[{"query":"type=GarbageCollector,name=*","attributes":["CollectionCount","CollectionTime"]},{"query":"type=Memory","attributes":["HeapMemoryUsage.Committed","HeapMemoryUsage.Init","HeapMemoryUsage.Max","HeapMemoryUsage.Used","NonHeapMemoryUsage.Committed","NonHeapMemoryUsage.Init","NonHeapMemoryUsage.Max","NonHeapMemoryUsage.Used"]},{"query":"type=Threading","attributes":["ThreadCount","TotalStartedThreadCount"]},{"query":"type=ClassLoading","attributes":["LoadedClassCount"]},{"query":"type=Compilation","attributes":["TotalCompilationTime"]}]}]}'

COLLECTION_CONFIG={"collect":[{"domain":"Catalina","event_type":"TomcatSample","beans":[{"query":"type=UtilityExecutor","attributes":["completedTaskCount"]}]}]}

The values for these settings can be defined in several ways:

• Adding the value directly in the config file. This is the most common way.
• Replacing the values from environment variables using the {{}} notation. This requires infrastructure agent v1.14.0 or higher. Read more here or see the example below.
• Using secrets management. Use this to protect sensible information (such as passwords) from being exposed in plain text on the configuration file. For more information, see secrets management.

Labels and custom attributes

You can further decorate your metrics using labels. Labels allow you to add key/value pair attributes to your metrics, which you can then use to query, filter or group your metrics on.

Our default sample config file includes examples of labels. These aren't mandatory, and you can remove, modify, or add new ones of your choice.

labels:
   env: production
   role: jmx

Metrics collection files

The metrics collection definition files are structured YAML files which tell the integration what metrics to collect. For an example configuration, see the metrics collection file example.

Default JVM metrics collection file: /etc/newrelic-infra/integrations.d/jvm-metrics.yml

Domains

The integration collects and organizes metrics according to domains. All metrics defined per domain will be sent to New Relic and can be found in a corresponding event type. This event type is either auto-generated or can be set by the user. Each file contains a single collect: block which contains an array of domains. For each domain, the following keys are defined:

• domain: The JMX domain; for example, java.lang. You can use wildcards to match multiple domains; for example, java.*. If you use a wildcard, event_type is required, and must be unique. This field is required.
• event_type: The event type name for a collection from this domain. If the domain is wildcarded, this is required, and must be unique. If the domain is not wildcarded and this is undefined by the user, this will be auto generated. For example, the domain java.lang will have event type JavaLangSample. For more information, see Naming tips.
• beans: An array of beans to collect in this domain.

Beans

Each domain contains an array of beans to be collected. For each bean, the following keys are defined:

• query: The bean name to collect; for example,type=GarbageCollector,name=YoungGen. You can use wildcards; for example, type=GarbageCollector,name=*. This field is required.
• exclude_regex: An optional list of regex patterns that match beans to exclude from collection; for example, type=GarbageCollector,name=.*.
• attributes: A list of attributes to collect. If unspecified, collects all attributes.

Attributes

Each bean can contain attributes, an optional list of beans that can be excluded from collection. For each attribute, the following keys are defined:

• attr: An exact match of the attribute name. Composite attributes can be collected by appending the composite member name to the attribute name with a dot; for example, HeapMemoryUsage.Max.

• attr_regex: A regex pattern that matches the attributes to be collected.

• metric_type: The New Relic metric type to collect this attribute as. Options are:

  • gauge: data will be collected as an instantaneous numeric measurement.
  • rate: data will be collected as the change in that metric per second.
  • delta: data will be collected as the change in that metric since the last measurement.
  • attribute: data will be collected as a string literal.

  If left unspecified, the JMX integration will attempt to infer the metric type based on the value returned. For example, if the metric is a number, it will collect it as gauge. If the metric is a string, it will collect it as attribute.

  If metrics are collected with an incorrect metric type, you can manually specify the correct metric type in the collection file.

• metric_name: The name under which the metric will appear in New Relic. If unspecified, it will default to the attribute name.

For more information about JMX queries, see the Oracle ObjectName documentation.

Collection configuration using Kubernetes annotations

You can use Kubernetes annotations to provide collection configuration. To achieve this, you need to deploy a configMap into the Kubernetes cluster that will create the configuration file for the nri-jmx application.

In this configuration file you need to specify the command for container auto-discovery, which will allow you to use placeholders in the integration configuration, including Kubernetes annotations.

Example of a configMap to monitor JVM in a Tomcat application:

nri-bundle:
  newrelic-infrastructure:
    integrations_config:
      - name: jmx-config.yml
        data:
          discovery:
            command:
              # Use the following optional arguments:
              # --namespaces: Comma separated list of namespaces to discover pods on
              # --tls: Use secure (TLS) connection
              # --port: Port used to connect to the kubelet. Default is 10255
              exec: /var/db/newrelic-infra/nri-discovery-kubernetes
              match:
                label.app: java
          integrations:
            - name: nri-jmx
              env:
                # Using the discovered IP as the host address
                JMX_HOST: ${discovery.ip}
                JMX_PORT: ${discovery.port}
                COLLECTION_CONFIG: ${discovery.annotation.newrelic.config}

apiVersion: v1
kind: ConfigMap
metadata:
  name: nri-integration-cfg
data:
  jmx-config.yml: |
    ---
    # Run auto discovery to find pods with label "app=java"
    discovery:
      command:
        # Use the following optional arguments:
        # --namespaces: Comma separated list of namespaces to discover pods on
        # --tls: Use secure (TLS) connection
        # --port: Port used to connect to the kubelet. Default is 10255
        exec: /var/db/newrelic-infra/nri-discovery-kubernetes
        match:
          label.app: java
    integrations:
      - name: nri-jmx
        env:
          # Using the discovered IP as the host address
          JMX_HOST: ${discovery.ip}
          JMX_PORT: ${discovery.port}
          COLLECTION_CONFIG: ${discovery.annotation.newrelic.config}

Next, use annotations to define collection configuration. For example, here's a Tomcat deployment with annotations:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tomcat-deployment
  labels:
    app: java
spec:
  replicas: 1
  selector:
    matchLabels:
      app: java
  template:
    metadata:
      annotations:
        newrelic.config: >-
          {
            "collect": [
              {
                "domain": "java.lang",
                "event_type": "JVMSample",
                "beans": [
                  {
                    "query": "type=GarbageCollector,name=*",
                    "attributes": [
                      "CollectionCount",
                      "CollectionTime"
                    ]
                  },
                  {
                    "query": "type=Memory",
                    "attributes": [
                      "HeapMemoryUsage.Committed",
                      "HeapMemoryUsage.Init",
                      "HeapMemoryUsage.Max",
                      "HeapMemoryUsage.Used",
                      "NonHeapMemoryUsage.Committed",
                      "NonHeapMemoryUsage.Init",
                      "NonHeapMemoryUsage.Max",
                      "NonHeapMemoryUsage.Used"
                    ]
                  },
                  {
                    "query": "type=Threading",
                    "attributes": [
                      "ThreadCount",
                      "TotalStartedThreadCount"
                    ]
                  },
                  {
                    "query": "type=ClassLoading",
                    "attributes": [
                      "LoadedClassCount"
                    ]
                  },
                  {
                    "query": "type=Compilation",
                    "attributes": [
                      "TotalCompilationTime"
                    ]
                  }
                ]
              }
            ]
          }
      labels:
        app: java
    spec:
      containers:
        - name: tomcat
          image: tomcat:10.0.12
          ports:
            - containerPort: 9999
          env:
            - name: CATALINA_OPTS
              value: '-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false'

Optional: Custom connector

JMX allows the use of custom connectors to communicate with the application. In order to use a custom connector, you have to include the custom connectors in the nrjmx classpath.

By default, the sub-folder connectors is in the classpath. If this folder does not exist, create it under the folder where nrjmx is installed.

For example, to add support for JBoss, create a folder named connectors under the default (Linux) library path /usr/lib/nrjmx/ (/usr/lib/nrjmx/connectors/) and copy the custom connector jar ($JBOSS_HOME/bin/client/jboss-cli-client.jar) into it. You can now execute JMX queries against JBoss.

Example configurations

Example file configurations for an on-host install:

integrations:
- name: nri-jmx
  env:
    COLLECTION_FILES: "/etc/newrelic-infra/integrations.d/jvm-metrics.yml,/etc/newrelic-infra/integrations.d/tomcat-metrics.yml"
    JMX_HOST: jmx-host.localnet
    JMX_PASS: admin
    JMX_PORT: 9999
    JMX_USER: admin
  interval: 15s
  labels:
    env: production

collect:
            # The event type for this domain will be JavaLangSample
          - domain: java.lang
            beans:
                # Collect all beans of type Threading
              - query: type=Threading
                # Attributes can be either a string or a map
                attribute:
                      # When unspecified, the metric_type is inferred
                      # and the metric name is just the attribute name
                    - ThreadCount
                      # If using a map attribute, a custom metric name can be set
                    - attr: TotalStartedThreadCount
                      metric_name: ThreadsStarted
                      # Attributes can be collected with regex matches and
                      # the metric type can be overridden if the integration
                      # can not correctly infer the type
                    - attr_regex: "ThreadCpu.*Enabled"
                      metric_type: attribute
              - query: type=Memory
                attributes:
                      # Composite attributes can be collected with this syntax
                    - HeapMemoryUsage.Max
                    - NonHeapMemoryUsage.Max
                # Queries can be wildcarded where
              - query: type=GarbageCollector,name=*
                # If a specific bean is unwanted, it can be excluded
                # with a regex match pattern. Useful if using a wildcard query
                exclude_regex:
                      # This will match any bean where the name is YoungGen
                    - name=YoungGen
                attributes:
                    - attr: LastGcInfo.GcThreadCount
                      metric_type: gauge
                      metric_name: GCThreadCount
            # Domains can be wildcarded
          - domain: java.util.*
            # If the domain is wildcarded, a custom event must be defined
            event_type: JavaUtilSample
            beans:
                  # If no attributes are defined, all are collected by default
                - query: type=Logging

For more about the general structure of on-host integration configuration, see Configuration.

Tips for naming your data

Metrics are sent and stored in the form of samples. This is a list of key-value pairs that include metric data and metadata. Each sample is stored as an event in New Relic’s event database.

You are responsible for creating and naming the JMX data reported to New Relic. For this reason, New Relic strongly recommends following these conventions when naming your event types. To ensure you have a consistent naming scheme:

• Use camel case.
• Use a name that clearly identifies what data it contains.

Example: MyorgApplicationSample

Recommendation: Use the same naming scheme for similar metrics across different applications.

Find and use data

Data from this service is reported to an integration dashboard.

JMX data is attached to the user-defined event type specified in the configuration file. For example, if you are interested in monitoring Tomcat using the JMX integration, define an event_type called TomcatSample, and query that event type.

You can query this data for troubleshooting purposes or to create custom charts and dashboards.

For more on how to find and use your data, see Understand integration data.

Metric data

The metrics generated by the integration include metadata associated with the MBean they are collecting from. You can use this metadata in NRQL queries to filter and facet the data so that the query returns only the data for the desired beans. It can also be used to uniquely identify the metrics, since the metric name is not necessarily unique between all beans.

Each event contains the following metadata:

Example NRQL query

Here's an example NRQL query taking advantage of metadata monitor all the collected JVM garbage collectors:

SELECT latest(CollectionTime)
FROM JVMSample
FACET `key:name`
WHERE `key:type` = 'GarbageCollector'

Metrics data attributes

The JMX integration collects the following metric data attributes:

Inventory data

The JMX integration captures the configuration parameters of the JMX integration. The data is available on the Inventory page, under the config/jmx source. For more about inventory data, see Understand integration data.

Troubleshooting

Troubleshooting tips:

"Domain x has n metrics, the current limit is 200. This domain will not be reported."

Check the source code

This integration is open source software. That means you can browse its source code and send improvements, or create your own fork and build it.