• Log in

Find and use your Kubernetes data

You can build your own charts and query all your Kubernetes integration data using the query builder and the NerdGraph API. Our integration collects Kubernetes data by instrumenting the container orchestration layer. For a simpler and more visual experience, use the cluster explorer.

one.newrelic.com > Dashboards: Using the query builder you can query your Kubernetes data and create clear visualizations.

Query Kubernetes data

The simplest way to query your Kubernetes data is using the query builder, which accepts NRQL queries. Alternatively, you can use the NerdGraph API to retrieve Kubernetes data.

Event types

Kubernetes data is attached to the following event types:

Event name

Type of Kubernetes data

Available since

K8sNodeSample

Node data

v1.0.0

K8sNamespaceSample

Namespace data

v1.0.0

K8sDeploymentSample

Deployment data

v1.0.0

K8sReplicasetSample

ReplicaSet data

v1.0.0

K8sDaemonsetSample

DaemonSet data

v1.13.0

K8sStatefulsetSample

StatefulSet data

v1.13.0

K8sPodSample

Pod data

v1.0.0

K8sClusterSample

Cluster data

v1.0.0

K8sContainerSample

Container data

v1.0.0

K8sVolumeSample

Volume data

v1.0.0

K8sApiServerSample

API server data

v1.11.0

K8sControllerManagerSample

Controller manager data

v1.11.0

K8sSchedulerSample

Scheduler data

v1.11.0

K8sEtcdSample

etcd data

v1.11.0

K8sEndpointSample

Endpoint data

v1.13.0

K8sServiceSample

Service data

v1.13.0

K8sHpaSample

Horizontal Pod Autoscaler data

v2.3.0

Manage alerts

You can be notified about alert violations for your Kubernetes data:

Kubernetes attributes and metrics

The Kubernetes integration collects the following metrics and other attributes.

Node data

Query the K8sNodeSample event for node data:

Node attribute

Description

allocatableCpuCores

Node allocatable CPU cores.

allocatableMemoryBytes

Node allocatable memory bytes.

allocatablePods

Node allocatable pods.

allocatableEphemeralStorageBytes

Node allocatable ephemeral-storage bytes.

capacityCpuCores

Node CPU capacity.

capacityMemoryBytes

Node memory capacity (in bytes).

capacityPods

Pod capacity of the node.

capacityEphemeralStorageBytes

Node ephemeral-storage capacity.

clusterName

Name that you assigned to the cluster when you installed the Kubernetes integration.

condition.{conditionName}={conditionValue}

Status of the current observed node condition. The reported conditions can vary depending on your Kubernetes flavor and installed operators.
Examples of common conditions are: Ready, DiskPressure, MemoryPressure, PIDPressure and NetworkUnavailable.
Condition values can be 1 (true), 0 (false), or -1 (unknown).

cpuUsedCoreMilliseconds

Node CPU usage measured in core milliseconds.

cpuUsedCores

Node CPU usage measured in cores.

cpuRequestedCores

Total amount of CPU cores requested.

allocatableCpuCoresUtilization

Percentage of CPU cores actually used with respect to the CPU cores allocatable.

fsAvailableBytes

Bytes available in the node filesystem.

fsCapacityBytes

Total capacity of the node filesystem in bytes.

fsInodes

Total number of inodes in the node filesystem.

fsInodesFree

Free inodes in the node filesystem.

fsInodesUsed

Used inodes in the node filesystem.

fsUsedBytes

Used bytes in the node filesystem.

fsCapacityUtilization

Percentage of used bytes in the node filesystem with respect to the capacity.

memoryAvailableBytes

Bytes of memory available in the node.

memoryMajorPageFaultsPerSecond

Number of major page faults per second in the node.

memoryPageFaults

Number of page faults in the node.

memoryRssBytes

Bytes of rss memory.

memoryUsedBytes

Bytes of memory used.

memoryWorkingSetBytes

Bytes of memory in the working set.

memoryRequestedBytes

Total amount of requested memory.

allocatableMemoryUtilization

Percentage of bytes of memory in the working set with respect to the node allocatable memory.

net.errorCountPerSecond

Number of errors per second while receiving/transmitting over the network.

nodeName

Host name that the pod is running on.

net.rxBytesPerSecond

Number of bytes per second received over the network.

net.txBytesPerSecond

Number of bytes per second transmitted over the network.

runtimeAvailableBytes

Bytes available to the container runtime filesystem.

runtimeCapacityBytes

Total capacity assigned to the container runtime filesystem in bytes.

runtimeInodes

Total number of inodes in the container runtime filesystem.

runtimeInodesFree

Free inodes in the container runtime filesystem.

runtimeInodesUsed

Used inodes in the container runtime filesystem.

runtimeUsedBytes

Used bytes in the container runtime filesystem.

unschedulable

Status of node schedulability of new pods. Its value can be 0 (false) or 1 (true).

label.LABEL_NAME

Labels associated with your node, so you can filter and query for specific nodes.

Namespace data

Query the K8sNamespaceSample event for namespace data:

Namespace attribute

Description

clusterName

Name that you assigned to the cluster when you installed the Kubernetes integration.

createdAt

Timestamp of the namespace when it was created.

namespaceName

Name of the namespace to be used as an identifier.

label.LABEL_NAME

Labels associated with your namespace, so you can filter and query for specific namespaces.

status

Current status of the namespace.

The value can be Active or Terminated

nrFiltered

Indicates whether the namespace is being filtered.

The value can be true or false.

Deployment data

Query the K8sDeploymentSample event for deployment data:

Deployment attribute

Description

clusterName

Name that you assigned to the cluster when you installed the Kubernetes integration.

createdAt

Timestamp of when the deployment was created.

deploymentName

Name of the deployment to be used as an identifier.

namespaceName

Name of the namespace that the deployment belongs to.

label.LABEL_NAME

Labels associated with your deployment, so you can filter and query for specific deployments.

podsAvailable

Number of replicas that are currently available.

podsDesired

Number of replicas that you defined in the deployment.

podsTotal

Total number of replicas that are currently running.

podsUnavailable

Number of replicas that are currently unavailable.

podsUpdated

Number of replicas that have been updated to achieve the desired state of the deployment.

podsMissing

Total number of replicas that are missing (number of desired replicas, podsDesired, minus the total number of replicas, podsTotal).

ReplicaSet data

Query the K8sReplicasetSample event for ReplicaSet data:

Replica attribute

Description

clusterName

Name that you assigned to the cluster when you installed the Kubernetes integration.

createdAt

Timestamp of when the ReplicaSet was created.

deploymentName

Name of the deployment to be used as an identifier.

namespaceName

Name of the namespace that the ReplicaSet belongs to.

observedGeneration

Integer representing generation observed by the ReplicaSet.

podsDesired

Number of replicas that you defined in the deployment.

podsFullyLabeled

Number of pods that have labels that match the ReplicaSet pod template labels.

podsReady

Number of replicas that are ready for this ReplicaSet.

podsTotal

Total number of replicas that are currently running.

podsMissing

Total number of replicas that are currently missing (number of desired replicas, podsDesired, minus the number of ready replicas, podsReady).

replicasetName

Name of the ReplicaSet to be used as an identifier.

DaemonSet data

Query the K8sDaemonsetSample event for DaemonSet data:

DaemonSet attribute

Description

clusterName

Name that you assigned to the cluster when you installed the Kubernetes integration.

createdAt

Timestamp of when the DaemonSet was created.

namespaceName

Name of the namespace that the DaemonSet belongs to.

label.LABEL_NAME

Labels associated with your DaemonSet, so you can filter and query for specific DaemonSet.

daemonsetName

Name associated with the DaemonSet.

podsDesired

The number of nodes that should be running the daemon pod.

podsScheduled

The number of nodes running at least one daemon pod and are supposed to.

podsAvailable

The number of nodes that should be running the daemon pod and have one or more of the daemon pod running and available.

podsReady

The number of nodes that should be running the daemon pod and have one or more of the daemon pod running and ready.

podsUnavailable

The number of nodes that should be running the daemon pod and have none of the daemon pod running and available.

podsMisscheduled

The number of nodes running a daemon pod but are not supposed to.

podsUpdatedScheduled

The total number of nodes that are running updated daemon pod.

podsMissing

Total number of replicas that are currently missing (number of desired replicas, podsDesired, minus the number of ready replicas, podsReady).

metadataGeneration

Sequence number representing a specific generation of the desired state.

StatefulSet data

Query the K8sStatefulsetSample event for StatefulSet data:

StatefulSet attribute

Description

clusterName

Name that you assigned to the cluster when you installed the Kubernetes integration.

createdAt

Timestamp of when the StatefulSet was created.

namespaceName

Name of the namespace that the StatefulSet belongs to.

label.LABEL_NAME

Labels associated with your StatefulSet, so you can filter and query for specific StatefulSet.

statefulsetName

Name associated with the StatefulSet.

podsDesired

Number of desired pods for a StatefulSet.

podsReady

The number of ready replicas per StatefulSet.

podsCurrent

The number of current replicas per StatefulSet.

podsTotal

The number of replicas per StatefulSet.

podsUpdated

The number of updated replicas per StatefulSet.

podsMissing

Total number of replicas that are currently missing (number of desired replicas, podsDesired, minus the number of ready replicas, podsReady).

observedGeneration

The generation observed by the StatefulSet controller.

metadataGeneration

Sequence number representing a specific generation of the desired state for the StatefulSet.

currentRevision

Indicates the version of the StatefulSet used to generate pods in the sequence. Value range: between 0 and podsCurrent.

updateRevision

Indicates the version of the StatefulSet used to generate pods in the sequence. Value range: between podsDesired-podsUpdated and podsDesired.

Pod data

Query the K8sPodSample event for pod data:

Pod attribute

Description

clusterName

Name that you assigned to the cluster when you installed the Kubernetes integration.

createdAt

Timestamp of when the pod was created in epoch seconds.

createdBy

Name of the Kubernetes object that created the pod.

For example, newrelic-infra

createdKind

Kind of Kubernetes object that created the pod.

For example, DaemonSet.

deploymentName

Name of the deployment to be used as an identifier.

isReady

Boolean representing whether or not the pod is ready to serve requests.

isScheduled

Boolean representing whether or not the pod has been scheduled to run on a node.

label.LABEL_NAME

Labels associated with your pod, so you can filter and query for specific pods.

message

Details related to the last pod status change.

namespaceName

Name of the namespace that the pod belongs to.

net.errorCountPerSecond

Number of errors per second while receiving/transmitting over the network.

net.errorsPerSecond

Number of errors per second.

net.rxBytesPerSecond

Number of bytes per second received over the network.

net.txBytesPerSecond

Number of bytes per second transmitted over the network.

nodeIP

Host IP address that the pod is running on.

nodeName

Host name that the pod is running on.

podIP

IP address of the pod. If it doesn't have an IP, it'll be empty.

podName

Name of the pod to be used as an identifier.

reason

Reason why the pod is in the current status.

startTime

Timestamp of when the pod started running in epoch seconds.

status

Current status of the pod.

Value can be Pending, Running, Succeeded, Failed, Unknown.

Cluster data

Query the K8sClusterSample event to see cluster data:

Cluster attribute

Description

clusterName

Name that you assigned to the cluster when you installed the Kubernetes integration.

clusterK8sVersion

Kubernetes version that the cluster is running.

Container data

Query the K8sContainerSample event for container data:

Container attribute

Description

clusterName

Name that you assigned to the cluster when you installed the Kubernetes integration.

containerID

Unique ID associated with the container. If you are running Docker, this is the Docker container id.

containerImage

Name of the image that the container is running.

containerImageID

Unique ID associated with the image that the container is running.

containerName

Name associated with the container.

cpuLimitCores

Integer representing limit CPU cores defined for the container in the pod specification.

cpuRequestedCores

Requested CPU cores defined for the container in the pod specification.

cpuUsedCores

CPU cores actually used by the container.

cpuCoresUtilization

Percentage of CPU cores actually used by the container with respect to the CPU limit specified. This percentage is based on this calculation: (cpuUsedCores / cpuLimitCores) * 100.

requestedCpuCoresUtilization

Percentage of CPU cores actually used by the container with respect to the CPU request specified.

deploymentName

Name of the deployment to be used as an identifier.

isReady

Boolean. Whether or not the container's readiness check succeeded.

label.LABEL_NAME

Labels associated with your container, so you can filter and query for specific containers.

memoryLimitBytes

Integer representing limit bytes of memory defined for the container in the pod specification.

memoryRequestedBytes

Integer. Requested bytes of memory defined for the container in the pod specification.

memoryUsedBytes

Integer. Bytes of memory actually used by the container.

memoryUtilization

Percentage of memory actually used by the container with respect to the memory limit specified.

requestedMemoryUtilization

Percentage of memory actually used by the container with respect to the memory request specified.

memoryWorkingSetBytes

Integer. Bytes of memory in the working set.

memoryWorkingSetUtilization

Percentage of working set memory actually used by the container with respect to the memory limit specified.

requestedMemoryWorkingSetUtilization

Percentage of working set memory actually used by the container with respect to the memory request specified.

namespaceName

Name of the namespace that the container belongs to.

nodeIP

Host IP address the container is running on.

nodeName

Host name that the container is running on.

podName

Name of the pod that the container is in, to be used as an identifier.

reason

Provides a reason why the container is in the current status.

restartCount

Number of times the container has been restarted.

restartCountDelta

Delta change of number of times the container has been restarted.

status

Current status of the container.

Value can be Running, Terminated, or Unknown

containerCpuCfsPeriodsDelta

Delta change of elapsed enforcement period intervals.

containerCpuCfsThrottledPeriodsDelta

Delta change of throttled period intervals.

containerCpuCfsThrottledSecondsDelta

Delta change of duration the container has been throttled, in seconds.

containerCpuCfsPeriodsTotal

Total number of elapsed enforcement period intervals.

containerCpuCfsThrottledPeriodsTotal

Total number of throttled period intervals.

containerCpuCfsThrottledSecondsTotal

Total time duration the container has been throttled, in seconds.

containerMemoryMappedFileBytes

Total size of memory mapped files used by this container, in bytes.

containerOOMEventsDelta

Number of "out of memory" events observed for the container (requires k8s v1.24+).

Volume data

Query the K8sVolumeSample event for volume data:

Volume attribute

Description

volumeName

Name that you assigned to the volume at creation.

clusterName

Cluster where the volume is configured.

namespaceName

Namespace where the volume is configured.

podName

The pod that the volume is attached to. The Kubernetes monitoring integration lists Volumes that are attached to a pod.

persistent

If this is a persistent volume, this value is set to true.

pvcNamespace

Namespace where the Persistent Volume Claim is configured.

pvcName

Name that you assigned to the Persistent Volume Claim at creation.

fsCapacityBytes

Capacity of the volume, in bytes.

fsUsedBytes

Usage of the volume, in bytes.

fsAvailableBytes

Capacity available of the volume, in bytes.

fsUsedPercent

Usage of the volume in percentage.

fsInodes

Total inodes of the volume.

fsInodesUsed

inodes used in the volume.

fsInodesFree

inodes available in the volume.

Volume data is available for volume plugins that implement the MetricsProvider interface:

  • AWSElasticBlockStore
  • AzureDisk
  • AzureFile
  • Cinder
  • Flexvolume
  • Flocker
  • GCEPersistentDisk
  • GlusterFS
  • iSCSI
  • NFS
  • StorageOS
  • VsphereVolume

API server data

Query the K8sApiServerSample event to see API Server data. For more information, see Configure control plane monitoring:

API server attribute

Description

processResidentMemoryBytes

Resident memory size, in bytes.

processCpuSecondsDelta

Difference of the user and system CPU time spent, in seconds.

goThreads

Number of OS threads created.

goGoroutines

Number of goroutines that currently exist.

apiserverRequestDelta_verb_VERB_code_CODE

This is a delta counter for each request verb (e.g. GET, WATCH, LIST, CREATE) and their response codes. The delta is between the current and the last polled value.

apiserverRequestRate_verb_VERB_code_CODE

Rate of apiserver requests per minute, broken out for each verb and HTTP response code.

restClientRequestsDelta_code_CODE_method_METHOD

Difference of the number of HTTP requests, partitioned by method and code.

restClientRequestsRate_code_CODE_method_METHOD

Rate of the number of HTTP requests, partitioned by method and code.

etcdObjectCounts_resource_RESOURCE-KIND

Number of stored objects at the time of last check, split by kind (alpha; deprecated in Kubernetes 1.22; replaced by apiserverStorageObjects).

apiserverStorageObjects_resource_RESOURCE-KIND

Number of stored objects at the time of last check, split by kind (requires k8s 1.23+; replaces etcdObjectCounts).

apiserverCurrentInflightRequestsMutating

Maximal number of currently used inflight request limit of the apiserver per request kind Mutating in the last second (requires k8s v1.23+).

apiserverCurrentInflightRequestsReadOnly

Maximal number of currently used inflight request limit of the apiserver per request ReadOnly kind in the last second (requires k8s v1.23+).

Controller manager data

Query the K8sControllerManagerSample event to see Controller manager data. For more information, see Configure control plane monitoring:

Controller manager attribute

Description

processResidentMemoryBytes

Resident memory size, in bytes.

processCpuSecondsDelta

Difference of the user and system CPU time spent in seconds.

goThreads

Number of OS threads created.

goGoroutines

Number of goroutines that currently exist.

workqueueAddsDelta_name_WORK-QUEUE-NAME

Difference of the total number of adds handled by workqueue.

workqueueDepth_name_WORK-QUEUE-NAME

Current depth of workqueue.

workqueueRetriesDelta_name_WORK-QUEUE-NAME

Difference of the total number of retries handled by workqueue.

leaderElectionMasterStatus

Gauge of if the reporting system is master of the relevant lease, 0 indicates backup, 1 indicates master.

nodeCollectorEvictionsDelta

Delta change of node evictions that happened since current instance of NodeController started (requires k8s v1.24+)

Scheduler data

Query the K8sSchedulerSample event to see Scheduler data. For more information, see Configure control plane monitoring:

Scheduler attribute

Description

processResidentMemoryBytes

Resident memory size, in bytes.

processCpuSecondsDelta

Difference of the user and system CPU time spent in seconds.

goThreads

Number of OS threads created.

goGoroutines

Number of goroutines that currently exist.

leaderElectionMasterStatus

Gauge of if the reporting system is master of the relevant lease, 0 indicates backup, 1 indicates master.

httpRequestDurationMicroseconds_handler_HANDLER_quantile_QUANTILE.

The HTTP request latencies in microseconds, per quantile.

httpRequestDurationMicroseconds_handler_HANDLER_sum

The sum of the HTTP request latencies, in microseconds.

httpRequestDurationMicroseconds_handler_HANDLER_count

The number of observed HTTP requests events.

restClientRequestsDelta_code_CODE_host_HOST_method_METHOD

Difference of the number of HTTP requests, partitioned by status code, method, and host.

restClientRequestsRate_code_CODE_host_HOST_method_METHOD

Rate of the number of HTTP requests, partitioned by status code, method, and host.

schedulerScheduleAttemptsDelta_result_RESULT

Difference of the number of attempts to schedule pods, by the result. unschedulable means a pod could not be scheduled, while error means an internal scheduler problem.

schedulerScheduleAttemptsRate_result_RESULT

Rate of the number of attempts to schedule pods, by the result. unschedulable means a pod could not be scheduled, while error means an internal scheduler problem.

schedulerSchedulingDurationSeconds_operation_OPERATION_quantile_QUANTILE

Scheduling latency in seconds split by sub-parts of the scheduling operation.

schedulerSchedulingDurationSeconds_operation_OPERATION_sum

The sum of scheduling latency in seconds split by sub-parts of the scheduling operation.

schedulerSchedulingDurationSeconds_operation_OPERATION_count

The number of observed events of schedulings split by sub-parts of the scheduling operation.

schedulerPreemptionAttemptsDelta

Difference of the total preemption attempts in the cluster till now.

schedulerPodPreemptionVictims

Number of selected preemption victims.

schedulerPendingPodsActive

Number of pending pods in the active queue (requires k8s v1.23+).

schedulerPendingPodsBackoff

Number of pending pods in the backoff queue (requires k8s v1.23+).

schedulerPendingPodsUnschedulable

Number of pending pods in the unschedulable queue (requires k8s v1.23+).

etcd data

Query the K8sEtcdSample event to see etcd data. For more information, see Configure control plane monitoring:

etcd attribute

Description

processResidentMemoryBytes

Resident memory size, in bytes.

processCpuSecondsDelta

Difference of the user and system CPU time spent in seconds.

goThreads

Number of OS threads created.

goGoroutines

Number of goroutines that currently exist.

etcdServerHasLeader

Whether or not a leader exists. 1 is existence, 0 is not.

etcdServerLeaderChangesSeenDelta

Difference of the number of leader changes seen.

etcdMvccDbTotalSizeInBytes

Total size of the underlying database physically allocated, in bytes.

etcdServerProposalsCommittedDelta

Difference of the total number of consensus proposals committed.

etcdServerProposalsCommittedRate

Rate of the total number of consensus proposals committed.

etcdServerProposalsAppliedDelta

Difference of the total number of consensus proposals applied.

etcdServerProposalsAppliedRate

Rate of the total number of consensus proposals applied.

etcdServerProposalsPending

The current number of pending proposals to commit.

etcdServerProposalsFailedDelta

Difference of the total number of failed proposals seen.

etcdServerProposalsFailedRate

Rate of the total number of failed proposals seen.

processOpenFds

Number of open file descriptors.

processMaxFds

Maximum number of open file descriptors.

processFdsUtilization

Percentage open file descriptors with respect to the maximum number that can be opened.

etcdNetworkClientGrpcReceivedBytesRate

Rate of the total number of bytes received from gRPC clients.

etcdNetworkClientGrpcSentBytesRate

Rate of the total number of bytes sent to gRPC clients.

Endpoint data

Query the K8sEndpointSample event for endpoint data:

Endpoint attribute

Description

clusterName

Name that you assigned to the cluster when you installed the Kubernetes integration.

createdAt

Timestamp of when the endpoint was created.

namespaceName

Name of the namespace that the endpoint belongs to.

endpointName

Name associated with the endpoint.

label.LABEL_NAME

Labels associated with your endpoint, so you can filter and query for specific endpoints.

addressAvailable

Number of addresses available in endpoint.

addressNotReady

Number of addresses not ready in endpoint.

Service data

Query the K8sServiceSample event for service data:

Service attribute

Description

clusterName

Name that you assigned to the cluster when you installed the Kubernetes integration.

createdAt

Timestamp of when the service was created.

namespaceName

Name of the namespace that the service belongs to.

label.LABEL_NAME

Labels associated with your service, so you can filter and query for specific service.

serviceName

Name associated with the service.

loadBalancerIP

The IP of the external load balancer, if Spectype is LoadBalancer.

externalName

The external name value, if Spectype is ExternalName.

clusterIP

The internal cluster IP, if Spectype is ClusterIP.

specType

Type of the service.

selector.LABEL_NAME

The label selector that this service targets.

Horizontal Pod Autoscaler data

Query the K8sHpaSample event for Horizontal Pod Autoscaler data:

HPA attribute

Description

clusterName

Name that you assigned to the cluster when you installed the Kubernetes integration.

label.LABEL_NAME

Labels associated with your HPA, so you can filter and query for specific autoscaler.

currentReplicas

Current number of replicas of pods managed by this autoscaler.

desiredReplicas

Desired number of replicas of pods managed by this autoscaler.

minReplicas

Lower limit for the number of pods that can be set by the autoscaler, 1 by default.

maxReplicas

Upper limit for the number of pods that can be set by the autoscaler; cannot be smaller than minReplicas.

targetMetric

The metric specifications used by this autoscaler when calculating the desired replica count.

isAble

Boolean representing whether or not the autoscaler is able to fetch and update scales, as well as whether or not any backoff-related conditions would prevent scaling.

isActive

Boolean representing whether or not the autoscaler is enabled (if it's able to calculate the desired scales).

isLimited

Boolean representing whether or not the autoscaler is capped, either up or down, by the maximum or minimum replicas configured.

labels

Number of Kubernetes labels converted to Prometheus labels.

metadataGeneration

The generation observed by the HorizontalPodAutoscaler controller.

Kubernetes metadata in APM-monitored applications

By linking your applications with Kubernetes, the following attributes are added to application trace and distributed trace:

  • nodeName
  • containerName
  • podName
  • clusterName
  • deploymentName
  • namespaceName

For more help

Copyright © 2022 New Relic Inc.

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.