When using synthetic monitoring's private locations with New Relic's alerts, you can be notified if a location is under-provisioned, mis-configured, or generally misbehaving.
This guide will help you answer the following basic questions regarding private location health by using New Relic dashboards and NRQL alerts:
- Are my private minions online?
- Does my private location need more minions?
- Can I check the status of a specific minion directly?
Prerequisites
Before following the instructions in this guide, ensure you have:
- A synthetic private location
- At least one private minion installed at that location
- Checks scheduled to run at that location
- An alert policy for the private location, with a configured notification channel to notify your team when a violation occurs
The following Private Minion dashboard example JSON can be imported to your account using:
{ "name": "Synthetics Private Minions", "description": "Details on events from SyntheticsPrivateLocationStatus, SyntheticsPrivateMinion, SyntheticCheck, SyntheticRequest, NrAuditEvent, plus a page for Performance Analysis.", "permissions": "PUBLIC_READ_WRITE", "pages": [ { "name": "SyntheticsPrivateLocationStatus", "description": "Private location queues.", "widgets": [ { "title": "pending checks by location", "layout": { "column": 1, "row": 1, "width": 3, "height": 5 }, "linkedEntityGuids": null, "visualization": { "id": "viz.bar" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateLocationStatus SELECT latest(checksPending) FACET name" } ] } }, { "title": "private location queue", "layout": { "column": 4, "row": 1, "width": 9, "height": 4 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateLocationStatus SELECT max(checksPending) FACET name TIMESERIES MAX SINCE 2 weeks ago" } ], "yAxisLeft": { "zero": true } } }, { "title": "rate of queue growth", "layout": { "column": 4, "row": 5, "width": 9, "height": 4 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateLocationStatus SELECT clamp_min(derivative(checksPending,1 minute),0) AS 'queue growth rate' FACET name TIMESERIES MAX SINCE 2 weeks ago" } ], "yAxisLeft": { "zero": true } } }, { "title": "Checks Pending", "layout": { "column": 1, "row": 6, "width": 3, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.billboard" }, "rawConfiguration": { "dataFormatters": [], "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateLocationStatus SELECT average(checksPending) SINCE 3 minutes ago UNTIL 2 minutes ago" } ], "thresholds": [ { "alertSeverity": "WARNING", "value": 1 }, { "alertSeverity": "CRITICAL", "value": 5 } ] } }, { "title": "audit events for private locations", "layout": { "column": 1, "row": 9, "width": 12, "height": 4 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "dataFormatters": [], "nrqlQueries": [ { "accountId": 0, "query": "FROM NrAuditEvent SELECT * WHERE targetType = 'PRIVATE_LOCATION' LIMIT MAX" } ] } } ] }, { "name": "SyntheticsPrivateMinion", "description": "Minion health.", "widgets": [ { "title": "private locations", "layout": { "column": 1, "row": 1, "width": 3, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "dataFormatters": [], "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateMinion SELECT uniqueCount(minionId) - 1 AS 'restarts',latest(timestamp) FACET minionLocation" } ] } }, { "title": "minion details", "layout": { "column": 4, "row": 1, "width": 9, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "dataFormatters": [], "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateMinion SELECT latest(minionId),latest(timestamp) AS 'last seen',latest(minionStartTimestamp) AS 'started',uniqueCount(minionId) - 1 AS 'restarts',latest(minionIpv4),max(minionJobsQueued),max(minionJobsFinished),100*max(minionJobsFailed)/max(minionJobsQueued) AS 'job failure rate',latest(minionJobsRunning),max(minionJobsTimedout),max(minionJobsSkipped),max(minionJobsInternalEngineError),latest(minionWorkers),latest(minionProcessors),latest(minionPhysicalMemoryUsedBytes/(1024*1024*1024)) AS 'used memory (GiB)',latest(minionPhysicalMemoryTotalBytes/(1024*1024*1024)) AS 'total memory (GiB)',latest(minionPhysicalMemoryTotalBytes/(1024*1024*1024))/latest(minionWorkers) AS 'memory (GiB) per heavy worker' FACET minionBuildNumber,minionLocation,minionOsVersion,minionContainerSystemVersion,minionHostname LIMIT 100" } ] } }, { "title": "minion instances over time", "layout": { "column": 1, "row": 4, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.billboard" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateMinion SELECT uniqueCount(minionId) AS 'minions'" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "restarts", "layout": { "column": 3, "row": 4, "width": 3, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.area" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateMinion SELECT uniqueCount(minionId) WHERE minionLocation NOT LIKE 'AWS_%' FACET minionId TIMESERIES LIMIT 50" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "CPU utilization (%)", "layout": { "column": 6, "row": 4, "width": 3, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateMinion SELECT average(minionProcessorsUsagePercentage) AS 'avg',max(minionProcessorsUsagePercentage) AS 'max' TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "memory utilization (%)", "layout": { "column": 9, "row": 4, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateMinion SELECT average(minionPhysicalMemoryUsedPercentage) AS 'average',max(minionPhysicalMemoryUsedPercentage) AS 'max' TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "used vs free swap (GiB)", "layout": { "column": 1, "row": 7, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.area" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateMinion SELECT max(minionSwapMemoryUsedBytes) / pow(1024,3) AS 'used',(max(minionSwapMemoryTotalBytes)-max(minionSwapMemoryUsedBytes)) / pow(1024,3) AS 'free' TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "used vs free memory (GiB)", "layout": { "column": 5, "row": 7, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.area" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateMinion SELECT max(minionPhysicalMemoryUsedBytes) / pow(1024,3) AS 'used',(max(minionPhysicalMemoryTotalBytes)-max(minionPhysicalMemoryUsedBytes)) / pow(1024,3) AS 'free' TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "count of Internal Engine Errors (IEE) vs failed jobs", "layout": { "column": 9, "row": 7, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateMinion SELECT max(minionJobsInternalEngineError) AS 'IEE',max(minionJobsFailed) AS 'failed jobs' TIMESERIES AUTO" } ], "yAxisLeft": { "zero": false } } }, { "title": "jobs running", "layout": { "column": 1, "row": 10, "width": 4, "height": 6 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateMinion SELECT max(minionJobsRunning) AS 'concurrent',clamp_min(derivative(minionJobsFinished,1 minute),0) AS 'finished per minute' TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "jobs finished", "layout": { "column": 5, "row": 10, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateMinion SELECT max(minionJobsFinished) - min(minionJobsFinished) AS 'total jobs' TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "job failure rates", "layout": { "column": 9, "row": 10, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.stacked-bar" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateMinion SELECT (max(minionJobsFailed/minionJobsReceived))*100 AS 'failed rate',(max(minionJobsInternalEngineError/minionJobsReceived))*100 AS 'IEE rate' TIMESERIES AUTO" } ] } }, { "title": "skipped jobs", "layout": { "column": 5, "row": 13, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateMinion SELECT max(minionJobsSkipped) - min(minionJobsSkipped) AS 'skipped jobs' FACET minionHostname TIMESERIES AUTO" } ], "yAxisLeft": { "zero": false } } }, { "title": "IEE as a percentage of failed jobs", "layout": { "column": 9, "row": 13, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.stacked-bar" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateMinion SELECT average(minionJobsInternalEngineError/minionJobsFailed)*100 AS 'IEE',(average((minionJobsFailed-minionJobsInternalEngineError)/minionJobsFailed))*100 AS 'failed jobs' TIMESERIES AUTO" } ] } }, { "title": "job timeout rates", "layout": { "column": 9, "row": 16, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.stacked-bar" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateMinion SELECT 100*latest(minionJobsTimedout)/latest(minionJobsReceived) AS 'job timeout rate' TIMESERIES AUTO" } ] } } ] }, { "name": "SyntheticCheck", "description": null, "widgets": [ { "title": "minion IDs by location", "layout": { "column": 1, "row": 1, "width": 6, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "dataFormatters": [], "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT latest(minion) WHERE location NOT LIKE 'AWS_%' FACET locationLabel,minionId LIMIT MAX" } ] } }, { "title": "minion count", "layout": { "column": 7, "row": 1, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT uniqueCount(minionId) WHERE location NOT LIKE 'AWS_%' TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "abnormal job duration by minion (s)", "layout": { "column": 9, "row": 1, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.stacked-bar" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT stddev(nr.internalQueueDuration+nr.executionDuration)/1e3 AS 'abnormal job duration (s)' FACET minion WHERE location NOT LIKE 'AWS_%' LIMIT 20 TIMESERIES AUTO" } ] } }, { "title": "avg monitor frequency by location (m)", "layout": { "column": 1, "row": 4, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "dataFormatters": [], "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId)/rate(uniqueCount(id),1 minute) AS 'avg frequency (m)' WHERE location NOT LIKE 'AWS_%' FACET locationLabel TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "avg monitor frequency by type (m)", "layout": { "column": 5, "row": 4, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "dataFormatters": [], "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId)/rate(uniqueCount(id),1 minute) AS 'avg frequency (m)' WHERE location NOT LIKE 'AWS_%' FACET type TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "avg monitor frequency by status (m)", "layout": { "column": 7, "row": 4, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "dataFormatters": [], "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId)/rate(uniqueCount(id),1 minute) AS 'avg frequency (m)' WHERE location NOT LIKE 'AWS_%' FACET result TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "abnormal job duration by monitor (s)", "layout": { "column": 9, "row": 4, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.stacked-bar" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT stddev(nr.internalQueueDuration+nr.executionDuration)/1e3 AS 'abnormal job duration (s)' FACET monitorId WHERE location NOT LIKE 'AWS_%' LIMIT 20 TIMESERIES AUTO" } ] } }, { "title": "monitors by location", "layout": { "column": 1, "row": 10, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.pie" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId) WHERE location NOT LIKE 'AWS_%' FACET locationLabel" } ] } }, { "title": "monitors by location", "layout": { "column": 5, "row": 10, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId) WHERE location NOT LIKE 'AWS_%' FACET locationLabel TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "job duration by location (s)", "layout": { "column": 7, "row": 10, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "dataFormatters": [], "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT average(nr.internalQueueDuration+nr.executionDuration)/1e3 AS 'avg job duration (s)' WHERE location NOT LIKE 'AWS_%' FACET locationLabel TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "abnormal job duration by location (s)", "layout": { "column": 9, "row": 10, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.stacked-bar" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT stddev(nr.internalQueueDuration+nr.executionDuration)/1e3 AS 'abnormal job duration (s)' FACET locationLabel WHERE location NOT LIKE 'AWS_%' LIMIT 20 TIMESERIES AUTO" } ] } }, { "title": "monitors by runtime type version", "layout": { "column": 1, "row": 13, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.pie" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId) WHERE location NOT LIKE 'AWS_%' FACET runtimeTypeVersion" } ] } }, { "title": "monitors by runtime type version", "layout": { "column": 5, "row": 13, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId) WHERE location NOT LIKE 'AWS_%' FACET runtimeTypeVersion TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "job duration by runtime type version (s)", "layout": { "column": 7, "row": 13, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "dataFormatters": [], "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT average(nr.internalQueueDuration+nr.executionDuration)/1e3 AS 'avg job duration (s)' WHERE location NOT LIKE 'AWS_%' FACET runtimeTypeVersion TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "abnormal job duration by runtime type version (s)", "layout": { "column": 9, "row": 13, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.stacked-bar" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT stddev(nr.internalQueueDuration+nr.executionDuration)/1e3 AS 'abnormal job duration (s)' FACET runtimeTypeVersion WHERE location NOT LIKE 'AWS_%' LIMIT 20 TIMESERIES AUTO" } ] } }, { "title": "monitors by type", "layout": { "column": 1, "row": 16, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.pie" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId) WHERE location NOT LIKE 'AWS_%' FACET type" } ] } }, { "title": "monitors by type", "layout": { "column": 5, "row": 16, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId) WHERE location NOT LIKE 'AWS_%' FACET type TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "job duration by type (s)", "layout": { "column": 7, "row": 16, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "dataFormatters": [], "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT average(nr.internalQueueDuration+nr.executionDuration)/1e3 AS 'avg job duration (s)' WHERE location NOT LIKE 'AWS_%' FACET type TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "abnormal job duration by type (s)", "layout": { "column": 9, "row": 16, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.stacked-bar" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT stddev(nr.internalQueueDuration+nr.executionDuration)/1e3 AS 'abnormal job duration (s)' FACET type WHERE location NOT LIKE 'AWS_%' TIMESERIES AUTO" } ] } }, { "title": "monitors by status", "layout": { "column": 1, "row": 19, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.pie" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId) WHERE location NOT LIKE 'AWS_%' FACET result" } ] } }, { "title": "monitors by status", "layout": { "column": 5, "row": 19, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId) WHERE location NOT LIKE 'AWS_%' FACET result TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "job duration by status (s)", "layout": { "column": 7, "row": 19, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "dataFormatters": [], "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT average(nr.internalQueueDuration+nr.executionDuration)/1e3 AS 'avg job duration (s)' WHERE location NOT LIKE 'AWS_%' FACET result TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "abnormal job duration by status (s)", "layout": { "column": 9, "row": 19, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.stacked-bar" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT stddev(nr.internalQueueDuration+nr.executionDuration)/1e3 AS 'abnormal job duration (s)' FACET result WHERE location NOT LIKE 'AWS_%' TIMESERIES AUTO" } ] } }, { "title": "successful monitors by type", "layout": { "column": 1, "row": 22, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.pie" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId) WHERE location NOT LIKE 'AWS_%' AND result = 'SUCCESS' FACET type" } ] } }, { "title": "successful monitors by type", "layout": { "column": 5, "row": 22, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId) WHERE location NOT LIKE 'AWS_%' AND result = 'SUCCESS' FACET type TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "job duration for successes by type (s)", "layout": { "column": 7, "row": 22, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "dataFormatters": [], "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT average(nr.internalQueueDuration+nr.executionDuration)/1e3 AS 'avg job duration (s)' WHERE location NOT LIKE 'AWS_%' AND result = 'SUCCESS' FACET type TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "abnormal job duration for successes by type (s)", "layout": { "column": 9, "row": 22, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.stacked-bar" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT stddev(nr.internalQueueDuration+nr.executionDuration)/1e3 AS 'abnormal job duration (s)' FACET type WHERE location NOT LIKE 'AWS_%' AND result = 'SUCCESS' TIMESERIES AUTO" } ] } }, { "title": "failed monitors by type", "layout": { "column": 1, "row": 25, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.pie" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId) WHERE location NOT LIKE 'AWS_%' AND result = 'FAILED' FACET type" } ] } }, { "title": "failed monitors by type", "layout": { "column": 5, "row": 25, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId) WHERE location NOT LIKE 'AWS_%' AND result = 'FAILED' FACET type TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "job duration for failures by type (s)", "layout": { "column": 7, "row": 25, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "dataFormatters": [], "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT average(nr.internalQueueDuration+nr.executionDuration)/1e3 AS 'avg job duration (s)' WHERE location NOT LIKE 'AWS_%' AND result = 'FAILED' FACET type TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "abnormal job duration for failures by type (s)", "layout": { "column": 9, "row": 25, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.stacked-bar" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT stddev(nr.internalQueueDuration+nr.executionDuration)/1e3 AS 'abnormal job duration (s)' FACET type WHERE location NOT LIKE 'AWS_%' AND result = 'FAILED' TIMESERIES AUTO" } ] } }, { "title": "ping monitors by location", "layout": { "column": 1, "row": 28, "width": 4, "height": 6 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "dataFormatters": [], "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId) AS 'monitors with results',uniqueCount(monitorId)/rate(uniqueCount(id),1 minute) AS 'avg job frequency (m)',average(nr.internalQueueDuration+nr.executionDuration)/1e3 AS 'avg duration (s)' WHERE location NOT LIKE 'AWS_%' AND type = 'SIMPLE' FACET locationLabel" } ] } }, { "title": "top 5 ping errors", "layout": { "column": 5, "row": 28, "width": 8, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "dataFormatters": [], "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT count(*),latest(monitorId) AS 'latest monitor id',latest(type),average((nr.internalQueueDuration+nr.executionDuration)/1e3) AS 'avg job duration (s)',max(timestamp) AS 'last occurred' WHERE type = 'SIMPLE' AND result = 'FAILED' AND location NOT LIKE 'AWS_%' FACET error LIMIT 5" } ] } }, { "title": "ping duration with monitor count (s)", "layout": { "column": 5, "row": 31, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT average(nr.internalQueueDuration+nr.executionDuration)/1e3 AS 'avg job duration (s)',uniqueCount(monitorId) AS 'monitor count' WHERE type = 'SIMPLE' AND location NOT LIKE 'AWS_%' TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "ping timeout vs other error durations", "layout": { "column": 7, "row": 31, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT average((nr.internalQueueDuration+nr.executionDuration)/1e3) WHERE type = 'SIMPLE' AND location NOT LIKE 'AWS_%' AND result = 'FAILED' FACET cases(WHERE error NOT LIKE '%timeout%' AS 'other error duration (s)',WHERE error LIKE '%timeout%' AS 'timeout duration (s)') TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "ping jobs per minute", "layout": { "column": 9, "row": 31, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT rate(uniqueCount(id),1 minute) WHERE location NOT LIKE 'AWS_%' AND type = 'SIMPLE' TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "ping timeout rate", "layout": { "column": 11, "row": 31, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT percentage(count(*),WHERE error LIKE '%timeout%' OR error LIKE '%timed-out%') WHERE location NOT LIKE 'AWS_%' AND type = 'SIMPLE' TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "non-ping monitors by location", "layout": { "column": 1, "row": 34, "width": 4, "height": 6 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "dataFormatters": [], "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId) AS 'monitors with results',uniqueCount(monitorId)/rate(uniqueCount(id),1 minute) AS 'avg job frequency (m)',average(nr.internalQueueDuration+nr.executionDuration)/1e3 AS 'avg duration (s)' WHERE location NOT LIKE 'AWS_%' AND type != 'SIMPLE' FACET locationLabel" } ] } }, { "title": "top 5 non-ping errors", "layout": { "column": 5, "row": 34, "width": 8, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "dataFormatters": [], "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT count(*),latest(monitorId) AS 'latest monitor id',latest(type),average((nr.internalQueueDuration+nr.executionDuration)/1e3) AS 'avg job duration (s)',max(timestamp) AS 'last occurred' WHERE type != 'SIMPLE' AND result = 'FAILED' AND location NOT LIKE 'AWS_%' FACET error LIMIT 5" } ] } }, { "title": "non-ping duration with monitor count (s)", "layout": { "column": 5, "row": 37, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT average(nr.internalQueueDuration+nr.executionDuration)/1e3 AS 'avg job duration (s)',uniqueCount(monitorId) AS 'monitor count' WHERE type != 'SIMPLE' AND location NOT LIKE 'AWS_%' TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "non-ping timeout vs other error durations", "layout": { "column": 7, "row": 37, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT average((nr.internalQueueDuration+nr.executionDuration)/1e3) WHERE type != 'SIMPLE' AND location NOT LIKE 'AWS_%' AND result = 'FAILED' FACET cases(WHERE error NOT LIKE '%timeout%' AS 'other error duration (s)',WHERE error LIKE '%timeout%' AS 'timeout duration (s)') TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "non-ping jobs per minute", "layout": { "column": 9, "row": 37, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT rate(uniqueCount(id),1 minute) WHERE location NOT LIKE 'AWS_%' AND type != 'SIMPLE' TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "non-ping timeout rate", "layout": { "column": 11, "row": 37, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT percentage(count(*),WHERE error LIKE '%timeout%' OR error LIKE '%timed-out%') WHERE location NOT LIKE 'AWS_%' AND type != 'SIMPLE' TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "jobs per minute by minion", "layout": { "column": 1, "row": 40, "width": 6, "height": 4 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT rate(uniqueCount(id),1 minute) WHERE location NOT LIKE 'AWS_%' FACET minion TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "errors per minute by type", "layout": { "column": 7, "row": 40, "width": 6, "height": 4 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT rate(uniqueCount(id),1 minute) WHERE location NOT LIKE 'AWS_%' AND error != '' AND error IS NOT NULL FACET type,result TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "jobs per minute by type", "layout": { "column": 1, "row": 44, "width": 6, "height": 4 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT rate(uniqueCount(id),1 minute) WHERE location NOT LIKE 'AWS_%' FACET type TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "error counts by class", "layout": { "column": 7, "row": 44, "width": 6, "height": 4 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "dataFormatters": [], "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT count(*) WHERE location NOT LIKE 'AWS_%' AND result != 'SUCCESS' FACET capture(error,r'(?P<firstWord>\\S+)\\s.+') AS 'error class' TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "monitor details", "layout": { "column": 1, "row": 48, "width": 12, "height": 4 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "dataFormatters": [], "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT uniqueCount(id) AS 'results',uniqueCount(location)/rate(uniqueCount(id),1 minute) AS 'avg job frequency (m)',average(nr.internalQueueDuration+nr.executionDuration)/1e3 AS 'avg duration (s)' WHERE location NOT LIKE 'AWS_%' FACET monitorId,type,apiVersion,runtimeTypeVersion LIMIT 50" } ] } } ] }, { "name": "SyntheticRequest", "description": null, "widgets": [ { "title": "locations", "layout": { "column": 1, "row": 1, "width": 5, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.pie" }, "rawConfiguration": { "facet": { "showOtherSeries": true }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT count(*) WHERE location NOT LIKE 'AWS_%' FACET locationLabel LIMIT 10" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "top 5 error response codes", "layout": { "column": 6, "row": 1, "width": 7, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.stacked-bar" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "dataFormatters": [], "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT count(responseCode) AS 'responses' WHERE responseCode NOT IN (200,201,202,203,204,300,301,302,303,304) AND responseCode IS NOT NULL AND location NOT LIKE 'AWS_%' FACET responseCode,responseStatus TIMESERIES AUTO LIMIT 5" } ], "yAxisLeft": { "zero": true } } }, { "title": "Average duration (ms) by location", "layout": { "column": 1, "row": 4, "width": 5, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT average(durationDNS) AS 'DNS',average(durationSSL) AS 'SSL',average(durationConnect) AS 'Connect',average(durationSend) AS 'Send',average(durationWait) AS 'Wait',average(durationBlocked) AS 'Blocked',average(durationReceive) AS 'Receive',average(duration) AS 'Duration' WHERE location NOT LIKE 'AWS_%' FACET locationLabel" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "max duration (ms) by URL", "layout": { "column": 6, "row": 4, "width": 7, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT max(durationDNS) AS 'DNS',max(durationSSL) AS 'SSL',max(durationConnect) AS 'Connect',max(durationSend) AS 'Send',max(durationWait) AS 'Wait',max(durationBlocked) AS 'Blocked',max(durationReceive) AS 'Receive',max(duration) AS 'Duration' WHERE location NOT LIKE 'AWS_%' FACET URL" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "Max duration (ms) by location", "layout": { "column": 1, "row": 7, "width": 5, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT max(durationDNS) AS 'DNS',max(durationSSL) AS 'SSL',max(durationConnect) AS 'Connect',max(durationSend) AS 'Send',max(durationWait) AS 'Wait',max(durationBlocked) AS 'Blocked',max(durationReceive) AS 'Receive',max(duration) AS 'Duration' WHERE location NOT LIKE 'AWS_%' FACET locationLabel" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "Average duration (ms) by URL", "layout": { "column": 6, "row": 7, "width": 7, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT average(durationDNS) AS 'DNS',average(durationSSL) AS 'SSL',average(durationConnect) AS 'Connect',average(durationSend) AS 'Send',average(durationWait) AS 'Wait',average(durationBlocked) AS 'Blocked',average(durationReceive) AS 'Receive',average(duration) AS 'Duration' WHERE location NOT LIKE 'AWS_%' FACET URL" } ] } }, { "title": "request rate", "layout": { "column": 1, "row": 10, "width": 5, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "dataFormatters": [], "legend": { "enabled": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT rate(uniqueCount(id),1 minute) AS 'request rate' WHERE location NOT LIKE 'AWS_%' TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "request rate by job", "layout": { "column": 6, "row": 10, "width": 7, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.stacked-bar" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "dataFormatters": [], "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT rate(uniqueCount(id),1 minute) AS 'request rate' WHERE location NOT LIKE 'AWS_%' FACET id TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "jobs per minion", "layout": { "column": 1, "row": 13, "width": 5, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "dataFormatters": [], "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT uniqueCount(id) AS 'jobs' WHERE location NOT LIKE 'AWS_%' FACET locationLabel,minion,minionId" } ] } }, { "title": "response status", "layout": { "column": 6, "row": 13, "width": 3, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.pie" }, "rawConfiguration": { "facet": { "showOtherSeries": true }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT count(*) WHERE location NOT LIKE 'AWS_%' FACET responseStatus LIMIT 10" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "content type", "layout": { "column": 9, "row": 13, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.pie" }, "rawConfiguration": { "facet": { "showOtherSeries": true }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT count(*) WHERE location NOT LIKE 'AWS_%' FACET contentType LIMIT 100" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "incomplete jobs", "layout": { "column": 1, "row": 16, "width": 5, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "dataFormatters": [], "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT percentage(uniqueCount(id),WHERE domComplete IS NULL) AS 'percent incomplete',uniqueCount(id) AS 'jobs' WHERE isNavigationRoot IS true AND location NOT LIKE 'AWS_%' FACET monitorName,monitorId,runtimeTypeVersion LIMIT 20" } ] } }, { "title": "response codes", "layout": { "column": 6, "row": 16, "width": 3, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.pie" }, "rawConfiguration": { "facet": { "showOtherSeries": true }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT count(*) WHERE location NOT LIKE 'AWS_%' FACET responseCode LIMIT 100" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "content category", "layout": { "column": 9, "row": 16, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.pie" }, "rawConfiguration": { "facet": { "showOtherSeries": true }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT count(*) WHERE location NOT LIKE 'AWS_%' FACET contentCategory LIMIT 100" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "percent of incomplete requests", "layout": { "column": 1, "row": 19, "width": 5, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "dataFormatters": [], "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT percentage(count(*),WHERE domComplete IS NULL) AS 'percent incomplete',count(*) AS 'requests',latest(responseCode),latest(responseStatus) WHERE isNavigationRoot IS true AND location NOT LIKE 'AWS_%' FACET domain" } ] } }, { "title": "complete vs incomplete jobs", "layout": { "column": 6, "row": 19, "width": 3, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.pie" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "dataFormatters": [], "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT uniqueCount(id) AS 'number of jobs' WHERE isNavigationRoot IS true AND location NOT LIKE 'AWS_%' FACET cases(WHERE domComplete IS NULL AS 'incomplete',WHERE domComplete IS NOT NULL AS 'complete')" } ] } }, { "title": "top 5 domains for incomplete jobs", "layout": { "column": 9, "row": 19, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "dataFormatters": [], "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT count(*) AS 'requests',latest(URL) WHERE domComplete IS NULL AND isNavigationRoot IS true AND location NOT LIKE 'AWS_%' FACET domain LIMIT 5" } ] } }, { "title": "page load duration for complete vs incomplete jobs", "layout": { "column": 1, "row": 22, "width": 3, "height": 4 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "dataFormatters": [], "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT max(onPageLoad/1e3) WHERE isNavigationRoot IS true AND location NOT LIKE 'AWS_%' FACET cases(WHERE domComplete IS NULL AS 'incomplete',WHERE domComplete IS NOT NULL AS 'complete') TIMESERIES AUTO" } ], "yAxisLeft": { "zero": true } } }, { "title": "page load duration for complete vs incomplete jobs by monitor", "layout": { "column": 4, "row": 22, "width": 9, "height": 4 }, "linkedEntityGuids": null, "visualization": { "id": "viz.line" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "dataFormatters": [], "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT max(onPageLoad/1e3) WHERE isNavigationRoot IS true AND location NOT LIKE 'AWS_%' FACET cases(WHERE domComplete IS NULL AS 'incomplete',WHERE domComplete IS NOT NULL AS 'complete'),monitorName TIMESERIES MAX LIMIT MAX" } ], "yAxisLeft": { "zero": true } } }, { "title": "domains", "layout": { "column": 1, "row": 26, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.pie" }, "rawConfiguration": { "facet": { "showOtherSeries": true }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT count(*) WHERE location NOT LIKE 'AWS_%' FACET domain LIMIT 100" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "request type", "layout": { "column": 5, "row": 26, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.pie" }, "rawConfiguration": { "facet": { "showOtherSeries": true }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT count(*) WHERE location NOT LIKE 'AWS_%' FACET verb LIMIT 100" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "non HTTP 200 by domain", "layout": { "column": 9, "row": 26, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.pie" }, "rawConfiguration": { "facet": { "showOtherSeries": true }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT count(*) WHERE responseCode != 200 AND location NOT LIKE 'AWS_%' FACET domain LIMIT 100" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "hosts", "layout": { "column": 1, "row": 29, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.pie" }, "rawConfiguration": { "facet": { "showOtherSeries": true }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT count(*) WHERE location NOT LIKE 'AWS_%' FACET host LIMIT 100" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "requests by URL", "layout": { "column": 5, "row": 29, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.pie" }, "rawConfiguration": { "facet": { "showOtherSeries": true }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT count(*) WHERE location NOT LIKE 'AWS_%' FACET hierarchicalURL LIMIT 10" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "non HTTP 200 by URL", "layout": { "column": 9, "row": 29, "width": 4, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.pie" }, "rawConfiguration": { "facet": { "showOtherSeries": true }, "legend": { "enabled": true }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT count(*) WHERE responseCode != 200 AND location NOT LIKE 'AWS_%' FACET hierarchicalURL LIMIT 100" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "average request and response size by URL", "layout": { "column": 1, "row": 32, "width": 6, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT average(requestBodySize),average(requestHeaderSize),average(responseBodySize),average(responseHeaderSize) WHERE location NOT LIKE 'AWS_%' FACET URL LIMIT 100" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "average request and response size by domain", "layout": { "column": 7, "row": 32, "width": 6, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT average(requestBodySize),average(requestHeaderSize),average(responseBodySize),average(responseHeaderSize) WHERE location NOT LIKE 'AWS_%' FACET domain LIMIT 100" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "max request and response size by URL", "layout": { "column": 1, "row": 35, "width": 6, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT max(requestBodySize),max(requestHeaderSize),max(responseBodySize),max(responseHeaderSize) WHERE location NOT LIKE 'AWS_%' FACET URL LIMIT 100" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "Max Request and Response size by Domain", "layout": { "column": 7, "row": 35, "width": 6, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticRequest SELECT max(requestBodySize),max(requestHeaderSize),max(responseBodySize),max(responseHeaderSize) WHERE location NOT LIKE 'AWS_%' FACET domain LIMIT 100" } ] } }, { "title": "audit events for monitors", "layout": { "column": 1, "row": 38, "width": 12, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "dataFormatters": [], "nrqlQueries": [ { "accountId": 0, "query": "FROM NrAuditEvent SELECT * WHERE targetType = 'MONITOR' LIMIT MAX" } ] } } ] }, { "name": "Performance Analysis", "description": "Assess how the minion is performing given the current job demand. Should we scale up by adding more heavy workers and a bigger VM,or should we scale out by adding more VMs?", "widgets": [ { "title": "", "layout": { "column": 1, "row": 1, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.markdown" }, "rawConfiguration": { "text": "See [this Google sheet](https://docs.google.com/spreadsheets/d/1k2Aw11r6-S8pIpXQUINQZ0T5qlxS8iSwSo9t9GKDvzc/edit?usp=sharing) to assist in calculating how many workers will be needed based on the values on this page. This will help you to assess the required size of each host (scaling up) and how many hosts you need (scaling out).\n\nFor more info: \nhttps://discuss.newrelic.com/t/relic-solution-scaling-and-rightsizing-for-the-cpm/123999" } }, { "title": "count of heavy worker threads and cpu cores", "layout": { "column": 3, "row": 1, "width": 6, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.table" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "dataFormatters": [], "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateMinion SELECT latest(minionWorkers),latest(minionProcessors) FACET minionId,minionLocation SINCE 5 minutes ago" } ] } }, { "title": "number of minions (1 per host)", "layout": { "column": 9, "row": 1, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.billboard" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticsPrivateMinion SELECT uniqueCount(minionId) AS 'number of minions' FACET minionLocation SINCE 5 minutes ago" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "non-ping monitors", "layout": { "column": 11, "row": 1, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.billboard" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT uniqueCount(monitorId) AS 'non-ping monitors' WHERE type != 'SIMPLE' AND location NOT LIKE 'AWS_%' FACET locationLabel SINCE 2 days ago" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "non-ping jobs per minute", "layout": { "column": 1, "row": 4, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.billboard" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT rate(uniqueCount(id),1 minute) AS 'jobs per minute' WHERE type != 'SIMPLE' AND location NOT LIKE 'AWS_%' FACET locationLabel SINCE 2 days ago" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "non-ping avg success duration", "layout": { "column": 3, "row": 4, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.billboard" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT average(nr.internalQueueDuration+nr.executionDuration)/1e3 AS 'avg success duration (s)' WHERE type != 'SIMPLE' AND location NOT LIKE 'AWS_%' AND result = 'SUCCESS' FACET locationLabel SINCE 2 days ago" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "non-ping avg timeout duration", "layout": { "column": 5, "row": 4, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.billboard" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT average(nr.internalQueueDuration+nr.executionDuration)/1e3 AS 'error duration (s)' WHERE type != 'SIMPLE' AND location NOT LIKE 'AWS_%' AND result = 'FAILED' FACET locationLabel,cases(WHERE error NOT LIKE '%timeout%' AS 'other error duration (s),just FYI',WHERE error LIKE '%timeout%' AS 'avg timeout duration (s)') SINCE 2 days ago" } ], "platformOptions": { "ignoreTimeRange": false } } }, { "title": "non-ping job timeout rate", "layout": { "column": 7, "row": 4, "width": 2, "height": 3 }, "linkedEntityGuids": null, "visualization": { "id": "viz.billboard" }, "rawConfiguration": { "facet": { "showOtherSeries": false }, "nrqlQueries": [ { "accountId": 0, "query": "FROM SyntheticCheck SELECT percentage(count(*),WHERE error LIKE '%timeout%' OR error LIKE '%timed-out%') WHERE type != 'SIMPLE' AND location NOT LIKE 'AWS_%' FACET locationLabel SINCE 2 days ago" } ], "platformOptions": { "ignoreTimeRange": false } } } ] } ]}
Steps to import:
- Copy the Dashboard JSON and paste into a text editor.
- Replace
"accountId": 0,
with your New Relic account ID for each occurrence in the JSON code. - Copy the Dashboard JSON from your text editor and import using one of the methods described above.
- Edit any charts that you'd like to use facet filtering with.
Tip
If your private locations exist in a parent account and Synthetics monitors in a sub account, insert the parent account ID for NRQL queries that use SyntheticPrivateLocationStatus
and SyntheticsPrivateMinion
, and the sub account ID for queries that use SyntheticCheck
and SyntheticRequest
.
Are my private minions online?
To answer this question, you can rely on attributes from the SyntheticsPrivateMinion
event. Private minions send this event to New Relic every 30 seconds. A simple way to check if your minions are online is to compare the unique count of minion IDs with the number of minions you expect to be online.
To understand how many minions are reporting, run this example NRQL query:
SELECT uniqueCount(minionId)
FROM SyntheticsPrivateMinion
WHERE minionLocation = '1-acme_okc_dc-309'
Using this query, you can create an alert condition to notify your team when fewer minions are reporting than expected. This condition is configured with a static threshold of 2 units
, which means you will receive an alert if any of your minions are offline.
You can verify that the alert policy works as expected by manually stopping one of your minions. Then, when the alert violation occurs, you will be notified by any notification channels that have been set up. Once the minion is restarted and it comes back online, the alert will recover.
There are more robust ways to check whether minions are functioning correctly, but this query and condition simply and successfully handle the case where a machine fails, is accidentally decommissioned, or the minion process crashes. It also ensures that the minion can communicate with New Relic.
Does my private location need more minions?
To answer this question, you can use the checksPending
attribute of the SyntheticsPrivateLocationStatus
event. The checksPending
attribute reflects the number of monitor checks that are scheduled (or "queued") but have yet to be accepted by a minion in the designated location. For a location with scheduled checks and no minions, this graph would grow linearly up and to the right.
This metric is more complicated to monitor than uniqueCount(minionId)
because a high value does not necessarily mean the location is in a bad state. As long as the metric is not growing linearly up and to the right (and checks are being run on schedule), the location is in a good state.
This use case is perfect for baseline NRQL alert conditions, which allow you to monitor the deviation of a metric rather than its static value. For example:
SELECT average(checksPending)
FROM SyntheticsPrivateLocationStatus
WHERE name = '1-acme_tokyo_dc-512'
To test this alert condition, schedule one-minute, browser-based monitors to run from your location. Browser-based jobs consume more resources than ping jobs, which is why they are a better fit for load simulation. New Relic will quickly notify you of a growing number of pending checks.
After doubling the number of minions to handle the load, the alert recovers. For example, using the Synthetics private location dashboard example, notice the growth and decline of pending checks over the course of the incident and recovery. By using the NRQL condition, New Relic will notify you if and when the location needs more minion capacity.
Can I check the status of a specific minion directly?
You can also check how a minion is operating by contacting it directly. You can use a set of HTTP endpoints exposed by the minion to determine what the application is doing. In order to access these endpoints, bind ports 8080
and 8180
to ports on the host. For example, for Docker, use docker run -p 80:8080 -p 81:8180 ...
):
:8080/status/check
: Details about internal health-checks the minion performs; HTTP 200 means "healthy.":8080/status
: Details about a minion's status; the same data is then reported as aSyntheticsPrivateMinion
event.:8180/
: JVM application admin endpoints; an advanced view of a minion's internal state.
This approach is not as automated or flexible as the checksPending
example. However, if you have total network connectivity failure, this manual approach can help troubleshoot the situation.