Export metrics for Prometheus
Returns the current state, result counts and response times of your checks, and the ping history and schedule of your heartbeats, in the Prometheus text format (version 0.0.4). Point Prometheus, Grafana Cloud, the Datadog Agent or another Prometheus-compatible scraper at this endpoint to chart OnlineOrNot data beside your own metrics.
Prometheus export requires a paid plan or trial.
The easiest way to get started is Settings → Integrations → Prometheus in the OnlineOrNot dashboard. It creates a token and gives you a ready-to-paste Prometheus configuration.
Authentication
Send an API token in the Authorization header. OAuth access tokens and session cookies are not accepted.
| Token permission | Exports |
|---|---|
UPTIME_CHECKS:READ | Uptime, browser, DNS and TCP checks |
HEARTBEAT_CHECKS:READ | Heartbeats |
| Both | Checks and heartbeats |
EDIT includes READ. A token with only one of the two permissions still returns a successful export, and oon_export_info says which monitors were left out.
Query parameters
- Name
project_id- Type
- string
- Description
Only export monitors in this project. Repeat it to select several projects, up to 100. Use
unassignedfor monitors that aren't in a project. Leave it out to export every monitor.
- Name
per_region- Type
- boolean
- Description
trueadds aregionlabel tooon_check_results_totalandoon_check_duration_seconds, with one set of series per region. Defaults tofalse, which combines every region into one set of series.
Any other parameter returns 400.
Responses
| Status | Body | Meaning |
|---|---|---|
| 200 | Metrics | Prometheus text format, Content-Type: text/plain; version=0.0.4; charset=utf-8 |
| 400 | invalid request | Unknown parameter, invalid per_region value, or a project that doesn't exist |
| 401 | unauthenticated | Missing, invalid, deleted or expired API token |
| 403 | insufficient permissions | The token can't read checks or heartbeats |
| 403 | Prometheus export requires a paid plan | The organization is on the free plan |
| 429 | rate limit exceeded | More than 10 requests in a minute with this token. Retry after the Retry-After header |
| 503 | metrics unavailable | Metrics can't be read right now. Prometheus records a failed scrape and tries again at the next interval |
A failed request never returns partial metrics.
Request
curl --request GET \
--url 'https://api.onlineornot.com/v1/metrics/prometheus?per_region=true' \
--header 'Authorization: Bearer {access-token}'
Response (excerpt)
# HELP oon_export_info Always 1. Says whether checks and heartbeats were included, based on the API token permissions.
# TYPE oon_export_info gauge
oon_export_info{checks="included",heartbeats="included"} 1
# HELP oon_monitor_status Current monitor state, 1 for the active state and 0 for the others.
# TYPE oon_monitor_status gauge
oon_monitor_status{monitor_id="a1b2c3d4",check_type="UPTIME",name="Checkout",state="UP"} 1
oon_monitor_status{monitor_id="a1b2c3d4",check_type="UPTIME",name="Checkout",state="DOWN"} 0
# HELP oon_check_results_total Check results by outcome since metrics collection began.
# TYPE oon_check_results_total counter
oon_check_results_total{monitor_id="a1b2c3d4",check_type="UPTIME",name="Checkout",region="us-east-1",outcome="passing"} 1440
oon_check_results_total{monitor_id="a1b2c3d4",check_type="UPTIME",name="Checkout",region="us-east-1",outcome="failing"} 2
Metrics
Every monitor metric has these labels:
- Name
monitor_id- Type
- string
- Description
The check or heartbeat ID, the same ID used in the API and in dashboard URLs.
- Name
check_type- Type
- string
- Description
UPTIME,BROWSER,DNS,TCPorHEARTBEAT. A check and a heartbeat can share an ID, so usemonitor_idandcheck_typetogether.
- Name
name- Type
- string
- Description
The monitor name. Renaming a monitor starts new series under the new name, so group by
monitor_idandcheck_typein queries and alerts that should survive a rename.
- Name
oon_export_info- Type
- gauge
- Description
Always 1. The
checksandheartbeatslabels areincluded, orno_permissionwhen the token can't read them.
- Name
oon_monitor_info- Type
- gauge
- Description
Always 1. Adds
project_idandproject_namelabels. Monitors without a project haveproject_id="unassigned".
- Name
oon_monitor_status- Type
- gauge
- Description
1 for the monitor's current state and 0 for the others.
stateis one ofUP,DOWN,PENDING,PAUSED,MUTED,MAINTENANCE,RECOVERINGorVERIFYING.
- Name
oon_monitor_last_result_timestamp_seconds- Type
- gauge
- Description
Unix time of the latest check result, or of the latest heartbeat ping.
- Name
oon_check_interval_seconds- Type
- gauge
- Description
How often the check runs.
- Name
oon_check_results_total- Type
- counter
- Description
Check results, with
outcome="passing"oroutcome="failing".
- Name
oon_check_duration_seconds- Type
- histogram
- Description
How long each check took, for passing and failing results. Buckets from 5 milliseconds to 10 minutes.
- Name
oon_heartbeat_pings_total- Type
- counter
- Description
Pings received.
- Name
oon_heartbeat_expected_interval_seconds- Type
- gauge
- Description
How often the heartbeat should ping. Not present for heartbeats on a cron schedule.
- Name
oon_heartbeat_grace_period_seconds- Type
- gauge
- Description
How long after the due time OnlineOrNot waits before marking the heartbeat down.
- Name
oon_heartbeat_next_due_timestamp_seconds- Type
- gauge
- Description
Unix time the next ping is due, not including the grace period.
- Name
oon_heartbeat_overdue- Type
- gauge
- Description
1 when the heartbeat is past its due time plus grace period, otherwise 0.
Counters start at zero when Prometheus export is turned on, by creating a token in the dashboard or by the first successful scrape, and don't include earlier results. If nothing scrapes the endpoint for 30 days, OnlineOrNot stops counting until the next successful scrape, and counters can start again from zero. Prometheus treats that as a counter reset, so rate() and increase() still work.
New results appear within about a minute. Each export reflects the latest results at the time of the scrape. If OnlineOrNot itself is having problems, counters and last-result times can stop updating for a while.
Number of series
Metrics services often charge by the number of series. Each check produces 32 series, or 11 plus 21 for each region when per_region=true. Each heartbeat produces 15 series, or 14 on a cron schedule. The Prometheus section in the dashboard shows the estimate for the projects and region setting you pick.
Set up Prometheus
Add a scrape job to prometheus.yml. Keep the token in a file that only Prometheus can read:
prometheus.yml
scrape_configs:
- job_name: onlineornot
scrape_interval: 60s
scrape_timeout: 10s
scheme: https
metrics_path: /v1/metrics/prometheus
authorization:
type: Bearer
credentials_file: /etc/prometheus/secrets/onlineornot-token
params:
per_region: ['true']
project_id: ['a1b2c3d4e5f6g7h8', 'unassigned']
static_configs:
- targets: ['api.onlineornot.com']
params is optional. Leave it out to export every monitor without region detail.
Set up Grafana Cloud
In Grafana Cloud, go to Connections, add a Metrics Endpoint connection and enter:
-
Scrape job URL, with any
project_idorper_regionparameters added:https://api.onlineornot.com/v1/metrics/prometheus -
Authentication: Bearer, with your API token
-
Scrape interval: 60 seconds
Set up Datadog
Use the Datadog Agent's OpenMetrics check. Create conf.d/openmetrics.d/conf.yaml:
conf.yaml
instances:
- openmetrics_endpoint: https://api.onlineornot.com/v1/metrics/prometheus
namespace: onlineornot
metrics:
- oon_.*
headers:
Authorization: Bearer <API token>
min_collection_interval: 60
Datadog counts these as custom metrics.
Example queries
Failure rate of each check over the last hour:
PromQL
sum by (monitor_id, check_type, name) (increase(oon_check_results_total{outcome="failing"}[1h]))
/
sum by (monitor_id, check_type, name) (increase(oon_check_results_total[1h]))
95th percentile check duration over the last hour:
PromQL
histogram_quantile(0.95, sum by (monitor_id, check_type, name, le) (rate(oon_check_duration_seconds_bucket[1h])))
Monitors that are down:
PromQL
oon_monitor_status{state="DOWN"} == 1
Timeouts and fast connection failures are included in the duration histogram, so an outage can move percentiles in either direction. Use oon_check_results_total to chart failures.
Grafana dashboard
Download the starter dashboard, then import it in Grafana from Dashboards → New → Import.
It shows how many monitors are in each state, which are down, failure ratios, response times, and when each heartbeat is next due. Every monitor links to its page in OnlineOrNot.
Use the pickers at the top to choose the data source and scrape job your OnlineOrNot metrics are in, and to filter by project, check type, monitor or region. The region filter needs per_region=true. Pick a longer window for checks that run less often than every few minutes. If your token can't read checks or heartbeats, the dashboard shows them as not exported.
Alerting
Download the example alert rules and add the file to rule_files in prometheus.yml:
prometheus.yml
rule_files:
- onlineornot-alerts.yml
- Name
OnlineOrNotMonitorDown- Type
- critical
- Description
OnlineOrNot has confirmed a check or heartbeat is down.
- Name
OnlineOrNotMultiRegionFailure- Type
- warning
- Description
A check had failing results in two or more regions in the last 15 minutes. Needs
per_region=true.
- Name
OnlineOrNotResultsStale- Type
- warning
- Description
A check hasn't reported a result for more than three check intervals. Paused checks, checks in maintenance and newly created checks are left out.
- Name
OnlineOrNotScrapeFailing- Type
- warning
- Description
Prometheus has failed to scrape OnlineOrNot for 5 minutes, for example because the token expired or was deleted. Assumes the scrape job is called
onlineornot.
Each alert includes an onlineornot_url annotation linking to the monitor in OnlineOrNot.
To write your own down alert, use oon_monitor_status{state="DOWN"} == 1. This is the same confirmed state OnlineOrNot uses for its own alerts, including for heartbeats. Use oon_heartbeat_overdue for dashboards: it can read 1 for a moment before OnlineOrNot marks the heartbeat down.
Prometheus alerts fire later than OnlineOrNot's own alerts. Results take up to a minute to appear in the export, and a 60-second scrape interval adds up to another minute.
If you alert from both OnlineOrNot and Prometheus, you'll be notified twice for the same outage. To alert only from Prometheus, remove the alert channels from those monitors in OnlineOrNot, or mute them. Muted checks keep running and keep exporting results.