GET/v1/metrics/prometheus

Export metrics for Prometheus

Returns the current state, result counts and response times of your checks, and the ping history and schedule of your heartbeats, in the Prometheus text format (version 0.0.4). Point Prometheus, Grafana Cloud, the Datadog Agent or another Prometheus-compatible scraper at this endpoint to chart OnlineOrNot data beside your own metrics.

Prometheus export requires a paid plan or trial.

The easiest way to get started is Settings → Integrations → Prometheus in the OnlineOrNot dashboard. It creates a token and gives you a ready-to-paste Prometheus configuration.

Authentication

Send an API token in the Authorization header. OAuth access tokens and session cookies are not accepted.

Token permissionExports
UPTIME_CHECKS:READUptime, browser, DNS and TCP checks
HEARTBEAT_CHECKS:READHeartbeats
BothChecks and heartbeats

EDIT includes READ. A token with only one of the two permissions still returns a successful export, and oon_export_info says which monitors were left out.

Query parameters

  • Name
    project_id
    Type
    string
    Description

    Only export monitors in this project. Repeat it to select several projects, up to 100. Use unassigned for monitors that aren't in a project. Leave it out to export every monitor.

  • Name
    per_region
    Type
    boolean
    Description

    true adds a region label to oon_check_results_total and oon_check_duration_seconds, with one set of series per region. Defaults to false, which combines every region into one set of series.

Any other parameter returns 400.

Responses

StatusBodyMeaning
200MetricsPrometheus text format, Content-Type: text/plain; version=0.0.4; charset=utf-8
400invalid requestUnknown parameter, invalid per_region value, or a project that doesn't exist
401unauthenticatedMissing, invalid, deleted or expired API token
403insufficient permissionsThe token can't read checks or heartbeats
403Prometheus export requires a paid planThe organization is on the free plan
429rate limit exceededMore than 10 requests in a minute with this token. Retry after the Retry-After header
503metrics unavailableMetrics can't be read right now. Prometheus records a failed scrape and tries again at the next interval

A failed request never returns partial metrics.

Request

GET
/v1/metrics/prometheus
curl --request GET \
  --url 'https://api.onlineornot.com/v1/metrics/prometheus?per_region=true' \
  --header 'Authorization: Bearer {access-token}'

Response (excerpt)

# HELP oon_export_info Always 1. Says whether checks and heartbeats were included, based on the API token permissions.
# TYPE oon_export_info gauge
oon_export_info{checks="included",heartbeats="included"} 1
# HELP oon_monitor_status Current monitor state, 1 for the active state and 0 for the others.
# TYPE oon_monitor_status gauge
oon_monitor_status{monitor_id="a1b2c3d4",check_type="UPTIME",name="Checkout",state="UP"} 1
oon_monitor_status{monitor_id="a1b2c3d4",check_type="UPTIME",name="Checkout",state="DOWN"} 0
# HELP oon_check_results_total Check results by outcome since metrics collection began.
# TYPE oon_check_results_total counter
oon_check_results_total{monitor_id="a1b2c3d4",check_type="UPTIME",name="Checkout",region="us-east-1",outcome="passing"} 1440
oon_check_results_total{monitor_id="a1b2c3d4",check_type="UPTIME",name="Checkout",region="us-east-1",outcome="failing"} 2

Metrics

Every monitor metric has these labels:

  • Name
    monitor_id
    Type
    string
    Description

    The check or heartbeat ID, the same ID used in the API and in dashboard URLs.

  • Name
    check_type
    Type
    string
    Description

    UPTIME, BROWSER, DNS, TCP or HEARTBEAT. A check and a heartbeat can share an ID, so use monitor_id and check_type together.

  • Name
    name
    Type
    string
    Description

    The monitor name. Renaming a monitor starts new series under the new name, so group by monitor_id and check_type in queries and alerts that should survive a rename.

  • Name
    oon_export_info
    Type
    gauge
    Description

    Always 1. The checks and heartbeats labels are included, or no_permission when the token can't read them.

  • Name
    oon_monitor_info
    Type
    gauge
    Description

    Always 1. Adds project_id and project_name labels. Monitors without a project have project_id="unassigned".

  • Name
    oon_monitor_status
    Type
    gauge
    Description

    1 for the monitor's current state and 0 for the others. state is one of UP, DOWN, PENDING, PAUSED, MUTED, MAINTENANCE, RECOVERING or VERIFYING.

  • Name
    oon_monitor_last_result_timestamp_seconds
    Type
    gauge
    Description

    Unix time of the latest check result, or of the latest heartbeat ping.

  • Name
    oon_check_interval_seconds
    Type
    gauge
    Description

    How often the check runs.

  • Name
    oon_check_results_total
    Type
    counter
    Description

    Check results, with outcome="passing" or outcome="failing".

  • Name
    oon_check_duration_seconds
    Type
    histogram
    Description

    How long each check took, for passing and failing results. Buckets from 5 milliseconds to 10 minutes.

  • Name
    oon_heartbeat_pings_total
    Type
    counter
    Description

    Pings received.

  • Name
    oon_heartbeat_expected_interval_seconds
    Type
    gauge
    Description

    How often the heartbeat should ping. Not present for heartbeats on a cron schedule.

  • Name
    oon_heartbeat_grace_period_seconds
    Type
    gauge
    Description

    How long after the due time OnlineOrNot waits before marking the heartbeat down.

  • Name
    oon_heartbeat_next_due_timestamp_seconds
    Type
    gauge
    Description

    Unix time the next ping is due, not including the grace period.

  • Name
    oon_heartbeat_overdue
    Type
    gauge
    Description

    1 when the heartbeat is past its due time plus grace period, otherwise 0.

Counters start at zero when Prometheus export is turned on, by creating a token in the dashboard or by the first successful scrape, and don't include earlier results. If nothing scrapes the endpoint for 30 days, OnlineOrNot stops counting until the next successful scrape, and counters can start again from zero. Prometheus treats that as a counter reset, so rate() and increase() still work.

New results appear within about a minute. Each export reflects the latest results at the time of the scrape. If OnlineOrNot itself is having problems, counters and last-result times can stop updating for a while.

Number of series

Metrics services often charge by the number of series. Each check produces 32 series, or 11 plus 21 for each region when per_region=true. Each heartbeat produces 15 series, or 14 on a cron schedule. The Prometheus section in the dashboard shows the estimate for the projects and region setting you pick.

Set up Prometheus

Add a scrape job to prometheus.yml. Keep the token in a file that only Prometheus can read:

prometheus.yml

scrape_configs:
  - job_name: onlineornot
    scrape_interval: 60s
    scrape_timeout: 10s
    scheme: https
    metrics_path: /v1/metrics/prometheus
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/secrets/onlineornot-token
    params:
      per_region: ['true']
      project_id: ['a1b2c3d4e5f6g7h8', 'unassigned']
    static_configs:
      - targets: ['api.onlineornot.com']

params is optional. Leave it out to export every monitor without region detail.

Set up Grafana Cloud

In Grafana Cloud, go to Connections, add a Metrics Endpoint connection and enter:

  • Scrape job URL, with any project_id or per_region parameters added:

    https://api.onlineornot.com/v1/metrics/prometheus
    
  • Authentication: Bearer, with your API token

  • Scrape interval: 60 seconds

Set up Datadog

Use the Datadog Agent's OpenMetrics check. Create conf.d/openmetrics.d/conf.yaml:

conf.yaml

instances:
  - openmetrics_endpoint: https://api.onlineornot.com/v1/metrics/prometheus
    namespace: onlineornot
    metrics:
      - oon_.*
    headers:
      Authorization: Bearer <API token>
    min_collection_interval: 60

Datadog counts these as custom metrics.

Example queries

Failure rate of each check over the last hour:

PromQL

sum by (monitor_id, check_type, name) (increase(oon_check_results_total{outcome="failing"}[1h]))
  /
sum by (monitor_id, check_type, name) (increase(oon_check_results_total[1h]))

95th percentile check duration over the last hour:

PromQL

histogram_quantile(0.95, sum by (monitor_id, check_type, name, le) (rate(oon_check_duration_seconds_bucket[1h])))

Monitors that are down:

PromQL

oon_monitor_status{state="DOWN"} == 1

Timeouts and fast connection failures are included in the duration histogram, so an outage can move percentiles in either direction. Use oon_check_results_total to chart failures.

Grafana dashboard

Download the starter dashboard, then import it in Grafana from Dashboards → New → Import.

It shows how many monitors are in each state, which are down, failure ratios, response times, and when each heartbeat is next due. Every monitor links to its page in OnlineOrNot.

Use the pickers at the top to choose the data source and scrape job your OnlineOrNot metrics are in, and to filter by project, check type, monitor or region. The region filter needs per_region=true. Pick a longer window for checks that run less often than every few minutes. If your token can't read checks or heartbeats, the dashboard shows them as not exported.

Alerting

Download the example alert rules and add the file to rule_files in prometheus.yml:

prometheus.yml

rule_files:
  - onlineornot-alerts.yml
  • Name
    OnlineOrNotMonitorDown
    Type
    critical
    Description

    OnlineOrNot has confirmed a check or heartbeat is down.

  • Name
    OnlineOrNotMultiRegionFailure
    Type
    warning
    Description

    A check had failing results in two or more regions in the last 15 minutes. Needs per_region=true.

  • Name
    OnlineOrNotResultsStale
    Type
    warning
    Description

    A check hasn't reported a result for more than three check intervals. Paused checks, checks in maintenance and newly created checks are left out.

  • Name
    OnlineOrNotScrapeFailing
    Type
    warning
    Description

    Prometheus has failed to scrape OnlineOrNot for 5 minutes, for example because the token expired or was deleted. Assumes the scrape job is called onlineornot.

Each alert includes an onlineornot_url annotation linking to the monitor in OnlineOrNot.

To write your own down alert, use oon_monitor_status{state="DOWN"} == 1. This is the same confirmed state OnlineOrNot uses for its own alerts, including for heartbeats. Use oon_heartbeat_overdue for dashboards: it can read 1 for a moment before OnlineOrNot marks the heartbeat down.

Prometheus alerts fire later than OnlineOrNot's own alerts. Results take up to a minute to appear in the export, and a 60-second scrape interval adds up to another minute.

If you alert from both OnlineOrNot and Prometheus, you'll be notified twice for the same outage. To alert only from Prometheus, remove the alert channels from those monitors in OnlineOrNot, or mute them. Muted checks keep running and keep exporting results.

Was this page helpful?