confdroid/confdroid_nagios
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

OP#792 initial commit after fork

Browse Source
This commit is contained in:
Arne Teuke
2026-09-27 11:03:48 +02:00
parent 1fc33f4f1f
commit 55b0890196
1338 changed files with 428609 additions and 57 deletions
Download Patch File Download Diff File Expand all files Collapse all files

195
docs/metrics/prometheus/README.md Normal file
View File

@@ -0,0 +1,195 @@
# How to monitor MinIO server with Prometheus? [![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io)
[Prometheus](https://prometheus.io) is a cloud-native monitoring platform. Prometheus offers a multi-dimensional data model with time series data identified by metric name and key/value pairs. The data collection happens via a pull model over HTTP/HTTPS. Users looking to monitor their MinIO instances can point Prometheus configuration to scrape data from following endpoints.
- MinIO exports Prometheus compatible data by default as an authorized endpoint at `/minio/v2/metrics/cluster`.
- MinIO exports Prometheus compatible data by default which is bucket centric as an authorized endpoint at `/minio/v2/metrics/bucket`.
This document explains how to setup Prometheus and configure it to scrape data from MinIO servers.
## Prerequisites
To get started with MinIO, refer [MinIO QuickStart Document](https://docs.min.io/community/minio-object-store/operations/deployments/baremetal-deploy-minio-on-redhat-linux.html).
Follow below steps to get started with MinIO monitoring using Prometheus.
### 1. Download Prometheus
[Download the latest release](https://prometheus.io/download) of Prometheus for your platform, then extract it
```sh
tar xvfz prometheus-*.tar.gz
cd prometheus-*
```
Prometheus server is a single binary called `prometheus` (or `prometheus.exe` on Microsoft Windows). Run the binary and pass `--help` flag to see available options
```sh
./prometheus --help
usage: prometheus [<flags>]
The Prometheus monitoring server
. . .
```
Refer [Prometheus documentation](https://prometheus.io/docs/introduction/first_steps/) for more details.
### 2. Configure authentication type for Prometheus metrics
MinIO supports two authentication modes for Prometheus either `jwt` or `public`, by default MinIO runs in `jwt` mode. To allow public access without authentication for prometheus metrics set environment as follows.
```
export MINIO_PROMETHEUS_AUTH_TYPE="public"
minio server ~/test
```
### 3. Configuring Prometheus
#### 3.1 Authenticated Prometheus config
> If MinIO is configured to expose metrics without authentication, you don't need to use `mc` to generate prometheus config. You can skip reading further and move to 3.2 section.
The Prometheus endpoint in MinIO requires authentication by default. Prometheus supports a bearer token approach to authenticate prometheus scrape requests, override the default Prometheus config with the one generated using mc. To generate a Prometheus config for an alias, use [mc](https://docs.min.io/community/minio-object-store/reference/minio-mc.html#quickstart) as follows `mc admin prometheus generate <alias> [METRIC-TYPE]`. The valid values for METRIC-TYPE are `cluster`, `node`, `bucket` and `resource` and if not mentioned, it defaults to `cluster`.
The command will generate the `scrape_configs` section of the prometheus.yml as follows:
##### Cluster
```yaml
scrape_configs:
- job_name: minio-job
bearer_token: <secret>
metrics_path: /minio/v2/metrics/cluster
scheme: http
static_configs:
- targets: ['localhost:9000']
```
##### Bucket centric
```yaml
- job_name: minio-job-bucket
bearer_token: <secret>
metrics_path: /minio/v2/metrics/bucket
scheme: http
static_configs:
- targets: ['localhost:9000']
```
##### Node centric (optional)
```yaml
- job_name: minio-job-node
bearer_token: <secret>
metrics_path: /minio/v2/metrics/node
scheme: http
static_configs:
- targets: ['localhost:9000']
```
##### Resource centric (optional)
```yaml
- job_name: minio-job-resource
bearer_token: <secret>
metrics_path: /minio/v2/metrics/resource
scheme: http
static_configs:
- targets: ['localhost:9000']
```
#### 3.2 Public Prometheus config
If Prometheus endpoint authentication type is set to `public`. Following prometheus config is sufficient to start scraping metrics data from MinIO.
This can be collected from any server once per collection.
##### Cluster
```yaml
scrape_configs:
- job_name: minio-job
metrics_path: /minio/v2/metrics/cluster
scheme: http
static_configs:
- targets: ['localhost:9000']
```
#### Bucket centric
```yaml
scrape_configs:
- job_name: minio-job-bucket
metrics_path: /minio/v2/metrics/bucket
scheme: http
static_configs:
- targets: ['localhost:9000']
```
##### Node (optional)
Optionally you can also collect per node metrics. This needs to be done on a per server instance.
The scrape configurations should use all the servers under `targets` so that graphing systems like
grafana can visualize them for all the nodes
```yaml
scrape_configs:
- job_name: minio-job
metrics_path: /minio/v2/metrics/node
scheme: http
static_configs:
- targets: ['server1:9000','server2:9000','server3:9000','server4:9000']
```
##### Resource (optional)
Optionally you can also collect resource metrics.
```yaml
scrape_configs:
- job_name: minio-job
metrics_path: /minio/v2/metrics/resource
scheme: http
static_configs:
- targets: ['localhost:9000']
```
### 4. Update `scrape_configs` section in prometheus.yml
To authorize every scrape request, copy and paste the generated `scrape_configs` section in the prometheus.yml and restart the Prometheus service.
### 5. Start Prometheus
Start (or) Restart Prometheus service by running
```sh
./prometheus --config.file=prometheus.yml
```
Here `prometheus.yml` is the name of configuration file. You can now see MinIO metrics in Prometheus dashboard. By default Prometheus dashboard is accessible at `http://localhost:9090`.
Prometheus sets the `Host` header to `domain:port` as part of HTTP operations against the MinIO metrics endpoint. For MinIO deployments behind a load balancer, reverse proxy, or other control plane (HAProxy, nginx, pfsense, opnsense, etc.), ensure the network service supports routing these requests to the deployment.
### 6. Configure Grafana
After Prometheus is configured, you can use Grafana to visualize MinIO metrics. Refer the [document here to setup Grafana with MinIO prometheus metrics](https://github.com/minio/minio/blob/master/docs/metrics/prometheus/grafana/README.md).
## List of metrics exposed by MinIO
- MinIO exports Prometheus compatible data by default as an authorized endpoint at `/minio/v2/metrics/cluster`.
- MinIO exports Prometheus compatible data by default which is bucket centric as an authorized endpoint at `/minio/v2/metrics/bucket`.
- MinIO exports Prometheus compatible data by default which is node centric as an authorized endpoint at `/minio/v2/metrics/node`.
- MinIO exports Prometheus compatible data by default which is resource centric as an authorized endpoint at `/minio/v2/metrics/resource`.
All of these can be accessed via Prometheus dashboard. A sample list of exposed metrics along with their definition is available on our public demo server at
```sh
curl https://play.min.io/minio/v2/metrics/cluster
```
### List of metrics reported Cluster and Bucket level
[The list of metrics reported can be here](https://github.com/minio/minio/blob/master/docs/metrics/prometheus/list.md)
### Configure Alerts for Prometheus
[The Prometheus AlertManager and alerts can be configured following this](https://github.com/minio/minio/blob/master/docs/metrics/prometheus/alerts.md)

120
docs/metrics/prometheus/alerts.md Normal file
View File

@@ -0,0 +1,120 @@
# How to configure Prometheus AlertManager
Alerting with prometheus is two step process. First we setup alerts in Prometheus server and then we need to send alerts to the AlertManager.
Prometheus AlertManager is the component that manages sending, inhibition and silencing of the alerts generated from Prometheus. The AlertManager can be configured to send alerts to variety of receivers. Refer [Prometheus AlertManager receivers](https://prometheus.io/docs/alerting/latest/configuration/#receiver) for more details.
Follow below steps to enable and use AlertManager.
## Deploy and start AlertManager
Install Prometheus AlertManager from https://prometheus.io/download/ and create configuration as below
```yaml
route:
group_by: ['alertname']
group_wait: 30s
group_interval: 5m
repeat_interval: 1h
receiver: 'web.hook'
receivers:
- name: 'web.hook'
webhook_configs:
- url: 'http://127.0.0.1:8010/webhook'
inhibit_rules:
- source_match:
severity: 'critical'
target_match:
severity: 'warning'
equal: ['alertname', 'dev', 'instance']
```
This sample configuration uses a `webhook` at http://127.0.0.1:8010/webhook to post the alerts.
Start the AlertManager and it listens on port `9093` by default. Make sure your webhook is up and listening for the alerts.
## Configure Prometheus to use AlertManager
Add below section to your `prometheus.yml`
```yaml
alerting:
alertmanagers:
- static_configs:
- targets: ['localhost:9093']
rule_files:
- rules.yml
```
Here `rules.yml` is the file which should contain the alerting rules defined.
## Add rules for your deployment
Below is a sample alerting rules configuration for MinIO. Refer https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/ for more instructions on writing alerting rules for Prometheus.
```yaml
groups:
- name: example
rules:
- alert: MinIOClusterTolerance
expr: minio_cluster_health_erasure_set_status < 1
for: 5m
labels:
severity: critical
annotations:
summary: "Instance {{ $labels.server }} has lost quorum on pool {{ $labels.pool }} on set {{ $labels.set }}"
description: "MinIO instance {{ $labels.server }} of job {{ $labels.job }} has lost quorum on pool {{ $labels.pool }} on set {{ $labels.set }} for more than 5 minutes."
```
## Verify the configuration and alerts
To verify the above sample alert follow below steps
1. Start a distributed MinIO instance (4 nodes setup)
2. Start Prometheus server and AlertManager
3. Bring down couple of MinIO instances to bring down the Erasure Set tolerance to -1 and verify the same with `mc admin prometheus metrics ALIAS | grep minio_cluster_health_erasure_set_status`
4. Wait for 5 mins (as alert is configured to be firing after 5 mins), and verify that you see an entry in webhook for the alert as well as in Prometheus console as shown below
```json
{
"receiver": "web\\.hook",
"status": "firing",
"alerts": [
{
"status": "firing",
"labels": {
"alertname": "MinIOClusterTolerance",
"instance": "localhost:9000",
"job": "minio-job-node",
"pool": "0",
"server": "127.0.0.1:9000",
"set": "0",
"severity": "critical"
},
"annotations": {
"description": "MinIO instance 127.0.0.1:9000 of job minio-job has tolerance <=0 for more than 5 minutes.",
"summary": "Instance 127.0.0.1:9000 unable to tolerate node failures"
},
"startsAt": "2023-11-18T06:20:09.456Z",
"endsAt": "0001-01-01T00:00:00Z",
"generatorURL": "http://fedora-minio:9090/graph?g0.expr=minio_cluster_health_erasure_set_tolerance+%3C%3D+0&g0.tab=1",
"fingerprint": "2255608b0da28ca3"
}
],
"groupLabels": {
"alertname": "MinIOClusterTolerance"
},
"commonLabels": {
"alertname": "MinIOClusterTolerance",
"instance": "localhost:9000",
"job": "minio-job-node",
"pool": "0",
"server": "127.0.0.1:9000",
"set": "0",
"severity": "critical"
},
"commonAnnotations": {
"description": "MinIO instance 127.0.0.1:9000 of job minio-job has lost quorum on pool 0 on set 0 for more than 5 minutes.",
"summary": "Instance 127.0.0.1:9000 has lost quorum on pool 0 on set 0"
},
"externalURL": "http://fedora-minio:9093",
"version": "4",
"groupKey": "{}:{alertname=\"MinIOClusterTolerance\"}",
"truncatedAlerts": 0
}
```
![Prometheus](https://raw.githubusercontent.com/minio/minio/master/docs/metrics/prometheus/minio-es-tolerance-alert.png)

34
docs/metrics/prometheus/grafana/README.md Normal file
View File

@@ -0,0 +1,34 @@
# How to monitor MinIO server with Grafana [![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io)
[Grafana](https://grafana.com/) allows you to query, visualize, alert on and understand your metrics no matter where they are stored. Create, explore, and share dashboards with your team and foster a data driven culture.
## Prerequisites
- Prometheus and MinIO configured as explained in [document here](https://github.com/minio/minio/blob/master/docs/metrics/prometheus/README.md).
- Grafana installed as explained [here](https://grafana.com/grafana/download).
## MinIO Grafana Dashboard
Visualize MinIO metrics with our official Grafana dashboard available on the [Grafana dashboard portal](https://grafana.com/grafana/dashboards/13502).
Refer to the dashboard [json file here](https://raw.githubusercontent.com/minio/minio/master/docs/metrics/prometheus/grafana/minio-dashboard.json).
![Grafana](https://raw.githubusercontent.com/minio/minio/master/docs/metrics/prometheus/grafana/grafana-minio.png)
Node level Replication metrics can be viewed in the Grafana dashboard using [json file here](https://raw.githubusercontent.com/minio/minio/master/docs/metrics/prometheus/grafana/replication/minio-replication-node.json)
![Grafana](https://raw.githubusercontent.com/minio/minio/master/docs/metrics/prometheus/grafana/replication/grafana-replication-node.png)
Cluster level Replication metrics can be viewed in the Grafana dashboard using [json file here](https://raw.githubusercontent.com/minio/minio/master/docs/metrics/prometheus/grafana/replication/minio-replication-cluster.json)
![Grafana](https://raw.githubusercontent.com/minio/minio/master/docs/metrics/prometheus/grafana/replication/grafana-replication-cluster.png)
Bucket metrics can be viewed in the Grafana dashboard using [json file here](https://raw.githubusercontent.com/minio/minio/master/docs/metrics/prometheus/grafana/bucket/minio-bucket.json)
![Grafana](https://raw.githubusercontent.com/minio/minio/master/docs/metrics/prometheus/grafana/bucket/grafana-bucket.png)
Node metrics can be viewed in the Grafana dashboard using [json file here](https://raw.githubusercontent.com/minio/minio/master/docs/metrics/prometheus/grafana/node/minio-node.json)
![Grafana](https://raw.githubusercontent.com/minio/minio/master/docs/metrics/prometheus/grafana/node/grafana-node.png)
Note: All these dashboards are provided as an example and need basis they should be customized as well as new graphs should be added.

BIN
docs/metrics/prometheus/grafana/bucket/grafana-bucket.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 355 KiB

4214
docs/metrics/prometheus/grafana/bucket/minio-bucket.json Normal file
View File

File diff suppressed because it is too large Load Diff

BIN
docs/metrics/prometheus/grafana/grafana-minio.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 591 KiB

3868
docs/metrics/prometheus/grafana/minio-dashboard.json Normal file
View File

File diff suppressed because it is too large Load Diff

BIN
docs/metrics/prometheus/grafana/node/grafana-node.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 136 KiB

955
docs/metrics/prometheus/grafana/node/minio-node.json Normal file
View File

@@ -0,0 +1,955 @@
{
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": {
"type": "datasource",
"uid": "grafana"
},
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"description": "MinIO Nodes Grafana Dashboard - https://min.io/",
"editable": true,
"fiscalYearStartMonth": 0,
"gnetId": 15306,
"graphTooltip": 0,
"id": 267,
"links": [
{
"icon": "external link",
"includeVars": true,
"keepTime": true,
"tags": [
"minio"
],
"type": "dashboards"
}
],
"liveNow": false,
"panels": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"description": "",
"fieldConfig": {
"defaults": {
"mappings": [
{
"options": {
"match": "null",
"result": {
"text": "N/A"
}
},
"type": "special"
}
],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "short"
},
"overrides": []
},
"gridPos": {
"h": 5,
"w": 6,
"x": 0,
"y": 0
},
"id": 21,
"maxDataPoints": 100,
"options": {
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"reduceOptions": {
"calcs": [
"mean"
],
"fields": "",
"values": false
},
"showPercentChange": false,
"text": {},
"textMode": "auto",
"wideLayout": true
},
"pluginVersion": "10.4.0",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"exemplar": true,
"expr": "max(minio_node_drive_total{job=~\"$scrape_jobs\",server=\"$server\"})",
"format": "table",
"hide": false,
"instant": true,
"interval": "",
"intervalFactor": 1,
"legendFormat": "",
"metric": "process_start_time_seconds",
"refId": "A",
"step": 60
}
],
"title": "Total Drives",
"type": "stat"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"description": "",
"fieldConfig": {
"defaults": {
"mappings": [],
"thresholds": {
"mode": "percentage",
"steps": [
{
"color": "green",
"value": null
}
]
}
},
"overrides": []
},
"gridPos": {
"h": 5,
"w": 6,
"x": 6,
"y": 0
},
"id": 22,
"maxDataPoints": 100,
"options": {
"minVizHeight": 75,
"minVizWidth": 75,
"orientation": "auto",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"showThresholdLabels": false,
"showThresholdMarkers": true,
"sizing": "auto"
},
"pluginVersion": "10.4.0",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"exemplar": false,
"expr": "max(minio_node_drive_online_total{job=~\"$scrape_jobs\",server=\"$server\"})",
"format": "time_series",
"hide": false,
"instant": true,
"interval": "",
"intervalFactor": 1,
"legendFormat": ".",
"metric": "process_start_time_seconds",
"range": false,
"refId": "A",
"step": 60
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"exemplar": false,
"expr": "max(minio_node_drive_offline_total{job=~\"$scrape_jobs\",server=\"$server\"})",
"format": "time_series",
"hide": false,
"instant": true,
"legendFormat": ".",
"range": false,
"refId": "B"
}
],
"title": "Total Online/Offline Drives",
"type": "gauge"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"description": "",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"links": [],
"mappings": [],
"min": 0,
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "bytes"
},
"overrides": []
},
"gridPos": {
"h": 5,
"w": 6,
"x": 12,
"y": 0
},
"id": 23,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "none"
}
},
"pluginVersion": "10.4.0",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"exemplar": true,
"expr": "minio_node_drive_total_bytes{job=~\"$scrape_jobs\",server=\"$server\"}",
"interval": "",
"legendFormat": "Total [{{drive}}]",
"refId": "A"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"exemplar": true,
"expr": "minio_node_drive_used_bytes{job=~\"$scrape_jobs\",server=\"$server\"}",
"interval": "",
"legendFormat": "Used [{{drive}}]",
"refId": "B"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"exemplar": true,
"expr": "minio_node_drive_free_bytes{job=~\"$scrape_jobs\",server=\"$server\"}",
"interval": "",
"legendFormat": "Free [{{drive}}]",
"refId": "C"
}
],
"title": "Drive Usage",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"description": "",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"links": [],
"mappings": [],
"min": 0,
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "none"
},
"overrides": []
},
"gridPos": {
"h": 5,
"w": 6,
"x": 18,
"y": 0
},
"id": 24,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "none"
}
},
"pluginVersion": "10.4.0",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"exemplar": true,
"expr": "minio_node_drive_free_inodes{job=~\"$scrape_jobs\",server=\"$server\"}",
"interval": "",
"legendFormat": "[{{drive}}]",
"refId": "B"
}
],
"title": "Free Inodes",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"description": "",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"links": [],
"mappings": [],
"min": 0,
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "none"
},
"overrides": []
},
"gridPos": {
"h": 5,
"w": 6,
"x": 0,
"y": 5
},
"id": 25,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "none"
}
},
"pluginVersion": "10.4.0",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"exemplar": true,
"expr": "minio_node_drive_latency_us{job=~\"$scrape_jobs\",server=\"$server\"}",
"interval": "",
"legendFormat": "[{{drive}}:{{api}}]",
"refId": "B"
}
],
"title": "Drive Latency (micro sec)",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"description": "",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"links": [],
"mappings": [],
"min": 0,
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "none"
},
"overrides": []
},
"gridPos": {
"h": 5,
"w": 6,
"x": 6,
"y": 5
},
"id": 26,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "none"
}
},
"pluginVersion": "10.4.0",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"exemplar": true,
"expr": "minio_node_drive_errors_availability{job=~\"$scrape_jobs\",server=\"$server\"}",
"interval": "",
"legendFormat": "[{{drive}}]",
"refId": "B"
}
],
"title": "Drive Errors",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"description": "",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"links": [],
"mappings": [],
"min": 0,
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "none"
},
"overrides": [
{
"matcher": {
"id": "byName",
"options": "available 10.13.1.25:9000"
},
"properties": [
{
"id": "color",
"value": {
"fixedColor": "green",
"mode": "fixed"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "used 10.13.1.25:9000"
},
"properties": [
{
"id": "color",
"value": {
"fixedColor": "blue",
"mode": "fixed"
}
}
]
}
]
},
"gridPos": {
"h": 5,
"w": 6,
"x": 12,
"y": 5
},
"id": 27,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "none"
}
},
"pluginVersion": "10.4.0",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"exemplar": true,
"expr": "minio_node_drive_errors_timeout{job=~\"$scrape_jobs\",server=\"$server\"}",
"interval": "",
"legendFormat": "[{{drive}}]",
"refId": "B"
}
],
"title": "Drive Timeout Errors",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"description": "",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"links": [],
"mappings": [],
"min": 0,
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "none"
},
"overrides": []
},
"gridPos": {
"h": 5,
"w": 6,
"x": 18,
"y": 5
},
"id": 28,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "none"
}
},
"pluginVersion": "10.4.0",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"exemplar": true,
"expr": "minio_node_drive_io_waiting{job=~\"$scrape_jobs\",server=\"$server\"}",
"interval": "",
"legendFormat": "[{{drive}}]",
"refId": "B"
}
],
"title": "IO Operations Waiting",
"type": "timeseries"
}
],
"refresh": "",
"schemaVersion": 39,
"tags": [
"minio"
],
"templating": {
"list": [
{
"current": {
"selected": false,
"text": "All",
"value": "$__all"
},
"label": "Data source",
"name": "DS_PROMETHEUS",
"options": [],
"query": "prometheus",
"refresh": 1,
"regex": "",
"type": "datasource"
},
{
"current": {},
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"definition": "label_values(job)",
"hide": 0,
"includeAll": false,
"multi": false,
"name": "scrape_jobs",
"options": [],
"query": {
"query": "label_values(job)",
"refId": "StandardVariableQuery"
},
"refresh": 1,
"regex": "",
"skipUrlSync": false,
"sort": 0,
"type": "query"
},
{
"current": {},
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"definition": "label_values(minio_node_drive_total,server)",
"hide": 0,
"includeAll": false,
"label": "Server",
"multi": false,
"name": "server",
"options": [],
"query": {
"qryType": 1,
"query": "label_values(minio_node_drive_total,server)",
"refId": "PrometheusVariableQueryEditor-VariableQuery"
},
"refresh": 1,
"regex": "",
"skipUrlSync": false,
"sort": 0,
"type": "query"
}
]
},
"time": {
"from": "now-1h",
"to": "now"
},
"timepicker": {
"refresh_intervals": [
"10s",
"30s",
"1m",
"5m",
"15m",
"30m",
"1h",
"2h",
"1d"
],
"time_options": [
"5m",
"15m",
"1h",
"6h",
"12h",
"24h",
"2d",
"7d",
"30d"
]
},
"timezone": "",
"title": "MinIO Node Dashboard",
"uid": "TgmJnnqn2k",
"version": 1,
"weekStart": ""
}

BIN
docs/metrics/prometheus/grafana/replication/grafana-replication-cluster.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 444 KiB

BIN
docs/metrics/prometheus/grafana/replication/grafana-replication-node.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 230 KiB

2952
docs/metrics/prometheus/grafana/replication/minio-replication-cluster.json Normal file
View File

File diff suppressed because it is too large Load Diff

2397
docs/metrics/prometheus/grafana/replication/minio-replication-node.json Normal file
View File

File diff suppressed because it is too large Load Diff

471
docs/metrics/prometheus/list.md Normal file
View File

@@ -0,0 +1,471 @@
# Cluster Metrics
MinIO collects the following metrics at the cluster level.
Metrics may include one or more labels, such as the server that calculated that metric.
These metrics can be obtained from any MinIO server once per collection by using the following URL:
```shell
https://HOSTNAME:PORT/minio/v2/metrics/cluster
```
Replace ``HOSTNAME:PORT`` with the hostname of your MinIO deployment.
For deployments behind a load balancer, use the load balancer hostname instead of a single node hostname.
## Audit Metrics
| Name | Description |
|:----------------------------------|:----------------------------------------------------------|
| `minio_audit_failed_messages` | Total number of messages that failed to send since start. |
| `minio_audit_target_queue_length` | Number of unsent messages in queue for target. |
| `minio_audit_total_messages` | Total number of messages sent since start. |
## Cluster Capacity Metrics
| Name | Description |
|:---------------------------------------------|:---------------------------------------------------------------|
| `minio_cluster_capacity_raw_free_bytes` | Total free capacity online in the cluster. |
| `minio_cluster_capacity_raw_total_bytes` | Total capacity online in the cluster. |
| `minio_cluster_capacity_usable_free_bytes` | Total free usable capacity online in the cluster. |
| `minio_cluster_capacity_usable_total_bytes` | Total usable capacity online in the cluster. |
| `minio_cluster_objects_size_distribution` | Distribution of object sizes across a cluster |
| `minio_cluster_objects_version_distribution` | Distribution of object versions across a cluster |
| `minio_cluster_usage_object_total` | Total number of objects in a cluster |
| `minio_cluster_usage_total_bytes` | Total cluster usage in bytes |
| `minio_cluster_usage_version_total` | Total number of versions (includes delete marker) in a cluster |
| `minio_cluster_usage_deletemarker_total` | Total number of delete markers in a cluster |
| `minio_cluster_bucket_total` | Total number of buckets in the cluster |
## Cluster Drive Metrics
| Name | Description |
|:------------------------------------|:--------------------------------------|
| `minio_cluster_drive_offline_total` | Total drives offline in this cluster. |
| `minio_cluster_drive_online_total` | Total drives online in this cluster. |
| `minio_cluster_drive_total` | Total drives in this cluster. |
## Cluster ILM Metrics
| Name | Description |
|:------------------------------------------|:-------------------------------------------------|
| `minio_cluster_ilm_transitioned_bytes` | Total bytes transitioned to a tier. |
| `minio_cluster_ilm_transitioned_objects` | Total number of objects transitioned to a tier. |
| `minio_cluster_ilm_transitioned_versions` | Total number of versions transitioned to a tier. |
## Cluster KMS Metrics
| Name | Description |
|:------------------------------------|:-----------------------------------------------------------------------------------------|
| `minio_cluster_kms_online` | Reports whether the KMS is online (1) or offline (0). |
| `minio_cluster_kms_request_error` | Number of KMS requests that failed due to some error. (HTTP 4xx status code). |
| `minio_cluster_kms_request_failure` | Number of KMS requests that failed due to some internal failure. (HTTP 5xx status code). |
| `minio_cluster_kms_request_success` | Number of KMS requests that succeeded. |
| `minio_cluster_kms_uptime` | The time the KMS has been up and running in seconds. |
## Cluster Health Metrics
| Name | Description |
|:--------------------------------------------------|:-----------------------------------------------|
| `minio_cluster_nodes_offline_total` | Total number of MinIO nodes offline. |
| `minio_cluster_nodes_online_total` | Total number of MinIO nodes online. |
| `minio_cluster_write_quorum` | Maximum write quorum across all pools and sets |
| `minio_cluster_health_status` | Get current cluster health status |
| `minio_cluster_health_erasure_set_healing_drives` | Count of healing drives in the erasure set |
| `minio_cluster_health_erasure_set_online_drives` | Count of online drives in the erasure set |
| `minio_cluster_health_erasure_set_read_quorum` | Get read quorum of the erasure set |
| `minio_cluster_health_erasure_set_write_quorum` | Get write quorum of the erasure set |
| `minio_cluster_health_erasure_set_status` | Get current health status of the erasure set |
## Cluster Replication Metrics
Metrics marked as ``Site Replication Only`` only populate on deployments with [Site Replication](https://docs.min.io/community/minio-object-store/operations/install-deploy-manage/multi-site-replication.html) configurations.
For deployments with [bucket](https://docs.min.io/community/minio-object-store/administration/bucket-replication.html) or [batch](https://docs.min.io/community/minio-object-store/administration/batch-framework.html#replicate) configurations, these metrics populate instead under the [Bucket Metrics](#bucket-metrics) endpoint.
| Name | Description
|:-----------------------------------------------------------|:---------------------------------------------------------------------------------------------------------|
| `minio_cluster_replication_last_hour_failed_bytes` | (_Site Replication Only_) Total number of bytes failed at least once to replicate in the last full hour. |
| `minio_cluster_replication_last_hour_failed_count` | (_Site Replication Only_) Total number of objects which failed replication in the last full hour. |
| `minio_cluster_replication_last_minute_failed_bytes` | Total number of bytes failed at least once to replicate in the last full minute. |
| `minio_cluster_replication_last_minute_failed_count` | Total number of objects which failed replication in the last full minute. |
| `minio_cluster_replication_total_failed_bytes` | (_Site Replication Only_) Total number of bytes failed at least once to replicate since server start. |
| `minio_cluster_replication_total_failed_count` | (_Site Replication Only_) Total number of objects which failed replication since server start. |
| `minio_cluster_replication_received_bytes` | (_Site Replication Only_) Total number of bytes replicated to this cluster from another source cluster. |
| `minio_cluster_replication_received_count` | (_Site Replication Only_) Total number of objects received by this cluster from another source cluster. |
| `minio_cluster_replication_sent_bytes` | (_Site Replication Only_) Total number of bytes replicated to the target cluster. |
| `minio_cluster_replication_sent_count` | (_Site Replication Only_) Total number of objects replicated to the target cluster. |
| `minio_cluster_replication_credential_errors` | (_Site Replication Only_) Total number of replication credential errors since server start |
| `minio_cluster_replication_proxied_get_requests_total` | (_Site Replication Only_)Number of GET requests proxied to replication target |
| `minio_cluster_replication_proxied_head_requests_total` | (_Site Replication Only_)Number of HEAD requests proxied to replication target |
| `minio_cluster_replication_proxied_delete_tagging_requests_total` | (_Site Replication Only_)Number of DELETE tagging requests proxied to replication target |
| `minio_cluster_replication_proxied_get_tagging_requests_total` | (_Site Replication Only_)Number of GET tagging requests proxied to replication target |
| `minio_cluster_replication_proxied_put_tagging_requests_total` | (_Site Replication Only_)Number of PUT tagging requests proxied to replication target |
| `minio_cluster_replication_proxied_get_requests_failures` | (_Site Replication Only_)Number of failures in GET requests proxied to replication target |
| `minio_cluster_replication_proxied_head_requests_failures` | (_Site Replication Only_)Number of failures in HEAD requests proxied to replication target |
| `minio_cluster_replication_proxied_delete_tagging_requests_failures` | (_Site Replication Only_)Number of failures proxying DELETE tagging requests to replication target |
| `minio_cluster_replication_proxied_get_tagging_requests_failures` | (_Site Replication Only_)Number of failures proxying GET tagging requests to replication target |
| `minio_cluster_replication_proxied_put_tagging_requests_failures` | (_Site Replication Only_)Number of failures proxying PUT tagging requests to replication target |
## Node Replication Metrics
Metrics marked as ``Site Replication Only`` only populate on deployments with [Site Replication](https://docs.min.io/community/minio-object-store/operations/install-deploy-manage/multi-site-replication.html) configurations.
For deployments with [bucket](https://docs.min.io/community/minio-object-store/administration/bucket-replication.html) or [batch](https://docs.min.io/community/minio-object-store/administration/batch-framework.html#replicate) configurations, these metrics populate instead under the [Bucket Metrics](#bucket-metrics) endpoint.
| Name | Description
|:-----------------------------------------------------------|:---------------------------------------------------------------------------------------------------------|
| `minio_node_replication_current_active_workers` | Total number of active replication workers |
| `minio_node_replication_average_active_workers` | Average number of active replication workers |
| `minio_node_replication_max_active_workers` | Maximum number of active replication workers seen since server start |
| `minio_node_replication_link_online` | Reports whether the replication link is online (1) or offline (0). |
| `minio_node_replication_link_offline_duration_seconds` | Total duration of replication link being offline in seconds since last offline event |
| `minio_node_replication_link_downtime_duration_seconds` | Total downtime of replication link in seconds since server start |
| `minio_node_replication_average_link_latency_ms` | Average replication link latency in milliseconds |
| `minio_node_replication_max_link_latency_ms` | Maximum replication link latency in milliseconds seen since server start |
| `minio_node_replication_current_link_latency_ms` | Current replication link latency in milliseconds |
| `minio_node_replication_current_transfer_rate` | Current replication transfer rate in bytes/sec |
| `minio_node_replication_average_transfer_rate` | Average replication transfer rate in bytes/sec |
| `minio_node_replication_max_transfer_rate` | Maximum replication transfer rate in bytes/sec seen since server start |
| `minio_node_replication_last_minute_queued_count` | Total number of objects queued for replication in the last full minute |
| `minio_node_replication_last_minute_queued_bytes` | Total number of bytes queued for replication in the last full minute |
| `minio_node_replication_average_queued_count` | Average number of objects queued for replication since server start |
| `minio_node_replication_average_queued_bytes` | Average number of bytes queued for replication since server start |
| `minio_node_replication_max_queued_bytes` | Maximum number of bytes queued for replication seen since server start |
| `minio_node_replication_max_queued_count` | Maximum number of objects queued for replication seen since server start |
| `minio_node_replication_recent_backlog_count` | Total number of objects seen in replication backlog in the last 5 minutes |
## Healing Metrics
| Name | Description |
|:---------------------------------------------|:-----------------------------------------------------------------|
| `minio_heal_objects_errors_total` | Objects for which healing failed in current self healing run. |
| `minio_heal_objects_heal_total` | Objects healed in current self healing run. |
| `minio_heal_objects_total` | Objects scanned in current self healing run. |
| `minio_heal_time_last_activity_nano_seconds` | Time elapsed (in nano seconds) since last self healing activity. |
## Inter Node Metrics
| Name | Description |
|:------------------------------------------|:--------------------------------------------------------|
| `minio_inter_node_traffic_dial_avg_time` | Average time of internodes TCP dial calls. |
| `minio_inter_node_traffic_dial_errors` | Total number of internode TCP dial timeouts and errors. |
| `minio_inter_node_traffic_errors_total` | Total number of failed internode calls. |
| `minio_inter_node_traffic_received_bytes` | Total number of bytes received from other peer nodes. |
| `minio_inter_node_traffic_sent_bytes` | Total number of bytes sent to the other peer nodes. |
## Bucket Notification Metrics
| Name | Description |
|:-----------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------|
| `minio_notify_current_send_in_progress` | Number of concurrent async Send calls active to all targets (deprecated, please use `minio_notify_target_current_send_in_progress` instead) |
| `minio_notify_events_errors_total` | Events that were failed to be sent to the targets (deprecated, please use `minio_notify_target_failed_events` instead) |
| `minio_notify_events_sent_total` | Total number of events sent to the targets (deprecated, please use `minio_notify_target_total_events` instead) |
| `minio_notify_events_skipped_total` | Events that were skipped to be sent to the targets due to the in-memory queue being full |
| `minio_notify_target_current_send_in_progress` | Number of concurrent async Send calls active to the target |
| `minio_notify_target_queue_length` | Number of events currently staged in the queue_dir configured for the target. |
| `minio_notify_target_total_events` | Total number of events sent (or) queued to the target |
## S3 API Request Metrics
| Name | Description |
|:----------------------------------------------|:---------------------------------------------------------|
| `minio_s3_requests_4xx_errors_total` | Total number S3 requests with (4xx) errors. |
| `minio_s3_requests_5xx_errors_total` | Total number S3 requests with (5xx) errors. |
| `minio_s3_requests_canceled_total` | Total number S3 requests canceled by the client. |
| `minio_s3_requests_errors_total` | Total number S3 requests with (4xx and 5xx) errors. |
| `minio_s3_requests_incoming_total` | Volatile number of total incoming S3 requests. |
| `minio_s3_requests_inflight_total` | Total number of S3 requests currently in flight. |
| `minio_s3_requests_rejected_auth_total` | Total number S3 requests rejected for auth failure. |
| `minio_s3_requests_rejected_header_total` | Total number S3 requests rejected for invalid header. |
| `minio_s3_requests_rejected_invalid_total` | Total number S3 invalid requests. |
| `minio_s3_requests_rejected_timestamp_total` | Total number S3 requests rejected for invalid timestamp. |
| `minio_s3_requests_total` | Total number S3 requests. |
| `minio_s3_requests_waiting_total` | Number of S3 requests in the waiting queue. |
| `minio_s3_requests_ttfb_seconds_distribution` | Distribution of the time to first byte across API calls. |
| `minio_s3_traffic_received_bytes` | Total number of s3 bytes received. |
| `minio_s3_traffic_sent_bytes` | Total number of s3 bytes sent. |
## Software Metrics
| Name | Description |
|:------------------------------|:---------------------------------------|
| `minio_software_commit_info` | Git commit hash for the MinIO release. |
| `minio_software_version_info` | MinIO Release tag for the server. |
## Drive Metrics
| Name | Description |
|:---------------------------------------|:--------------------------------------------------------------------|
| `minio_node_drive_free_bytes` | Total storage available on a drive. |
| `minio_node_drive_free_inodes` | Total free inodes. |
| `minio_node_drive_latency_us` | Average last minute latency in µs for drive API storage operations. |
| `minio_node_drive_offline_total` | Total drives offline in this node. |
| `minio_node_drive_online_total` | Total drives online in this node. |
| `minio_node_drive_total` | Total drives in this node. |
| `minio_node_drive_total_bytes` | Total storage on a drive. |
| `minio_node_drive_used_bytes` | Total storage used on a drive. |
| `minio_node_drive_errors_timeout` | Total number of drive timeout errors since server start |
| `minio_node_drive_errors_ioerror` | Total number of drive I/O errors since server start |
| `minio_node_drive_errors_availability` | Total number of drive I/O errors, timeouts since server start |
| `minio_node_drive_io_waiting` | Total number I/O operations waiting on drive |
## Identity and Access Management (IAM) Metrics
| Name | Description |
|:-------------------------------------------|:------------------------------------------------------------|
| `minio_node_iam_last_sync_duration_millis` | Last successful IAM data sync duration in milliseconds. |
| `minio_node_iam_since_last_sync_millis` | Time (in milliseconds) since last successful IAM data sync. |
| `minio_node_iam_sync_failures` | Number of failed IAM data syncs since server start. |
| `minio_node_iam_sync_successes` | Number of successful IAM data syncs since server start. |
## Information Lifecycle Management (ILM) Metrics
| Name | Description |
|:-------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------|
| `minio_node_ilm_expiry_pending_tasks` | Number of pending ILM expiry tasks in the queue. |
| `minio_node_ilm_transition_active_tasks` | Number of active ILM transition tasks. |
| `minio_node_ilm_transition_pending_tasks` | Number of pending ILM transition tasks in the queue. |
| `minio_node_ilm_transition_missed_immediate_tasks` | Number of missed immediate ILM transition tasks. |
| `minio_node_ilm_versions_scanned` | Total number of object versions checked for ilm actions since server start. |
| `minio_node_ilm_action_count_delete_action` | Total action outcome of lifecycle checks since server start for deleting object |
| `minio_node_ilm_action_count_delete_version_action` | Total action outcome of lifecycle checks since server start for deleting a version |
| `minio_node_ilm_action_count_transition_action` | Total action outcome of lifecycle checks since server start for transition of an object |
| `minio_node_ilm_action_count_transition_version_action` | Total action outcome of lifecycle checks since server start for transition of a particular object version |
| `minio_node_ilm_action_count_delete_restored_action` | Total action outcome of lifecycle checks since server start for deletion of temporarily restored object |
| `minio_node_ilm_action_count_delete_restored_version_action` | Total action outcome of lifecycle checks since server start for deletion of a temporarily restored version |
| `minio_node_ilm_action_count_delete_all_versions_action` | Total action outcome of lifecycle checks since server start for deletion of all versions |
## Tier Metrics
| Name | Description |
|:---------------------------------------------------|:----------------------------------------------------------------------------|
| `minio_node_tier_tier_ttlb_seconds_distribution` | Distribution of time to last byte for objects downloaded from warm tier |
| `minio_node_tier_requests_success` | Number of requests to download object from warm tier that were successful |
| `minio_node_tier_requests_failure` | Number of requests to download object from warm tier that were failure |
## System Metrics
| Name | Description |
|:-------------------------------------------|:----------------------------------------------------------------------------------------------------------------|
| `minio_node_file_descriptor_limit_total` | Limit on total number of open file descriptors for the MinIO Server process. |
| `minio_node_file_descriptor_open_total` | Total number of open file descriptors by the MinIO Server process. |
| `minio_node_go_routine_total` | Total number of go routines running. |
| `minio_node_io_rchar_bytes` | Total bytes read by the process from the underlying storage system including cache, /proc/[pid]/io rchar. |
| `minio_node_io_read_bytes` | Total bytes read by the process from the underlying storage system, /proc/[pid]/io read_bytes. |
| `minio_node_io_wchar_bytes` | Total bytes written by the process to the underlying storage system including page cache, /proc/[pid]/io wchar. |
| `minio_node_io_write_bytes` | Total bytes written by the process to the underlying storage system, /proc/[pid]/io write_bytes. |
| `minio_node_process_cpu_total_seconds` | Total user and system CPU time spent in seconds by the process. |
| `minio_node_process_resident_memory_bytes` | Resident memory size in bytes. |
| `minio_node_process_virtual_memory_bytes` | Virtual memory size in bytes. |
| `minio_node_process_starttime_seconds` | Start time for MinIO process per node, time in seconds since Unix epoc. |
| `minio_node_process_uptime_seconds` | Uptime for MinIO process per node in seconds. |
## Scanner Metrics
| Name | Description |
|:-------------------------------------------|:------------------------------------------------------------|
| `minio_node_scanner_bucket_scans_finished` | Total number of bucket scans finished since server start. |
| `minio_node_scanner_bucket_scans_started` | Total number of bucket scans started since server start. |
| `minio_node_scanner_directories_scanned` | Total number of directories scanned since server start. |
| `minio_node_scanner_objects_scanned` | Total number of unique objects scanned since server start. |
| `minio_node_scanner_versions_scanned` | Total number of object versions scanned since server start. |
| `minio_node_syscall_read_total` | Total read SysCalls to the kernel. /proc/[pid]/io syscr. |
| `minio_node_syscall_write_total` | Total write SysCalls to the kernel. /proc/[pid]/io syscw. |
| `minio_usage_last_activity_nano_seconds` | Time elapsed (in nano seconds) since last scan activity. |
# Bucket Metrics
MinIO collects the following metrics at the bucket level.
Each metric includes the ``bucket`` label to identify the corresponding bucket.
Metrics may include one or more additional labels, such as the server that calculated that metric.
These metrics can be obtained from any MinIO server once per collection by using the following URL:
```shell
https://HOSTNAME:PORT/minio/v2/metrics/bucket
```
Replace ``HOSTNAME:PORT`` with the hostname of your MinIO deployment.
For deployments behind a load balancer, use the load balancer hostname instead of a single node hostname.
## Distribution Metrics
| Name | Description |
|:--------------------------------------------|:--------------------------------------------------------------------------------|
| `minio_bucket_objects_size_distribution` | Distribution of object sizes in the bucket, includes label for the bucket name. |
| `minio_bucket_objects_version_distribution` | Distribution of object sizes in a bucket, by number of versions |
## Replication Metrics
These metrics only populate on deployments with [Bucket Replication](https://docs.min.io/community/minio-object-store/administration/bucket-replication.html) or [Batch Replication](https://docs.min.io/community/minio-object-store/administration/batch-framework.html) configurations.
For deployments with [Site Replication](https://docs.min.io/community/minio-object-store/operations/install-deploy-manage/multi-site-replication.html) configured, select metrics populate under the [Cluster Metrics](#cluster-metrics) endpoint.
| Name | Description |
|:----------------------------------------------------|:---------------------------------------------------------------------------------|
| `minio_bucket_replication_last_minute_failed_bytes` | Total number of bytes failed at least once to replicate in the last full minute. |
| `minio_bucket_replication_last_minute_failed_count` | Total number of objects which failed replication in the last full minute. |
| `minio_bucket_replication_last_hour_failed_bytes` | Total number of bytes failed at least once to replicate in the last full hour. |
| `minio_bucket_replication_last_hour_failed_count` | Total number of objects which failed replication in the last full hour. |
| `minio_bucket_replication_total_failed_bytes` | Total number of bytes failed at least once to replicate since server start. |
| `minio_bucket_replication_total_failed_count` | Total number of objects which failed replication since server start. |
| `minio_bucket_replication_latency_ms` | Replication latency in milliseconds. |
| `minio_bucket_replication_received_bytes` | Total number of bytes replicated to this bucket from another source bucket. |
| `minio_bucket_replication_received_count` | Total number of objects received by this bucket from another source bucket. |
| `minio_bucket_replication_sent_bytes` | Total number of bytes replicated to the target bucket. |
| `minio_bucket_replication_sent_count` | Total number of objects replicated to the target bucket. |
| `minio_bucket_replication_credential_errors` | Total number of replication credential errors since server start |
| `minio_bucket_replication_proxied_get_requests_total` | Number of GET requests proxied to replication target |
| `minio_bucket_replication_proxied_head_requests_total` | Number of HEAD requests proxied to replication target |
| `minio_bucket_replication_proxied_delete_tagging_requests_total` | Number of DELETE tagging requests proxied to replication target |
| `minio_bucket_replication_proxied_get_tagging_requests_total` | Number of GET tagging requests proxied to replication target |
| `minio_bucket_replication_proxied_put_tagging_requests_total` | Number of PUT tagging requests proxied to replication target |
| `minio_bucket_replication_proxied_get_requests_failures` | Number of failures in GET requests proxied to replication target |
| `minio_bucket_replication_proxied_head_requests_failures` | Number of failures in HEAD requests proxied to replication target |
| `minio_bucket_replication_proxied_delete_tagging_requests_failures` | Number of failures in DELETE tagging proxy requests to replication target |
| `minio_bucket_replication_proxied_get_tagging_requests_failures` |Number of failures in GET tagging proxy requests to replication target |
| `minio_bucket_replication_proxied_put_tagging_requests_failures` | Number of failures in PUT tagging proxy requests to replication target |
## Traffic Metrics
| Name | Description |
|:--------------------------------------|:---------------------------------------------------|
| `minio_bucket_traffic_received_bytes` | Total number of S3 bytes received for this bucket. |
| `minio_bucket_traffic_sent_bytes` | Total number of S3 bytes sent for this bucket. |
## Usage Metrics
| Name | Description |
|:----------------------------------------|:--------------------------------------------------|
| `minio_bucket_usage_object_total` | Total number of objects. |
| `minio_bucket_usage_version_total` | Total number of versions (includes delete marker) |
| `minio_bucket_usage_deletemarker_total` | Total number of delete markers. |
| `minio_bucket_usage_total_bytes` | Total bucket size in bytes. |
| `minio_bucket_quota_total_bytes` | Total bucket quota size in bytes. |
## Requests Metrics
| Name | Description |
|:--------------------------------------------------|:----------------------------------------------------------------|
| `minio_bucket_requests_4xx_errors_total` | Total number of S3 requests with (4xx) errors on a bucket. |
| `minio_bucket_requests_5xx_errors_total` | Total number of S3 requests with (5xx) errors on a bucket. |
| `minio_bucket_requests_inflight_total` | Total number of S3 requests currently in flight on a bucket. |
| `minio_bucket_requests_total` | Total number of S3 requests on a bucket. |
| `minio_bucket_requests_canceled_total` | Total number S3 requests canceled by the client. |
| `minio_bucket_requests_ttfb_seconds_distribution` | Distribution of time to first byte across API calls per bucket. |
# Resource Metrics
MinIO collects the following resource metrics at the node level.
Each metric includes the `server` label to identify the corresponding node.
Metrics may include one or more additional labels, such as the drive path, interface name, etc.
These metrics can be obtained from any MinIO server once per collection by using the following URL:
```shell
https://HOSTNAME:PORT/minio/v2/metrics/resource
```
Replace `HOSTNAME:PORT` with the hostname of your MinIO deployment.
For deployments behind a load balancer, use the load balancer hostname instead of a single node hostname.
## Drive Resource Metrics
| Name | Description |
|:-------------------------------------|:---------------------------------------------------------|
| `minio_node_drive_total_bytes` | Total bytes on a drive. |
| `minio_node_drive_used_bytes` | Used bytes on a drive. |
| `minio_node_drive_total_inodes` | Total inodes on a drive. |
| `minio_node_drive_used_inodes` | Total inodes used on a drive. |
| `minio_node_drive_reads_per_sec` | Reads per second on a drive. |
| `minio_node_drive_reads_kb_per_sec` | Kilobytes read per second on a drive. |
| `minio_node_drive_reads_await` | Average time for read requests to be served on a drive. |
| `minio_node_drive_writes_per_sec` | Writes per second on a drive. |
| `minio_node_drive_writes_kb_per_sec` | Kilobytes written per second on a drive. |
| `minio_node_drive_writes_await` | Average time for write requests to be served on a drive. |
| `minio_node_drive_perc_util` | Percentage of time the disk was busy since uptime. |
## Network Interface Metrics
| Name | Description |
|:------------------------------|:-----------------------------------------------------------|
| `minio_node_if_rx_bytes` | Bytes received on the interface in 60s. |
| `minio_node_if_rx_bytes_avg` | Bytes received on the interface in 60s (avg) since uptime. |
| `minio_node_if_rx_bytes_max` | Bytes received on the interface in 60s (max) since uptime. |
| `minio_node_if_rx_errors` | Receive errors in 60s. |
| `minio_node_if_rx_errors_avg` | Receive errors in 60s (avg). |
| `minio_node_if_rx_errors_max` | Receive errors in 60s (max). |
| `minio_node_if_tx_bytes` | Bytes transmitted in 60s. |
| `minio_node_if_tx_bytes_avg` | Bytes transmitted in 60s (avg). |
| `minio_node_if_tx_bytes_max` | Bytes transmitted in 60s (max). |
| `minio_node_if_tx_errors` | Transmit errors in 60s. |
| `minio_node_if_tx_errors_avg` | Transmit errors in 60s (avg). |
| `minio_node_if_tx_errors_max` | Transmit errors in 60s (max). |
## CPU Metrics
| Name | Description |
|:-------------------------------------|:-------------------------------------------|
| `minio_node_cpu_avg_user` | CPU user time. |
| `minio_node_cpu_avg_user_avg` | CPU user time (avg). |
| `minio_node_cpu_avg_user_max` | CPU user time (max). |
| `minio_node_cpu_avg_system` | CPU system time. |
| `minio_node_cpu_avg_system_avg` | CPU system time (avg). |
| `minio_node_cpu_avg_system_max` | CPU system time (max). |
| `minio_node_cpu_avg_idle` | CPU idle time. |
| `minio_node_cpu_avg_idle_avg` | CPU idle time (avg). |
| `minio_node_cpu_avg_idle_max` | CPU idle time (max). |
| `minio_node_cpu_avg_iowait` | CPU ioWait time. |
| `minio_node_cpu_avg_iowait_avg` | CPU ioWait time (avg). |
| `minio_node_cpu_avg_iowait_max` | CPU ioWait time (max). |
| `minio_node_cpu_avg_nice` | CPU nice time. |
| `minio_node_cpu_avg_nice_avg` | CPU nice time (avg). |
| `minio_node_cpu_avg_nice_max` | CPU nice time (max). |
| `minio_node_cpu_avg_steal` | CPU steam time. |
| `minio_node_cpu_avg_steal_avg` | CPU steam time (avg). |
| `minio_node_cpu_avg_steal_max` | CPU steam time (max). |
| `minio_node_cpu_avg_load1` | CPU load average 1min. |
| `minio_node_cpu_avg_load1_avg` | CPU load average 1min (avg). |
| `minio_node_cpu_avg_load1_max` | CPU load average 1min (max). |
| `minio_node_cpu_avg_load1_perc` | CPU load average 1min (percentage). |
| `minio_node_cpu_avg_load1_perc_avg` | CPU load average 1min (percentage) (avg). |
| `minio_node_cpu_avg_load1_perc_max` | CPU load average 1min (percentage) (max). |
| `minio_node_cpu_avg_load5` | CPU load average 5min. |
| `minio_node_cpu_avg_load5_avg` | CPU load average 5min (avg). |
| `minio_node_cpu_avg_load5_max` | CPU load average 5min (max). |
| `minio_node_cpu_avg_load5_perc` | CPU load average 5min (percentage). |
| `minio_node_cpu_avg_load5_perc_avg` | CPU load average 5min (percentage) (avg). |
| `minio_node_cpu_avg_load5_perc_max` | CPU load average 5min (percentage) (max). |
| `minio_node_cpu_avg_load15` | CPU load average 15min. |
| `minio_node_cpu_avg_load15_avg` | CPU load average 15min (avg). |
| `minio_node_cpu_avg_load15_max` | CPU load average 15min (max). |
| `minio_node_cpu_avg_load15_perc` | CPU load average 15min (percentage). |
| `minio_node_cpu_avg_load15_perc_avg` | CPU load average 15min (percentage) (avg). |
| `minio_node_cpu_avg_load15_perc_max` | CPU load average 15min (percentage) (max). |
## Memory Metrics
| Name | Description |
|:-------------------------------|:------------------------------------------|
| `minio_node_mem_available` | Available memory on the node. |
| `minio_node_mem_available_avg` | Available memory on the node (avg). |
| `minio_node_mem_available_max` | Available memory on the node (max). |
| `minio_node_mem_buffers` | Buffers memory on the node. |
| `minio_node_mem_buffers_avg` | Buffers memory on the node (avg). |
| `minio_node_mem_buffers_max` | Buffers memory on the node (max). |
| `minio_node_mem_cache` | Cache memory on the node. |
| `minio_node_mem_cache_avg` | Cache memory on the node (avg). |
| `minio_node_mem_cache_max` | Cache memory on the node (max). |
| `minio_node_mem_free` | Free memory on the node. |
| `minio_node_mem_free_avg` | Free memory on the node (avg). |
| `minio_node_mem_free_max` | Free memory on the node (max). |
| `minio_node_mem_shared` | Shared memory on the node. |
| `minio_node_mem_shared_avg` | Shared memory on the node (avg). |
| `minio_node_mem_shared_max` | Shared memory on the node (max). |
| `minio_node_mem_total` | Total memory on the node. |
| `minio_node_mem_total_avg` | Total memory on the node (avg). |
| `minio_node_mem_total_max` | Total memory on the node (max). |
| `minio_node_mem_used` | Used memory on the node. |
| `minio_node_mem_used_avg` | Used memory on the node (avg). |
| `minio_node_mem_used_max` | Used memory on the node (max). |
| `minio_node_mem_used_perc` | Used memory percentage on the node. |
| `minio_node_mem_used_perc_avg` | Used memory percentage on the node (avg). |
| `minio_node_mem_used_perc_max` | Used memory percentage on the node (max). |

BIN
docs/metrics/prometheus/minio-es-tolerance-alert.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 80 KiB