OP#792 initial commit after fork
This commit is contained in:
32
docs/metrics/README.md
Normal file
32
docs/metrics/README.md
Normal file
@@ -0,0 +1,32 @@
|
||||
# MinIO Monitoring Guide
|
||||
|
||||
MinIO server exposes monitoring data over endpoints. Monitoring tools can pick the data from these endpoints. This document lists the monitoring endpoints and relevant documentation.
|
||||
|
||||
## Healthcheck Probe
|
||||
|
||||
MinIO server has two healthcheck related un-authenticated endpoints, a liveness probe to indicate if server is responding, cluster probe to check if server can be taken down for maintenance.
|
||||
|
||||
- Liveness probe available at `/minio/health/live`
|
||||
- Cluster probe available at `/minio/health/cluster`
|
||||
|
||||
Read more on how to use these endpoints in [MinIO healthcheck guide](https://github.com/minio/minio/blob/master/docs/metrics/healthcheck/README.md).
|
||||
|
||||
## Prometheus Probe
|
||||
|
||||
MinIO allows reading metrics for the entire cluster from any single node. This allows for metrics collection for a MinIO instance across all servers. Thus, metrics collection for instances behind a load balancer can be done without any knowledge of the individual node addresses. The cluster wide metrics can be read at
|
||||
`<Address for MinIO Service>/minio/v2/metrics/cluster`.
|
||||
|
||||
The additional node specific metrics which include additional go metrics or process metrics are exposed at
|
||||
`<Address for MinIO Node>/minio/v2/metrics/node`.
|
||||
|
||||
The additional bucket specific metrics which include additional go metrics or process metrics are exposed at
|
||||
`<Address for MinIO Node>/minio/v2/metrics/bucket`.
|
||||
|
||||
The additional resource specific metrics which include additional go metrics or process metrics are exposed at
|
||||
`<Address for MinIO Node>/minio/v2/metrics/resource`.
|
||||
|
||||
To use this endpoint, setup Prometheus to scrape data from this endpoint. Read more on how to configure and use Prometheus to monitor MinIO server in [How to monitor MinIO server with Prometheus](https://github.com/minio/minio/blob/master/docs/metrics/prometheus/README.md).
|
||||
|
||||
### **Deprecated metrics monitoring**
|
||||
|
||||
- Prometheus' data available at `/minio/prometheus/metrics` is deprecated
|
||||
93
docs/metrics/healthcheck/README.md
Normal file
93
docs/metrics/healthcheck/README.md
Normal file
@@ -0,0 +1,93 @@
|
||||
# MinIO Healthcheck
|
||||
|
||||
MinIO server exposes three un-authenticated, healthcheck endpoints liveness probe and a cluster probe at `/minio/health/live` and `/minio/health/cluster` respectively.
|
||||
|
||||
## Liveness probe
|
||||
|
||||
This probe always responds with '200 OK'. Only fails if 'etcd' is configured and unreachable. When liveness probe fails, Kubernetes like platforms restart the container.
|
||||
|
||||
```
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /minio/health/live
|
||||
port: 9000
|
||||
scheme: HTTP
|
||||
initialDelaySeconds: 120
|
||||
periodSeconds: 30
|
||||
timeoutSeconds: 10
|
||||
successThreshold: 1
|
||||
failureThreshold: 3
|
||||
```
|
||||
|
||||
## Readiness probe
|
||||
|
||||
This probe always responds with '200 OK'. Only fails if 'etcd' is configured and unreachable. When readiness probe fails, Kubernetes like platforms turn-off routing to the container.
|
||||
|
||||
```
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /minio/health/ready
|
||||
port: 9000
|
||||
scheme: HTTP
|
||||
initialDelaySeconds: 120
|
||||
periodSeconds: 15
|
||||
timeoutSeconds: 10
|
||||
successThreshold: 1
|
||||
failureThreshold: 3
|
||||
```
|
||||
|
||||
## Cluster probe
|
||||
|
||||
### Cluster-writeable probe
|
||||
|
||||
The reply is '200 OK' if cluster has write quorum if not it returns '503 Service Unavailable'.
|
||||
|
||||
```
|
||||
curl http://minio1:9001/minio/health/cluster
|
||||
HTTP/1.1 503 Service Unavailable
|
||||
Accept-Ranges: bytes
|
||||
Content-Length: 0
|
||||
Server: MinIO
|
||||
Vary: Origin
|
||||
X-Amz-Bucket-Region: us-east-1
|
||||
X-Minio-Write-Quorum: 3
|
||||
X-Amz-Request-Id: 16239D6AB80EBECF
|
||||
X-Xss-Protection: 1; mode=block
|
||||
Date: Tue, 21 Jul 2020 00:36:14 GMT
|
||||
```
|
||||
|
||||
### Cluster-readable probe
|
||||
|
||||
The reply is '200 OK' if cluster has read quorum if not it returns '503 Service Unavailable'.
|
||||
|
||||
```
|
||||
curl http://minio1:9001/minio/health/cluster/read
|
||||
HTTP/1.1 503 Service Unavailable
|
||||
Accept-Ranges: bytes
|
||||
Content-Length: 0
|
||||
Server: MinIO
|
||||
Vary: Origin
|
||||
X-Amz-Bucket-Region: us-east-1
|
||||
X-Minio-Write-Quorum: 3
|
||||
X-Amz-Request-Id: 16239D6AB80EBECF
|
||||
X-Xss-Protection: 1; mode=block
|
||||
Date: Tue, 21 Jul 2020 00:36:14 GMT
|
||||
```
|
||||
|
||||
### Checking cluster health for maintenance
|
||||
|
||||
You may query the cluster probe endpoint to check if the node which received the request can be taken down for maintenance, if the server replies back '412 Precondition Failed' this means you will lose HA. '200 OK' means you are okay to proceed.
|
||||
|
||||
```
|
||||
curl http://minio1:9001/minio/health/cluster?maintenance=true
|
||||
HTTP/1.1 412 Precondition Failed
|
||||
Accept-Ranges: bytes
|
||||
Content-Length: 0
|
||||
Server: MinIO
|
||||
Vary: Origin
|
||||
X-Amz-Bucket-Region: us-east-1
|
||||
X-Amz-Request-Id: 16239D63820C6E76
|
||||
X-Xss-Protection: 1; mode=block
|
||||
X-Minio-Write-Quorum: 3
|
||||
Date: Tue, 21 Jul 2020 00:35:43 GMT
|
||||
```
|
||||
195
docs/metrics/prometheus/README.md
Normal file
195
docs/metrics/prometheus/README.md
Normal file
@@ -0,0 +1,195 @@
|
||||
# How to monitor MinIO server with Prometheus? [](https://slack.min.io)
|
||||
|
||||
[Prometheus](https://prometheus.io) is a cloud-native monitoring platform. Prometheus offers a multi-dimensional data model with time series data identified by metric name and key/value pairs. The data collection happens via a pull model over HTTP/HTTPS. Users looking to monitor their MinIO instances can point Prometheus configuration to scrape data from following endpoints.
|
||||
|
||||
- MinIO exports Prometheus compatible data by default as an authorized endpoint at `/minio/v2/metrics/cluster`.
|
||||
- MinIO exports Prometheus compatible data by default which is bucket centric as an authorized endpoint at `/minio/v2/metrics/bucket`.
|
||||
|
||||
This document explains how to setup Prometheus and configure it to scrape data from MinIO servers.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
To get started with MinIO, refer [MinIO QuickStart Document](https://docs.min.io/community/minio-object-store/operations/deployments/baremetal-deploy-minio-on-redhat-linux.html).
|
||||
Follow below steps to get started with MinIO monitoring using Prometheus.
|
||||
|
||||
### 1. Download Prometheus
|
||||
|
||||
[Download the latest release](https://prometheus.io/download) of Prometheus for your platform, then extract it
|
||||
|
||||
```sh
|
||||
tar xvfz prometheus-*.tar.gz
|
||||
cd prometheus-*
|
||||
```
|
||||
|
||||
Prometheus server is a single binary called `prometheus` (or `prometheus.exe` on Microsoft Windows). Run the binary and pass `--help` flag to see available options
|
||||
|
||||
```sh
|
||||
./prometheus --help
|
||||
usage: prometheus [<flags>]
|
||||
|
||||
The Prometheus monitoring server
|
||||
|
||||
. . .
|
||||
```
|
||||
|
||||
Refer [Prometheus documentation](https://prometheus.io/docs/introduction/first_steps/) for more details.
|
||||
|
||||
### 2. Configure authentication type for Prometheus metrics
|
||||
|
||||
MinIO supports two authentication modes for Prometheus either `jwt` or `public`, by default MinIO runs in `jwt` mode. To allow public access without authentication for prometheus metrics set environment as follows.
|
||||
|
||||
```
|
||||
export MINIO_PROMETHEUS_AUTH_TYPE="public"
|
||||
minio server ~/test
|
||||
```
|
||||
|
||||
### 3. Configuring Prometheus
|
||||
|
||||
#### 3.1 Authenticated Prometheus config
|
||||
|
||||
> If MinIO is configured to expose metrics without authentication, you don't need to use `mc` to generate prometheus config. You can skip reading further and move to 3.2 section.
|
||||
|
||||
The Prometheus endpoint in MinIO requires authentication by default. Prometheus supports a bearer token approach to authenticate prometheus scrape requests, override the default Prometheus config with the one generated using mc. To generate a Prometheus config for an alias, use [mc](https://docs.min.io/community/minio-object-store/reference/minio-mc.html#quickstart) as follows `mc admin prometheus generate <alias> [METRIC-TYPE]`. The valid values for METRIC-TYPE are `cluster`, `node`, `bucket` and `resource` and if not mentioned, it defaults to `cluster`.
|
||||
|
||||
The command will generate the `scrape_configs` section of the prometheus.yml as follows:
|
||||
|
||||
##### Cluster
|
||||
|
||||
```yaml
|
||||
scrape_configs:
|
||||
- job_name: minio-job
|
||||
bearer_token: <secret>
|
||||
metrics_path: /minio/v2/metrics/cluster
|
||||
scheme: http
|
||||
static_configs:
|
||||
- targets: ['localhost:9000']
|
||||
```
|
||||
|
||||
##### Bucket centric
|
||||
|
||||
```yaml
|
||||
- job_name: minio-job-bucket
|
||||
bearer_token: <secret>
|
||||
metrics_path: /minio/v2/metrics/bucket
|
||||
scheme: http
|
||||
static_configs:
|
||||
- targets: ['localhost:9000']
|
||||
```
|
||||
|
||||
##### Node centric (optional)
|
||||
|
||||
```yaml
|
||||
- job_name: minio-job-node
|
||||
bearer_token: <secret>
|
||||
metrics_path: /minio/v2/metrics/node
|
||||
scheme: http
|
||||
static_configs:
|
||||
- targets: ['localhost:9000']
|
||||
```
|
||||
|
||||
##### Resource centric (optional)
|
||||
|
||||
```yaml
|
||||
- job_name: minio-job-resource
|
||||
bearer_token: <secret>
|
||||
metrics_path: /minio/v2/metrics/resource
|
||||
scheme: http
|
||||
static_configs:
|
||||
- targets: ['localhost:9000']
|
||||
```
|
||||
|
||||
#### 3.2 Public Prometheus config
|
||||
|
||||
If Prometheus endpoint authentication type is set to `public`. Following prometheus config is sufficient to start scraping metrics data from MinIO.
|
||||
This can be collected from any server once per collection.
|
||||
|
||||
##### Cluster
|
||||
|
||||
```yaml
|
||||
scrape_configs:
|
||||
- job_name: minio-job
|
||||
metrics_path: /minio/v2/metrics/cluster
|
||||
scheme: http
|
||||
static_configs:
|
||||
- targets: ['localhost:9000']
|
||||
```
|
||||
|
||||
#### Bucket centric
|
||||
|
||||
```yaml
|
||||
scrape_configs:
|
||||
- job_name: minio-job-bucket
|
||||
metrics_path: /minio/v2/metrics/bucket
|
||||
scheme: http
|
||||
static_configs:
|
||||
- targets: ['localhost:9000']
|
||||
```
|
||||
|
||||
##### Node (optional)
|
||||
|
||||
Optionally you can also collect per node metrics. This needs to be done on a per server instance.
|
||||
The scrape configurations should use all the servers under `targets` so that graphing systems like
|
||||
grafana can visualize them for all the nodes
|
||||
|
||||
```yaml
|
||||
scrape_configs:
|
||||
- job_name: minio-job
|
||||
metrics_path: /minio/v2/metrics/node
|
||||
scheme: http
|
||||
static_configs:
|
||||
- targets: ['server1:9000','server2:9000','server3:9000','server4:9000']
|
||||
```
|
||||
|
||||
##### Resource (optional)
|
||||
|
||||
Optionally you can also collect resource metrics.
|
||||
|
||||
```yaml
|
||||
scrape_configs:
|
||||
- job_name: minio-job
|
||||
metrics_path: /minio/v2/metrics/resource
|
||||
scheme: http
|
||||
static_configs:
|
||||
- targets: ['localhost:9000']
|
||||
```
|
||||
|
||||
### 4. Update `scrape_configs` section in prometheus.yml
|
||||
|
||||
To authorize every scrape request, copy and paste the generated `scrape_configs` section in the prometheus.yml and restart the Prometheus service.
|
||||
|
||||
### 5. Start Prometheus
|
||||
|
||||
Start (or) Restart Prometheus service by running
|
||||
|
||||
```sh
|
||||
./prometheus --config.file=prometheus.yml
|
||||
```
|
||||
|
||||
Here `prometheus.yml` is the name of configuration file. You can now see MinIO metrics in Prometheus dashboard. By default Prometheus dashboard is accessible at `http://localhost:9090`.
|
||||
|
||||
Prometheus sets the `Host` header to `domain:port` as part of HTTP operations against the MinIO metrics endpoint. For MinIO deployments behind a load balancer, reverse proxy, or other control plane (HAProxy, nginx, pfsense, opnsense, etc.), ensure the network service supports routing these requests to the deployment.
|
||||
|
||||
### 6. Configure Grafana
|
||||
|
||||
After Prometheus is configured, you can use Grafana to visualize MinIO metrics. Refer the [document here to setup Grafana with MinIO prometheus metrics](https://github.com/minio/minio/blob/master/docs/metrics/prometheus/grafana/README.md).
|
||||
|
||||
## List of metrics exposed by MinIO
|
||||
|
||||
- MinIO exports Prometheus compatible data by default as an authorized endpoint at `/minio/v2/metrics/cluster`.
|
||||
- MinIO exports Prometheus compatible data by default which is bucket centric as an authorized endpoint at `/minio/v2/metrics/bucket`.
|
||||
- MinIO exports Prometheus compatible data by default which is node centric as an authorized endpoint at `/minio/v2/metrics/node`.
|
||||
- MinIO exports Prometheus compatible data by default which is resource centric as an authorized endpoint at `/minio/v2/metrics/resource`.
|
||||
|
||||
All of these can be accessed via Prometheus dashboard. A sample list of exposed metrics along with their definition is available on our public demo server at
|
||||
|
||||
```sh
|
||||
curl https://play.min.io/minio/v2/metrics/cluster
|
||||
```
|
||||
|
||||
### List of metrics reported Cluster and Bucket level
|
||||
|
||||
[The list of metrics reported can be here](https://github.com/minio/minio/blob/master/docs/metrics/prometheus/list.md)
|
||||
|
||||
### Configure Alerts for Prometheus
|
||||
|
||||
[The Prometheus AlertManager and alerts can be configured following this](https://github.com/minio/minio/blob/master/docs/metrics/prometheus/alerts.md)
|
||||
120
docs/metrics/prometheus/alerts.md
Normal file
120
docs/metrics/prometheus/alerts.md
Normal file
@@ -0,0 +1,120 @@
|
||||
# How to configure Prometheus AlertManager
|
||||
|
||||
Alerting with prometheus is two step process. First we setup alerts in Prometheus server and then we need to send alerts to the AlertManager.
|
||||
Prometheus AlertManager is the component that manages sending, inhibition and silencing of the alerts generated from Prometheus. The AlertManager can be configured to send alerts to variety of receivers. Refer [Prometheus AlertManager receivers](https://prometheus.io/docs/alerting/latest/configuration/#receiver) for more details.
|
||||
|
||||
Follow below steps to enable and use AlertManager.
|
||||
|
||||
## Deploy and start AlertManager
|
||||
Install Prometheus AlertManager from https://prometheus.io/download/ and create configuration as below
|
||||
|
||||
```yaml
|
||||
route:
|
||||
group_by: ['alertname']
|
||||
group_wait: 30s
|
||||
group_interval: 5m
|
||||
repeat_interval: 1h
|
||||
receiver: 'web.hook'
|
||||
receivers:
|
||||
- name: 'web.hook'
|
||||
webhook_configs:
|
||||
- url: 'http://127.0.0.1:8010/webhook'
|
||||
inhibit_rules:
|
||||
- source_match:
|
||||
severity: 'critical'
|
||||
target_match:
|
||||
severity: 'warning'
|
||||
equal: ['alertname', 'dev', 'instance']
|
||||
```
|
||||
|
||||
This sample configuration uses a `webhook` at http://127.0.0.1:8010/webhook to post the alerts.
|
||||
Start the AlertManager and it listens on port `9093` by default. Make sure your webhook is up and listening for the alerts.
|
||||
|
||||
## Configure Prometheus to use AlertManager
|
||||
|
||||
Add below section to your `prometheus.yml`
|
||||
```yaml
|
||||
alerting:
|
||||
alertmanagers:
|
||||
- static_configs:
|
||||
- targets: ['localhost:9093']
|
||||
rule_files:
|
||||
- rules.yml
|
||||
```
|
||||
Here `rules.yml` is the file which should contain the alerting rules defined.
|
||||
|
||||
## Add rules for your deployment
|
||||
Below is a sample alerting rules configuration for MinIO. Refer https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/ for more instructions on writing alerting rules for Prometheus.
|
||||
|
||||
```yaml
|
||||
groups:
|
||||
- name: example
|
||||
rules:
|
||||
- alert: MinIOClusterTolerance
|
||||
expr: minio_cluster_health_erasure_set_status < 1
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "Instance {{ $labels.server }} has lost quorum on pool {{ $labels.pool }} on set {{ $labels.set }}"
|
||||
description: "MinIO instance {{ $labels.server }} of job {{ $labels.job }} has lost quorum on pool {{ $labels.pool }} on set {{ $labels.set }} for more than 5 minutes."
|
||||
```
|
||||
|
||||
## Verify the configuration and alerts
|
||||
To verify the above sample alert follow below steps
|
||||
|
||||
1. Start a distributed MinIO instance (4 nodes setup)
|
||||
2. Start Prometheus server and AlertManager
|
||||
3. Bring down couple of MinIO instances to bring down the Erasure Set tolerance to -1 and verify the same with `mc admin prometheus metrics ALIAS | grep minio_cluster_health_erasure_set_status`
|
||||
4. Wait for 5 mins (as alert is configured to be firing after 5 mins), and verify that you see an entry in webhook for the alert as well as in Prometheus console as shown below
|
||||
|
||||
```json
|
||||
{
|
||||
"receiver": "web\\.hook",
|
||||
"status": "firing",
|
||||
"alerts": [
|
||||
{
|
||||
"status": "firing",
|
||||
"labels": {
|
||||
"alertname": "MinIOClusterTolerance",
|
||||
"instance": "localhost:9000",
|
||||
"job": "minio-job-node",
|
||||
"pool": "0",
|
||||
"server": "127.0.0.1:9000",
|
||||
"set": "0",
|
||||
"severity": "critical"
|
||||
},
|
||||
"annotations": {
|
||||
"description": "MinIO instance 127.0.0.1:9000 of job minio-job has tolerance <=0 for more than 5 minutes.",
|
||||
"summary": "Instance 127.0.0.1:9000 unable to tolerate node failures"
|
||||
},
|
||||
"startsAt": "2023-11-18T06:20:09.456Z",
|
||||
"endsAt": "0001-01-01T00:00:00Z",
|
||||
"generatorURL": "http://fedora-minio:9090/graph?g0.expr=minio_cluster_health_erasure_set_tolerance+%3C%3D+0&g0.tab=1",
|
||||
"fingerprint": "2255608b0da28ca3"
|
||||
}
|
||||
],
|
||||
"groupLabels": {
|
||||
"alertname": "MinIOClusterTolerance"
|
||||
},
|
||||
"commonLabels": {
|
||||
"alertname": "MinIOClusterTolerance",
|
||||
"instance": "localhost:9000",
|
||||
"job": "minio-job-node",
|
||||
"pool": "0",
|
||||
"server": "127.0.0.1:9000",
|
||||
"set": "0",
|
||||
"severity": "critical"
|
||||
},
|
||||
"commonAnnotations": {
|
||||
"description": "MinIO instance 127.0.0.1:9000 of job minio-job has lost quorum on pool 0 on set 0 for more than 5 minutes.",
|
||||
"summary": "Instance 127.0.0.1:9000 has lost quorum on pool 0 on set 0"
|
||||
},
|
||||
"externalURL": "http://fedora-minio:9093",
|
||||
"version": "4",
|
||||
"groupKey": "{}:{alertname=\"MinIOClusterTolerance\"}",
|
||||
"truncatedAlerts": 0
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
34
docs/metrics/prometheus/grafana/README.md
Normal file
34
docs/metrics/prometheus/grafana/README.md
Normal file
@@ -0,0 +1,34 @@
|
||||
# How to monitor MinIO server with Grafana [](https://slack.min.io)
|
||||
|
||||
[Grafana](https://grafana.com/) allows you to query, visualize, alert on and understand your metrics no matter where they are stored. Create, explore, and share dashboards with your team and foster a data driven culture.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Prometheus and MinIO configured as explained in [document here](https://github.com/minio/minio/blob/master/docs/metrics/prometheus/README.md).
|
||||
- Grafana installed as explained [here](https://grafana.com/grafana/download).
|
||||
|
||||
## MinIO Grafana Dashboard
|
||||
|
||||
Visualize MinIO metrics with our official Grafana dashboard available on the [Grafana dashboard portal](https://grafana.com/grafana/dashboards/13502).
|
||||
|
||||
Refer to the dashboard [json file here](https://raw.githubusercontent.com/minio/minio/master/docs/metrics/prometheus/grafana/minio-dashboard.json).
|
||||
|
||||
|
||||
|
||||
Node level Replication metrics can be viewed in the Grafana dashboard using [json file here](https://raw.githubusercontent.com/minio/minio/master/docs/metrics/prometheus/grafana/replication/minio-replication-node.json)
|
||||
|
||||
|
||||
|
||||
Cluster level Replication metrics can be viewed in the Grafana dashboard using [json file here](https://raw.githubusercontent.com/minio/minio/master/docs/metrics/prometheus/grafana/replication/minio-replication-cluster.json)
|
||||
|
||||
|
||||
|
||||
Bucket metrics can be viewed in the Grafana dashboard using [json file here](https://raw.githubusercontent.com/minio/minio/master/docs/metrics/prometheus/grafana/bucket/minio-bucket.json)
|
||||
|
||||
|
||||
|
||||
Node metrics can be viewed in the Grafana dashboard using [json file here](https://raw.githubusercontent.com/minio/minio/master/docs/metrics/prometheus/grafana/node/minio-node.json)
|
||||
|
||||
|
||||
|
||||
Note: All these dashboards are provided as an example and need basis they should be customized as well as new graphs should be added.
|
||||
BIN
docs/metrics/prometheus/grafana/bucket/grafana-bucket.png
Normal file
BIN
docs/metrics/prometheus/grafana/bucket/grafana-bucket.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 355 KiB |
4214
docs/metrics/prometheus/grafana/bucket/minio-bucket.json
Normal file
4214
docs/metrics/prometheus/grafana/bucket/minio-bucket.json
Normal file
File diff suppressed because it is too large
Load Diff
BIN
docs/metrics/prometheus/grafana/grafana-minio.png
Normal file
BIN
docs/metrics/prometheus/grafana/grafana-minio.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 591 KiB |
3868
docs/metrics/prometheus/grafana/minio-dashboard.json
Normal file
3868
docs/metrics/prometheus/grafana/minio-dashboard.json
Normal file
File diff suppressed because it is too large
Load Diff
BIN
docs/metrics/prometheus/grafana/node/grafana-node.png
Normal file
BIN
docs/metrics/prometheus/grafana/node/grafana-node.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 136 KiB |
955
docs/metrics/prometheus/grafana/node/minio-node.json
Normal file
955
docs/metrics/prometheus/grafana/node/minio-node.json
Normal file
@@ -0,0 +1,955 @@
|
||||
{
|
||||
"annotations": {
|
||||
"list": [
|
||||
{
|
||||
"builtIn": 1,
|
||||
"datasource": {
|
||||
"type": "datasource",
|
||||
"uid": "grafana"
|
||||
},
|
||||
"enable": true,
|
||||
"hide": true,
|
||||
"iconColor": "rgba(0, 211, 255, 1)",
|
||||
"name": "Annotations & Alerts",
|
||||
"type": "dashboard"
|
||||
}
|
||||
]
|
||||
},
|
||||
"description": "MinIO Nodes Grafana Dashboard - https://min.io/",
|
||||
"editable": true,
|
||||
"fiscalYearStartMonth": 0,
|
||||
"gnetId": 15306,
|
||||
"graphTooltip": 0,
|
||||
"id": 267,
|
||||
"links": [
|
||||
{
|
||||
"icon": "external link",
|
||||
"includeVars": true,
|
||||
"keepTime": true,
|
||||
"tags": [
|
||||
"minio"
|
||||
],
|
||||
"type": "dashboards"
|
||||
}
|
||||
],
|
||||
"liveNow": false,
|
||||
"panels": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"description": "",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"mappings": [
|
||||
{
|
||||
"options": {
|
||||
"match": "null",
|
||||
"result": {
|
||||
"text": "N/A"
|
||||
}
|
||||
},
|
||||
"type": "special"
|
||||
}
|
||||
],
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 80
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 5,
|
||||
"w": 6,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
"id": 21,
|
||||
"maxDataPoints": 100,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"justifyMode": "auto",
|
||||
"orientation": "auto",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"mean"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"showPercentChange": false,
|
||||
"text": {},
|
||||
"textMode": "auto",
|
||||
"wideLayout": true
|
||||
},
|
||||
"pluginVersion": "10.4.0",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"exemplar": true,
|
||||
"expr": "max(minio_node_drive_total{job=~\"$scrape_jobs\",server=\"$server\"})",
|
||||
"format": "table",
|
||||
"hide": false,
|
||||
"instant": true,
|
||||
"interval": "",
|
||||
"intervalFactor": 1,
|
||||
"legendFormat": "",
|
||||
"metric": "process_start_time_seconds",
|
||||
"refId": "A",
|
||||
"step": 60
|
||||
}
|
||||
],
|
||||
"title": "Total Drives",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"description": "",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"mappings": [],
|
||||
"thresholds": {
|
||||
"mode": "percentage",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 5,
|
||||
"w": 6,
|
||||
"x": 6,
|
||||
"y": 0
|
||||
},
|
||||
"id": 22,
|
||||
"maxDataPoints": 100,
|
||||
"options": {
|
||||
"minVizHeight": 75,
|
||||
"minVizWidth": 75,
|
||||
"orientation": "auto",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"showThresholdLabels": false,
|
||||
"showThresholdMarkers": true,
|
||||
"sizing": "auto"
|
||||
},
|
||||
"pluginVersion": "10.4.0",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"exemplar": false,
|
||||
"expr": "max(minio_node_drive_online_total{job=~\"$scrape_jobs\",server=\"$server\"})",
|
||||
"format": "time_series",
|
||||
"hide": false,
|
||||
"instant": true,
|
||||
"interval": "",
|
||||
"intervalFactor": 1,
|
||||
"legendFormat": ".",
|
||||
"metric": "process_start_time_seconds",
|
||||
"range": false,
|
||||
"refId": "A",
|
||||
"step": 60
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"exemplar": false,
|
||||
"expr": "max(minio_node_drive_offline_total{job=~\"$scrape_jobs\",server=\"$server\"})",
|
||||
"format": "time_series",
|
||||
"hide": false,
|
||||
"instant": true,
|
||||
"legendFormat": ".",
|
||||
"range": false,
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"title": "Total Online/Offline Drives",
|
||||
"type": "gauge"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"description": "",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisBorderShow": false,
|
||||
"axisCenteredZero": false,
|
||||
"axisColorMode": "text",
|
||||
"axisLabel": "",
|
||||
"axisPlacement": "auto",
|
||||
"barAlignment": 0,
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"gradientMode": "none",
|
||||
"hideFrom": {
|
||||
"legend": false,
|
||||
"tooltip": false,
|
||||
"viz": false
|
||||
},
|
||||
"insertNulls": false,
|
||||
"lineInterpolation": "linear",
|
||||
"lineWidth": 1,
|
||||
"pointSize": 5,
|
||||
"scaleDistribution": {
|
||||
"type": "linear"
|
||||
},
|
||||
"showPoints": "never",
|
||||
"spanNulls": false,
|
||||
"stacking": {
|
||||
"group": "A",
|
||||
"mode": "none"
|
||||
},
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"links": [],
|
||||
"mappings": [],
|
||||
"min": 0,
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 80
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "bytes"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 5,
|
||||
"w": 6,
|
||||
"x": 12,
|
||||
"y": 0
|
||||
},
|
||||
"id": 23,
|
||||
"options": {
|
||||
"legend": {
|
||||
"calcs": [],
|
||||
"displayMode": "list",
|
||||
"placement": "bottom",
|
||||
"showLegend": true
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "none"
|
||||
}
|
||||
},
|
||||
"pluginVersion": "10.4.0",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"exemplar": true,
|
||||
"expr": "minio_node_drive_total_bytes{job=~\"$scrape_jobs\",server=\"$server\"}",
|
||||
"interval": "",
|
||||
"legendFormat": "Total [{{drive}}]",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"exemplar": true,
|
||||
"expr": "minio_node_drive_used_bytes{job=~\"$scrape_jobs\",server=\"$server\"}",
|
||||
"interval": "",
|
||||
"legendFormat": "Used [{{drive}}]",
|
||||
"refId": "B"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"exemplar": true,
|
||||
"expr": "minio_node_drive_free_bytes{job=~\"$scrape_jobs\",server=\"$server\"}",
|
||||
"interval": "",
|
||||
"legendFormat": "Free [{{drive}}]",
|
||||
"refId": "C"
|
||||
}
|
||||
],
|
||||
"title": "Drive Usage",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"description": "",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisBorderShow": false,
|
||||
"axisCenteredZero": false,
|
||||
"axisColorMode": "text",
|
||||
"axisLabel": "",
|
||||
"axisPlacement": "auto",
|
||||
"barAlignment": 0,
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"gradientMode": "none",
|
||||
"hideFrom": {
|
||||
"legend": false,
|
||||
"tooltip": false,
|
||||
"viz": false
|
||||
},
|
||||
"insertNulls": false,
|
||||
"lineInterpolation": "linear",
|
||||
"lineWidth": 1,
|
||||
"pointSize": 5,
|
||||
"scaleDistribution": {
|
||||
"type": "linear"
|
||||
},
|
||||
"showPoints": "never",
|
||||
"spanNulls": false,
|
||||
"stacking": {
|
||||
"group": "A",
|
||||
"mode": "none"
|
||||
},
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"links": [],
|
||||
"mappings": [],
|
||||
"min": 0,
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 80
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "none"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 5,
|
||||
"w": 6,
|
||||
"x": 18,
|
||||
"y": 0
|
||||
},
|
||||
"id": 24,
|
||||
"options": {
|
||||
"legend": {
|
||||
"calcs": [],
|
||||
"displayMode": "list",
|
||||
"placement": "bottom",
|
||||
"showLegend": true
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "none"
|
||||
}
|
||||
},
|
||||
"pluginVersion": "10.4.0",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"exemplar": true,
|
||||
"expr": "minio_node_drive_free_inodes{job=~\"$scrape_jobs\",server=\"$server\"}",
|
||||
"interval": "",
|
||||
"legendFormat": "[{{drive}}]",
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"title": "Free Inodes",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"description": "",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisBorderShow": false,
|
||||
"axisCenteredZero": false,
|
||||
"axisColorMode": "text",
|
||||
"axisLabel": "",
|
||||
"axisPlacement": "auto",
|
||||
"barAlignment": 0,
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"gradientMode": "none",
|
||||
"hideFrom": {
|
||||
"legend": false,
|
||||
"tooltip": false,
|
||||
"viz": false
|
||||
},
|
||||
"insertNulls": false,
|
||||
"lineInterpolation": "linear",
|
||||
"lineWidth": 1,
|
||||
"pointSize": 5,
|
||||
"scaleDistribution": {
|
||||
"type": "linear"
|
||||
},
|
||||
"showPoints": "never",
|
||||
"spanNulls": false,
|
||||
"stacking": {
|
||||
"group": "A",
|
||||
"mode": "none"
|
||||
},
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"links": [],
|
||||
"mappings": [],
|
||||
"min": 0,
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 80
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "none"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 5,
|
||||
"w": 6,
|
||||
"x": 0,
|
||||
"y": 5
|
||||
},
|
||||
"id": 25,
|
||||
"options": {
|
||||
"legend": {
|
||||
"calcs": [],
|
||||
"displayMode": "list",
|
||||
"placement": "bottom",
|
||||
"showLegend": true
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "none"
|
||||
}
|
||||
},
|
||||
"pluginVersion": "10.4.0",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"exemplar": true,
|
||||
"expr": "minio_node_drive_latency_us{job=~\"$scrape_jobs\",server=\"$server\"}",
|
||||
"interval": "",
|
||||
"legendFormat": "[{{drive}}:{{api}}]",
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"title": "Drive Latency (micro sec)",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"description": "",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisBorderShow": false,
|
||||
"axisCenteredZero": false,
|
||||
"axisColorMode": "text",
|
||||
"axisLabel": "",
|
||||
"axisPlacement": "auto",
|
||||
"barAlignment": 0,
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"gradientMode": "none",
|
||||
"hideFrom": {
|
||||
"legend": false,
|
||||
"tooltip": false,
|
||||
"viz": false
|
||||
},
|
||||
"insertNulls": false,
|
||||
"lineInterpolation": "linear",
|
||||
"lineWidth": 1,
|
||||
"pointSize": 5,
|
||||
"scaleDistribution": {
|
||||
"type": "linear"
|
||||
},
|
||||
"showPoints": "never",
|
||||
"spanNulls": false,
|
||||
"stacking": {
|
||||
"group": "A",
|
||||
"mode": "none"
|
||||
},
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"links": [],
|
||||
"mappings": [],
|
||||
"min": 0,
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 80
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "none"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 5,
|
||||
"w": 6,
|
||||
"x": 6,
|
||||
"y": 5
|
||||
},
|
||||
"id": 26,
|
||||
"options": {
|
||||
"legend": {
|
||||
"calcs": [],
|
||||
"displayMode": "list",
|
||||
"placement": "bottom",
|
||||
"showLegend": true
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "none"
|
||||
}
|
||||
},
|
||||
"pluginVersion": "10.4.0",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"exemplar": true,
|
||||
"expr": "minio_node_drive_errors_availability{job=~\"$scrape_jobs\",server=\"$server\"}",
|
||||
"interval": "",
|
||||
"legendFormat": "[{{drive}}]",
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"title": "Drive Errors",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"description": "",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisBorderShow": false,
|
||||
"axisCenteredZero": false,
|
||||
"axisColorMode": "text",
|
||||
"axisLabel": "",
|
||||
"axisPlacement": "auto",
|
||||
"barAlignment": 0,
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"gradientMode": "none",
|
||||
"hideFrom": {
|
||||
"legend": false,
|
||||
"tooltip": false,
|
||||
"viz": false
|
||||
},
|
||||
"insertNulls": false,
|
||||
"lineInterpolation": "linear",
|
||||
"lineWidth": 1,
|
||||
"pointSize": 5,
|
||||
"scaleDistribution": {
|
||||
"type": "linear"
|
||||
},
|
||||
"showPoints": "never",
|
||||
"spanNulls": false,
|
||||
"stacking": {
|
||||
"group": "A",
|
||||
"mode": "none"
|
||||
},
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"links": [],
|
||||
"mappings": [],
|
||||
"min": 0,
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 80
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "none"
|
||||
},
|
||||
"overrides": [
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "available 10.13.1.25:9000"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "color",
|
||||
"value": {
|
||||
"fixedColor": "green",
|
||||
"mode": "fixed"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "used 10.13.1.25:9000"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "color",
|
||||
"value": {
|
||||
"fixedColor": "blue",
|
||||
"mode": "fixed"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 5,
|
||||
"w": 6,
|
||||
"x": 12,
|
||||
"y": 5
|
||||
},
|
||||
"id": 27,
|
||||
"options": {
|
||||
"legend": {
|
||||
"calcs": [],
|
||||
"displayMode": "list",
|
||||
"placement": "bottom",
|
||||
"showLegend": true
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "none"
|
||||
}
|
||||
},
|
||||
"pluginVersion": "10.4.0",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"exemplar": true,
|
||||
"expr": "minio_node_drive_errors_timeout{job=~\"$scrape_jobs\",server=\"$server\"}",
|
||||
"interval": "",
|
||||
"legendFormat": "[{{drive}}]",
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"title": "Drive Timeout Errors",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"description": "",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisBorderShow": false,
|
||||
"axisCenteredZero": false,
|
||||
"axisColorMode": "text",
|
||||
"axisLabel": "",
|
||||
"axisPlacement": "auto",
|
||||
"barAlignment": 0,
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"gradientMode": "none",
|
||||
"hideFrom": {
|
||||
"legend": false,
|
||||
"tooltip": false,
|
||||
"viz": false
|
||||
},
|
||||
"insertNulls": false,
|
||||
"lineInterpolation": "linear",
|
||||
"lineWidth": 1,
|
||||
"pointSize": 5,
|
||||
"scaleDistribution": {
|
||||
"type": "linear"
|
||||
},
|
||||
"showPoints": "never",
|
||||
"spanNulls": false,
|
||||
"stacking": {
|
||||
"group": "A",
|
||||
"mode": "none"
|
||||
},
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"links": [],
|
||||
"mappings": [],
|
||||
"min": 0,
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 80
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "none"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 5,
|
||||
"w": 6,
|
||||
"x": 18,
|
||||
"y": 5
|
||||
},
|
||||
"id": 28,
|
||||
"options": {
|
||||
"legend": {
|
||||
"calcs": [],
|
||||
"displayMode": "list",
|
||||
"placement": "bottom",
|
||||
"showLegend": true
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "none"
|
||||
}
|
||||
},
|
||||
"pluginVersion": "10.4.0",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"exemplar": true,
|
||||
"expr": "minio_node_drive_io_waiting{job=~\"$scrape_jobs\",server=\"$server\"}",
|
||||
"interval": "",
|
||||
"legendFormat": "[{{drive}}]",
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"title": "IO Operations Waiting",
|
||||
"type": "timeseries"
|
||||
}
|
||||
],
|
||||
"refresh": "",
|
||||
"schemaVersion": 39,
|
||||
"tags": [
|
||||
"minio"
|
||||
],
|
||||
"templating": {
|
||||
"list": [
|
||||
{
|
||||
"current": {
|
||||
"selected": false,
|
||||
"text": "All",
|
||||
"value": "$__all"
|
||||
},
|
||||
"label": "Data source",
|
||||
"name": "DS_PROMETHEUS",
|
||||
"options": [],
|
||||
"query": "prometheus",
|
||||
"refresh": 1,
|
||||
"regex": "",
|
||||
"type": "datasource"
|
||||
},
|
||||
{
|
||||
"current": {},
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"definition": "label_values(job)",
|
||||
"hide": 0,
|
||||
"includeAll": false,
|
||||
"multi": false,
|
||||
"name": "scrape_jobs",
|
||||
"options": [],
|
||||
"query": {
|
||||
"query": "label_values(job)",
|
||||
"refId": "StandardVariableQuery"
|
||||
},
|
||||
"refresh": 1,
|
||||
"regex": "",
|
||||
"skipUrlSync": false,
|
||||
"sort": 0,
|
||||
"type": "query"
|
||||
},
|
||||
{
|
||||
"current": {},
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"definition": "label_values(minio_node_drive_total,server)",
|
||||
"hide": 0,
|
||||
"includeAll": false,
|
||||
"label": "Server",
|
||||
"multi": false,
|
||||
"name": "server",
|
||||
"options": [],
|
||||
"query": {
|
||||
"qryType": 1,
|
||||
"query": "label_values(minio_node_drive_total,server)",
|
||||
"refId": "PrometheusVariableQueryEditor-VariableQuery"
|
||||
},
|
||||
"refresh": 1,
|
||||
"regex": "",
|
||||
"skipUrlSync": false,
|
||||
"sort": 0,
|
||||
"type": "query"
|
||||
}
|
||||
]
|
||||
},
|
||||
"time": {
|
||||
"from": "now-1h",
|
||||
"to": "now"
|
||||
},
|
||||
"timepicker": {
|
||||
"refresh_intervals": [
|
||||
"10s",
|
||||
"30s",
|
||||
"1m",
|
||||
"5m",
|
||||
"15m",
|
||||
"30m",
|
||||
"1h",
|
||||
"2h",
|
||||
"1d"
|
||||
],
|
||||
"time_options": [
|
||||
"5m",
|
||||
"15m",
|
||||
"1h",
|
||||
"6h",
|
||||
"12h",
|
||||
"24h",
|
||||
"2d",
|
||||
"7d",
|
||||
"30d"
|
||||
]
|
||||
},
|
||||
"timezone": "",
|
||||
"title": "MinIO Node Dashboard",
|
||||
"uid": "TgmJnnqn2k",
|
||||
"version": 1,
|
||||
"weekStart": ""
|
||||
}
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 444 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 230 KiB |
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
471
docs/metrics/prometheus/list.md
Normal file
471
docs/metrics/prometheus/list.md
Normal file
@@ -0,0 +1,471 @@
|
||||
# Cluster Metrics
|
||||
|
||||
MinIO collects the following metrics at the cluster level.
|
||||
Metrics may include one or more labels, such as the server that calculated that metric.
|
||||
|
||||
These metrics can be obtained from any MinIO server once per collection by using the following URL:
|
||||
|
||||
```shell
|
||||
https://HOSTNAME:PORT/minio/v2/metrics/cluster
|
||||
```
|
||||
|
||||
Replace ``HOSTNAME:PORT`` with the hostname of your MinIO deployment.
|
||||
For deployments behind a load balancer, use the load balancer hostname instead of a single node hostname.
|
||||
|
||||
## Audit Metrics
|
||||
|
||||
| Name | Description |
|
||||
|:----------------------------------|:----------------------------------------------------------|
|
||||
| `minio_audit_failed_messages` | Total number of messages that failed to send since start. |
|
||||
| `minio_audit_target_queue_length` | Number of unsent messages in queue for target. |
|
||||
| `minio_audit_total_messages` | Total number of messages sent since start. |
|
||||
|
||||
## Cluster Capacity Metrics
|
||||
|
||||
| Name | Description |
|
||||
|:---------------------------------------------|:---------------------------------------------------------------|
|
||||
| `minio_cluster_capacity_raw_free_bytes` | Total free capacity online in the cluster. |
|
||||
| `minio_cluster_capacity_raw_total_bytes` | Total capacity online in the cluster. |
|
||||
| `minio_cluster_capacity_usable_free_bytes` | Total free usable capacity online in the cluster. |
|
||||
| `minio_cluster_capacity_usable_total_bytes` | Total usable capacity online in the cluster. |
|
||||
| `minio_cluster_objects_size_distribution` | Distribution of object sizes across a cluster |
|
||||
| `minio_cluster_objects_version_distribution` | Distribution of object versions across a cluster |
|
||||
| `minio_cluster_usage_object_total` | Total number of objects in a cluster |
|
||||
| `minio_cluster_usage_total_bytes` | Total cluster usage in bytes |
|
||||
| `minio_cluster_usage_version_total` | Total number of versions (includes delete marker) in a cluster |
|
||||
| `minio_cluster_usage_deletemarker_total` | Total number of delete markers in a cluster |
|
||||
| `minio_cluster_bucket_total` | Total number of buckets in the cluster |
|
||||
|
||||
## Cluster Drive Metrics
|
||||
|
||||
| Name | Description |
|
||||
|:------------------------------------|:--------------------------------------|
|
||||
| `minio_cluster_drive_offline_total` | Total drives offline in this cluster. |
|
||||
| `minio_cluster_drive_online_total` | Total drives online in this cluster. |
|
||||
| `minio_cluster_drive_total` | Total drives in this cluster. |
|
||||
|
||||
## Cluster ILM Metrics
|
||||
|
||||
| Name | Description |
|
||||
|:------------------------------------------|:-------------------------------------------------|
|
||||
| `minio_cluster_ilm_transitioned_bytes` | Total bytes transitioned to a tier. |
|
||||
| `minio_cluster_ilm_transitioned_objects` | Total number of objects transitioned to a tier. |
|
||||
| `minio_cluster_ilm_transitioned_versions` | Total number of versions transitioned to a tier. |
|
||||
|
||||
## Cluster KMS Metrics
|
||||
|
||||
| Name | Description |
|
||||
|:------------------------------------|:-----------------------------------------------------------------------------------------|
|
||||
| `minio_cluster_kms_online` | Reports whether the KMS is online (1) or offline (0). |
|
||||
| `minio_cluster_kms_request_error` | Number of KMS requests that failed due to some error. (HTTP 4xx status code). |
|
||||
| `minio_cluster_kms_request_failure` | Number of KMS requests that failed due to some internal failure. (HTTP 5xx status code). |
|
||||
| `minio_cluster_kms_request_success` | Number of KMS requests that succeeded. |
|
||||
| `minio_cluster_kms_uptime` | The time the KMS has been up and running in seconds. |
|
||||
|
||||
## Cluster Health Metrics
|
||||
|
||||
| Name | Description |
|
||||
|:--------------------------------------------------|:-----------------------------------------------|
|
||||
| `minio_cluster_nodes_offline_total` | Total number of MinIO nodes offline. |
|
||||
| `minio_cluster_nodes_online_total` | Total number of MinIO nodes online. |
|
||||
| `minio_cluster_write_quorum` | Maximum write quorum across all pools and sets |
|
||||
| `minio_cluster_health_status` | Get current cluster health status |
|
||||
| `minio_cluster_health_erasure_set_healing_drives` | Count of healing drives in the erasure set |
|
||||
| `minio_cluster_health_erasure_set_online_drives` | Count of online drives in the erasure set |
|
||||
| `minio_cluster_health_erasure_set_read_quorum` | Get read quorum of the erasure set |
|
||||
| `minio_cluster_health_erasure_set_write_quorum` | Get write quorum of the erasure set |
|
||||
| `minio_cluster_health_erasure_set_status` | Get current health status of the erasure set |
|
||||
|
||||
## Cluster Replication Metrics
|
||||
|
||||
Metrics marked as ``Site Replication Only`` only populate on deployments with [Site Replication](https://docs.min.io/community/minio-object-store/operations/install-deploy-manage/multi-site-replication.html) configurations.
|
||||
For deployments with [bucket](https://docs.min.io/community/minio-object-store/administration/bucket-replication.html) or [batch](https://docs.min.io/community/minio-object-store/administration/batch-framework.html#replicate) configurations, these metrics populate instead under the [Bucket Metrics](#bucket-metrics) endpoint.
|
||||
|
||||
| Name | Description
|
||||
|:-----------------------------------------------------------|:---------------------------------------------------------------------------------------------------------|
|
||||
| `minio_cluster_replication_last_hour_failed_bytes` | (_Site Replication Only_) Total number of bytes failed at least once to replicate in the last full hour. |
|
||||
| `minio_cluster_replication_last_hour_failed_count` | (_Site Replication Only_) Total number of objects which failed replication in the last full hour. |
|
||||
| `minio_cluster_replication_last_minute_failed_bytes` | Total number of bytes failed at least once to replicate in the last full minute. |
|
||||
| `minio_cluster_replication_last_minute_failed_count` | Total number of objects which failed replication in the last full minute. |
|
||||
| `minio_cluster_replication_total_failed_bytes` | (_Site Replication Only_) Total number of bytes failed at least once to replicate since server start. |
|
||||
| `minio_cluster_replication_total_failed_count` | (_Site Replication Only_) Total number of objects which failed replication since server start. |
|
||||
| `minio_cluster_replication_received_bytes` | (_Site Replication Only_) Total number of bytes replicated to this cluster from another source cluster. |
|
||||
| `minio_cluster_replication_received_count` | (_Site Replication Only_) Total number of objects received by this cluster from another source cluster. |
|
||||
| `minio_cluster_replication_sent_bytes` | (_Site Replication Only_) Total number of bytes replicated to the target cluster. |
|
||||
| `minio_cluster_replication_sent_count` | (_Site Replication Only_) Total number of objects replicated to the target cluster. |
|
||||
| `minio_cluster_replication_credential_errors` | (_Site Replication Only_) Total number of replication credential errors since server start |
|
||||
| `minio_cluster_replication_proxied_get_requests_total` | (_Site Replication Only_)Number of GET requests proxied to replication target |
|
||||
| `minio_cluster_replication_proxied_head_requests_total` | (_Site Replication Only_)Number of HEAD requests proxied to replication target |
|
||||
| `minio_cluster_replication_proxied_delete_tagging_requests_total` | (_Site Replication Only_)Number of DELETE tagging requests proxied to replication target |
|
||||
| `minio_cluster_replication_proxied_get_tagging_requests_total` | (_Site Replication Only_)Number of GET tagging requests proxied to replication target |
|
||||
| `minio_cluster_replication_proxied_put_tagging_requests_total` | (_Site Replication Only_)Number of PUT tagging requests proxied to replication target |
|
||||
| `minio_cluster_replication_proxied_get_requests_failures` | (_Site Replication Only_)Number of failures in GET requests proxied to replication target |
|
||||
| `minio_cluster_replication_proxied_head_requests_failures` | (_Site Replication Only_)Number of failures in HEAD requests proxied to replication target |
|
||||
| `minio_cluster_replication_proxied_delete_tagging_requests_failures` | (_Site Replication Only_)Number of failures proxying DELETE tagging requests to replication target |
|
||||
| `minio_cluster_replication_proxied_get_tagging_requests_failures` | (_Site Replication Only_)Number of failures proxying GET tagging requests to replication target |
|
||||
| `minio_cluster_replication_proxied_put_tagging_requests_failures` | (_Site Replication Only_)Number of failures proxying PUT tagging requests to replication target |
|
||||
|
||||
|
||||
## Node Replication Metrics
|
||||
|
||||
Metrics marked as ``Site Replication Only`` only populate on deployments with [Site Replication](https://docs.min.io/community/minio-object-store/operations/install-deploy-manage/multi-site-replication.html) configurations.
|
||||
For deployments with [bucket](https://docs.min.io/community/minio-object-store/administration/bucket-replication.html) or [batch](https://docs.min.io/community/minio-object-store/administration/batch-framework.html#replicate) configurations, these metrics populate instead under the [Bucket Metrics](#bucket-metrics) endpoint.
|
||||
|
||||
| Name | Description
|
||||
|:-----------------------------------------------------------|:---------------------------------------------------------------------------------------------------------|
|
||||
| `minio_node_replication_current_active_workers` | Total number of active replication workers |
|
||||
| `minio_node_replication_average_active_workers` | Average number of active replication workers |
|
||||
| `minio_node_replication_max_active_workers` | Maximum number of active replication workers seen since server start |
|
||||
| `minio_node_replication_link_online` | Reports whether the replication link is online (1) or offline (0). |
|
||||
| `minio_node_replication_link_offline_duration_seconds` | Total duration of replication link being offline in seconds since last offline event |
|
||||
| `minio_node_replication_link_downtime_duration_seconds` | Total downtime of replication link in seconds since server start |
|
||||
| `minio_node_replication_average_link_latency_ms` | Average replication link latency in milliseconds |
|
||||
| `minio_node_replication_max_link_latency_ms` | Maximum replication link latency in milliseconds seen since server start |
|
||||
| `minio_node_replication_current_link_latency_ms` | Current replication link latency in milliseconds |
|
||||
| `minio_node_replication_current_transfer_rate` | Current replication transfer rate in bytes/sec |
|
||||
| `minio_node_replication_average_transfer_rate` | Average replication transfer rate in bytes/sec |
|
||||
| `minio_node_replication_max_transfer_rate` | Maximum replication transfer rate in bytes/sec seen since server start |
|
||||
| `minio_node_replication_last_minute_queued_count` | Total number of objects queued for replication in the last full minute |
|
||||
| `minio_node_replication_last_minute_queued_bytes` | Total number of bytes queued for replication in the last full minute |
|
||||
| `minio_node_replication_average_queued_count` | Average number of objects queued for replication since server start |
|
||||
| `minio_node_replication_average_queued_bytes` | Average number of bytes queued for replication since server start |
|
||||
| `minio_node_replication_max_queued_bytes` | Maximum number of bytes queued for replication seen since server start |
|
||||
| `minio_node_replication_max_queued_count` | Maximum number of objects queued for replication seen since server start |
|
||||
| `minio_node_replication_recent_backlog_count` | Total number of objects seen in replication backlog in the last 5 minutes |
|
||||
|
||||
## Healing Metrics
|
||||
|
||||
| Name | Description |
|
||||
|:---------------------------------------------|:-----------------------------------------------------------------|
|
||||
| `minio_heal_objects_errors_total` | Objects for which healing failed in current self healing run. |
|
||||
| `minio_heal_objects_heal_total` | Objects healed in current self healing run. |
|
||||
| `minio_heal_objects_total` | Objects scanned in current self healing run. |
|
||||
| `minio_heal_time_last_activity_nano_seconds` | Time elapsed (in nano seconds) since last self healing activity. |
|
||||
|
||||
## Inter Node Metrics
|
||||
|
||||
| Name | Description |
|
||||
|:------------------------------------------|:--------------------------------------------------------|
|
||||
| `minio_inter_node_traffic_dial_avg_time` | Average time of internodes TCP dial calls. |
|
||||
| `minio_inter_node_traffic_dial_errors` | Total number of internode TCP dial timeouts and errors. |
|
||||
| `minio_inter_node_traffic_errors_total` | Total number of failed internode calls. |
|
||||
| `minio_inter_node_traffic_received_bytes` | Total number of bytes received from other peer nodes. |
|
||||
| `minio_inter_node_traffic_sent_bytes` | Total number of bytes sent to the other peer nodes. |
|
||||
|
||||
## Bucket Notification Metrics
|
||||
|
||||
| Name | Description |
|
||||
|:-----------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| `minio_notify_current_send_in_progress` | Number of concurrent async Send calls active to all targets (deprecated, please use `minio_notify_target_current_send_in_progress` instead) |
|
||||
| `minio_notify_events_errors_total` | Events that were failed to be sent to the targets (deprecated, please use `minio_notify_target_failed_events` instead) |
|
||||
| `minio_notify_events_sent_total` | Total number of events sent to the targets (deprecated, please use `minio_notify_target_total_events` instead) |
|
||||
| `minio_notify_events_skipped_total` | Events that were skipped to be sent to the targets due to the in-memory queue being full |
|
||||
| `minio_notify_target_current_send_in_progress` | Number of concurrent async Send calls active to the target |
|
||||
| `minio_notify_target_queue_length` | Number of events currently staged in the queue_dir configured for the target. |
|
||||
| `minio_notify_target_total_events` | Total number of events sent (or) queued to the target |
|
||||
|
||||
## S3 API Request Metrics
|
||||
|
||||
| Name | Description |
|
||||
|:----------------------------------------------|:---------------------------------------------------------|
|
||||
| `minio_s3_requests_4xx_errors_total` | Total number S3 requests with (4xx) errors. |
|
||||
| `minio_s3_requests_5xx_errors_total` | Total number S3 requests with (5xx) errors. |
|
||||
| `minio_s3_requests_canceled_total` | Total number S3 requests canceled by the client. |
|
||||
| `minio_s3_requests_errors_total` | Total number S3 requests with (4xx and 5xx) errors. |
|
||||
| `minio_s3_requests_incoming_total` | Volatile number of total incoming S3 requests. |
|
||||
| `minio_s3_requests_inflight_total` | Total number of S3 requests currently in flight. |
|
||||
| `minio_s3_requests_rejected_auth_total` | Total number S3 requests rejected for auth failure. |
|
||||
| `minio_s3_requests_rejected_header_total` | Total number S3 requests rejected for invalid header. |
|
||||
| `minio_s3_requests_rejected_invalid_total` | Total number S3 invalid requests. |
|
||||
| `minio_s3_requests_rejected_timestamp_total` | Total number S3 requests rejected for invalid timestamp. |
|
||||
| `minio_s3_requests_total` | Total number S3 requests. |
|
||||
| `minio_s3_requests_waiting_total` | Number of S3 requests in the waiting queue. |
|
||||
| `minio_s3_requests_ttfb_seconds_distribution` | Distribution of the time to first byte across API calls. |
|
||||
| `minio_s3_traffic_received_bytes` | Total number of s3 bytes received. |
|
||||
| `minio_s3_traffic_sent_bytes` | Total number of s3 bytes sent. |
|
||||
|
||||
## Software Metrics
|
||||
|
||||
| Name | Description |
|
||||
|:------------------------------|:---------------------------------------|
|
||||
| `minio_software_commit_info` | Git commit hash for the MinIO release. |
|
||||
| `minio_software_version_info` | MinIO Release tag for the server. |
|
||||
|
||||
## Drive Metrics
|
||||
|
||||
| Name | Description |
|
||||
|:---------------------------------------|:--------------------------------------------------------------------|
|
||||
| `minio_node_drive_free_bytes` | Total storage available on a drive. |
|
||||
| `minio_node_drive_free_inodes` | Total free inodes. |
|
||||
| `minio_node_drive_latency_us` | Average last minute latency in µs for drive API storage operations. |
|
||||
| `minio_node_drive_offline_total` | Total drives offline in this node. |
|
||||
| `minio_node_drive_online_total` | Total drives online in this node. |
|
||||
| `minio_node_drive_total` | Total drives in this node. |
|
||||
| `minio_node_drive_total_bytes` | Total storage on a drive. |
|
||||
| `minio_node_drive_used_bytes` | Total storage used on a drive. |
|
||||
| `minio_node_drive_errors_timeout` | Total number of drive timeout errors since server start |
|
||||
| `minio_node_drive_errors_ioerror` | Total number of drive I/O errors since server start |
|
||||
| `minio_node_drive_errors_availability` | Total number of drive I/O errors, timeouts since server start |
|
||||
| `minio_node_drive_io_waiting` | Total number I/O operations waiting on drive |
|
||||
|
||||
## Identity and Access Management (IAM) Metrics
|
||||
|
||||
| Name | Description |
|
||||
|:-------------------------------------------|:------------------------------------------------------------|
|
||||
| `minio_node_iam_last_sync_duration_millis` | Last successful IAM data sync duration in milliseconds. |
|
||||
| `minio_node_iam_since_last_sync_millis` | Time (in milliseconds) since last successful IAM data sync. |
|
||||
| `minio_node_iam_sync_failures` | Number of failed IAM data syncs since server start. |
|
||||
| `minio_node_iam_sync_successes` | Number of successful IAM data syncs since server start. |
|
||||
|
||||
## Information Lifecycle Management (ILM) Metrics
|
||||
|
||||
| Name | Description |
|
||||
|:-------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------|
|
||||
| `minio_node_ilm_expiry_pending_tasks` | Number of pending ILM expiry tasks in the queue. |
|
||||
| `minio_node_ilm_transition_active_tasks` | Number of active ILM transition tasks. |
|
||||
| `minio_node_ilm_transition_pending_tasks` | Number of pending ILM transition tasks in the queue. |
|
||||
| `minio_node_ilm_transition_missed_immediate_tasks` | Number of missed immediate ILM transition tasks. |
|
||||
| `minio_node_ilm_versions_scanned` | Total number of object versions checked for ilm actions since server start. |
|
||||
| `minio_node_ilm_action_count_delete_action` | Total action outcome of lifecycle checks since server start for deleting object |
|
||||
| `minio_node_ilm_action_count_delete_version_action` | Total action outcome of lifecycle checks since server start for deleting a version |
|
||||
| `minio_node_ilm_action_count_transition_action` | Total action outcome of lifecycle checks since server start for transition of an object |
|
||||
| `minio_node_ilm_action_count_transition_version_action` | Total action outcome of lifecycle checks since server start for transition of a particular object version |
|
||||
| `minio_node_ilm_action_count_delete_restored_action` | Total action outcome of lifecycle checks since server start for deletion of temporarily restored object |
|
||||
| `minio_node_ilm_action_count_delete_restored_version_action` | Total action outcome of lifecycle checks since server start for deletion of a temporarily restored version |
|
||||
| `minio_node_ilm_action_count_delete_all_versions_action` | Total action outcome of lifecycle checks since server start for deletion of all versions |
|
||||
|
||||
## Tier Metrics
|
||||
|
||||
| Name | Description |
|
||||
|:---------------------------------------------------|:----------------------------------------------------------------------------|
|
||||
| `minio_node_tier_tier_ttlb_seconds_distribution` | Distribution of time to last byte for objects downloaded from warm tier |
|
||||
| `minio_node_tier_requests_success` | Number of requests to download object from warm tier that were successful |
|
||||
| `minio_node_tier_requests_failure` | Number of requests to download object from warm tier that were failure |
|
||||
|
||||
## System Metrics
|
||||
|
||||
| Name | Description |
|
||||
|:-------------------------------------------|:----------------------------------------------------------------------------------------------------------------|
|
||||
| `minio_node_file_descriptor_limit_total` | Limit on total number of open file descriptors for the MinIO Server process. |
|
||||
| `minio_node_file_descriptor_open_total` | Total number of open file descriptors by the MinIO Server process. |
|
||||
| `minio_node_go_routine_total` | Total number of go routines running. |
|
||||
| `minio_node_io_rchar_bytes` | Total bytes read by the process from the underlying storage system including cache, /proc/[pid]/io rchar. |
|
||||
| `minio_node_io_read_bytes` | Total bytes read by the process from the underlying storage system, /proc/[pid]/io read_bytes. |
|
||||
| `minio_node_io_wchar_bytes` | Total bytes written by the process to the underlying storage system including page cache, /proc/[pid]/io wchar. |
|
||||
| `minio_node_io_write_bytes` | Total bytes written by the process to the underlying storage system, /proc/[pid]/io write_bytes. |
|
||||
| `minio_node_process_cpu_total_seconds` | Total user and system CPU time spent in seconds by the process. |
|
||||
| `minio_node_process_resident_memory_bytes` | Resident memory size in bytes. |
|
||||
| `minio_node_process_virtual_memory_bytes` | Virtual memory size in bytes. |
|
||||
| `minio_node_process_starttime_seconds` | Start time for MinIO process per node, time in seconds since Unix epoc. |
|
||||
| `minio_node_process_uptime_seconds` | Uptime for MinIO process per node in seconds. |
|
||||
|
||||
## Scanner Metrics
|
||||
|
||||
| Name | Description |
|
||||
|:-------------------------------------------|:------------------------------------------------------------|
|
||||
| `minio_node_scanner_bucket_scans_finished` | Total number of bucket scans finished since server start. |
|
||||
| `minio_node_scanner_bucket_scans_started` | Total number of bucket scans started since server start. |
|
||||
| `minio_node_scanner_directories_scanned` | Total number of directories scanned since server start. |
|
||||
| `minio_node_scanner_objects_scanned` | Total number of unique objects scanned since server start. |
|
||||
| `minio_node_scanner_versions_scanned` | Total number of object versions scanned since server start. |
|
||||
| `minio_node_syscall_read_total` | Total read SysCalls to the kernel. /proc/[pid]/io syscr. |
|
||||
| `minio_node_syscall_write_total` | Total write SysCalls to the kernel. /proc/[pid]/io syscw. |
|
||||
| `minio_usage_last_activity_nano_seconds` | Time elapsed (in nano seconds) since last scan activity. |
|
||||
|
||||
# Bucket Metrics
|
||||
|
||||
MinIO collects the following metrics at the bucket level.
|
||||
Each metric includes the ``bucket`` label to identify the corresponding bucket.
|
||||
Metrics may include one or more additional labels, such as the server that calculated that metric.
|
||||
|
||||
These metrics can be obtained from any MinIO server once per collection by using the following URL:
|
||||
|
||||
```shell
|
||||
https://HOSTNAME:PORT/minio/v2/metrics/bucket
|
||||
```
|
||||
|
||||
Replace ``HOSTNAME:PORT`` with the hostname of your MinIO deployment.
|
||||
For deployments behind a load balancer, use the load balancer hostname instead of a single node hostname.
|
||||
|
||||
## Distribution Metrics
|
||||
|
||||
| Name | Description |
|
||||
|:--------------------------------------------|:--------------------------------------------------------------------------------|
|
||||
| `minio_bucket_objects_size_distribution` | Distribution of object sizes in the bucket, includes label for the bucket name. |
|
||||
| `minio_bucket_objects_version_distribution` | Distribution of object sizes in a bucket, by number of versions |
|
||||
|
||||
## Replication Metrics
|
||||
|
||||
These metrics only populate on deployments with [Bucket Replication](https://docs.min.io/community/minio-object-store/administration/bucket-replication.html) or [Batch Replication](https://docs.min.io/community/minio-object-store/administration/batch-framework.html) configurations.
|
||||
For deployments with [Site Replication](https://docs.min.io/community/minio-object-store/operations/install-deploy-manage/multi-site-replication.html) configured, select metrics populate under the [Cluster Metrics](#cluster-metrics) endpoint.
|
||||
|
||||
| Name | Description |
|
||||
|:----------------------------------------------------|:---------------------------------------------------------------------------------|
|
||||
| `minio_bucket_replication_last_minute_failed_bytes` | Total number of bytes failed at least once to replicate in the last full minute. |
|
||||
| `minio_bucket_replication_last_minute_failed_count` | Total number of objects which failed replication in the last full minute. |
|
||||
| `minio_bucket_replication_last_hour_failed_bytes` | Total number of bytes failed at least once to replicate in the last full hour. |
|
||||
| `minio_bucket_replication_last_hour_failed_count` | Total number of objects which failed replication in the last full hour. |
|
||||
| `minio_bucket_replication_total_failed_bytes` | Total number of bytes failed at least once to replicate since server start. |
|
||||
| `minio_bucket_replication_total_failed_count` | Total number of objects which failed replication since server start. |
|
||||
| `minio_bucket_replication_latency_ms` | Replication latency in milliseconds. |
|
||||
| `minio_bucket_replication_received_bytes` | Total number of bytes replicated to this bucket from another source bucket. |
|
||||
| `minio_bucket_replication_received_count` | Total number of objects received by this bucket from another source bucket. |
|
||||
| `minio_bucket_replication_sent_bytes` | Total number of bytes replicated to the target bucket. |
|
||||
| `minio_bucket_replication_sent_count` | Total number of objects replicated to the target bucket. |
|
||||
| `minio_bucket_replication_credential_errors` | Total number of replication credential errors since server start |
|
||||
| `minio_bucket_replication_proxied_get_requests_total` | Number of GET requests proxied to replication target |
|
||||
| `minio_bucket_replication_proxied_head_requests_total` | Number of HEAD requests proxied to replication target |
|
||||
| `minio_bucket_replication_proxied_delete_tagging_requests_total` | Number of DELETE tagging requests proxied to replication target |
|
||||
| `minio_bucket_replication_proxied_get_tagging_requests_total` | Number of GET tagging requests proxied to replication target |
|
||||
| `minio_bucket_replication_proxied_put_tagging_requests_total` | Number of PUT tagging requests proxied to replication target |
|
||||
| `minio_bucket_replication_proxied_get_requests_failures` | Number of failures in GET requests proxied to replication target |
|
||||
| `minio_bucket_replication_proxied_head_requests_failures` | Number of failures in HEAD requests proxied to replication target |
|
||||
| `minio_bucket_replication_proxied_delete_tagging_requests_failures` | Number of failures in DELETE tagging proxy requests to replication target |
|
||||
| `minio_bucket_replication_proxied_get_tagging_requests_failures` |Number of failures in GET tagging proxy requests to replication target |
|
||||
| `minio_bucket_replication_proxied_put_tagging_requests_failures` | Number of failures in PUT tagging proxy requests to replication target |
|
||||
|
||||
## Traffic Metrics
|
||||
|
||||
| Name | Description |
|
||||
|:--------------------------------------|:---------------------------------------------------|
|
||||
| `minio_bucket_traffic_received_bytes` | Total number of S3 bytes received for this bucket. |
|
||||
| `minio_bucket_traffic_sent_bytes` | Total number of S3 bytes sent for this bucket. |
|
||||
|
||||
## Usage Metrics
|
||||
|
||||
| Name | Description |
|
||||
|:----------------------------------------|:--------------------------------------------------|
|
||||
| `minio_bucket_usage_object_total` | Total number of objects. |
|
||||
| `minio_bucket_usage_version_total` | Total number of versions (includes delete marker) |
|
||||
| `minio_bucket_usage_deletemarker_total` | Total number of delete markers. |
|
||||
| `minio_bucket_usage_total_bytes` | Total bucket size in bytes. |
|
||||
| `minio_bucket_quota_total_bytes` | Total bucket quota size in bytes. |
|
||||
|
||||
## Requests Metrics
|
||||
|
||||
| Name | Description |
|
||||
|:--------------------------------------------------|:----------------------------------------------------------------|
|
||||
| `minio_bucket_requests_4xx_errors_total` | Total number of S3 requests with (4xx) errors on a bucket. |
|
||||
| `minio_bucket_requests_5xx_errors_total` | Total number of S3 requests with (5xx) errors on a bucket. |
|
||||
| `minio_bucket_requests_inflight_total` | Total number of S3 requests currently in flight on a bucket. |
|
||||
| `minio_bucket_requests_total` | Total number of S3 requests on a bucket. |
|
||||
| `minio_bucket_requests_canceled_total` | Total number S3 requests canceled by the client. |
|
||||
| `minio_bucket_requests_ttfb_seconds_distribution` | Distribution of time to first byte across API calls per bucket. |
|
||||
|
||||
# Resource Metrics
|
||||
|
||||
MinIO collects the following resource metrics at the node level.
|
||||
Each metric includes the `server` label to identify the corresponding node.
|
||||
Metrics may include one or more additional labels, such as the drive path, interface name, etc.
|
||||
|
||||
These metrics can be obtained from any MinIO server once per collection by using the following URL:
|
||||
|
||||
```shell
|
||||
https://HOSTNAME:PORT/minio/v2/metrics/resource
|
||||
```
|
||||
|
||||
Replace `HOSTNAME:PORT` with the hostname of your MinIO deployment.
|
||||
For deployments behind a load balancer, use the load balancer hostname instead of a single node hostname.
|
||||
|
||||
## Drive Resource Metrics
|
||||
|
||||
| Name | Description |
|
||||
|:-------------------------------------|:---------------------------------------------------------|
|
||||
| `minio_node_drive_total_bytes` | Total bytes on a drive. |
|
||||
| `minio_node_drive_used_bytes` | Used bytes on a drive. |
|
||||
| `minio_node_drive_total_inodes` | Total inodes on a drive. |
|
||||
| `minio_node_drive_used_inodes` | Total inodes used on a drive. |
|
||||
| `minio_node_drive_reads_per_sec` | Reads per second on a drive. |
|
||||
| `minio_node_drive_reads_kb_per_sec` | Kilobytes read per second on a drive. |
|
||||
| `minio_node_drive_reads_await` | Average time for read requests to be served on a drive. |
|
||||
| `minio_node_drive_writes_per_sec` | Writes per second on a drive. |
|
||||
| `minio_node_drive_writes_kb_per_sec` | Kilobytes written per second on a drive. |
|
||||
| `minio_node_drive_writes_await` | Average time for write requests to be served on a drive. |
|
||||
| `minio_node_drive_perc_util` | Percentage of time the disk was busy since uptime. |
|
||||
|
||||
## Network Interface Metrics
|
||||
|
||||
| Name | Description |
|
||||
|:------------------------------|:-----------------------------------------------------------|
|
||||
| `minio_node_if_rx_bytes` | Bytes received on the interface in 60s. |
|
||||
| `minio_node_if_rx_bytes_avg` | Bytes received on the interface in 60s (avg) since uptime. |
|
||||
| `minio_node_if_rx_bytes_max` | Bytes received on the interface in 60s (max) since uptime. |
|
||||
| `minio_node_if_rx_errors` | Receive errors in 60s. |
|
||||
| `minio_node_if_rx_errors_avg` | Receive errors in 60s (avg). |
|
||||
| `minio_node_if_rx_errors_max` | Receive errors in 60s (max). |
|
||||
| `minio_node_if_tx_bytes` | Bytes transmitted in 60s. |
|
||||
| `minio_node_if_tx_bytes_avg` | Bytes transmitted in 60s (avg). |
|
||||
| `minio_node_if_tx_bytes_max` | Bytes transmitted in 60s (max). |
|
||||
| `minio_node_if_tx_errors` | Transmit errors in 60s. |
|
||||
| `minio_node_if_tx_errors_avg` | Transmit errors in 60s (avg). |
|
||||
| `minio_node_if_tx_errors_max` | Transmit errors in 60s (max). |
|
||||
|
||||
## CPU Metrics
|
||||
|
||||
| Name | Description |
|
||||
|:-------------------------------------|:-------------------------------------------|
|
||||
| `minio_node_cpu_avg_user` | CPU user time. |
|
||||
| `minio_node_cpu_avg_user_avg` | CPU user time (avg). |
|
||||
| `minio_node_cpu_avg_user_max` | CPU user time (max). |
|
||||
| `minio_node_cpu_avg_system` | CPU system time. |
|
||||
| `minio_node_cpu_avg_system_avg` | CPU system time (avg). |
|
||||
| `minio_node_cpu_avg_system_max` | CPU system time (max). |
|
||||
| `minio_node_cpu_avg_idle` | CPU idle time. |
|
||||
| `minio_node_cpu_avg_idle_avg` | CPU idle time (avg). |
|
||||
| `minio_node_cpu_avg_idle_max` | CPU idle time (max). |
|
||||
| `minio_node_cpu_avg_iowait` | CPU ioWait time. |
|
||||
| `minio_node_cpu_avg_iowait_avg` | CPU ioWait time (avg). |
|
||||
| `minio_node_cpu_avg_iowait_max` | CPU ioWait time (max). |
|
||||
| `minio_node_cpu_avg_nice` | CPU nice time. |
|
||||
| `minio_node_cpu_avg_nice_avg` | CPU nice time (avg). |
|
||||
| `minio_node_cpu_avg_nice_max` | CPU nice time (max). |
|
||||
| `minio_node_cpu_avg_steal` | CPU steam time. |
|
||||
| `minio_node_cpu_avg_steal_avg` | CPU steam time (avg). |
|
||||
| `minio_node_cpu_avg_steal_max` | CPU steam time (max). |
|
||||
| `minio_node_cpu_avg_load1` | CPU load average 1min. |
|
||||
| `minio_node_cpu_avg_load1_avg` | CPU load average 1min (avg). |
|
||||
| `minio_node_cpu_avg_load1_max` | CPU load average 1min (max). |
|
||||
| `minio_node_cpu_avg_load1_perc` | CPU load average 1min (percentage). |
|
||||
| `minio_node_cpu_avg_load1_perc_avg` | CPU load average 1min (percentage) (avg). |
|
||||
| `minio_node_cpu_avg_load1_perc_max` | CPU load average 1min (percentage) (max). |
|
||||
| `minio_node_cpu_avg_load5` | CPU load average 5min. |
|
||||
| `minio_node_cpu_avg_load5_avg` | CPU load average 5min (avg). |
|
||||
| `minio_node_cpu_avg_load5_max` | CPU load average 5min (max). |
|
||||
| `minio_node_cpu_avg_load5_perc` | CPU load average 5min (percentage). |
|
||||
| `minio_node_cpu_avg_load5_perc_avg` | CPU load average 5min (percentage) (avg). |
|
||||
| `minio_node_cpu_avg_load5_perc_max` | CPU load average 5min (percentage) (max). |
|
||||
| `minio_node_cpu_avg_load15` | CPU load average 15min. |
|
||||
| `minio_node_cpu_avg_load15_avg` | CPU load average 15min (avg). |
|
||||
| `minio_node_cpu_avg_load15_max` | CPU load average 15min (max). |
|
||||
| `minio_node_cpu_avg_load15_perc` | CPU load average 15min (percentage). |
|
||||
| `minio_node_cpu_avg_load15_perc_avg` | CPU load average 15min (percentage) (avg). |
|
||||
| `minio_node_cpu_avg_load15_perc_max` | CPU load average 15min (percentage) (max). |
|
||||
|
||||
## Memory Metrics
|
||||
|
||||
| Name | Description |
|
||||
|:-------------------------------|:------------------------------------------|
|
||||
| `minio_node_mem_available` | Available memory on the node. |
|
||||
| `minio_node_mem_available_avg` | Available memory on the node (avg). |
|
||||
| `minio_node_mem_available_max` | Available memory on the node (max). |
|
||||
| `minio_node_mem_buffers` | Buffers memory on the node. |
|
||||
| `minio_node_mem_buffers_avg` | Buffers memory on the node (avg). |
|
||||
| `minio_node_mem_buffers_max` | Buffers memory on the node (max). |
|
||||
| `minio_node_mem_cache` | Cache memory on the node. |
|
||||
| `minio_node_mem_cache_avg` | Cache memory on the node (avg). |
|
||||
| `minio_node_mem_cache_max` | Cache memory on the node (max). |
|
||||
| `minio_node_mem_free` | Free memory on the node. |
|
||||
| `minio_node_mem_free_avg` | Free memory on the node (avg). |
|
||||
| `minio_node_mem_free_max` | Free memory on the node (max). |
|
||||
| `minio_node_mem_shared` | Shared memory on the node. |
|
||||
| `minio_node_mem_shared_avg` | Shared memory on the node (avg). |
|
||||
| `minio_node_mem_shared_max` | Shared memory on the node (max). |
|
||||
| `minio_node_mem_total` | Total memory on the node. |
|
||||
| `minio_node_mem_total_avg` | Total memory on the node (avg). |
|
||||
| `minio_node_mem_total_max` | Total memory on the node (max). |
|
||||
| `minio_node_mem_used` | Used memory on the node. |
|
||||
| `minio_node_mem_used_avg` | Used memory on the node (avg). |
|
||||
| `minio_node_mem_used_max` | Used memory on the node (max). |
|
||||
| `minio_node_mem_used_perc` | Used memory percentage on the node. |
|
||||
| `minio_node_mem_used_perc_avg` | Used memory percentage on the node (avg). |
|
||||
| `minio_node_mem_used_perc_max` | Used memory percentage on the node (max). |
|
||||
BIN
docs/metrics/prometheus/minio-es-tolerance-alert.png
Normal file
BIN
docs/metrics/prometheus/minio-es-tolerance-alert.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 80 KiB |
417
docs/metrics/v3.md
Normal file
417
docs/metrics/v3.md
Normal file
@@ -0,0 +1,417 @@
|
||||
# Metrics Version 3
|
||||
|
||||
In metrics version 3, all metrics are available under the following base endpoint:
|
||||
|
||||
```
|
||||
/minio/metrics/v3
|
||||
```
|
||||
|
||||
To query metrics of a specific type, append the appropriate path to the base endpoint.
|
||||
Querying the base endpoint returns "404 Not Found."
|
||||
|
||||
Metrics are organized into groups at paths **relative** to the top-level endpoint above.
|
||||
|
||||
Metrics are also available using the [MinIO Admin Client](https://docs.min.io/community/minio-object-store/reference/minio-mc-admin.html) and the `mc admin prometheus metrics` command. For more information, see [Metrics and Alerts](https://docs.min.io/community/minio-object-store/operations/monitoring/metrics-and-alerts.html) in the MinIO Documentation.
|
||||
|
||||
## Metrics Request Handling
|
||||
|
||||
Each endpoint can be queried as needed via a scrape configuration in Prometheus or a compatible metrics collection tool. You should schedule scrape operations so a prior scrape completes before the next one begins.
|
||||
|
||||
For ease of configuration, each (non-empty) parent of the path serves all the metric endpoints at its child paths. For example, to query all system metrics scrape `/minio/metrics/v3/system/`.
|
||||
|
||||
Each metric endpoint may support multiple child endpoints. For example, the `/v3/system/` metric has multiple child groups as `/v3/system/<child>`. Querying the parent endpoint returns metrics for all child groups. Querying a child group returns only metrics for that child.
|
||||
|
||||
### Per-bucket Metrics
|
||||
|
||||
Metrics with a `/bucket` component in the path return results for each specified bucket in the deployment. These endpoints **require** providing a list of buckets as a `bucket` query parameter. The endpoint then returns only metrics for the given buckets, with the bucket name in a `bucket` label.
|
||||
|
||||
For example, to query API metrics for buckets `test1` and `test2`, make a scrape request to `/minio/metrics/v3/api/bucket?buckets=test1,test2`.
|
||||
|
||||
### List Available Metrics
|
||||
|
||||
Instead of a metrics scrape, you can list the metrics that would be returned by a path by adding a `list` query parameter. The MinIO server then lists all available metrics that could be returned. Note that during an actual metrics scrape only metrics with available _values_ are returned. Metrics with null values are omitted from the scrape results.
|
||||
|
||||
To set the output format, set the request `Content-Type` to the desired format. Accepted values are `application/json` for JSON output or `text/plain` for a Markdown-formatted table. The default is Markdown.
|
||||
|
||||
For example, the following returns a list of all available bucket metrics:
|
||||
```
|
||||
/minio/metrics/v3/api/bucket?list
|
||||
```
|
||||
|
||||
## Metric Categories
|
||||
|
||||
At a high level, metrics are grouped into categories as described in the following sections. The path in each of the tables is relative to the top-level endpoint. Note that the standard GoCollector metrics are not shown.
|
||||
|
||||
### Request metrics
|
||||
|
||||
Metrics about requests served by the current node.
|
||||
|
||||
| Path | Description |
|
||||
|-----------------|-----------------------------------------------|
|
||||
| `/api/requests` | Metrics over all requests. |
|
||||
| `/bucket/api` | Metrics over all requests for a given bucket. |
|
||||
|
||||
#### `/api/requests`
|
||||
|
||||
| Name | Description | Labels |
|
||||
|------------------------------------------------|--------------------------------------------------------------------------------|----------------------------------------------|
|
||||
| `minio_api_requests_rejected_auth_total` | Total number of requests rejected for auth failure. <br><br>Type: counter | `type`, `pool_index`, `server` |
|
||||
| `minio_api_requests_rejected_header_total` | Total number of requests rejected for invalid header. <br><br>Type: counter | `type`, `pool_index`, `server` |
|
||||
| `minio_api_requests_rejected_timestamp_total` | Total number of requests rejected for invalid timestamp. <br><br>Type: counter | `type`, `pool_index`, `server` |
|
||||
| `minio_api_requests_rejected_invalid_total` | Total number of invalid requests. <br><br>Type: counter | `type`, `pool_index`, `server` |
|
||||
| `minio_api_requests_waiting_total` | Total number of requests in the waiting queue. <br><br>Type: gauge | `type`, `pool_index`, `server` |
|
||||
| `minio_api_requests_incoming_total` | Total number of incoming requests. <br><br>Type: gauge | `type`, `pool_index`, `server` |
|
||||
| `minio_api_requests_inflight_total` | Total number of requests currently in flight. <br><br>Type: gauge | `name`, `type`, `pool_index`, `server` |
|
||||
| `minio_api_requests_total` | Total number of requests. <br><br>Type: counter | `name`, `type`, `pool_index`, `server` |
|
||||
| `minio_api_requests_errors_total` | Total number of requests with 4xx or 5xx errors. <br><br>Type: counter | `name`, `type`, `pool_index`, `server` |
|
||||
| `minio_api_requests_5xx_errors_total` | Total number of requests with 5xx errors. <br><br>Type: counter | `name`, `type`, `pool_index`, `server` |
|
||||
| `minio_api_requests_4xx_errors_total` | Total number of requests with 4xx errors. <br><br>Type: counter | `name`, `type`, `pool_index`, `server` |
|
||||
| `minio_api_requests_canceled_total` | Total number of requests canceled by the client. <br><br>Type: counter | `name`, `type`, `pool_index`, `server` |
|
||||
| `minio_api_requests_ttfb_seconds_distribution` | Distribution of time to first byte across API calls. <br><br>Type: counter | `name`, `type`, `le`, `pool_index`, `server` |
|
||||
| `minio_api_requests_traffic_sent_bytes` | Total number of bytes sent. <br><br>Type: counter | `type`, `pool_index`, `server` |
|
||||
| `minio_api_requests_traffic_received_bytes` | Total number of bytes received. <br><br>Type: counter | `type`, `pool_index`, `server` |
|
||||
|
||||
#### `/bucket/api`
|
||||
|
||||
| Name | Description | Labels |
|
||||
|----------------------------------------------|-----------------------------------------------------------------------------------------|--------------------------------------------------------|
|
||||
| `minio_bucket_api_traffic_received_bytes` | Total number of bytes sent for a bucket. <br><br>Type: counter | `bucket`, `type`, `server`, `pool_index` |
|
||||
| `minio_bucket_api_traffic_sent_bytes` | Total number of bytes received for a bucket. <br><br>Type: counter | `bucket`, `type`, `server`, `pool_index` |
|
||||
| `minio_bucket_api_inflight_total` | Total number of requests currently in flight for a bucket. <br><br>Type: gauge | `bucket`, `name`, `type`, `server`, `pool_index` |
|
||||
| `minio_bucket_api_total` | Total number of requests for a bucket. <br><br>Type: counter | `bucket`, `name`, `type`, `server`, `pool_index` |
|
||||
| `minio_bucket_api_canceled_total` | Total number of requests canceled by the client for a bucket. <br><br>Type: counter | `bucket`, `name`, `type`, `server`, `pool_index` |
|
||||
| `minio_bucket_api_4xx_errors_total` | Total number of requests with 4xx errors for a bucket. <br><br>Type: counter | `bucket`, `name`, `type`, `server`, `pool_index` |
|
||||
| `minio_bucket_api_5xx_errors_total` | Total number of requests with 5xx errors for a bucket. <br><br>Type: counter | `bucket`, `name`, `type`, `server`, `pool_index` |
|
||||
| `minio_bucket_api_ttfb_seconds_distribution` | Distribution of time to first byte across API calls for a bucket. <br><br>Type: counter | `bucket`, `name`, `le`, `type`, `server`, `pool_index` |
|
||||
|
||||
### Audit metrics
|
||||
|
||||
Metrics about the MinIO audit functionality.
|
||||
|
||||
| Path | Description |
|
||||
|----------|-----------------------------------------|
|
||||
| `/audit` | Metrics related to audit functionality. |
|
||||
|
||||
#### `/audit`
|
||||
|
||||
| Name | Description | Labels |
|
||||
|-----------------------------------|---------------------------------------------------------------------------------|-----------------------|
|
||||
| `minio_audit_failed_messages` | Total number of messages that failed to send since start. <br><br>Type: counter | `target_id`, `server` |
|
||||
| `minio_audit_target_queue_length` | Number of unsent messages in queue for target. <br><br>Type: gauge | `target_id`, `server` |
|
||||
| `minio_audit_total_messages` | Total number of messages sent since start. <br><br>Type: counter | `target_id`, `server` |
|
||||
|
||||
### Cluster metrics
|
||||
|
||||
Metrics about an entire MinIO cluster.
|
||||
|
||||
| Path | Description |
|
||||
|--------------------------|--------------------------------|
|
||||
| `/cluster/config` | Cluster configuration metrics. |
|
||||
| `/cluster/erasure-set` | Erasure set metrics. |
|
||||
| `/cluster/health` | Cluster health metrics. |
|
||||
| `/cluster/iam` | Cluster iam metrics. |
|
||||
| `/cluster/usage/buckets` | Object statistics by bucket. |
|
||||
| `/cluster/usage/objects` | Object statistics. |
|
||||
|
||||
#### `/cluster/config`
|
||||
|
||||
| Name | Description | Labels |
|
||||
|----------------------------------------|--------------------------------------------------------------|--------|
|
||||
| `minio_cluster_config_rrs_parity` | Reduced redundancy storage class parity. <br><br>Type: gauge | |
|
||||
| `minio_cluster_config_standard_parity` | Standard storage class parity. <br><br>Type: gauge | |
|
||||
|
||||
#### `/cluster/erasure-set`
|
||||
|
||||
| Name | Description | Labels |
|
||||
|--------------------------------------------------|---------------------------------------------------------------------------------------------------------|---------------------|
|
||||
| `minio_cluster_erasure_set_overall_write_quorum` | Overall write quorum across pools and sets. <br><br>Type: gauge | |
|
||||
| `minio_cluster_erasure_set_overall_health` | Overall health across pools and sets (1=healthy, 0=unhealthy). <br><br>Type: gauge | |
|
||||
| `minio_cluster_erasure_set_read_quorum` | Read quorum for the erasure set in a pool. <br><br>Type: gauge | `pool_id`, `set_id` |
|
||||
| `minio_cluster_erasure_set_write_quorum` | Write quorum for the erasure set in a pool. <br><br>Type: gauge | `pool_id`, `set_id` |
|
||||
| `minio_cluster_erasure_set_online_drives_count` | Count of online drives in the erasure set in a pool. <br><br>Type: gauge | `pool_id`, `set_id` |
|
||||
| `minio_cluster_erasure_set_healing_drives_count` | Count of healing drives in the erasure set in a pool. <br><br>Type: gauge | `pool_id`, `set_id` |
|
||||
| `minio_cluster_erasure_set_health` | Health of the erasure set in a pool (1=healthy, 0=unhealthy). <br><br>Type: gauge | `pool_id`, `set_id` |
|
||||
| `minio_cluster_erasure_set_read_tolerance` | Number of drive failures that can be tolerated without disrupting read operations. <br><br>Type: gauge | `pool_id`, `set_id` |
|
||||
| `minio_cluster_erasure_set_write_tolerance` | Number of drive failures that can be tolerated without disrupting write operations. <br><br>Type: gauge | `pool_id`, `set_id` |
|
||||
| `minio_cluster_erasure_set_read_health` | Health of the erasure set in a pool for read operations (1=healthy, 0=unhealthy). <br><br>Type: gauge | `pool_id`, `set_id` |
|
||||
| `minio_cluster_erasure_set_write_health` | Health of the erasure set in a pool for write operations (1=healthy, 0=unhealthy). <br><br>Type: gauge | `pool_id`, `set_id` |
|
||||
|
||||
#### `/cluster/health`
|
||||
|
||||
| Name | Description | Labels |
|
||||
|----------------------------------------------------|---------------------------------------------------------------------|--------|
|
||||
| `minio_cluster_health_drives_offline_count` | Count of offline drives in the cluster. <br><br>Type: gauge | |
|
||||
| `minio_cluster_health_drives_online_count` | Count of online drives in the cluster. <br><br>Type: gauge | |
|
||||
| `minio_cluster_health_drives_count` | Count of all drives in the cluster. <br><br>Type: gauge | |
|
||||
| `minio_cluster_health_nodes_offline_count` | Count of offline nodes in the cluster. <br><br>Type: gauge | |
|
||||
| `minio_cluster_health_nodes_online_count` | Count of online nodes in the cluster. <br><br>Type: gauge | |
|
||||
| `minio_cluster_health_capacity_raw_total_bytes` | Total cluster raw storage capacity in bytes. <br><br>Type: gauge | |
|
||||
| `minio_cluster_health_capacity_raw_free_bytes` | Total cluster raw storage free in bytes. <br><br>Type: gauge | |
|
||||
| `minio_cluster_health_capacity_usable_total_bytes` | Total cluster usable storage capacity in bytes. <br><br>Type: gauge | |
|
||||
| `minio_cluster_health_capacity_usable_free_bytes` | Total cluster usable storage free in bytes. <br><br>Type: gauge | |
|
||||
|
||||
#### `/cluster/iam`
|
||||
|
||||
| Name | Description | Labels |
|
||||
|-----------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------|--------|
|
||||
| `minio_cluster_iam_last_sync_duration_millis` | Last successful IAM data sync duration in milliseconds. <br><br>Type: counter | |
|
||||
| `minio_cluster_iam_plugin_authn_service_failed_requests_minute` | When plugin authentication is configured, returns failed requests count in the last full minute. <br><br>Type: counter | |
|
||||
| `minio_cluster_iam_plugin_authn_service_last_fail_seconds` | When plugin authentication is configured, returns time (in seconds) since the last failed request to the service. <br><br>Type: counter | |
|
||||
| `minio_cluster_iam_plugin_authn_service_last_succ_seconds` | When plugin authentication is configured, returns time (in seconds) since the last successful request to the service. <br><br>Type: counter | |
|
||||
| `minio_cluster_iam_plugin_authn_service_succ_avg_rtt_ms_minute` | When plugin authentication is configured, returns average round-trip time of successful requests in the last full minute. <br><br>Type: counter | |
|
||||
| `minio_cluster_iam_plugin_authn_service_succ_max_rtt_ms_minute` | When plugin authentication is configured, returns maximum round-trip time of successful requests in the last full minute. <br><br>Type: counter | |
|
||||
| `minio_cluster_iam_plugin_authn_service_total_requests_minute` | When plugin authentication is configured, returns total requests count in the last full minute. <br><br>Type: counter | |
|
||||
| `minio_cluster_iam_since_last_sync_millis` | Time (in milliseconds) since last successful IAM data sync. <br><br>Type: counter | |
|
||||
| `minio_cluster_iam_sync_failures` | Number of failed IAM data syncs since server start. <br><br>Type: counter | |
|
||||
| `minio_cluster_iam_sync_successes` | Number of successful IAM data syncs since server start. <br><br>Type: counter | |
|
||||
|
||||
#### `/cluster/usage/buckets`
|
||||
|
||||
| Name | Description | Labels |
|
||||
|-----------------------------------------------------------------|--------------------------------------------------------------------------------------|-------------------|
|
||||
| `minio_cluster_usage_buckets_since_last_update_seconds` | Time since last update of usage metrics in seconds. <br><br>Type: gauge | |
|
||||
| `minio_cluster_usage_buckets_total_bytes` | Total bucket size in bytes. <br><br>Type: gauge | `bucket` |
|
||||
| `minio_cluster_usage_buckets_objects_count` | Total object count in bucket. <br><br>Type: gauge | `bucket` |
|
||||
| `minio_cluster_usage_buckets_versions_count` | Total object versions count in bucket, including delete markers. <br><br>Type: gauge | `bucket` |
|
||||
| `minio_cluster_usage_buckets_delete_markers_count` | Total delete markers count in bucket. <br><br>Type: gauge | `bucket` |
|
||||
| `minio_cluster_usage_buckets_quota_total_bytes` | Total bucket quota in bytes. <br><br>Type: gauge | `bucket` |
|
||||
| `minio_cluster_usage_buckets_object_size_distribution` | Bucket object size distribution. <br><br>Type: gauge | `range`, `bucket` |
|
||||
| `minio_cluster_usage_buckets_object_version_count_distribution` | Bucket object version count distribution. <br><br>Type: gauge | `range`, `bucket` |
|
||||
|
||||
#### `/cluster/usage/objects`
|
||||
|
||||
| Name | Description | Labels |
|
||||
|----------------------------------------------------------|------------------------------------------------------------------------------------|---------|
|
||||
| `minio_cluster_usage_objects_since_last_update_seconds` | Time since last update of usage metrics in seconds. <br><br>Type: gauge | |
|
||||
| `minio_cluster_usage_objects_total_bytes` | Total cluster usage in bytes. <br><br>Type: gauge | |
|
||||
| `minio_cluster_usage_objects_count` | Total cluster objects count. <br><br>Type: gauge | |
|
||||
| `minio_cluster_usage_objects_versions_count` | Total cluster object versions count, including delete markers. <br><br>Type: gauge | |
|
||||
| `minio_cluster_usage_objects_delete_markers_count` | Total cluster delete markers count. <br><br>Type: gauge | |
|
||||
| `minio_cluster_usage_objects_buckets_count` | Total cluster buckets count. <br><br>Type: gauge | |
|
||||
| `minio_cluster_usage_objects_size_distribution` | Cluster object size distribution. <br><br>Type: gauge | `range` |
|
||||
| `minio_cluster_usage_objects_version_count_distribution` | Cluster object version count distribution. <br><br>Type: gauge | `range` |
|
||||
|
||||
### Debug metrics
|
||||
|
||||
Standard Go runtime metrics from the [Prometheus Go Client base collector](https://github.com/prometheus/client_golang).
|
||||
|
||||
| Path | Description |
|
||||
|-------------|---------------------|
|
||||
| `/debug/go` | Go runtime metrics. |
|
||||
|
||||
### ILM metrics
|
||||
|
||||
Metrics about the MinIO ILM functionality.
|
||||
|
||||
| Path | Description |
|
||||
|--------|---------------------------------------|
|
||||
| `/ilm` | Metrics related to ILM functionality. |
|
||||
|
||||
#### `/ilm`
|
||||
|
||||
| Name | Description | Labels |
|
||||
|-------------------------------------------------------|---------------------------------------------------------------------------------------------------|----------|
|
||||
| `minio_cluster_ilm_expiry_pending_tasks` | Number of pending ILM expiry tasks in the queue. <br><br>Type: gauge | `server` |
|
||||
| `minio_cluster_ilm_transition_active_tasks` | Number of active ILM transition tasks. <br><br>Type: gauge | `server` |
|
||||
| `minio_cluster_ilm_transition_pending_tasks` | Number of pending ILM transition tasks in the queue. <br><br>Type: gauge | `server` |
|
||||
| `minio_cluster_ilm_transition_missed_immediate_tasks` | Number of missed immediate ILM transition tasks. <br><br>Type: counter | `server` |
|
||||
| `minio_cluster_ilm_versions_scanned` | Total number of object versions checked for ILM actions since server start. <br><br>Type: counter | `server` |
|
||||
|
||||
### Logger webhook metrics
|
||||
|
||||
Metrics about MinIO logger webhooks.
|
||||
|
||||
| Path | Description |
|
||||
|-------------------|-------------------------------------|
|
||||
| `/logger/webhook` | Metrics related to logger webhooks. |
|
||||
|
||||
#### `/logger/webhook`
|
||||
|
||||
| Name | Description | Labels |
|
||||
|-----------------------------------------|---------------------------------------------------------------------|------------------------------|
|
||||
| `minio_logger_webhook_failed_messages` | Number of messages that failed to send. <br><br>Type: counter | `server`, `name`, `endpoint` |
|
||||
| `minio_logger_webhook_queue_length` | Webhook queue length. <br><br>Type: gauge | `server`, `name`, `endpoint` |
|
||||
| `minio_logger_webhook_total_message` | Total number of messages sent to this target. <br><br>Type: counter | `server`, `name`, `endpoint` |
|
||||
|
||||
### Notification metrics
|
||||
|
||||
Metrics about the MinIO notification functionality.
|
||||
|
||||
| Path | Description |
|
||||
|-----------------|------------------------------------------------|
|
||||
| `/notification` | Metrics related to notification functionality. |
|
||||
|
||||
#### `/notification`
|
||||
|
||||
| Name | Description | Labels |
|
||||
|-----------------------------------------------|-------------------------------------------------------------------------------------------------------|----------|
|
||||
| `minio_notification_current_send_in_progress` | Number of concurrent async Send calls active to all targets. <br><br>Type: counter | `server` |
|
||||
| `minio_notification_events_errors_total` | Total number of events that failed to send to the targets. <br><br>Type: counter | `server` |
|
||||
| `minio_notification_events_sent_total` | Total number of events sent to the targets. <br><br>Type: counter | `server` |
|
||||
| `minio_notification_events_skipped_total` | Number of events not sent to the targets due to the in-memory queue being full. <br><br>Type: counter | `server` |
|
||||
|
||||
### Replication metrics
|
||||
|
||||
Metrics about MinIO site and bucket replication.
|
||||
|
||||
| Path | Description |
|
||||
|-----------------------|----------------------------------------|
|
||||
| `/bucket/replication` | Metrics related to bucket replication. |
|
||||
| `/replication` | Metrics related to site replication. |
|
||||
|
||||
#### `/replication`
|
||||
|
||||
| Name | Description | Labels |
|
||||
|---------------------------------------------------|---------------------------------------------------------------------------------------------|----------|
|
||||
| `minio_replication_average_active_workers` | Average number of active replication workers. <br><br>Type: gauge | `server` |
|
||||
| `minio_replication_average_queued_bytes` | Average number of bytes queued for replication since server start. <br><br>Type: gauge | `server` |
|
||||
| `minio_replication_average_queued_count` | Average number of objects queued for replication since server start. <br><br>Type: gauge | `server` |
|
||||
| `minio_replication_average_data_transfer_rate` | Average replication data transfer rate in bytes/sec. <br><br>Type: gauge | `server` |
|
||||
| `minio_replication_current_active_workers` | Total number of active replication workers. <br><br>Type: gauge | `server` |
|
||||
| `minio_replication_current_data_transfer_rate` | Current replication data transfer rate in bytes/sec. <br><br>Type: gauge | `server` |
|
||||
| `minio_replication_last_minute_queued_bytes` | Number of bytes queued for replication in the last full minute. <br><br>Type: gauge | `server` |
|
||||
| `minio_replication_last_minute_queued_count` | Number of objects queued for replication in the last full minute. <br><br>Type: gauge | `server` |
|
||||
| `minio_replication_max_active_workers` | Maximum number of active replication workers seen since server start. <br><br>Type: gauge | `server` |
|
||||
| `minio_replication_max_queued_bytes` | Maximum number of bytes queued for replication since server start. <br><br>Type: gauge | `server` |
|
||||
| `minio_replication_max_queued_count` | Maximum number of objects queued for replication since server start. <br><br>Type: gauge | `server` |
|
||||
| `minio_replication_max_data_transfer_rate` | Maximum replication data transfer rate in bytes/sec since server start. <br><br>Type: gauge | `server` |
|
||||
| `minio_replication_recent_backlog_count` | Total number of objects seen in replication backlog in the last 5 minutes <br><br>Type: gauge | `server` |
|
||||
#### `/bucket/replication`
|
||||
|
||||
| Name | Description | Labels |
|
||||
|---------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------|-------------------------------------------------------|
|
||||
| `minio_bucket_replication_last_hour_failed_bytes` | Total number of bytes on a bucket which failed to replicate at least once in the last hour. <br><br>Type: gauge | `bucket`, `server` |
|
||||
| `minio_bucket_replication_last_hour_failed_count` | Total number of objects on a bucket which failed to replicate in the last hour. <br><br>Type: gauge | `bucket`, `server` |
|
||||
| `minio_bucket_replication_last_minute_failed_bytes` | Total number of bytes on a bucket which failed at least once in the last full minute. <br><br>Type: gauge | `bucket`, `server` |
|
||||
| `minio_bucket_replication_last_minute_failed_count` | Total number of objects on a bucket which failed to replicate in the last full minute. <br><br>Type: gauge | `bucket`, `server` |
|
||||
| `minio_bucket_replication_latency_ms` | Replication latency on a bucket in milliseconds. <br><br>Type: gauge | `bucket`, `operation`, `range`, `targetArn`, `server` |
|
||||
| `minio_bucket_replication_proxied_delete_tagging_requests_total` | Number of DELETE tagging requests proxied to replication target. <br><br>Type: counter | `bucket`, `server` |
|
||||
| `minio_bucket_replication_proxied_get_requests_failures` | Number of failures in GET requests proxied to replication target. <br><br>Type: counter | `bucket`, `server` |
|
||||
| `minio_bucket_replication_proxied_get_requests_total` | Number of GET requests proxied to replication target. <br><br>Type: counter | `bucket`, `server` |
|
||||
| `minio_bucket_replication_proxied_get_tagging_requests_failures` | Number of failures in GET tagging requests proxied to replication target. <br><br>Type: counter | `bucket`, `server` |
|
||||
| `minio_bucket_replication_proxied_get_tagging_requests_total` | Number of GET tagging requests proxied to replication target. <br><br>Type: counter | `bucket`, `server` |
|
||||
| `minio_bucket_replication_proxied_head_requests_failures` | Number of failures in HEAD requests proxied to replication target. <br><br>Type: counter | `bucket`, `server` |
|
||||
| `minio_bucket_replication_proxied_head_requests_total` | Number of HEAD requests proxied to replication target. <br><br>Type: counter | `bucket`, `server` |
|
||||
| `minio_bucket_replication_proxied_put_tagging_requests_failures` | Number of failures in PUT tagging requests proxied to replication target. <br><br>Type: counter | `bucket`, `server` |
|
||||
| `minio_bucket_replication_proxied_put_tagging_requests_total` | Number of PUT tagging requests proxied to replication target. <br><br>Type: counter | `bucket`, `server` |
|
||||
| `minio_bucket_replication_sent_bytes` | Total number of bytes replicated to the target. <br><br>Type: counter | `bucket`, `server` |
|
||||
| `minio_bucket_replication_sent_count` | Total number of objects replicated to the target. <br><br>Type: counter | `bucket`, `server` |
|
||||
| `minio_bucket_replication_total_failed_bytes` | Total number of bytes failed to replicate at least once since server start. <br><br>Type: counter | `bucket`, `server` |
|
||||
| `minio_bucket_replication_total_failed_count` | Total number of objects that failed to replicate since server start. <br><br>Type: counter | `bucket`, `server` |
|
||||
| `minio_bucket_replication_proxied_delete_tagging_requests_failures` | Number of failures in DELETE tagging requests proxied to replication target. <br><br>Type: counter | `bucket`, `server` |
|
||||
|
||||
### Scanner metrics
|
||||
|
||||
Metrics about the MinIO scanner.
|
||||
|
||||
| Path | Description |
|
||||
|------------|---------------------------------------|
|
||||
| `/scanner` | Metrics related to the MinIO scanner. |
|
||||
|
||||
#### `/scanner`
|
||||
|
||||
| Name | Description | Labels |
|
||||
|--------------------------------------------|-----------------------------------------------------------------------------------|----------|
|
||||
| `minio_scanner_bucket_scans_finished` | Total number of bucket scans completed since server start. <br><br>Type: counter | `server` |
|
||||
| `minio_scanner_bucket_scans_started` | Total number of bucket scans started since server start. <br><br>Type: counter | `server` |
|
||||
| `minio_scanner_directories_scanned` | Total number of directories scanned since server start. <br><br>Type: counter | `server` |
|
||||
| `minio_scanner_last_activity_seconds` | Time elapsed (in seconds) since last scan activity. <br><br>Type: gauge | `server` |
|
||||
| `minio_scanner_objects_scanned` | Total number of unique objects scanned since server start. <br><br>Type: counter | `server` |
|
||||
| `minio_scanner_versions_scanned` | Total number of object versions scanned since server start. <br><br>Type: counter | `server` |
|
||||
|
||||
### System metrics
|
||||
|
||||
Metrics about the MinIO process and the node.
|
||||
|
||||
| Path | Description |
|
||||
|-----------------------------|----------------------------------------------------|
|
||||
| `/system/cpu` | Metrics about CPUs on the system. |
|
||||
| `/system/drive` | Metrics about drives on the system. |
|
||||
| `/system/network/internode` | Metrics about internode requests made by the node. |
|
||||
| `/system/memory` | Metrics about memory on the system. |
|
||||
| `/system/process` | Standard process metrics. |
|
||||
|
||||
#### `/system/drive`
|
||||
|
||||
| Name | Description | Labels |
|
||||
|------------------------------------------------|-----------------------------------------------------------------------------------------|--------------------------------------------------------------------|
|
||||
| `minio_system_drive_used_bytes` | Total storage used on a drive in bytes. <br><br>Type: gauge | `drive`, `set_index`, `drive_index`, `pool_index`, `server` |
|
||||
| `minio_system_drive_free_bytes` | Total storage free on a drive in bytes. <br><br>Type: gauge | `drive`, `set_index`, `drive_index`, `pool_index`, `server` |
|
||||
| `minio_system_drive_total_bytes` | Total storage available on a drive in bytes. <br><br>Type: gauge | `drive`, `set_index`, `drive_index`, `pool_index`, `server` |
|
||||
| `minio_system_drive_used_inodes` | Total used inodes on a drive. <br><br>Type: gauge | `drive`, `set_index`, `drive_index`, `pool_index`, `server` |
|
||||
| `minio_system_drive_free_inodes` | Total free inodes on a drive. <br><br>Type: gauge | `drive`, `set_index`, `drive_index`, `pool_index`, `server` |
|
||||
| `minio_system_drive_total_inodes` | Total inodes available on a drive. <br><br>Type: gauge | `drive`, `set_index`, `drive_index`, `pool_index`, `server` |
|
||||
| `minio_system_drive_timeout_errors_total` | Total timeout errors on a drive. <br><br>Type: counter | `drive`, `set_index`, `drive_index`, `pool_index`, `server` |
|
||||
| `minio_system_drive_io_errors_total` | Total I/O errors on a drive. <br><br>Type: counter | `drive`, `set_index`, `drive_index`, `pool_index`, `server` |
|
||||
| `minio_system_drive_availability_errors_total` | Total availability errors (I/O errors, timeouts) on a drive. <br><br>Type: counter | `drive`, `set_index`, `drive_index`, `pool_index`, `server` |
|
||||
| `minio_system_drive_waiting_io` | Total waiting I/O operations on a drive. <br><br>Type: gauge | `drive`, `set_index`, `drive_index`, `pool_index`, `server` |
|
||||
| `minio_system_drive_api_latency_micros` | Average last minute latency in µs for drive API storage operations. <br><br>Type: gauge | `drive`, `api`, `set_index`, `drive_index`, `pool_index`, `server` |
|
||||
| `minio_system_drive_offline_count` | Count of offline drives. <br><br>Type: gauge | `pool_index`, `server` |
|
||||
| `minio_system_drive_online_count` | Count of online drives. <br><br>Type: gauge | `pool_index`, `server` |
|
||||
| `minio_system_drive_count` | Count of all drives. <br><br>Type: gauge | `pool_index`, `server` |
|
||||
| `minio_system_drive_health` | Drive health (0 = offline, 1 = healthy, 2 = healing). <br><br>Type: gauge | `drive`, `set_index`, `drive_index`, `pool_index`, `server` |
|
||||
| `minio_system_drive_reads_per_sec` | Reads per second on a drive. <br><br>Type: gauge | `drive`, `set_index`, `drive_index`, `pool_index`, `server` |
|
||||
| `minio_system_drive_reads_kb_per_sec` | Kilobytes read per second on a drive. <br><br>Type: gauge | `drive`, `set_index`, `drive_index`, `pool_index`, `server` |
|
||||
| `minio_system_drive_reads_await` | Average time for read requests served on a drive. <br><br>Type: gauge | `drive`, `set_index`, `drive_index`, `pool_index`, `server` |
|
||||
| `minio_system_drive_writes_per_sec` | Writes per second on a drive. <br><br>Type: gauge | `drive`, `set_index`, `drive_index`, `pool_index`, `server` |
|
||||
| `minio_system_drive_writes_kb_per_sec` | Kilobytes written per second on a drive. <br><br>Type: gauge | `drive`, `set_index`, `drive_index`, `pool_index`, `server` |
|
||||
| `minio_system_drive_writes_await` | Average time for write requests served on a drive. <br><br>Type: gauge | `drive`, `set_index`, `drive_index`, `pool_index`, `server` |
|
||||
| `minio_system_drive_perc_util` | Percentage of time the disk was busy. <br><br>Type: gauge | `drive`, `set_index`, `drive_index`, `pool_index`, `server` |
|
||||
|
||||
#### `/system/memory`
|
||||
|
||||
| Name | Description | Labels |
|
||||
|----------------------------------|---------------------------------------------------------|----------|
|
||||
| `minio_system_memory_used` | Used memory on the node. <br><br>Type: gauge | `server` |
|
||||
| `minio_system_memory_used_perc` | Used memory percentage on the node. <br><br>Type: gauge | `server` |
|
||||
| `minio_system_memory_free` | Free memory on the node. <br><br>Type: gauge | `server` |
|
||||
| `minio_system_memory_total` | Total memory on the node. <br><br>Type: gauge | `server` |
|
||||
| `minio_system_memory_buffers` | Buffers memory on the node. <br><br>Type: gauge | `server` |
|
||||
| `minio_system_memory_cache` | Cache memory on the node. <br><br>Type: gauge | `server` |
|
||||
| `minio_system_memory_shared` | Shared memory on the node. <br><br>Type: gauge | `server` |
|
||||
| `minio_system_memory_available` | Available memory on the node. <br><br>Type: gauge | `server` |
|
||||
|
||||
#### `/system/cpu`
|
||||
|
||||
| Name | Description | Labels |
|
||||
|-------------------------------|---------------------------------------------------------|----------|
|
||||
| `minio_system_cpu_avg_idle` | Average CPU idle time. <br><br>Type: gauge | `server` |
|
||||
| `minio_system_cpu_avg_iowait` | Average CPU IOWait time. <br><br>Type: gauge | `server` |
|
||||
| `minio_system_cpu_load` | CPU load average 1min. <br><br>Type: gauge | `server` |
|
||||
| `minio_system_cpu_load_perc` | CPU load average 1min (percentage). <br><br>Type: gauge | `server` |
|
||||
| `minio_system_cpu_nice` | CPU nice time. <br><br>Type: gauge | `server` |
|
||||
| `minio_system_cpu_steal` | CPU steal time. <br><br>Type: gauge | `server` |
|
||||
| `minio_system_cpu_system` | CPU system time. <br><br>Type: gauge | `server` |
|
||||
| `minio_system_cpu_user` | CPU user time. <br><br>Type: gauge | `server` |
|
||||
|
||||
#### `/system/network/internode`
|
||||
|
||||
| Name | Description | Labels |
|
||||
|------------------------------------------------------|-------------------------------------------------------------------------------|------------------------|
|
||||
| `minio_system_network_internode_errors_total` | Total number of failed internode calls. <br><br>Type: counter | `server`, `pool_index` |
|
||||
| `minio_system_network_internode_dial_errors_total` | Total number of internode TCP dial timeouts and errors. <br><br>Type: counter | `server`, `pool_index` |
|
||||
| `minio_system_network_internode_dial_avg_time_nanos` | Average dial time of internodes TCP calls in nanoseconds. <br><br>Type: gauge | `server`, `pool_index` |
|
||||
| `minio_system_network_internode_sent_bytes_total` | Total number of bytes sent to other peer nodes. <br><br>Type: counter | `server`, `pool_index` |
|
||||
| `minio_system_network_internode_recv_bytes_total` | Total number of bytes received from other peer nodes. <br><br>Type: counter | `server`, `pool_index` |
|
||||
|
||||
#### `/system/process`
|
||||
|
||||
| Name | Description | Labels |
|
||||
|----------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------|----------|
|
||||
| `minio_system_process_locks_read_total` | Number of current READ locks on this peer. <br><br>Type: gauge | `server` |
|
||||
| `minio_system_process_locks_write_total` | Number of current WRITE locks on this peer. <br><br>Type: gauge | `server` |
|
||||
| `minio_system_process_cpu_total_seconds` | Total user and system CPU time spent in seconds. <br><br>Type: counter | `server` |
|
||||
| `minio_system_process_go_routine_total` | Total number of go routines running. <br><br>Type: gauge | `server` |
|
||||
| `minio_system_process_io_rchar_bytes` | Total bytes read by the process from the underlying storage system including cache, /proc/[pid]/io rchar. <br><br>Type: counter | `server` |
|
||||
| `minio_system_process_io_read_bytes` | Total bytes read by the process from the underlying storage system, /proc/[pid]/io read_bytes. <br><br>Type: counter | `server` |
|
||||
| `minio_system_process_io_wchar_bytes` | Total bytes written by the process to the underlying storage system including page cache, /proc/[pid]/io wchar. <br><br>Type: counter | `server` |
|
||||
| `minio_system_process_io_write_bytes` | Total bytes written by the process to the underlying storage system, /proc/[pid]/io write_bytes. <br><br>Type: counter | `server` |
|
||||
| `minio_system_process_start_time_seconds` | Start time for MinIO process in seconds since Unix epoch. <br><br>Type: gauge | `server` |
|
||||
| `minio_system_process_uptime_seconds` | Uptime for MinIO process in seconds. <br><br>Type: gauge | `server` |
|
||||
| `minio_system_process_file_descriptor_limit_total` | Limit on total number of open file descriptors for the MinIO Server process. <br><br>Type: gauge | `server` |
|
||||
| `minio_system_process_file_descriptor_open_total` | Total number of open file descriptors by the MinIO Server process. <br><br>Type: gauge | `server` |
|
||||
| `minio_system_process_syscall_read_total` | Total read SysCalls to the kernel. /proc/[pid]/io syscr. <br><br>Type: counter | `server` |
|
||||
| `minio_system_process_syscall_write_total` | Total write SysCalls to the kernel. /proc/[pid]/io syscw. <br><br>Type: counter | `server` |
|
||||
| `minio_system_process_resident_memory_bytes` | Resident memory size in bytes. <br><br>Type: gauge | `server` |
|
||||
| `minio_system_process_virtual_memory_bytes` | Virtual memory size in bytes. <br><br>Type: gauge | `server` |
|
||||
| `minio_system_process_virtual_memory_max_bytes` | Maximum virtual memory size in bytes. <br><br>Type: gauge | `server` |
|
||||
Reference in New Issue
Block a user