confdroid/confdroid_nagios
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

OP#792 initial commit after fork

Browse Source
This commit is contained in:
Arne Teuke
2026-09-27 11:03:48 +02:00
parent 1fc33f4f1f
commit 55b0890196
1338 changed files with 428609 additions and 57 deletions
Download Patch File Download Diff File Expand all files Collapse all files

395
docs/LICENSE Normal file
View File

@@ -0,0 +1,395 @@
Attribution 4.0 International
=======================================================================
Creative Commons Corporation ("Creative Commons") is not a law firm and
does not provide legal services or legal advice. Distribution of
Creative Commons public licenses does not create a lawyer-client or
other relationship. Creative Commons makes its licenses and related
information available on an "as-is" basis. Creative Commons gives no
warranties regarding its licenses, any material licensed under their
terms and conditions, or any related information. Creative Commons
disclaims all liability for damages resulting from their use to the
fullest extent possible.
Using Creative Commons Public Licenses
Creative Commons public licenses provide a standard set of terms and
conditions that creators and other rights holders may use to share
original works of authorship and other material subject to copyright
and certain other rights specified in the public license below. The
following considerations are for informational purposes only, are not
exhaustive, and do not form part of our licenses.
Considerations for licensors: Our public licenses are
intended for use by those authorized to give the public
permission to use material in ways otherwise restricted by
copyright and certain other rights. Our licenses are
irrevocable. Licensors should read and understand the terms
and conditions of the license they choose before applying it.
Licensors should also secure all rights necessary before
applying our licenses so that the public can reuse the
material as expected. Licensors should clearly mark any
material not subject to the license. This includes other CC-
licensed material, or material used under an exception or
limitation to copyright. More considerations for licensors:
wiki.creativecommons.org/Considerations_for_licensors
Considerations for the public: By using one of our public
licenses, a licensor grants the public permission to use the
licensed material under specified terms and conditions. If
the licensor's permission is not necessary for any reason--for
example, because of any applicable exception or limitation to
copyright--then that use is not regulated by the license. Our
licenses grant only permissions under copyright and certain
other rights that a licensor has authority to grant. Use of
the licensed material may still be restricted for other
reasons, including because others have copyright or other
rights in the material. A licensor may make special requests,
such as asking that all changes be marked or described.
Although not required by our licenses, you are encouraged to
respect those requests where reasonable. More_considerations
for the public:
wiki.creativecommons.org/Considerations_for_licensees
=======================================================================
Creative Commons Attribution 4.0 International Public License
By exercising the Licensed Rights (defined below), You accept and agree
to be bound by the terms and conditions of this Creative Commons
Attribution 4.0 International Public License ("Public License"). To the
extent this Public License may be interpreted as a contract, You are
granted the Licensed Rights in consideration of Your acceptance of
these terms and conditions, and the Licensor grants You such rights in
consideration of benefits the Licensor receives from making the
Licensed Material available under these terms and conditions.
Section 1 -- Definitions.
a. Adapted Material means material subject to Copyright and Similar
Rights that is derived from or based upon the Licensed Material
and in which the Licensed Material is translated, altered,
arranged, transformed, or otherwise modified in a manner requiring
permission under the Copyright and Similar Rights held by the
Licensor. For purposes of this Public License, where the Licensed
Material is a musical work, performance, or sound recording,
Adapted Material is always produced where the Licensed Material is
synched in timed relation with a moving image.
b. Adapter's License means the license You apply to Your Copyright
and Similar Rights in Your contributions to Adapted Material in
accordance with the terms and conditions of this Public License.
c. Copyright and Similar Rights means copyright and/or similar rights
closely related to copyright including, without limitation,
performance, broadcast, sound recording, and Sui Generis Database
Rights, without regard to how the rights are labeled or
categorized. For purposes of this Public License, the rights
specified in Section 2(b)(1)-(2) are not Copyright and Similar
Rights.
d. Effective Technological Measures means those measures that, in the
absence of proper authority, may not be circumvented under laws
fulfilling obligations under Article 11 of the WIPO Copyright
Treaty adopted on December 20, 1996, and/or similar international
agreements.
e. Exceptions and Limitations means fair use, fair dealing, and/or
any other exception or limitation to Copyright and Similar Rights
that applies to Your use of the Licensed Material.
f. Licensed Material means the artistic or literary work, database,
or other material to which the Licensor applied this Public
License.
g. Licensed Rights means the rights granted to You subject to the
terms and conditions of this Public License, which are limited to
all Copyright and Similar Rights that apply to Your use of the
Licensed Material and that the Licensor has authority to license.
h. Licensor means the individual(s) or entity(ies) granting rights
under this Public License.
i. Share means to provide material to the public by any means or
process that requires permission under the Licensed Rights, such
as reproduction, public display, public performance, distribution,
dissemination, communication, or importation, and to make material
available to the public including in ways that members of the
public may access the material from a place and at a time
individually chosen by them.
j. Sui Generis Database Rights means rights other than copyright
resulting from Directive 96/9/EC of the European Parliament and of
the Council of 11 March 1996 on the legal protection of databases,
as amended and/or succeeded, as well as other essentially
equivalent rights anywhere in the world.
k. You means the individual or entity exercising the Licensed Rights
under this Public License. Your has a corresponding meaning.
Section 2 -- Scope.
a. License grant.
1. Subject to the terms and conditions of this Public License,
the Licensor hereby grants You a worldwide, royalty-free,
non-sublicensable, non-exclusive, irrevocable license to
exercise the Licensed Rights in the Licensed Material to:
a. reproduce and Share the Licensed Material, in whole or
in part; and
b. produce, reproduce, and Share Adapted Material.
2. Exceptions and Limitations. For the avoidance of doubt, where
Exceptions and Limitations apply to Your use, this Public
License does not apply, and You do not need to comply with
its terms and conditions.
3. Term. The term of this Public License is specified in Section
6(a).
4. Media and formats; technical modifications allowed. The
Licensor authorizes You to exercise the Licensed Rights in
all media and formats whether now known or hereafter created,
and to make technical modifications necessary to do so. The
Licensor waives and/or agrees not to assert any right or
authority to forbid You from making technical modifications
necessary to exercise the Licensed Rights, including
technical modifications necessary to circumvent Effective
Technological Measures. For purposes of this Public License,
simply making modifications authorized by this Section 2(a)
(4) never produces Adapted Material.
5. Downstream recipients.
a. Offer from the Licensor -- Licensed Material. Every
recipient of the Licensed Material automatically
receives an offer from the Licensor to exercise the
Licensed Rights under the terms and conditions of this
Public License.
b. No downstream restrictions. You may not offer or impose
any additional or different terms or conditions on, or
apply any Effective Technological Measures to, the
Licensed Material if doing so restricts exercise of the
Licensed Rights by any recipient of the Licensed
Material.
6. No endorsement. Nothing in this Public License constitutes or
may be construed as permission to assert or imply that You
are, or that Your use of the Licensed Material is, connected
with, or sponsored, endorsed, or granted official status by,
the Licensor or others designated to receive attribution as
provided in Section 3(a)(1)(A)(i).
b. Other rights.
1. Moral rights, such as the right of integrity, are not
licensed under this Public License, nor are publicity,
privacy, and/or other similar personality rights; however, to
the extent possible, the Licensor waives and/or agrees not to
assert any such rights held by the Licensor to the limited
extent necessary to allow You to exercise the Licensed
Rights, but not otherwise.
2. Patent and trademark rights are not licensed under this
Public License.
3. To the extent possible, the Licensor waives any right to
collect royalties from You for the exercise of the Licensed
Rights, whether directly or through a collecting society
under any voluntary or waivable statutory or compulsory
licensing scheme. In all other cases the Licensor expressly
reserves any right to collect such royalties.
Section 3 -- License Conditions.
Your exercise of the Licensed Rights is expressly made subject to the
following conditions.
a. Attribution.
1. If You Share the Licensed Material (including in modified
form), You must:
a. retain the following if it is supplied by the Licensor
with the Licensed Material:
i. identification of the creator(s) of the Licensed
Material and any others designated to receive
attribution, in any reasonable manner requested by
the Licensor (including by pseudonym if
designated);
ii. a copyright notice;
iii. a notice that refers to this Public License;
iv. a notice that refers to the disclaimer of
warranties;
v. a URI or hyperlink to the Licensed Material to the
extent reasonably practicable;
b. indicate if You modified the Licensed Material and
retain an indication of any previous modifications; and
c. indicate the Licensed Material is licensed under this
Public License, and include the text of, or the URI or
hyperlink to, this Public License.
2. You may satisfy the conditions in Section 3(a)(1) in any
reasonable manner based on the medium, means, and context in
which You Share the Licensed Material. For example, it may be
reasonable to satisfy the conditions by providing a URI or
hyperlink to a resource that includes the required
information.
3. If requested by the Licensor, You must remove any of the
information required by Section 3(a)(1)(A) to the extent
reasonably practicable.
4. If You Share Adapted Material You produce, the Adapter's
License You apply must not prevent recipients of the Adapted
Material from complying with this Public License.
Section 4 -- Sui Generis Database Rights.
Where the Licensed Rights include Sui Generis Database Rights that
apply to Your use of the Licensed Material:
a. for the avoidance of doubt, Section 2(a)(1) grants You the right
to extract, reuse, reproduce, and Share all or a substantial
portion of the contents of the database;
b. if You include all or a substantial portion of the database
contents in a database in which You have Sui Generis Database
Rights, then the database in which You have Sui Generis Database
Rights (but not its individual contents) is Adapted Material; and
c. You must comply with the conditions in Section 3(a) if You Share
all or a substantial portion of the contents of the database.
For the avoidance of doubt, this Section 4 supplements and does not
replace Your obligations under this Public License where the Licensed
Rights include other Copyright and Similar Rights.
Section 5 -- Disclaimer of Warranties and Limitation of Liability.
a. UNLESS OTHERWISE SEPARATELY UNDERTAKEN BY THE LICENSOR, TO THE
EXTENT POSSIBLE, THE LICENSOR OFFERS THE LICENSED MATERIAL AS-IS
AND AS-AVAILABLE, AND MAKES NO REPRESENTATIONS OR WARRANTIES OF
ANY KIND CONCERNING THE LICENSED MATERIAL, WHETHER EXPRESS,
IMPLIED, STATUTORY, OR OTHER. THIS INCLUDES, WITHOUT LIMITATION,
WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR
PURPOSE, NON-INFRINGEMENT, ABSENCE OF LATENT OR OTHER DEFECTS,
ACCURACY, OR THE PRESENCE OR ABSENCE OF ERRORS, WHETHER OR NOT
KNOWN OR DISCOVERABLE. WHERE DISCLAIMERS OF WARRANTIES ARE NOT
ALLOWED IN FULL OR IN PART, THIS DISCLAIMER MAY NOT APPLY TO YOU.
b. TO THE EXTENT POSSIBLE, IN NO EVENT WILL THE LICENSOR BE LIABLE
TO YOU ON ANY LEGAL THEORY (INCLUDING, WITHOUT LIMITATION,
NEGLIGENCE) OR OTHERWISE FOR ANY DIRECT, SPECIAL, INDIRECT,
INCIDENTAL, CONSEQUENTIAL, PUNITIVE, EXEMPLARY, OR OTHER LOSSES,
COSTS, EXPENSES, OR DAMAGES ARISING OUT OF THIS PUBLIC LICENSE OR
USE OF THE LICENSED MATERIAL, EVEN IF THE LICENSOR HAS BEEN
ADVISED OF THE POSSIBILITY OF SUCH LOSSES, COSTS, EXPENSES, OR
DAMAGES. WHERE A LIMITATION OF LIABILITY IS NOT ALLOWED IN FULL OR
IN PART, THIS LIMITATION MAY NOT APPLY TO YOU.
c. The disclaimer of warranties and limitation of liability provided
above shall be interpreted in a manner that, to the extent
possible, most closely approximates an absolute disclaimer and
waiver of all liability.
Section 6 -- Term and Termination.
a. This Public License applies for the term of the Copyright and
Similar Rights licensed here. However, if You fail to comply with
this Public License, then Your rights under this Public License
terminate automatically.
b. Where Your right to use the Licensed Material has terminated under
Section 6(a), it reinstates:
1. automatically as of the date the violation is cured, provided
it is cured within 30 days of Your discovery of the
violation; or
2. upon express reinstatement by the Licensor.
For the avoidance of doubt, this Section 6(b) does not affect any
right the Licensor may have to seek remedies for Your violations
of this Public License.
c. For the avoidance of doubt, the Licensor may also offer the
Licensed Material under separate terms or conditions or stop
distributing the Licensed Material at any time; however, doing so
will not terminate this Public License.
d. Sections 1, 5, 6, 7, and 8 survive termination of this Public
License.
Section 7 -- Other Terms and Conditions.
a. The Licensor shall not be bound by any additional or different
terms or conditions communicated by You unless expressly agreed.
b. Any arrangements, understandings, or agreements regarding the
Licensed Material not stated herein are separate from and
independent of the terms and conditions of this Public License.
Section 8 -- Interpretation.
a. For the avoidance of doubt, this Public License does not, and
shall not be interpreted to, reduce, limit, restrict, or impose
conditions on any use of the Licensed Material that could lawfully
be made without permission under this Public License.
b. To the extent possible, if any provision of this Public License is
deemed unenforceable, it shall be automatically reformed to the
minimum extent necessary to make it enforceable. If the provision
cannot be reformed, it shall be severed from this Public License
without affecting the enforceability of the remaining terms and
conditions.
c. No term or condition of this Public License will be waived and no
failure to comply consented to unless expressly agreed to by the
Licensor.
d. Nothing in this Public License constitutes or may be interpreted
as a limitation upon, or waiver of, any privileges and immunities
that apply to the Licensor or You, including from the legal
processes of any jurisdiction or authority.
=======================================================================
Creative Commons is not a party to its public
licenses. Notwithstanding, Creative Commons may elect to apply one of
its public licenses to material it publishes and in those instances
will be considered the “Licensor.” The text of the Creative Commons
public licenses is dedicated to the public domain under the CC0 Public
Domain Dedication. Except for the limited purpose of indicating that
material is shared under a Creative Commons public license or as
otherwise permitted by the Creative Commons policies published at
creativecommons.org/policies, Creative Commons does not authorize the
use of the trademark "Creative Commons" or any other trademark or logo
of Creative Commons without its prior written consent including,
without limitation, in connection with any unauthorized modifications
to any of its public licenses or any other arrangements,
understandings, or agreements concerning use of licensed material. For
the avoidance of doubt, this paragraph does not form part of the
public licenses.
Creative Commons may be contacted at creativecommons.org.

62
docs/auditlog/auditlog-echo.go Normal file
View File

@@ -0,0 +1,62 @@
//go:build ignore
// +build ignore
// Copyright (c) 2015-2024 MinIO, Inc.
//
// This file is part of MinIO Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package main
import (
"bytes"
"encoding/json"
"flag"
"fmt"
"io"
"log"
"net/http"
)
var port int
func init() {
flag.IntVar(&port, "port", 8080, "Port to listen on")
}
func mainHandler(w http.ResponseWriter, r *http.Request) {
body, err := io.ReadAll(r.Body)
defer r.Body.Close()
if err != nil {
log.Printf("Error reading request body: %v", err)
w.WriteHeader(http.StatusBadRequest)
return
}
log.Printf(">>> %s %s\n", r.Method, r.URL.Path)
var out bytes.Buffer
json.Indent(&out, body, "", " ")
log.Printf("%s\n", out.String())
w.WriteHeader(http.StatusOK)
}
func main() {
flag.Parse()
http.HandleFunc("/", mainHandler)
log.Printf("Listening on :%d\n", port)
log.Fatal(http.ListenAndServe(fmt.Sprintf(":%d", port), nil))
}

17
docs/auditlog/auditlog-echo.md Normal file
View File

@@ -0,0 +1,17 @@
# `auditlog-echo`: A tool to view MinIO Audit logs on the console
1. Run the tool with:
```
go run docs/auditlog/auditlog-echo.go
```
The listen port has a default value (8080), but can be set with the `-port` flag.
2. Configure audit logging in MinIO with for example:
```
mc admin config set myminio audit_webhook enable=on endpoint=http://localhost:8080
```
3. Make any requests to MinIO and see audit logs printed to the tool's console.

152
docs/batch-jobs/README.md Normal file
View File

@@ -0,0 +1,152 @@
# MinIO Batch Job
MinIO Batch jobs is an MinIO object management feature that lets you manage objects at scale. Jobs currently supported by MinIO
- Replicate objects between buckets on multiple sites
Upcoming Jobs
- Copy objects from NAS to MinIO
- Copy objects from HDFS to MinIO
## Replication Job
To perform replication via batch jobs, you create a job. The job consists of a job description YAML that describes
- Source location from where the objects must be copied from
- Target location from where the objects must be copied to
- Fine grained filtering is available to pick relevant objects from source to copy from
MinIO batch jobs framework also provides
- Retrying a failed job automatically driven by user input
- Monitoring job progress in real-time
- Send notifications upon completion or failure to user configured target
Following YAML describes the structure of a replication job, each value is documented and self-describing.
```yaml
replicate:
apiVersion: v1
# source of the objects to be replicated
source:
type: TYPE # valid values are "minio"
bucket: BUCKET
prefix: PREFIX
# NOTE: if source is remote then target must be "local"
# endpoint: ENDPOINT
# credentials:
# accessKey: ACCESS-KEY
# secretKey: SECRET-KEY
# sessionToken: SESSION-TOKEN # Available when rotating credentials are used
# target where the objects must be replicated
target:
type: TYPE # valid values are "minio"
bucket: BUCKET
prefix: PREFIX
# NOTE: if target is remote then source must be "local"
# endpoint: ENDPOINT
# credentials:
# accessKey: ACCESS-KEY
# secretKey: SECRET-KEY
# sessionToken: SESSION-TOKEN # Available when rotating credentials are used
# optional flags based filtering criteria
# for all source objects
flags:
filter:
newerThan: "7d" # match objects newer than this value (e.g. 7d10h31s)
olderThan: "7d" # match objects older than this value (e.g. 7d10h31s)
createdAfter: "date" # match objects created after "date"
createdBefore: "date" # match objects created before "date"
## NOTE: tags are not supported when "source" is remote.
# tags:
# - key: "name"
# value: "pick*" # match objects with tag 'name', with all values starting with 'pick'
## NOTE: metadata filter not supported when "source" is non MinIO.
# metadata:
# - key: "content-type"
# value: "image/*" # match objects with 'content-type', with all values starting with 'image/'
notify:
endpoint: "https://notify.endpoint" # notification endpoint to receive job status events
token: "Bearer xxxxx" # optional authentication token for the notification endpoint
retry:
attempts: 10 # number of retries for the job before giving up
delay: "500ms" # least amount of delay between each retry
```
You can create and run multiple 'replication' jobs at a time there are no predefined limits set.
## Batch Jobs Terminology
### Job
A job is the basic unit of work for MinIO Batch Job. A job is a self describing YAML, once this YAML is submitted and evaluated - MinIO performs the requested actions on each of the objects obtained under the described criteria in job YAML file.
### Type
Type describes the job type, such as replicating objects between MinIO sites. Each job performs a single type of operation across all objects that match the job description criteria.
## Batch Jobs via Commandline
[mc](http://github.com/minio/mc) provides 'mc batch' command to create, start and manage submitted jobs.
```
NAME:
mc batch - manage batch jobs
USAGE:
mc batch COMMAND [COMMAND FLAGS | -h] [ARGUMENTS...]
COMMANDS:
generate generate a new batch job definition
start start a new batch job
list, ls list all current batch jobs
status summarize job events on MinIO server in real-time
describe describe job definition for a job
```
### Generate a job yaml
```
mc batch generate alias/ replicate
```
### Start the batch job (returns back the JID)
```
mc batch start alias/ ./replicate.yaml
Successfully start 'replicate' job `E24HH4nNMcgY5taynaPfxu` on '2022-09-26 17:19:06.296974771 -0700 PDT'
```
### List all batch jobs
```
mc batch list alias/
ID TYPE USER STARTED
E24HH4nNMcgY5taynaPfxu replicate minioadmin 1 minute ago
```
### List all 'replicate' batch jobs
```
mc batch list alias/ --type replicate
ID TYPE USER STARTED
E24HH4nNMcgY5taynaPfxu replicate minioadmin 1 minute ago
```
### Real-time 'status' for a batch job
```
mc batch status myminio/ E24HH4nNMcgY5taynaPfxu
●∙∙
Objects: 28766
Versions: 28766
Throughput: 3.0 MiB/s
Transferred: 406 MiB
Elapsed: 2m14.227222868s
CurrObjName: share/doc/xml-core/examples/foo.xmlcatalogs
```
### 'describe' the batch job yaml.
```
mc batch describe myminio/ E24HH4nNMcgY5taynaPfxu
replicate:
apiVersion: v1
...
```

306
docs/bigdata/README.md Normal file
View File

@@ -0,0 +1,306 @@
# **Disaggregated HDP Spark and Hive with MinIO**
## **1. Cloud-native Architecture**
![cloud-native](https://github.com/minio/minio/blob/master/docs/bigdata/images/image1.png?raw=true "cloud native architecture")
Kubernetes manages stateless Spark and Hive containers elastically on the compute nodes. Spark has native scheduler integration with Kubernetes. Hive, for legacy reasons, uses YARN scheduler on top of Kubernetes.
All access to MinIO object storage is via S3/SQL SELECT API. In addition to the compute nodes, MinIO containers are also managed by Kubernetes as stateful containers with local storage (JBOD/JBOF) mapped as persistent local volumes. This architecture enables multi-tenant MinIO, allowing isolation of data between customers.
MinIO also supports multi-cluster, multi-site federation similar to AWS regions and tiers. Using MinIO Information Lifecycle Management (ILM), you can configure data to be tiered between NVMe based hot storage, and HDD based warm storage. All data is encrypted with per-object key. Access Control and Identity Management between the tenants are managed by MinIO using OpenID Connect or Kerberos/LDAP/AD.
## **2. Prerequisites**
- Install Hortonworks Distribution using this [guide.](https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.1.0/bk_ambari-installation/content/ch_Installing_Ambari.html)
- [Setup Ambari](https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.1.0/bk_ambari-installation/content/set_up_the_ambari_server.html) which automatically sets up YARN
- [Installing Spark](https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.1/installing-spark/content/installing_spark.html)
- Install MinIO Distributed Server using one of the guides below.
- [Deployment based on Kubernetes](https://docs.min.io/community/minio-object-store/operations/deployments/kubernetes.html)
- [Deployment based on MinIO Helm Chart](https://github.com/helm/charts/tree/master/stable/minio)
## **3. Configure Hadoop, Spark, Hive to use MinIO**
After successful installation navigate to the Ambari UI `http://<ambari-server>:8080/` and login using the default credentials: [**_username: admin, password: admin_**]
![ambari-login](https://github.com/minio/minio/blob/master/docs/bigdata/images/image3.png?raw=true "ambari login")
### **3.1 Configure Hadoop**
Navigate to **Services** -> **HDFS** -> **CONFIGS** -> **ADVANCED** as shown below
![hdfs-configs](https://github.com/minio/minio/blob/master/docs/bigdata/images/image2.png?raw=true "hdfs advanced configs")
Navigate to **Custom core-site** to configure MinIO parameters for `_s3a_` connector
![s3a-config](https://github.com/minio/minio/blob/master/docs/bigdata/images/image5.png?raw=true "custom core-site")
```
sudo pip install yq
alias kv-pairify='yq ".configuration[]" | jq ".[]" | jq -r ".name + \"=\" + .value"'
```
Let's take for example a set of 12 compute nodes with an aggregate memory of _1.2TiB_, we need to do following settings for optimal results. Add the following optimal entries for _core-site.xml_ to configure _s3a_ with **MinIO**. Most important options here are
```
cat ${HADOOP_CONF_DIR}/core-site.xml | kv-pairify | grep "mapred"
mapred.maxthreads.generate.mapoutput=2 # Num threads to write map outputs
mapred.maxthreads.partition.closer=0 # Asynchronous map flushers
mapreduce.fileoutputcommitter.algorithm.version=2 # Use the latest committer version
mapreduce.job.reduce.slowstart.completedmaps=0.99 # 99% map, then reduce
mapreduce.reduce.shuffle.input.buffer.percent=0.9 # Min % buffer in RAM
mapreduce.reduce.shuffle.merge.percent=0.9 # Minimum % merges in RAM
mapreduce.reduce.speculative=false # Disable speculation for reducing
mapreduce.task.io.sort.factor=999 # Threshold before writing to disk
mapreduce.task.sort.spill.percent=0.9 # Minimum % before spilling to disk
```
S3A is the connector to use S3 and other S3-compatible object stores such as MinIO. MapReduce workloads typically interact with object stores in the same way they do with HDFS. These workloads rely on HDFS atomic rename functionality to complete writing data to the datastore. Object storage operations are atomic by nature and they do not require/implement rename API. The default S3A committer emulates renames through copy and delete APIs. This interaction pattern causes significant loss of performance because of the write amplification. _Netflix_, for example, developed two new staging committers - the Directory staging committer and the Partitioned staging committer - to take full advantage of native object storage operations. These committers do not require rename operation. The two staging committers were evaluated, along with another new addition called the Magic committer for benchmarking.
It was found that the directory staging committer was the fastest among the three, S3A connector should be configured with the following parameters for optimal results:
```
cat ${HADOOP_CONF_DIR}/core-site.xml | kv-pairify | grep "s3a"
fs.s3a.access.key=minio
fs.s3a.secret.key=minio123
fs.s3a.path.style.access=true
fs.s3a.block.size=512M
fs.s3a.buffer.dir=${hadoop.tmp.dir}/s3a
fs.s3a.committer.magic.enabled=false
fs.s3a.committer.name=directory
fs.s3a.committer.staging.abort.pending.uploads=true
fs.s3a.committer.staging.conflict-mode=append
fs.s3a.committer.staging.tmp.path=/tmp/staging
fs.s3a.committer.staging.unique-filenames=true
fs.s3a.connection.establish.timeout=5000
fs.s3a.connection.ssl.enabled=false
fs.s3a.connection.timeout=200000
fs.s3a.endpoint=http://minio:9000
fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
fs.s3a.committer.threads=2048 # Number of threads writing to MinIO
fs.s3a.connection.maximum=8192 # Maximum number of concurrent conns
fs.s3a.fast.upload.active.blocks=2048 # Number of parallel uploads
fs.s3a.fast.upload.buffer=disk # Use disk as the buffer for uploads
fs.s3a.fast.upload=true # Turn on fast upload mode
fs.s3a.max.total.tasks=2048 # Maximum number of parallel tasks
fs.s3a.multipart.size=512M # Size of each multipart chunk
fs.s3a.multipart.threshold=512M # Size before using multipart uploads
fs.s3a.socket.recv.buffer=65536 # Read socket buffer hint
fs.s3a.socket.send.buffer=65536 # Write socket buffer hint
fs.s3a.threads.max=2048 # Maximum number of threads for S3A
```
The rest of the other optimization options are discussed in the links below
- [https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html](https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html)
- [https://hadoop.apache.org/docs/r3.1.1/hadoop-aws/tools/hadoop-aws/committers.html](https://hadoop.apache.org/docs/r3.1.1/hadoop-aws/tools/hadoop-aws/committers.html)
Once the config changes are applied, proceed to restart **Hadoop** services.
![hdfs-services](https://github.com/minio/minio/blob/master/docs/bigdata/images/image7.png?raw=true "hdfs restart services")
### **3.2 Configure Spark2**
Navigate to **Services** -> **Spark2** -> **CONFIGS** as shown below
![spark-config](https://github.com/minio/minio/blob/master/docs/bigdata/images/image6.png?raw=true "spark config")
Navigate to “**Custom spark-defaults**” to configure MinIO parameters for `_s3a_` connector
![spark-config](https://github.com/minio/minio/blob/master/docs/bigdata/images/image9.png?raw=true "spark defaults")
Add the following optimal entries for _spark-defaults.conf_ to configure Spark with **MinIO**.
```
spark.hadoop.fs.s3a.access.key minio
spark.hadoop.fs.s3a.secret.key minio123
spark.hadoop.fs.s3a.path.style.access true
spark.hadoop.fs.s3a.block.size 512M
spark.hadoop.fs.s3a.buffer.dir ${hadoop.tmp.dir}/s3a
spark.hadoop.fs.s3a.committer.magic.enabled false
spark.hadoop.fs.s3a.committer.name directory
spark.hadoop.fs.s3a.committer.staging.abort.pending.uploads true
spark.hadoop.fs.s3a.committer.staging.conflict-mode append
spark.hadoop.fs.s3a.committer.staging.tmp.path /tmp/staging
spark.hadoop.fs.s3a.committer.staging.unique-filenames true
spark.hadoop.fs.s3a.committer.threads 2048 # number of threads writing to MinIO
spark.hadoop.fs.s3a.connection.establish.timeout 5000
spark.hadoop.fs.s3a.connection.maximum 8192 # maximum number of concurrent conns
spark.hadoop.fs.s3a.connection.ssl.enabled false
spark.hadoop.fs.s3a.connection.timeout 200000
spark.hadoop.fs.s3a.endpoint http://minio:9000
spark.hadoop.fs.s3a.fast.upload.active.blocks 2048 # number of parallel uploads
spark.hadoop.fs.s3a.fast.upload.buffer disk # use disk as the buffer for uploads
spark.hadoop.fs.s3a.fast.upload true # turn on fast upload mode
spark.hadoop.fs.s3a.impl org.apache.hadoop.spark.hadoop.fs.s3a.S3AFileSystem
spark.hadoop.fs.s3a.max.total.tasks 2048 # maximum number of parallel tasks
spark.hadoop.fs.s3a.multipart.size 512M # size of each multipart chunk
spark.hadoop.fs.s3a.multipart.threshold 512M # size before using multipart uploads
spark.hadoop.fs.s3a.socket.recv.buffer 65536 # read socket buffer hint
spark.hadoop.fs.s3a.socket.send.buffer 65536 # write socket buffer hint
spark.hadoop.fs.s3a.threads.max 2048 # maximum number of threads for S3A
```
Once the config changes are applied, proceed to restart **Spark** services.
![spark-config](https://github.com/minio/minio/blob/master/docs/bigdata/images/image12.png?raw=true "spark restart services")
### **3.3 Configure Hive**
Navigate to **Services** -> **Hive** -> **CONFIGS**-> **ADVANCED** as shown below
![hive-config](https://github.com/minio/minio/blob/master/docs/bigdata/images/image10.png?raw=true "hive advanced config")
Navigate to “**Custom hive-site**” to configure MinIO parameters for `_s3a_` connector
![hive-config](https://github.com/minio/minio/blob/master/docs/bigdata/images/image11.png?raw=true "hive advanced config")
Add the following optimal entries for `hive-site.xml` to configure Hive with **MinIO**.
```
hive.blobstore.use.blobstore.as.scratchdir=true
hive.exec.input.listing.max.threads=50
hive.load.dynamic.partitions.thread=25
hive.metastore.fshandler.threads=50
hive.mv.files.threads=40
mapreduce.input.fileinputformat.list-status.num-threads=50
```
For more information about these options please visit [https://www.cloudera.com/documentation/enterprise/5-11-x/topics/admin_hive_on_s3_tuning.html](https://www.cloudera.com/documentation/enterprise/5-11-x/topics/admin_hive_on_s3_tuning.html)
![hive-config](https://github.com/minio/minio/blob/master/docs/bigdata/images/image13.png?raw=true "hive advanced custom config")
Once the config changes are applied, proceed to restart all Hive services.
![hive-config](https://github.com/minio/minio/blob/master/docs/bigdata/images/image14.png?raw=true "restart hive services")
## **4. Run Sample Applications**
After installing Hive, Hadoop and Spark successfully, we can now proceed to run some sample applications to see if they are configured appropriately. We can use Spark Pi and Spark WordCount programs to validate our Spark installation. We can also explore how to run Spark jobs from the command line and Spark shell.
### **4.1 Spark Pi**
Test the Spark installation by running the following compute intensive example, which calculates pi by “throwing darts” at a circle. The program generates points in the unit square ((0,0) to (1,1)) and counts how many points fall within the unit circle within the square. The result approximates pi.
Follow these steps to run the Spark Pi example:
- Login as user **‘spark’**.
- When the job runs, the library can now use **MinIO** during intermediate processing.
- Navigate to a node with the Spark client and access the spark2-client directory:
```
cd /usr/hdp/current/spark2-client
su spark
```
- Run the Apache Spark Pi job in yarn-client mode, using code from **org.apache.spark**:
```
./bin/spark-submit --class org.apache.spark.examples.SparkPi \
--master yarn-client \
--num-executors 1 \
--driver-memory 512m \
--executor-memory 512m \
--executor-cores 1 \
examples/jars/spark-examples*.jar 10
```
The job should produce an output as shown below. Note the value of pi in the output.
```
17/03/22 23:21:10 INFO DAGScheduler: Job 0 finished: reduce at SparkPi.scala:38, took 1.302805 s
Pi is roughly 3.1445191445191445
```
Job status can also be viewed in a browser by navigating to the YARN ResourceManager Web UI and clicking on job history server information.
### **4.2 WordCount**
WordCount is a simple program that counts how often a word occurs in a text file. The code builds a dataset of (String, Int) pairs called counts, and saves the dataset to a file.
The following example submits WordCount code to the Scala shell. Select an input file for the Spark WordCount example. We can use any text file as input.
- Login as user **‘spark’**.
- When the job runs, the library can now use **MinIO** during intermediate processing.
- Navigate to a node with Spark client and access the spark2-client directory:
```
cd /usr/hdp/current/spark2-client
su spark
```
The following example uses _log4j.properties_ as the input file:
#### **4.2.1 Upload the input file to HDFS:**
```
hadoop fs -copyFromLocal /etc/hadoop/conf/log4j.properties
s3a://testbucket/testdata
```
#### **4.2.2 Run the Spark shell:**
```
./bin/spark-shell --master yarn-client --driver-memory 512m --executor-memory 512m
```
The command should produce an output as shown below. (with additional status messages):
```
Spark context Web UI available at http://172.26.236.247:4041
Spark context available as 'sc' (master = yarn, app id = application_1490217230866_0002).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.1.0.2.6.0.0-598
/_/
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_112)
Type in expressions to have them evaluated.
Type :help for more information.
scala>
```
- At the _scala>_ prompt, submit the job by typing the following commands, Replace node names, file name, and file location with your values:
```
scala> val file = sc.textFile("s3a://testbucket/testdata")
file: org.apache.spark.rdd.RDD[String] = s3a://testbucket/testdata MapPartitionsRDD[1] at textFile at <console>:24
scala> val counts = file.flatMap(line => line.split(" ")).map(word => (word, 1)).reduceByKey(_ + _)
counts: org.apache.spark.rdd.RDD[(String, Int)] = ShuffledRDD[4] at reduceByKey at <console>:25
scala> counts.saveAsTextFile("s3a://testbucket/wordcount")
```
Use one of the following approaches to view job output:
View output in the Scala shell:
```
scala> counts.count()
364
```
To view the output from MinIO exit the Scala shell. View WordCount job status:
```
hadoop fs -ls s3a://testbucket/wordcount
```
The output should be similar to the following:
```
Found 3 items
-rw-rw-rw- 1 spark spark 0 2019-05-04 01:36 s3a://testbucket/wordcount/_SUCCESS
-rw-rw-rw- 1 spark spark 4956 2019-05-04 01:36 s3a://testbucket/wordcount/part-00000
-rw-rw-rw- 1 spark spark 5616 2019-05-04 01:36 s3a://testbucket/wordcount/part-00001
```

BIN
docs/bigdata/images/image1.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 322 KiB

BIN
docs/bigdata/images/image10.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 53 KiB

BIN
docs/bigdata/images/image11.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 45 KiB

BIN
docs/bigdata/images/image12.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 42 KiB

BIN
docs/bigdata/images/image13.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 74 KiB

BIN
docs/bigdata/images/image14.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 45 KiB

BIN
docs/bigdata/images/image2.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 67 KiB

BIN
docs/bigdata/images/image3.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 11 KiB

BIN
docs/bigdata/images/image4.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 82 KiB

BIN
docs/bigdata/images/image5.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 64 KiB

BIN
docs/bigdata/images/image6.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 48 KiB

BIN
docs/bigdata/images/image7.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 58 KiB

BIN
docs/bigdata/images/image8.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 78 KiB

BIN
docs/bigdata/images/image9.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 70 KiB

55
docs/bucket/lifecycle/DESIGN.md Normal file
View File

@@ -0,0 +1,55 @@
# ILM Tiering Design [![slack](https://slack.min.io/slack?type=svg)](https://slack.min.io) [![Docker Pulls](https://img.shields.io/docker/pulls/minio/minio.svg?maxAge=604800)](https://hub.docker.com/r/minio/minio/)
Lifecycle transition functionality provided in [bucket lifecycle guide](https://github.com/minio/minio/master/docs/bucket/lifecycle/README.md) allows tiering of content from MinIO object store to public clouds or other MinIO clusters.
Transition tiers can be added to MinIO using `mc admin tier add` command to associate a `gcs`, `s3` or `azure` bucket or prefix path on a bucket to the tier name.
Lifecycle transition rules can be applied to buckets (both versioned and un-versioned) by specifying the tier name defined above as the transition storage class for the lifecycle rule.
## Implementation
ILM tiering takes place when a object placed in the bucket meets lifecycle transition rules and becomes eligible for tiering. MinIO scanner (which runs at one minute intervals, each time scanning one sixteenth of the namespace), picks up the object for tiering. The data is moved to the remote tier in entirety, leaving only the object metadata on MinIO.
The data on the backend is stored under the `bucket/prefix` specified in the tier configuration with a custom name derived from a randomly generated uuid - e.g. `0b/c4/0bc4fab7-2daf-4d2f-8e39-5c6c6fb7e2d3`. The first two prefixes are characters 1-2,3-4 from the uuid. This format allows tiering to any cloud irrespective of whether the cloud in question supports versioning. The reference to the transitioned object name and transitioned tier is stored as part of the internal metadata for the object (or its version) on MinIO.
Extra metadata maintained internally in `xl.meta` for a transitioned object
```
...
"MetaSys": {
"x-minio-internal-transition-status": "Y29tcGxldGU=",
"x-minio-internal-transition-tier": "R0NTVElFUjE=",
"x-minio-internal-transitioned-object": "ZDIvN2MvZDI3Y2MwYWMtZGIzNC00ZGM1LWIxNDUtYjI5MGNjZjU1MjY5"
},
```
When a transitioned object is restored temporarily to local MinIO instance via PostRestoreObject API, the object data is copied back from the remote tier, and additional metadata for the restored object is maintained as referenced below. Once the restore period expires, the local copy of the object is removed by the scanner during its periodic runs.
```
...
"MetaUsr": {
"X-Amz-Restore-Expiry-Days": "4",
"X-Amz-Restore-Request-Date": "Mon, 22 Feb 2021 21:10:09 GMT",
"x-amz-restore": "ongoing-request=false, expiry-date=Sat, 27 Feb 2021 00:00:00 GMT",
...
```
### Encrypted/Object locked objects
For objects under SSE-S3 or SSE-C encryption, the encrypted content from MinIO cluster is copied as is to the remote tier without any decryption. The content is decrypted as it is streamed from remote tier on `GET/HEAD`. Objects under retention are protected because the metadata present on MinIO server ensures that the object (version) is not deleted until retention period is over. Administrators need to ensure that the remote tier bucket is under proper access control.
### Transition Status
MinIO specific extension header `X-Minio-Transition` is displayed on `HEAD/GET` to predict expected transition date on the object. Once object is transitioned to the remote tier,`x-amz-storage-class` shows the tier name to which object transitioned. Additional headers such as "X-Amz-Restore-Expiry-Days", "x-amz-restore", and "X-Amz-Restore-Request-Date" are displayed when a object is under restore/has been restored to local MinIO cluster.
### Expiry or removal events
An object that is in transition tier will be deleted once the object hits expiry date or if removed via `mc rm` (`mc rm --vid` in the case of delete of a specific object version). Other rules specific to legal hold and object locking precede any lifecycle rules.
### Additional notes
Tiering and lifecycle transition are applicable only to erasure/distributed MinIO.
## Explore Further
- [MinIO | Golang Client API Reference](https://docs.min.io/community/minio-object-store/developers/go/API.html#SetBucketLifecycle)
- [Object Lifecycle Management](https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html)

232
docs/bucket/lifecycle/README.md Normal file
View File

@@ -0,0 +1,232 @@
# Bucket Lifecycle Configuration Quickstart Guide [![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io) [![Docker Pulls](https://img.shields.io/docker/pulls/minio/minio.svg?maxAge=604800)](https://hub.docker.com/r/minio/minio/)
Enable object lifecycle configuration on buckets to setup automatic deletion of objects after a specified number of days or a specified date.
## 1. Prerequisites
- Install MinIO - [MinIO Quickstart Guide](https://docs.min.io/community/minio-object-store/operations/deployments/baremetal-deploy-minio-on-redhat-linux.html).
- Install `mc` - [mc Quickstart Guide](https://docs.min.io/community/minio-object-store/reference/minio-mc.html#quickstart)
## 2. Enable bucket lifecycle configuration
- Create a bucket lifecycle configuration which expires the objects under the prefix `old/` on `2020-01-01T00:00:00.000Z` date and the objects under `temp/` after 7 days.
- Enable bucket lifecycle configuration using `mc`:
```sh
$ mc ilm import play/testbucket <<EOF
{
"Rules": [
{
"Expiration": {
"Date": "2020-01-01T00:00:00.000Z"
},
"ID": "OldPictures",
"Filter": {
"Prefix": "old/"
},
"Status": "Enabled"
},
{
"Expiration": {
"Days": 7
},
"ID": "TempUploads",
"Filter": {
"Prefix": "temp/"
},
"Status": "Enabled"
}
]
}
EOF
```
```
Lifecycle configuration imported successfully to `play/testbucket`.
```
- List the current settings
```
$ mc ilm ls play/testbucket
ID | Prefix | Enabled | Expiry | Date/Days | Transition | Date/Days | Storage-Class | Tags
------------|----------|------------|--------|--------------|--------------|------------------|------------------|------------------
OldPictures | old/ | ✓ | ✓ | 1 Jan 2020 | ✗ | | |
------------|----------|------------|--------|--------------|--------------|------------------|------------------|------------------
TempUploads | temp/ | ✓ | ✓ | 7 day(s) | ✗ | | |
------------|----------|------------|--------|--------------|--------------|------------------|------------------|------------------
```
## 3. Activate ILM versioning features
This will only work with a versioned bucket, take a look at [Bucket Versioning Guide](https://docs.min.io/community/minio-object-store/administration/object-management/object-versioning.html) for more understanding.
### 3.1 Automatic removal of non current objects versions
A non-current object version is a version which is not the latest for a given object. It is possible to set up an automatic removal of non-current versions when a version becomes older than a given number of days.
e.g., To scan objects stored under `user-uploads/` prefix and remove versions older than one year.
```
{
"Rules": [
{
"ID": "Removing all old versions",
"Filter": {
"Prefix": "users-uploads/"
},
"NoncurrentVersionExpiration": {
"NoncurrentDays": 365
},
"Status": "Enabled"
}
]
}
```
This JSON rule is equivalent to the following MinIO Client command:
```
mc ilm rule add --noncurrent-expire-days 365 --prefix "user-uploads/" myminio/mydata
```
### 3.2 Automatic removal of noncurrent versions keeping only most recent ones after noncurrent days
It is possible to configure automatic removal of older noncurrent versions keeping only the most recent `N` using `NewerNoncurrentVersions`.
e.g, To remove noncurrent versions of all objects keeping the most recent 5 noncurrent versions under the prefix `user-uploads/` 30 days after they become noncurrent ,
```
{
"Rules": [
{
"ID": "Keep only most recent 5 noncurrent versions",
"Status": "Enabled",
"Filter": {
"Prefix": "users-uploads/"
},
"NoncurrentVersionExpiration": {
"NewerNoncurrentVersions": 5,
"NoncurrentDays": 30
}
}
]
}
```
This JSON rule is equivalent to the following MinIO Client command:
```
mc ilm rule add --noncurrent-expire-days 30 --noncurrent-expire-newer 5 --prefix "user-uploads/" myminio/mydata
```
#### 3.2.a Automatic removal of noncurrent versions keeping only most recent ones immediately (MinIO only extension)
This is available only on MinIO as an extension to the NewerNoncurrentVersions feature. The following rule makes it possible to remove older noncurrent versions
of objects under the prefix `user-uploads/` as soon as there are more than `N` noncurrent versions of an object.
```
{
"Rules": [
{
"ID": "Keep only most recent 5 noncurrent versions",
"Status": "Enabled",
"Filter": {
"Prefix": "users-uploads/"
},
"NoncurrentVersionExpiration": {
"NewerNoncurrentVersions": 5
}
}
]
}
```
Note: This rule has an implicit zero NoncurrentDays, which makes the expiry of those 'extra' noncurrent versions immediate.
#### 3.2.b Automatic removal of all versions (MinIO only extension)
This is available only on MinIO as an extension to the Expiration feature. The following rule makes it possible to remove all versions of an object under
the prefix `user-uploads/` as soon as the latest object satisfies the expiration criteria.
> NOTE: If the latest object is a delete marker then filtering based on `Filter.Tags` is ignored and
> if the DELETE marker modTime satisfies the `Expiration.Days` then all versions of the object are
> immediately purged.
```
{
"Rules": [
{
"ID": "Purge all versions of an expired object",
"Status": "Enabled",
"Filter": {
"Prefix": "users-uploads/"
},
"Expiration": {
"Days": 7,
"ExpiredObjectAllVersions": true
}
}
]
}
```
### 3.3 Automatic removal of delete markers with no other versions
When an object has only one version as a delete marker, the latter can be automatically removed after a certain number of days using the following configuration:
```
{
"Rules": [
{
"ID": "Removing all delete markers",
"Expiration": {
"ExpiredObjectDeleteMarker": true
},
"Status": "Enabled"
}
]
}
```
## 4. Enable ILM transition feature
In Erasure mode, MinIO supports tiering to public cloud providers such as GCS, AWS and Azure as well as to other MinIO clusters via the ILM transition feature. This will allow transitioning of older objects to a different cluster or the public cloud by setting up transition rules in the bucket lifecycle configuration. This feature enables applications to optimize storage costs by moving less frequently accessed data to a cheaper storage without compromising accessibility of data.
To transition objects in a bucket to a destination bucket on a different cluster, applications need to specify a transition tier defined on MinIO instead of storage class while setting up the ILM lifecycle rule.
> To create a transition tier for transitioning objects to a prefix `testprefix` in `azurebucket` on Azure blob using `mc`:
```
mc admin tier add azure source AZURETIER --endpoint https://blob.core.windows.net --access-key AZURE_ACCOUNT_NAME --secret-key AZURE_ACCOUNT_KEY --bucket azurebucket --prefix testprefix1/
```
> The admin user running this command needs the "admin:SetTier" and "admin:ListTier" permissions if not running as root.
Using above tier, set up a lifecycle rule with transition:
```
mc ilm add --expiry-days 365 --transition-days 45 --storage-class "AZURETIER" myminio/srcbucket
```
Note: In the case of S3, it is possible to create a tier from MinIO running in EC2 to S3 using AWS role attached to EC2 as credentials instead of accesskey/secretkey:
```
mc admin tier add s3 source S3TIER --bucket s3bucket --prefix testprefix/ --use-aws-role
```
Once transitioned, GET or HEAD on the object will stream the content from the transitioned tier. In the event that the object needs to be restored temporarily to the local cluster, the AWS [RestoreObject API](https://docs.aws.amazon.com/AmazonS3/latest/API/API_RestoreObject.html) can be utilized.
```
aws s3api restore-object --bucket srcbucket \
--key object \
--restore-request Days=3
```
### 4.1 Monitoring transition events
`s3:ObjectTransition:Complete` and `s3:ObjectTransition:Failed` events can be used to monitor transition events between the source cluster and transition tier. To watch lifecycle events, you can enable bucket notification on the source bucket with `mc event add` and specify `--event ilm` flag.
Note that transition event notification is a MinIO extension.
## Explore Further
- [MinIO | Golang Client API Reference](https://docs.min.io/community/minio-object-store/developers/go/API.html)
- [Object Lifecycle Management](https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html)

90
docs/bucket/lifecycle/setup_ilm_transition.sh Executable file
View File

@@ -0,0 +1,90 @@
#!/usr/bin/env bash
set -x
trap 'catch $LINENO' ERR
# shellcheck disable=SC2120
catch() {
if [ $# -ne 0 ]; then
echo "error on line $1"
for site in sitea siteb; do
echo "$site server logs ========="
cat "/tmp/${site}_1.log"
echo "==========================="
cat "/tmp/${site}_2.log"
done
fi
echo "Cleaning up instances of MinIO"
pkill minio
pkill -9 minio
rm -rf /tmp/multisitea
rm -rf /tmp/multisiteb
if [ $# -ne 0 ]; then
exit $#
fi
}
catch
export MINIO_CI_CD=1
export MINIO_BROWSER=off
export MINIO_KMS_AUTO_ENCRYPTION=off
export MINIO_PROMETHEUS_AUTH_TYPE=public
export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
unset MINIO_KMS_KES_CERT_FILE
unset MINIO_KMS_KES_KEY_FILE
unset MINIO_KMS_KES_ENDPOINT
unset MINIO_KMS_KES_KEY_NAME
if [ ! -f ./mc ]; then
wget --quiet -O mc https://dl.minio.io/client/mc/release/linux-amd64/mc &&
chmod +x mc
fi
minio server --address 127.0.0.1:9001 "http://127.0.0.1:9001/tmp/multisitea/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9002/tmp/multisitea/data/disterasure/xl{5...8}" >/tmp/sitea_1.log 2>&1 &
minio server --address 127.0.0.1:9002 "http://127.0.0.1:9001/tmp/multisitea/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9002/tmp/multisitea/data/disterasure/xl{5...8}" >/tmp/sitea_2.log 2>&1 &
minio server --address 127.0.0.1:9003 "http://127.0.0.1:9003/tmp/multisiteb/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9004/tmp/multisiteb/data/disterasure/xl{5...8}" >/tmp/siteb_1.log 2>&1 &
minio server --address 127.0.0.1:9004 "http://127.0.0.1:9003/tmp/multisiteb/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9004/tmp/multisiteb/data/disterasure/xl{5...8}" >/tmp/siteb_2.log 2>&1 &
# Wait to make sure all MinIO instances are up
export MC_HOST_sitea=http://minioadmin:minioadmin@127.0.0.1:9001
export MC_HOST_siteb=http://minioadmin:minioadmin@127.0.0.1:9004
./mc ready sitea
./mc ready siteb
./mc mb --ignore-existing sitea/bucket
./mc mb --ignore-existing siteb/bucket
sleep 10s
## Add warm tier
./mc ilm tier add minio sitea WARM-TIER --endpoint http://localhost:9004 --access-key minioadmin --secret-key minioadmin --bucket bucket
## Add ILM rules
./mc ilm add sitea/bucket --transition-days 0 --transition-tier WARM-TIER
./mc ilm rule list sitea/bucket
./mc cp README.md sitea/bucket/README.md
until $(./mc stat sitea/bucket/README.md --json | jq -r '.metadata."X-Amz-Storage-Class"' | grep -q WARM-TIER); do
echo "waiting until the object is tiered to run heal"
sleep 1s
done
./mc stat sitea/bucket/README.md
success=$(./mc admin heal -r sitea/bucket/README.md --json --force | jq -r 'select((.name == "bucket/README.md") and (.after.color == "green")) | .after.color == "green"')
if [ "${success}" != "true" ]; then
echo "Found bug expected transitioned object to report 'green'"
exit 1
fi
catch

1443
docs/bucket/notifications/README.md Normal file
View File

File diff suppressed because it is too large Load Diff

30
docs/bucket/quota/README.md Normal file
View File

@@ -0,0 +1,30 @@
# Bucket Quota Configuration Quickstart Guide [![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io) [![Docker Pulls](https://img.shields.io/docker/pulls/minio/minio.svg?maxAge=604800)](https://hub.docker.com/r/minio/minio/)
![quota](https://raw.githubusercontent.com/minio/minio/master/docs/bucket/quota/bucketquota.png)
Buckets can be configured to have `Hard` quota - it disallows writes to the bucket after configured quota limit is reached.
## Prerequisites
- Install MinIO - [MinIO Quickstart Guide](https://docs.min.io/community/minio-object-store/operations/deployments/baremetal-deploy-minio-on-redhat-linux.html#procedure).
- [Use `mc` with MinIO Server](https://docs.min.io/community/minio-object-store/reference/minio-mc.html#quickstart)
## Set bucket quota configuration
### Set a hard quota of 1GB for a bucket `mybucket` on MinIO object storage
```sh
mc admin bucket quota myminio/mybucket --hard 1gb
```
### Verify the quota configured on `mybucket` on MinIO
```sh
mc admin bucket quota myminio/mybucket
```
### Clear bucket quota configuration for `mybucket` on MinIO
```sh
mc admin bucket quota myminio/mybucket --clear
```

BIN
docs/bucket/quota/bucketquota.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 95 KiB

BIN
docs/bucket/replication/DELETE_bucket_replication.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 37 KiB

160
docs/bucket/replication/DESIGN.md Normal file
View File

@@ -0,0 +1,160 @@
# Bucket Replication Design [![slack](https://slack.min.io/slack?type=svg)](https://slack.min.io) [![Docker Pulls](https://img.shields.io/docker/pulls/minio/minio.svg?maxAge=604800)](https://hub.docker.com/r/minio/minio/)
This document explains the design approach of server side bucket replication. If you're looking to get started with replication, we suggest you go through the [Bucket replication guide](https://github.com/minio/minio/blob/master/docs/bucket/replication/README.md) first.
## Overview
Replication relies on immutability provided by versioning to sync objects between the configured source and replication target. Replication results in the object data, metadata, last modification time and version ID all being identical between the source and target. Thus version ordering is automatically guaranteed on the source and target clusters.
### Replication of object version and metadata
If an object meets replication rules as set in the replication configuration, `X-Amz-Replication-Status` is first set to `PENDING` as the PUT operation completes and replication is queued (unless synchronous replication is in place). After replication is performed, the metadata on the source object version changes to `COMPLETED` or `FAILED` depending on whether replication succeeded. The object version on the target shows `X-Amz-Replication-Status` of `REPLICA`
All replication failures are picked up by the scanner which runs at a one minute frequency, each time scanning up to a sixteenth of the namespace. Object versions marked `PENDING` or `FAILED` are re-queued for replication.
Replication speed depends on the cluster load, number of objects in the object store as well as storage speed. In addition, any bandwidth limits set via `mc admin bucket remote add` could also contribute to replication speed. The number of workers used for replication defaults to 100. Based on network bandwidth and system load, the number of workers used in replication can be configured using `mc admin config set alias api` to set the `replication_workers`. The prometheus metrics exposed by MinIO can be used to plan resource allocation and bandwidth management to optimize replication speed.
If synchronous replication is configured above, replication is attempted right away prior to returning the PUT object response. In the event that the replication target is down, the `X-Amz-Replication-Status` is marked as `FAILED` and resynced with target when the scanner runs again.
Any metadata changes on the source object version, such as metadata updates via PutObjectTagging, PutObjectRetention, PutObjectLegalHold and COPY api are replicated in a similar manner to target version, with the `X-Amz-Replication-Status` again cycling through the same states.
The description above details one way replication from source to target w.r.t incoming object uploads and metadata changes to source object version. If active-active replication is configured, any incoming uploads and metadata changes to versions created on the target, will sync back to the source and be marked as `REPLICA` on the source. AWS, as well as MinIO do not by default sync metadata changes on a object version marked `REPLICA` back to source. This requires a setting in the replication configuration called [replica modification sync](https://aws.amazon.com/about-aws/whats-new/2020/12/amazon-s3-replication-adds-support-two-way-replication/).
For active-active replication, automatic failover occurs on `GET/HEAD` operations if object or object version requested qualifies for replication and is missing on one site, but present on the other. This allows the applications to take full advantage of two-way replication even before the two sites get fully synced.
In the case of multi destination replication, the replication status shows `COMPLETED` only after the replication operation succeeds on each of the targets specified in the replication configuration. If multiple targets are configured to use active-active replication and multi destination replication, the administrator should ensure that the replication features enabled (such as replica metadata sync, delete marker replication etc) are identical to avoid asymmetric state. This is because all replication activity is inherently a one-way operation from source to target, irrespective of the number of targets.
### Replication of DeleteMarker and versioned Delete
MinIO allows DeleteMarker replication and versioned delete replication by setting `--replicate delete,delete-marker` while setting up replication configuration using `mc replicate add`. The MinIO implementation is based on V2 configuration, however it has been extended to allow both DeleteMarker replication and replication of versioned deletes with the `DeleteMarkerReplication` and `DeleteReplication` fields in the replication configuration. By default, this is set to `Disabled` unless the user specifies it while adding a replication rule.
Similar to object version replication, DeleteMarker replication also cycles through `PENDING` to `COMPLETED` or `FAILED` states for the `X-Amz-Replication-Status` on the source when a delete marker is set (i.e. performing `mc rm` on an object without specifying a version).After replication syncs the delete marker on the target, the DeleteMarker on the target shows `X-Amz-Replication-Status` of `REPLICA`. The status of DeleteMarker replication is returned by `X-Minio-Replication-DeleteMarker-Status` header on `HEAD/GET` calls for the delete marker version in question - i.e with `mc stat --version-id dm-version-id`
It must be noted that if active-active replication is set up with delete marker replication, there is potential for duplicate delete markers to be created if both source and target concurrently set a Delete Marker or if one/both of the clusters went down at tandem before the replication event was synced.This is an unavoidable side-effect in active-active replication caused by allowing delete markers set on a object version with `REPLICA` status back to source.
In the case of versioned deletes a.k.a permanent delete of a version by doing a `mc rm --version-id` on a object, replication implementation marks a object version permanently deleted as `PENDING` purge and deletes the version from source after syncing to the target and ensuring target version is deleted. The delete marker being deleted or object version being deleted will still be visible on listing with `mc ls --versions` until the sync is completed. Objects marked as deleted will not be accessible via `GET` or `HEAD` requests and would return a http response code of `405`. The status of versioned delete replication on the source can be queried by `HEAD` request on the delete marker versionID or object versionID in question.
An additional header `X-Minio-Replication-Delete-Status` is returned which would show `PENDING` or `FAILED` status if the replication is still not caught up.
Note that synchronous replication, i.e. when remote target is configured with --sync mode in `mc admin bucket remote add` does not apply to `DELETE` operations. The version being deleted on the source cluster needs to maintain state and ensure that the operation is mirrored to the target cluster prior to completing on the source object version. Since this needs to account for the target cluster availability and the need to serialize concurrent DELETE operations on different versions of the same object during multi DELETE operations, the current implementation queues the `DELETE` operations in both sync and async modes.
### Existing object replication
Existing object replication works similar to regular replication. Objects qualifying for existing object replication are detected when scanner runs, and will be replicated if existing object replication is enabled and applicable replication rules are satisfied. Because replication depends on the immutability of versions, only pre-existing objects created while versioning was enabled can be replicated. Even if replication rules are disabled and re-enabled later, the objects created during the interim will be synced as the scanner queues them. For saving iops, objects qualifying for
existing object replication are not marked as `PENDING` prior to replication.
Note that objects with `null` versions, i.e. objects created prior to enabling versioning break the immutability guarantees provided by versioning. When existing object replication is enabled, these objects will be replicated as `null` versions to the remote targets provided they are not present on the target or if `null` version of object on source is newer than the `null` version of object on target.
If the remote site is fully lost and objects previously replicated need to be re-synced, the `mc replicate resync start` command with optional flag of `--older-than` needs to be used to trigger re-syncing of previously replicated objects. This command generates a ResetID which is a unique UUID saved to the remote target config along with the applicable date(defaults to time of initiating the reset). All objects created prior to this date are eligible for re-replication if existing object replication is enabled for the replication rule the object satisfies. At the time of completion of replication, `x-minio-internal-replication-reset-arn:<arn>` is set in the metadata with the timestamp of replication and ResetID. For saving iops, the objects which are re-replicated are not first set to `PENDING` state.
This is a slower operation that does not use replication queues and is designed to walk the namespace and replicate objects one at a time so as not to impede server load. Ideally, resync should not be initiated for multiple buckets simultaneously - progress of the syncing can be monitored by looking at `mc replicate resync status alias/bucket --remote-bucket <arn>`. In the event that resync operation failed to replicate some versions, they would be picked up by the healing mechanism in-built as part of the scanner. If the resync operation reports a failed status or in the event of a cluster restart while resync is in progress, a fresh `resync start` can be issued - this will replicate previously unsynced content at the cost of additional overhead in additional metadata updates.
### Multi destination replication
The replication design for multiple sites works in a similar manner as described above for two site scenario. However there are some
important exceptions.
Replication status on the source cluster will be marked as `COMPLETED` only after replication is completed on all targets. If one or more targets failed replication, the replication status is reflected as `PENDING`.
If 3 or more targets are participating in active-active replication, the replication configuration for replica metadata sync, delete marker replication and delete replication should match to avoid inconsistent picture between the clusters. It is not recommended to turn on asymmetric replication - for e.g. if three sites A,B,C are participating in replication, it would be better to avoid replication setups like A -> [B, C], B -> A. In this particular example, an object uploaded to A will be replicated to B,C. If replica metadata sync is turned on in site B, any metadata updates on a replica version made in B would reflect in A, but not in C.
### Internal metadata for replication
`xl.meta` that is in use for [versioning](https://github.com/minio/minio/blob/master/docs/bucket/versioning/DESIGN.md) has additional metadata for replication of objects,delete markers and versioned deletes.
### Metadata for object replication - on source
```
...
"MetaSys": {
"x-minio-internal-inline-data": "dHJ1ZQ==",
"x-minio-internal-replication-status": "YXJuOm1pbmlvOnJlcGxpY2F0aW9uOjo2YjdmYzFlMS0wNmU4LTQxMTUtYjYxNy00YTgzZGIyODhmNTM6YnVja2V0PUNPTVBMRVRFRDthcm46bWluaW86cmVwbGljYXRpb246OmI5MGYxZWEzLWMzYWQtNDEyMy1iYWE2LWZjMDZhYmEyMjA2MjpidWNrZXQ9Q09NUExFVEVEOw==",
"x-minio-internal-replication-timestamp": "MjAyMS0wOS0xN1QwMTo0MzozOC40MDQwMDA0ODNa",
"x-minio-internal-tier-free-versionID": "OWZlZjk5N2QtMjMzZi00N2U3LTlkZmMtNWYxNzc3NzdlZTM2"
},
"MetaUsr": {
"X-Amz-Replication-Status": "COMPLETED",
"content-type": "application/octet-stream",
"etag": "8315e643ed6a5d7c9962fc0a8ef9c11f"
},
...
```
### Metadata for object replication - on target
```
...
"MetaSys": {
"x-minio-internal-inline-data": "dHJ1ZQ==",
"x-minio-internal-replica-status": "UkVQTElDQQ==",
"x-minio-internal-replica-timestamp": "MjAyMS0wOS0xN1QwMTo0MzozOC4zODg5ODU4ODRa"
},
"MetaUsr": {
"X-Amz-Replication-Status": "REPLICA",
"content-type": "application/octet-stream",
"etag": "8315e643ed6a5d7c9962fc0a8ef9c11f",
"x-amz-storage-class": "STANDARD"
},
...
```
### Additional replication metadata for DeleteMarker
```
...
{
"DelObj": {
"ID": "u8H5pYQFRMKgkIgkpSKIkQ==",
"MTime": 1631843124147668389,
"MetaSys": {
"x-minio-internal-replication-status": "YXJuOm1pbmlvOnJlcGxpY2F0aW9uOjpiOTBmMWVhMy1jM2FkLTQxMjMtYmFhNi1mYzA2YWJhMjIwNjI6YnVja2V0PUNPTVBMRVRFRDthcm46bWluaW86cmVwbGljYXRpb246OjZiN2ZjMWUxLTA2ZTgtNDExNS1iNjE3LTRhODNkYjI4OGY1MzpidWNrZXQ9Q09NUExFVEVEOw==",
"x-minio-internal-replication-timestamp": "U3VuLCAzMSBEZWMgMDAwMCAxOTowMzo1OCBHTVQ="
}
},
"Type": 2
}
```
### Additional replication metadata for versioned delete
```
{
"DelObj": {
"ID": "u8H5pYQFRMKgkIgkpSKIkQ==",
"MTime": 1631843124147668389,
"MetaSys": {
"purgestatus": "YXJuOm1pbmlvOnJlcGxpY2F0aW9uOjpiOTBmMWVhMy1jM2FkLTQxMjMtYmFhNi1mYzA2YWJhMjIwNjI6YnVja2V0PUNPTVBMRVRFO2FybjptaW5pbzpyZXBsaWNhdGlvbjo6NmI3ZmMxZTEtMDZlOC00MTE1LWI2MTctNGE4M2RiMjg4ZjUzOmJ1Y2tldD1GQUlMRUQ7",
"x-minio-internal-replication-status": "YXJuOm1pbmlvOnJlcGxpY2F0aW9uOjpiOTBmMWVhMy1jM2FkLTQxMjMtYmFhNi1mYzA2YWJhMjIwNjI6YnVja2V0PTthcm46bWluaW86cmVwbGljYXRpb246OjZiN2ZjMWUxLTA2ZTgtNDExNS1iNjE3LTRhODNkYjI4OGY1MzpidWNrZXQ9Ow==",
"x-minio-internal-replication-timestamp": "U3VuLCAzMSBEZWMgMDAwMCAxOTowMzo1OCBHTVQ="
}
},
"Type": 2
}
```
### Additional Metadata for object replication resync - on source
```
...
"MetaSys": {
...
"x-minio-internal-replication-reset-arn:minio:replication::af470089-d354-4473-934c-9e1f52f6da89:bucket": "TW9uLCAwNyBGZWIgMjAyMiAyMDowMzo0MCBHTVQ7ZGMxMWQzNDgtMTAwMS00ODA3LWFhNjEtOGY2MmFiNWQ5ZjU2",
...
},
...
```
### Additional Metadata for resync replication of delete-markers - on source
```
...
"MetaSys": {
"x-minio-internal-replication-reset-arn:minio:replication::af470089-d354-4473-934c-9e1f52f6da89:bucket": "TW9uLCAwNyBGZWIgMjAyMiAyMDowMzo0MCBHTVQ7ZGMxMWQzNDgtMTAwMS00ODA3LWFhNjEtOGY2MmFiNWQ5ZjU2",
...
}
...
```
## Explore Further
- [MinIO Bucket Versioning Implementation](https://docs.min.io/community/minio-object-store/administration/object-management/object-versioning.html)
- [MinIO Client Quickstart Guide](https://docs.min.io/community/minio-object-store/reference/minio-mc.html#quickstart)

BIN
docs/bucket/replication/HEAD_bucket_replication.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 22 KiB

BIN
docs/bucket/replication/PUT_bucket_replication.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 89 KiB

281
docs/bucket/replication/README.md Normal file
View File

@@ -0,0 +1,281 @@
# Bucket Replication Guide [![slack](https://slack.min.io/slack?type=svg)](https://slack.min.io) [![Docker Pulls](https://img.shields.io/docker/pulls/minio/minio.svg?maxAge=604800)](https://hub.docker.com/r/minio/minio/)
Bucket replication is designed to replicate selected objects in a bucket to a destination bucket.
The contents of this page have been migrated to the new [MinIO Documentation: Bucket Replication](https://docs.min.io/community/minio-object-store/administration/bucket-replication.html) page. The [Bucket Replication](https://docs.min.io/community/minio-object-store/administration/bucket-replication/bucket-replication-requirements.html) page references dedicated tutorials for configuring one-way "Active-Passive" and two-way "Active-Active" bucket replication.
To replicate objects in a bucket to a destination bucket on a target site either in the same cluster or a different cluster, start by enabling [versioning](https://docs.min.io/community/minio-object-store/administration/object-management/object-versioning.html) for both source and destination buckets. Finally, the target site and the destination bucket need to be configured on the source MinIO server.
## Highlights
- Supports source and destination buckets to have the same name unlike AWS S3, addresses variety of use-cases such as *Splunk*, *Veeam* site to site DR.
- Supports object locking/retention across source and destination buckets natively out of the box, unlike AWS S3.
- Simpler implementation than [AWS S3 Bucket Replication Config](https://docs.aws.amazon.com/AmazonS3/latest/dev/replication-add-config.html) with requirements such as IAM Role, AccessControlTranslation, Metrics and SourceSelectionCriteria are not needed with MinIO.
- Active-Active replication
- Multi destination replication
## How to use?
Ensure that versioning is enabled on the source and target buckets with `mc version` command. If object locking is required, the buckets should have been created with `mc mb --with-lock`
The user setting up replication needs *s3:GetReplicationConfiguration* and *s3:GetBucketVersioning* permission on the source cluster. We do not recommend running root credentials/super admin with replication, instead create a dedicated user. The access credentials used at the destination requires *s3:ReplicateObject* permission.
The following minimal permission policy is needed by admin user setting up replication on the `source`:
```
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"admin:SetBucketTarget",
"admin:GetBucketTarget"
],
"Effect": "Allow",
"Sid": ""
},
{
"Effect": "Allow",
"Action": [
"s3:GetReplicationConfiguration",
"s3:PutReplicationConfiguration",
"s3:ListBucket",
"s3:ListBucketMultipartUploads",
"s3:GetBucketLocation",
"s3:GetBucketVersioning"
],
"Resource": [
"arn:aws:s3:::srcbucket"
]
}
]
}
```
The access key provided for the replication *target* cluster should have these minimal permissions:
```
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetReplicationConfiguration",
"s3:ListBucket",
"s3:ListBucketMultipartUploads",
"s3:GetBucketLocation",
"s3:GetBucketVersioning",
"s3:GetBucketObjectLockConfiguration"
],
"Resource": [
"arn:aws:s3:::destbucket"
]
},
{
"Effect": "Allow",
"Action": [
"s3:GetReplicationConfiguration",
"s3:ReplicateTags",
"s3:AbortMultipartUpload",
"s3:GetObject",
"s3:GetObjectVersion",
"s3:GetObjectVersionTagging",
"s3:PutObject",
"s3:DeleteObject",
"s3:ReplicateObject",
"s3:ReplicateDelete"
],
"Resource": [
"arn:aws:s3:::destbucket/*"
]
}
]
}
```
Please note that the permissions required by the admin user on the target cluster can be more fine grained to exclude permissions like "s3:ReplicateDelete", "s3:GetBucketObjectLockConfiguration" etc depending on whether delete replication rules are set up or if object locking is disabled on `destbucket`. The above policies assume that replication of objects, tags and delete marker replication are all enabled on object lock enabled buckets. A sample script to setup replication is provided [here](https://github.com/minio/minio/blob/master/docs/bucket/replication/setup_replication.sh)
To set up replication from a source bucket `srcbucket` on myminio cluster to a bucket `destbucket` on the target minio cluster with endpoint https://replica-endpoint:9000, use:
```
mc replicate add myminio/srcbucket --priority 1 --remote-bucket https://accessKey:secretKey@replica-endpoint:9000/destbucket
Replication configuration applied successfully to myminio/srcbucket.
```
Internally, this creates an ARN for the remote target associating the remote bucket as a replication target to the srcbucket on myminio.By default, if --replicate flag is not specified, replication of delete marker, permanent deletes, existing object replication and replica modification sync are all enabled. If you are using older mc versions, the ARN needs to be generated as a separate step before adding a replication rule.
> NOTE: If you are using a mc version below `RELEASE.2022-12-24T15-21-38Z`, the --remote-bucket flag needs an ARN generated by `mc admin bucket remote add` command. For mc versions RELEASE.2021-09-02T09-21-27Z and older, the remote target ARN needs to be passed in the --arn flag and actual remote bucket name in --remote-bucket flag of `mc replicate add`. For example, in older releases of mc replication configuration used to be added with:
```
mc admin bucket remote add myminio/srcbucket https://accessKey:secretKey@replica-endpoint:9000/destbucket --service replication --region us-east-1
Remote ARN = 'arn:minio:replication:us-east-1:c5be6b16-769d-432a-9ef1-4567081f3566:destbucket'
mc replicate add myminio/srcbucket/Tax --priority 1 --remote-bucket destbucket --remote-bucket "arn:minio:replication:us-east-1:c5be6b16-769d-432a-9ef1-4567081f3566:destbucket" --tags "Year=2019&Company=AcmeCorp" --storage-class "STANDARD" --replicate "delete,delete-marker"
Replication configuration applied successfully to myminio/srcbucket.
```
The replication configuration generated has the following format and can be exported with `mc replicate export` command:
```json
{
"Role" :"",
"Rules": [
{
"Status": "Enabled",
"Priority": 1,
"DeleteMarkerReplication": { "Status": "Disabled" },
"DeleteReplication": { "Status": "Disabled" },
"Filter" : {
"And": {
"Prefix": "Tax",
"Tags": [
{
"Key": "Year",
"Value": "2019"
},
{
"Key": "Company",
"Value": "AcmeCorp"
}
]
}
},
"Destination": {
"Bucket": "arn:minio:replication:us-east-1:c5be6b16-769d-432a-9ef1-4567081f3566:destbucket",
"StorageClass": "STANDARD"
},
"SourceSelectionCriteria": {
"ReplicaModifications": {
"Status": "Enabled"
}
}
}
]
}
```
The replication configuration follows [AWS S3 Spec](https://docs.aws.amazon.com/AmazonS3/latest/dev/replication-add-config.html). Any objects uploaded to the source bucket that meet replication criteria will now be automatically replicated by the MinIO server to the remote destination bucket. Replication can be disabled at any time by disabling specific rules in the configuration or deleting the replication configuration entirely.
When object locking is used in conjunction with replication, both source and destination buckets needs to have [object locking](https://docs.min.io/community/minio-object-store/administration/object-management/object-retention.html) enabled. Similarly objects encrypted on the server side, will be replicated if destination also supports encryption.
Replication status can be seen in the metadata on the source and destination objects. On the source side, the `X-Amz-Replication-Status` changes from `PENDING` to `COMPLETED` or `FAILED` after replication attempt either succeeded or failed respectively. On the destination side, a `X-Amz-Replication-Status` status of `REPLICA` indicates that the object was replicated successfully. Any replication failures are automatically re-attempted during a periodic disk scanner cycle.
To perform bi-directional replication, repeat the above process on the target site - this time setting the source bucket as the replication target. It is recommended that replication be run in a system with at least two CPU's available to the process, so that replication can run in its own thread.
![put](https://raw.githubusercontent.com/minio/minio/master/docs/bucket/replication/PUT_bucket_replication.png)
![head](https://raw.githubusercontent.com/minio/minio/master/docs/bucket/replication/HEAD_bucket_replication.png)
## Replica Modification sync
If bi-directional replication is set up between two clusters, any metadata update on the REPLICA object is by default reflected back in the source object when `ReplicaModifications` status in the `SourceSelectionCriteria` is `Enabled`. In MinIO, this is enabled by default. If a metadata update is performed on the "REPLICA" object, its `X-Amz-Replication-Status` will change from `PENDING` to `COMPLETE` or `FAILED`, and the source object version will show `X-Amz-Replication-Status` of `REPLICA` once the replication operation is complete.
The replication configuration in use on a bucket can be viewed using the `mc replicate export alias/bucket` command.
To disable replica metadata modification syncing, use `mc replicate edit` with the --replicate flag.
```
mc replicate edit alias/bucket --id xyz.id --replicate "delete,delete-marker"
```
To re-enable replica metadata modification syncing,
```
mc replicate edit alias/bucket --id xyz.id --replicate "delete,delete-marker,replica-metadata-sync"
```
## MinIO Extension
### Replicating Deletes
Delete marker replication is allowed in [AWS V1 Configuration](https://aws.amazon.com/blogs/storage/managing-delete-marker-replication-in-amazon-s3/) but not in V2 configuration. The MinIO implementation above is based on V2 configuration, however it has been extended to allow both DeleteMarker replication and replication of versioned deletes with the `DeleteMarkerReplication` and `DeleteReplication` fields in the replication configuration above. By default, this is set to `Disabled` unless the user specifies it while adding a replication rule.
When an object is deleted from the source bucket, the corresponding replica version will be marked deleted if delete marker replication is enabled in the replication configuration. Replication of deletes that specify a version id (a.k.a hard deletes) can be enabled by setting the `DeleteReplication` status to enabled in the replication configuration. This is a MinIO specific extension that can be enabled using the `mc replicate add` or `mc replicate edit` command with the --replicate "delete" flag.
Note that due to this extension behavior, AWS SDK's may not support the extension functionality pertaining to replicating versioned deletes.
Note that just like with [AWS](https://docs.aws.amazon.com/AmazonS3/latest/userguide/delete-marker-replication.html), Delete marker replication is disallowed in MinIO when the replication rule has tags.
To add a replication rule allowing both delete marker replication, versioned delete replication or both specify the --replicate flag with comma separated values as in the example below.
Additional permission of "s3:ReplicateDelete" action would need to be specified on the access key configured for the target cluster if Delete Marker replication or versioned delete replication is enabled.
```
mc replicate add myminio/srcbucket/Tax --priority 1 --remote-bucket `remote-target` --tags "Year=2019&Company=AcmeCorp" --storage-class "STANDARD" --replicate "delete,delete-marker"
Replication configuration applied successfully to myminio/srcbucket.
```
> NOTE: In mc versions `RELEASE.2022-12-24T15-21-38Z` and above `remote-target` should be of the format `https://accessKey:secretKey@replica-endpoint:9000/destbucket` which earlier used to be set during `mc admin bucket remote add`. For older releases, use the arn generated with `mc admin bucket remote add` command - e.g."arn:minio:replication:us-east-1:c5be6b16-769d-432a-9ef1-4567081f3566:destbucket" as the `remote-target`.
Also note that `mc` version `RELEASE.2021-09-02T09-21-27Z` or older supports only a single remote target per bucket. To take advantage of multiple destination replication, use the latest version of `mc`
Status of delete marker replication can be viewed by doing a GET/HEAD on the object version - it will return a `X-Minio-Replication-DeleteMarker-Status` header and http response code of `405`. In the case of permanent deletes, if the delete replication is pending or failed to propagate to the target cluster, GET/HEAD will return additional `X-Minio-Replication-Delete-Status` header and a http response code of `405`.
![delete](https://raw.githubusercontent.com/minio/minio/master/docs/bucket/replication/DELETE_bucket_replication.png)
The status of replication can be monitored by configuring event notifications on the source and target buckets using `mc event add`.On the source side, the `s3:PutObject`, `s3:Replication:OperationCompletedReplication` and `s3:Replication:OperationFailedReplication` events show the status of replication in the `X-Amz-Replication-Status` metadata.
On the target bucket, `s3:PutObject` event shows `X-Amz-Replication-Status` status of `REPLICA` in the metadata. Additional metrics to monitor backlog state for the purpose of bandwidth management and resource allocation are exposed via Prometheus - see <https://github.com/minio/minio/blob/master/docs/metrics/prometheus/list.md> for more details.
### Sync/Async Replication
By default, replication is completed asynchronously. If synchronous replication is desired, set the --sync flag while adding a
remote replication target using the `mc admin bucket remote add` command. For mc releases on or after `RELEASE.2022-12-24T15-21-38Z`, the
--sync, --health-check and --bandwidth flags can be specified in `mc replicate add|update` command
```
mc admin bucket remote add myminio/srcbucket https://accessKey:secretKey@replica-endpoint:9000/destbucket --service replication --region us-east-1 --sync --healthcheck-seconds 100
```
### Existing object replication
Existing object replication as detailed [here](https://aws.amazon.com/blogs/storage/replicating-existing-objects-between-s3-buckets/) can be enabled by passing `existing-objects` as a value to `--replicate` flag while adding or editing a replication rule.
Once existing object replication is enabled, all objects or object prefixes that satisfy the replication rules and were created prior to adding replication configuration OR while replication rules were disabled will be synced to the target cluster. Depending on the number of previously existing objects, the existing objects that are now eligible to be replicated will eventually be synced to the target cluster as the scanner schedules them. This may be slower depending on the load on the cluster, latency and size of the namespace.
In the rare event that target DR site is entirely lost and previously replicated objects to the DR cluster need to be re-replicated, `mc replicate resync start alias/bucket --remote-bucket <arn>` can be used to initiate a reset. This would initiate a re-sync between the two clusters by walking the bucket namespace and replicating eligible objects that satisfy the existing objects replication rule specified in the replication config. The status of the resync operation can be viewed with `mc replicate resync status alias/bucket --remote-bucket <arn>`.
Note that ExistingObjectReplication needs to be enabled in the config via `mc replicate [add|edit]` by passing `existing-objects` as one of the values to `--replicate` flag. Only those objects meeting replication rules and having existing object replication enabled will be re-synced.
### Multi destination replication
Replication from a source bucket to multiple destination buckets is supported. For each of the targets, repeat the steps to configure a remote target ARN and add replication rules to the source bucket's replication config.
Note that on the source side, the `X-Amz-Replication-Status` changes from `PENDING` to `COMPLETED` after replication succeeds to each of the targets. On the destination side, a `X-Amz-Replication-Status` status of `REPLICA` indicates that the object was replicated successfully. Any replication failures are automatically re-attempted during a periodic disk scanner cycle.
### Interaction with extended Bucket Versioning configuration
When Bucket Versioning with excluded prefixes are configured objects matching these prefixes are excluded from being versioned and replicated.
```
<VersioningConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Status>Enabled</Status>
<ExcludeFolders>true</ExcludeFolders>
<ExcludedPrefixes>
<Prefix>app1-jobs/*/_temporary/</Prefix>
</ExcludedPrefixes>
<ExcludedPrefixes>
<Prefix>app2-jobs/*/_magic/</Prefix>
</ExcludedPrefixes>
<!-- .. up to 10 prefixes in all -->
</VersioningConfiguration>
```
In the above sample config, objects under prefixes matching any of the `ExcludedPrefixes` glob patterns will neither be versioned nor replicated.
### SSE-C Encryption
MinIO does not support SSE-C encrypted objects on replicated buckets, any application uploading SSE-C encrypted objects will be rejected with an error on replicated buckets.
#### Rationale
- SSE-C requires application to remember the keys for all GET/PUT operations, any unfortunate loss of keys would automatically mean the objects cannot be accessed anymore.
- SSE-C is hardly adopted by most widely used applications, applications prefer server to manage the keys via SSE-KMS or SSE-S3.
- MinIO recommends applications to use SSE-KMS, SSE-S3 for simpler, safer and robust encryption mechanism for replicated buckets.
## Explore Further
- [MinIO Bucket Replication Design](https://github.com/minio/minio/blob/master/docs/bucket/replication/DESIGN.md)
- [MinIO Bucket Versioning Implementation](https://docs.min.io/community/minio-object-store/administration/object-management/object-retention.html)
- [MinIO Client Quickstart Guide](https://docs.min.io/community/minio-object-store/reference/minio-mc.html#quickstart)

150
docs/bucket/replication/delete-replication.sh Executable file
View File

@@ -0,0 +1,150 @@
#!/usr/bin/env bash
echo "Running $0"
if [ -n "$TEST_DEBUG" ]; then
set -x
fi
trap 'catch $LINENO' ERR
# shellcheck disable=SC2120
catch() {
if [ $# -ne 0 ]; then
echo "error on line $1"
echo "dc1 server logs ========="
cat /tmp/dc1.log
echo "dc2 server logs ========="
cat /tmp/dc2.log
fi
echo "Cleaning up instances of MinIO"
set +e
pkill minio
pkill mc
rm -rf /tmp/xl/
if [ $# -ne 0 ]; then
exit $#
fi
}
catch
set -e
export MINIO_CI_CD=1
export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="minio123"
export MINIO_KMS_AUTO_ENCRYPTION=off
export MINIO_PROMETHEUS_AUTH_TYPE=public
export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
unset MINIO_KMS_KES_CERT_FILE
unset MINIO_KMS_KES_KEY_FILE
unset MINIO_KMS_KES_ENDPOINT
unset MINIO_KMS_KES_KEY_NAME
if [ ! -f ./mc ]; then
wget --quiet -O mc https://dl.minio.io/client/mc/release/linux-amd64/mc &&
chmod +x mc
fi
mkdir -p /tmp/xl/1/ /tmp/xl/2/
export MINIO_KMS_SECRET_KEY="my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw="
export MINIO_ROOT_USER="minioadmin"
export MINIO_ROOT_PASSWORD="minioadmin"
./minio server --address ":9001" /tmp/xl/1/{1...4}/ 2>&1 >/tmp/dc1.log &
pid1=$!
./minio server --address ":9002" /tmp/xl/2/{1...4}/ 2>&1 >/tmp/dc2.log &
pid2=$!
sleep 3
export MC_HOST_myminio1=http://minioadmin:minioadmin@localhost:9001
export MC_HOST_myminio2=http://minioadmin:minioadmin@localhost:9002
./mc ready myminio1
./mc ready myminio2
./mc mb myminio1/testbucket/
./mc version enable myminio1/testbucket/
./mc mb myminio2/testbucket/
./mc version enable myminio2/testbucket/
./mc replicate add myminio1/testbucket --remote-bucket http://minioadmin:minioadmin@localhost:9002/testbucket/ --priority 1
# Test replication of delete markers and permanent deletes
./mc cp README.md myminio1/testbucket/dir/file
./mc cp README.md myminio1/testbucket/dir/file
sleep 1s
echo "=== myminio1"
./mc ls --versions myminio1/testbucket/dir/file
echo "=== myminio2"
./mc ls --versions myminio2/testbucket/dir/file
versionId="$(./mc ls --json --versions myminio1/testbucket/dir/ | tail -n1 | jq -r .versionId)"
export AWS_ACCESS_KEY_ID=minioadmin
export AWS_SECRET_ACCESS_KEY=minioadmin
export AWS_REGION=us-east-1
aws s3api --endpoint-url http://localhost:9001 delete-object --bucket testbucket --key dir/file --version-id "$versionId"
./mc ls -r --versions myminio1/testbucket >/tmp/myminio1.txt
./mc ls -r --versions myminio2/testbucket >/tmp/myminio2.txt
out=$(diff -qpruN /tmp/myminio1.txt /tmp/myminio2.txt)
ret=$?
if [ $ret -ne 0 ]; then
echo "BUG: expected no missing entries after replication: $out"
exit 1
fi
./mc rm myminio1/testbucket/dir/file
sleep 1s
./mc ls -r --versions myminio1/testbucket >/tmp/myminio1.txt
./mc ls -r --versions myminio2/testbucket >/tmp/myminio2.txt
out=$(diff -qpruN /tmp/myminio1.txt /tmp/myminio2.txt)
ret=$?
if [ $ret -ne 0 ]; then
echo "BUG: expected no missing entries after replication: $out"
exit 1
fi
# Test listing of non replicated permanent deletes
set -x
./mc mb myminio1/foobucket/ myminio2/foobucket/ --with-versioning
./mc replicate add myminio1/foobucket --remote-bucket http://minioadmin:minioadmin@localhost:9002/foobucket/ --priority 1
./mc cp README.md myminio1/foobucket/dir/file
versionId="$(./mc ls --json --versions myminio1/foobucket/dir/ | jq -r .versionId)"
kill ${pid2} && wait ${pid2} || true
aws s3api --endpoint-url http://localhost:9001 delete-object --bucket foobucket --key dir/file --version-id "$versionId"
out="$(./mc ls myminio1/foobucket/dir/)"
if [ "$out" != "" ]; then
echo "BUG: non versioned listing should not show pending/failed replicated delete:"
echo "$out"
exit 1
fi
out="$(./mc ls --versions myminio1/foobucket/dir/)"
if [ "$out" != "" ]; then
echo "BUG: versioned listing should not show pending/failed replicated deletes:"
echo "$out"
exit 1
fi
echo "Success"
catch

205
docs/bucket/replication/setup_2site_existing_replication.sh Executable file
View File

@@ -0,0 +1,205 @@
#!/usr/bin/env bash
echo "Running $0"
set -x
trap 'catch $LINENO' ERR
# shellcheck disable=SC2120
catch() {
if [ $# -ne 0 ]; then
echo "error on line $1"
for site in sitea siteb; do
echo "$site server logs ========="
cat "/tmp/${site}_1.log"
echo "==========================="
cat "/tmp/${site}_2.log"
done
fi
echo "Cleaning up instances of MinIO"
pkill minio
pkill -9 minio
rm -rf /tmp/multisitea
rm -rf /tmp/multisiteb
rm -rf /tmp/data
if [ $# -ne 0 ]; then
exit $#
fi
}
catch
set -e
export MINIO_CI_CD=1
export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="minio123"
export MINIO_KMS_AUTO_ENCRYPTION=off
export MINIO_PROMETHEUS_AUTH_TYPE=public
export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
unset MINIO_KMS_KES_CERT_FILE
unset MINIO_KMS_KES_KEY_FILE
unset MINIO_KMS_KES_ENDPOINT
unset MINIO_KMS_KES_KEY_NAME
if [ ! -f ./mc ]; then
wget --quiet -O mc https://dl.minio.io/client/mc/release/linux-amd64/mc &&
chmod +x mc
fi
minio server --address 127.0.0.1:9001 "http://127.0.0.1:9001/tmp/multisitea/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9002/tmp/multisitea/data/disterasure/xl{5...8}" >/tmp/sitea_1.log 2>&1 &
minio server --address 127.0.0.1:9002 "http://127.0.0.1:9001/tmp/multisitea/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9002/tmp/multisitea/data/disterasure/xl{5...8}" >/tmp/sitea_2.log 2>&1 &
minio server --address 127.0.0.1:9003 "http://127.0.0.1:9003/tmp/multisiteb/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9004/tmp/multisiteb/data/disterasure/xl{5...8}" >/tmp/siteb_1.log 2>&1 &
minio server --address 127.0.0.1:9004 "http://127.0.0.1:9003/tmp/multisiteb/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9004/tmp/multisiteb/data/disterasure/xl{5...8}" >/tmp/siteb_2.log 2>&1 &
export MC_HOST_sitea=http://minio:minio123@127.0.0.1:9001
export MC_HOST_siteb=http://minio:minio123@127.0.0.1:9004
./mc ready sitea
./mc ready siteb
./mc mb sitea/bucket
## Create 100 files
mkdir -p /tmp/data
for i in $(seq 1 10); do
echo "T" >/tmp/data/file_${i}.txt
done
./mc mirror /tmp/data sitea/bucket/
./mc version enable sitea/bucket
./mc cp /tmp/data/file_1.txt sitea/bucket/marker
./mc rm sitea/bucket/marker
./mc mb siteb/bucket/
./mc version enable siteb/bucket/
echo "adding replication rule for site a -> site b"
./mc replicate add sitea/bucket/ \
--remote-bucket http://minio:minio123@127.0.0.1:9004/bucket
remote_arn=$(./mc replicate ls sitea/bucket --json | jq -r .rule.Destination.Bucket)
sleep 1
./mc replicate resync start sitea/bucket/ --remote-bucket "${remote_arn}"
sleep 30s ## sleep for 30s idea is that we give 300ms per object.
./mc ls -r --versions sitea/bucket >/tmp/sitea.txt
./mc ls -r --versions siteb/bucket >/tmp/siteb.txt
out=$(diff -qpruN /tmp/sitea.txt /tmp/siteb.txt)
ret=$?
if [ $ret -ne 0 ]; then
echo "BUG: expected no missing entries after replication: $out"
exit 1
fi
./mc cp /tmp/data/file_1.txt sitea/bucket/marker_new
./mc rm sitea/bucket/marker_new
sleep 12s ## sleep for 12s idea is that we give 100ms per object.
./mc ls -r --versions sitea/bucket >/tmp/sitea.txt
./mc ls -r --versions siteb/bucket >/tmp/siteb.txt
out=$(diff -qpruN /tmp/sitea.txt /tmp/siteb.txt)
ret=$?
if [ $ret -ne 0 ]; then
echo "BUG: expected no 'diff' after replication: $out"
exit 1
fi
./mc rm -r --force --versions sitea/bucket/marker
sleep 14s ## sleep for 14s idea is that we give 100ms per object.
./mc ls -r --versions sitea/bucket >/tmp/sitea.txt
./mc ls -r --versions siteb/bucket >/tmp/siteb.txt
out=$(diff -qpruN /tmp/sitea.txt /tmp/siteb.txt)
ret=$?
if [ $ret -ne 0 ]; then
echo "BUG: expected no 'diff' after replication: $out"
exit 1
fi
./mc mb sitea/bucket-version/
./mc mb siteb/bucket-version
./mc version enable sitea/bucket-version/
./mc version enable siteb/bucket-version/
echo "adding replication rule for site a -> site b"
./mc replicate add sitea/bucket-version/ \
--remote-bucket http://minio:minio123@127.0.0.1:9004/bucket-version
./mc mb sitea/bucket-version/directory/
sleep 2s
./mc ls -r --versions sitea/bucket-version/ >/tmp/sitea_dirs.txt
./mc ls -r --versions siteb/bucket-version/ >/tmp/siteb_dirs.txt
out=$(diff -qpruN /tmp/sitea_dirs.txt /tmp/siteb_dirs.txt)
ret=$?
if [ $ret -ne 0 ]; then
echo "BUG: expected no 'diff' after replication: $out"
exit 1
fi
./mc rm -r --versions --force sitea/bucket-version/
sleep 2s
./mc ls -r --versions sitea/bucket-version/ >/tmp/sitea_dirs.txt
./mc ls -r --versions siteb/bucket-version/ >/tmp/siteb_dirs.txt
out=$(diff -qpruN /tmp/sitea_dirs.txt /tmp/siteb_dirs.txt)
ret=$?
if [ $ret -ne 0 ]; then
echo "BUG: expected no 'diff' after replication: $out"
exit 1
fi
## check if we don't create delete markers on the directory objects, its always permanent delete.
./mc mb sitea/bucket-version/directory/
sleep 2s
./mc rm -r --force sitea/bucket-version/
sleep 2s
./mc ls -r --versions sitea/bucket-version/ >/tmp/sitea_dirs.txt
./mc ls -r --versions siteb/bucket-version/ >/tmp/siteb_dirs.txt
out=$(diff -qpruN /tmp/sitea_dirs.txt /tmp/siteb_dirs.txt)
ret=$?
if [ $ret -ne 0 ]; then
echo "BUG: expected no 'diff' after replication: $out"
exit 1
fi
sitea_count=$(cat /tmp/sitea_dirs.txt | wc -l) # need to do it this way to avoid filename in the output
siteb_count=$(cat /tmp/siteb_dirs.txt | wc -l) # need to do it this way to avoid filename in the output
sitea_out=$(cat /tmp/sitea_dirs.txt)
siteb_out=$(cat /tmp/siteb_dirs.txt)
if [ $sitea_count -ne 0 ]; then
echo "BUG: expected no 'directory objects' left after deletion: ${sitea_out}"
exit 1
fi
if [ $siteb_count -ne 0 ]; then
echo "BUG: expected no 'directory objects' left after deletion: ${siteb_out}"
exit 1
fi
catch

244
docs/bucket/replication/setup_3site_replication.sh Executable file
View File

@@ -0,0 +1,244 @@
#!/usr/bin/env bash
echo "Running $0"
if [ -n "$TEST_DEBUG" ]; then
set -x
fi
trap 'catch $LINENO' ERR
# shellcheck disable=SC2120
catch() {
if [ $# -ne 0 ]; then
echo "error on line $1"
for site in sitea siteb sitec; do
echo "$site server logs ========="
cat "/tmp/${site}_1.log"
echo "==========================="
cat "/tmp/${site}_2.log"
done
fi
echo "Cleaning up instances of MinIO"
pkill minio
pkill -9 minio
rm -rf /tmp/multisitea
rm -rf /tmp/multisiteb
rm -rf /tmp/multisitec
if [ $# -ne 0 ]; then
exit $#
fi
}
catch
set -e
export MINIO_CI_CD=1
export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="minio123"
export MINIO_KMS_AUTO_ENCRYPTION=off
export MINIO_PROMETHEUS_AUTH_TYPE=public
export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
unset MINIO_KMS_KES_CERT_FILE
unset MINIO_KMS_KES_KEY_FILE
unset MINIO_KMS_KES_ENDPOINT
unset MINIO_KMS_KES_KEY_NAME
go install -v github.com/minio/mc@master
cp -a $(go env GOPATH)/bin/mc ./mc
if [ ! -f mc.RELEASE.2021-03-12T03-36-59Z ]; then
wget -q -O mc.RELEASE.2021-03-12T03-36-59Z https://dl.minio.io/client/mc/release/linux-amd64/archive/mc.RELEASE.2021-03-12T03-36-59Z &&
chmod +x mc.RELEASE.2021-03-12T03-36-59Z
fi
minio server --address 127.0.0.1:9001 "http://127.0.0.1:9001/tmp/multisitea/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9002/tmp/multisitea/data/disterasure/xl{5...8}" >/tmp/sitea_1.log 2>&1 &
minio server --address 127.0.0.1:9002 "http://127.0.0.1:9001/tmp/multisitea/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9002/tmp/multisitea/data/disterasure/xl{5...8}" >/tmp/sitea_2.log 2>&1 &
minio server --address 127.0.0.1:9003 "http://127.0.0.1:9003/tmp/multisiteb/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9004/tmp/multisiteb/data/disterasure/xl{5...8}" >/tmp/siteb_1.log 2>&1 &
minio server --address 127.0.0.1:9004 "http://127.0.0.1:9003/tmp/multisiteb/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9004/tmp/multisiteb/data/disterasure/xl{5...8}" >/tmp/siteb_2.log 2>&1 &
minio server --address 127.0.0.1:9005 "http://127.0.0.1:9005/tmp/multisitec/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9006/tmp/multisitec/data/disterasure/xl{5...8}" >/tmp/sitec_1.log 2>&1 &
minio server --address 127.0.0.1:9006 "http://127.0.0.1:9005/tmp/multisitec/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9006/tmp/multisitec/data/disterasure/xl{5...8}" >/tmp/sitec_2.log 2>&1 &
export MC_HOST_sitea=http://minio:minio123@127.0.0.1:9001
export MC_HOST_siteb=http://minio:minio123@127.0.0.1:9004
export MC_HOST_sitec=http://minio:minio123@127.0.0.1:9006
./mc ready sitea
./mc ready siteb
./mc ready sitec
./mc mb sitea/bucket
./mc version enable sitea/bucket
./mc mb -l sitea/olockbucket
./mc mb siteb/bucket/
./mc version enable siteb/bucket/
./mc mb -l siteb/olockbucket/
./mc mb sitec/bucket/
./mc version enable sitec/bucket/
./mc mb -l sitec/olockbucket
echo "adding replication rule for a -> b : ${remote_arn}"
sleep 1
./mc replicate add sitea/bucket/ \
--remote-bucket http://minio:minio123@127.0.0.1:9004/bucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync"
sleep 1
echo "adding replication rule for b -> a : ${remote_arn}"
./mc replicate add siteb/bucket/ \
--remote-bucket http://minio:minio123@127.0.0.1:9001/bucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync"
sleep 1
echo "adding replication rule for a -> c : ${remote_arn}"
./mc replicate add sitea/bucket/ \
--remote-bucket http://minio:minio123@127.0.0.1:9006/bucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 2
sleep 1
echo "adding replication rule for c -> a : ${remote_arn}"
./mc replicate add sitec/bucket/ \
--remote-bucket http://minio:minio123@127.0.0.1:9001/bucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 2
sleep 1
echo "adding replication rule for b -> c : ${remote_arn}"
./mc replicate add siteb/bucket/ \
--remote-bucket http://minio:minio123@127.0.0.1:9006/bucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 3
sleep 1
echo "adding replication rule for c -> b : ${remote_arn}"
./mc replicate add sitec/bucket/ \
--remote-bucket http://minio:minio123@127.0.0.1:9004/bucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 3
sleep 1
echo "adding replication rule for olockbucket a -> b : ${remote_arn}"
./mc replicate add sitea/olockbucket/ \
--remote-bucket http://minio:minio123@127.0.0.1:9004/olockbucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync"
sleep 1
echo "adding replication rule for olockbucket b -> a : ${remote_arn}"
./mc replicate add siteb/olockbucket/ \
--remote-bucket http://minio:minio123@127.0.0.1:9001/olockbucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync"
sleep 1
echo "adding replication rule for olockbucket a -> c : ${remote_arn}"
./mc replicate add sitea/olockbucket/ \
--remote-bucket http://minio:minio123@127.0.0.1:9006/olockbucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 2
sleep 1
echo "adding replication rule for olockbucket c -> a : ${remote_arn}"
./mc replicate add sitec/olockbucket/ \
--remote-bucket http://minio:minio123@127.0.0.1:9001/olockbucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 2
sleep 1
echo "adding replication rule for olockbucket b -> c : ${remote_arn}"
./mc replicate add siteb/olockbucket/ \
--remote-bucket http://minio:minio123@127.0.0.1:9006/olockbucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 3
sleep 1
echo "adding replication rule for olockbucket c -> b : ${remote_arn}"
./mc replicate add sitec/olockbucket/ \
--remote-bucket http://minio:minio123@127.0.0.1:9004/olockbucket \
--replicate "existing-objects,delete,delete-marker,replica-metadata-sync" --priority 3
sleep 1
echo "Set default governance retention 30d"
./mc retention set --default governance 30d sitea/olockbucket
echo "Copying data to source sitea/bucket"
./mc cp --enc-s3 "sitea/" --quiet /etc/hosts sitea/bucket
sleep 1
echo "Copying data to source sitea/olockbucket"
./mc cp --quiet /etc/hosts sitea/olockbucket
sleep 1
echo "Verifying the metadata difference between source and target"
if diff -pruN <(./mc stat --no-list --json sitea/bucket/hosts | jq .) <(./mc stat --no-list --json siteb/bucket/hosts | jq .) | grep -q 'COMPLETED\|REPLICA'; then
echo "verified sitea-> COMPLETED, siteb-> REPLICA"
fi
if diff -pruN <(./mc stat --no-list --json sitea/bucket/hosts | jq .) <(./mc stat --no-list --json sitec/bucket/hosts | jq .) | grep -q 'COMPLETED\|REPLICA'; then
echo "verified sitea-> COMPLETED, sitec-> REPLICA"
fi
echo "Verifying the metadata difference between source and target"
if diff -pruN <(./mc stat --no-list --json sitea/olockbucket/hosts | jq .) <(./mc stat --no-list --json siteb/olockbucket/hosts | jq .) | grep -q 'COMPLETED\|REPLICA'; then
echo "verified sitea-> COMPLETED, siteb-> REPLICA"
fi
if diff -pruN <(./mc stat --no-list --json sitea/olockbucket/hosts | jq .) <(./mc stat --no-list --json sitec/olockbucket/hosts | jq .) | grep -q 'COMPLETED\|REPLICA'; then
echo "verified sitea-> COMPLETED, sitec-> REPLICA"
fi
sleep 5
head -c 221227088 </dev/urandom >200M
./mc.RELEASE.2021-03-12T03-36-59Z cp --config-dir ~/.mc --encrypt "sitea" --quiet 200M "sitea/bucket/200M-enc-v1"
./mc.RELEASE.2021-03-12T03-36-59Z cp --config-dir ~/.mc --quiet 200M "sitea/bucket/200M-v1"
./mc cp --enc-s3 "sitea" --quiet 200M "sitea/bucket/200M-enc-v2"
./mc cp --quiet 200M "sitea/bucket/200M-v2"
sleep 10
echo "Verifying ETag for all objects"
./s3-check-md5 -versions -access-key minio -secret-key minio123 -endpoint http://127.0.0.1:9001/ -bucket bucket
./s3-check-md5 -versions -access-key minio -secret-key minio123 -endpoint http://127.0.0.1:9002/ -bucket bucket
./s3-check-md5 -versions -access-key minio -secret-key minio123 -endpoint http://127.0.0.1:9003/ -bucket bucket
./s3-check-md5 -versions -access-key minio -secret-key minio123 -endpoint http://127.0.0.1:9004/ -bucket bucket
./s3-check-md5 -versions -access-key minio -secret-key minio123 -endpoint http://127.0.0.1:9005/ -bucket bucket
./s3-check-md5 -versions -access-key minio -secret-key minio123 -endpoint http://127.0.0.1:9006/ -bucket bucket
./s3-check-md5 -versions -access-key minio -secret-key minio123 -endpoint http://127.0.0.1:9001/ -bucket olockbucket
./s3-check-md5 -versions -access-key minio -secret-key minio123 -endpoint http://127.0.0.1:9002/ -bucket olockbucket
./s3-check-md5 -versions -access-key minio -secret-key minio123 -endpoint http://127.0.0.1:9003/ -bucket olockbucket
./s3-check-md5 -versions -access-key minio -secret-key minio123 -endpoint http://127.0.0.1:9004/ -bucket olockbucket
./s3-check-md5 -versions -access-key minio -secret-key minio123 -endpoint http://127.0.0.1:9005/ -bucket olockbucket
./s3-check-md5 -versions -access-key minio -secret-key minio123 -endpoint http://127.0.0.1:9006/ -bucket olockbucket
# additional tests for encryption object alignment
go install -v github.com/minio/multipart-debug@latest
upload_id=$(multipart-debug --endpoint 127.0.0.1:9001 --accesskey minio --secretkey minio123 multipart new --bucket bucket --object new-test-encrypted-object --encrypt)
dd if=/dev/urandom bs=1 count=7048531 of=/tmp/7048531.txt
dd if=/dev/urandom bs=1 count=2847391 of=/tmp/2847391.txt
sudo apt install jq -y
etag_1=$(multipart-debug --endpoint 127.0.0.1:9002 --accesskey minio --secretkey minio123 multipart upload --bucket bucket --object new-test-encrypted-object --uploadid ${upload_id} --file /tmp/7048531.txt --number 1 | jq -r .ETag)
etag_2=$(multipart-debug --endpoint 127.0.0.1:9001 --accesskey minio --secretkey minio123 multipart upload --bucket bucket --object new-test-encrypted-object --uploadid ${upload_id} --file /tmp/2847391.txt --number 2 | jq -r .ETag)
multipart-debug --endpoint 127.0.0.1:9002 --accesskey minio --secretkey minio123 multipart complete --bucket bucket --object new-test-encrypted-object --uploadid ${upload_id} 1.${etag_1} 2.${etag_2}
sleep 10
./mc stat --no-list sitea/bucket/new-test-encrypted-object
./mc stat --no-list siteb/bucket/new-test-encrypted-object
./mc stat --no-list sitec/bucket/new-test-encrypted-object
./mc ls -r sitea/bucket/
./mc ls -r siteb/bucket/
./mc ls -r sitec/bucket/
catch

314
docs/bucket/replication/setup_ilm_expiry_replication.sh Executable file
View File

@@ -0,0 +1,314 @@
#!/usr/bin/env bash
set -x
trap 'catch $LINENO' ERR
# shellcheck disable=SC2120
catch() {
if [ $# -ne 0 ]; then
echo "error on line $1"
for site in sitea siteb sitec sited; do
echo "$site server logs ========="
cat "/tmp/${site}_1.log"
echo "==========================="
cat "/tmp/${site}_2.log"
done
fi
echo "Cleaning up instances of MinIO"
pkill minio
pkill -9 minio
rm -rf /tmp/multisitea
rm -rf /tmp/multisiteb
rm -rf /tmp/multisitec
rm -rf /tmp/multisited
rm -rf /tmp/data
if [ $# -ne 0 ]; then
exit $#
fi
}
catch
set -e
export MINIO_CI_CD=1
export MINIO_BROWSER=off
export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="minio123"
export MINIO_KMS_AUTO_ENCRYPTION=off
export MINIO_PROMETHEUS_AUTH_TYPE=public
export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
unset MINIO_KMS_KES_CERT_FILE
unset MINIO_KMS_KES_KEY_FILE
unset MINIO_KMS_KES_ENDPOINT
unset MINIO_KMS_KES_KEY_NAME
if [ ! -f ./mc ]; then
wget --quiet -O mc https://dl.minio.io/client/mc/release/linux-amd64/mc &&
chmod +x mc
fi
minio server --address 127.0.0.1:9001 "http://127.0.0.1:9001/tmp/multisitea/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9002/tmp/multisitea/data/disterasure/xl{5...8}" >/tmp/sitea_1.log 2>&1 &
minio server --address 127.0.0.1:9002 "http://127.0.0.1:9001/tmp/multisitea/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9002/tmp/multisitea/data/disterasure/xl{5...8}" >/tmp/sitea_2.log 2>&1 &
minio server --address 127.0.0.1:9003 "http://127.0.0.1:9003/tmp/multisiteb/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9004/tmp/multisiteb/data/disterasure/xl{5...8}" >/tmp/siteb_1.log 2>&1 &
minio server --address 127.0.0.1:9004 "http://127.0.0.1:9003/tmp/multisiteb/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9004/tmp/multisiteb/data/disterasure/xl{5...8}" >/tmp/siteb_2.log 2>&1 &
minio server --address 127.0.0.1:9005 "http://127.0.0.1:9005/tmp/multisitec/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9006/tmp/multisitec/data/disterasure/xl{5...8}" >/tmp/sitec_1.log 2>&1 &
minio server --address 127.0.0.1:9006 "http://127.0.0.1:9005/tmp/multisitec/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9006/tmp/multisitec/data/disterasure/xl{5...8}" >/tmp/sitec_2.log 2>&1 &
minio server --address 127.0.0.1:9007 "http://127.0.0.1:9007/tmp/multisited/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9008/tmp/multisited/data/disterasure/xl{5...8}" >/tmp/sited_1.log 2>&1 &
minio server --address 127.0.0.1:9008 "http://127.0.0.1:9007/tmp/multisited/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9008/tmp/multisited/data/disterasure/xl{5...8}" >/tmp/sited_2.log 2>&1 &
# Wait to make sure all MinIO instances are up
export MC_HOST_sitea=http://minio:minio123@127.0.0.1:9001
export MC_HOST_siteb=http://minio:minio123@127.0.0.1:9004
export MC_HOST_sitec=http://minio:minio123@127.0.0.1:9006
export MC_HOST_sited=http://minio:minio123@127.0.0.1:9008
./mc ready sitea
./mc ready siteb
./mc ready sitec
./mc ready sited
./mc mb sitea/bucket
./mc mb sitec/bucket
## Setup site replication
./mc admin replicate add sitea siteb --replicate-ilm-expiry
sleep 10s
## Add warm tier
./mc ilm tier add minio sitea WARM-TIER --endpoint http://localhost:9006 --access-key minio --secret-key minio123 --bucket bucket
## Add ILM rules
./mc ilm add sitea/bucket --transition-days 0 --transition-tier WARM-TIER --transition-days 0 --noncurrent-expire-days 2 --expire-days 3 --prefix "myprefix" --tags "tag1=val1&tag2=val2"
./mc ilm rule list sitea/bucket
## Check ilm expiry flag
./mc admin replicate info sitea --json
flag1=$(./mc admin replicate info sitea --json | jq '.sites[0]."replicate-ilm-expiry"')
flag2=$(./mc admin replicate info sitea --json | jq '.sites[1]."replicate-ilm-expiry"')
if [ "$flag1" != "true" ]; then
echo "BUG: Expected ILM expiry replication not set for 'sitea'"
exit 1
fi
if [ "$flag2" != "true" ]; then
echo "BUG: Expected ILM expiry replication not set for 'siteb'"
exit 1
fi
## Check if ILM expiry rules replicated
sleep 30s
./mc ilm rule list siteb/bucket
count=$(./mc ilm rule list siteb/bucket --json | jq '.config.Rules | length')
if [ $count -ne 1 ]; then
echo "BUG: ILM expiry rules not replicated to 'siteb'"
exit 1
fi
## Check replication of rules content
expDays=$(./mc ilm rule list siteb/bucket --json | jq '.config.Rules[0].Expiration.Days')
noncurrentDays=$(./mc ilm rule list siteb/bucket --json | jq '.config.Rules[0].NoncurrentVersionExpiration.NoncurrentDays')
if [ $expDays -ne 3 ]; then
echo "BUG: Incorrect expiry days '${expDays}' set for 'siteb'"
exit 1
fi
if [ $noncurrentDays -ne 2 ]; then
echo "BUG: Incorrect non current expiry days '${noncurrentDays}' set for siteb"
exit 1
fi
## Make sure transition rule not replicated to siteb
tranDays=$(./mc ilm rule list siteb/bucket --json | jq '.config.Rules[0].Transition.Days')
if [ "${tranDays}" != "null" ]; then
echo "BUG: Transition rules as well copied to siteb"
exit 1
fi
## Check replication of rules prefix and tags
prefix=$(./mc ilm rule list siteb/bucket --json | jq '.config.Rules[0].Filter.And.Prefix' | sed 's/"//g')
tagName1=$(./mc ilm rule list siteb/bucket --json | jq '.config.Rules[0].Filter.And.Tags[0].Key' | sed 's/"//g')
tagVal1=$(./mc ilm rule list siteb/bucket --json | jq '.config.Rules[0].Filter.And.Tags[0].Value' | sed 's/"//g')
tagName2=$(./mc ilm rule list siteb/bucket --json | jq '.config.Rules[0].Filter.And.Tags[1].Key' | sed 's/"//g')
tagVal2=$(./mc ilm rule list siteb/bucket --json | jq '.config.Rules[0].Filter.And.Tags[1].Value' | sed 's/"//g')
if [ "${prefix}" != "myprefix" ]; then
echo "BUG: ILM expiry rules prefix not replicated to 'siteb'"
exit 1
fi
if [ "${tagName1}" != "tag1" ] || [ "${tagVal1}" != "val1" ] || [ "${tagName2}" != "tag2" ] || [ "${tagVal2}" != "val2" ]; then
echo "BUG: ILM expiry rules tags not replicated to 'siteb'"
exit 1
fi
## Check edit of ILM expiry rule and its replication
id=$(./mc ilm rule list sitea/bucket --json | jq '.config.Rules[] | select(.Expiration.Days==3) | .ID' | sed 's/"//g')
./mc ilm edit --id "${id}" --expire-days "100" sitea/bucket
sleep 30s
count1=$(./mc ilm rule list sitea/bucket --json | jq '.config.Rules[0].Expiration.Days')
count2=$(./mc ilm rule list siteb/bucket --json | jq '.config.Rules[0].Expiration.Days')
if [ $count1 -ne 100 ]; then
echo "BUG: Expiration days not changed on 'sitea'"
exit 1
fi
if [ $count2 -ne 100 ]; then
echo "BUG: Modified ILM expiry rule not replicated to 'siteb'"
exit 1
fi
## Check disabling of ILM expiry rules replication
./mc admin replicate update sitea --disable-ilm-expiry-replication
flag=$(./mc admin replicate info sitea --json | jq '.sites[] | select (.name=="sitea") | ."replicate-ilm-expiry"')
if [ "$flag" != "false" ]; then
echo "BUG: ILM expiry replication not disabled for 'sitea'"
exit 1
fi
flag=$(./mc admin replicate info siteb --json | jq '.sites[] | select (.name=="sitea") | ."replicate-ilm-expiry"')
if [ "$flag" != "false" ]; then
echo "BUG: ILM expiry replication not disabled for 'siteb'"
exit 1
fi
## Perform individual updates of rules to sites
./mc ilm edit --id "${id}" --expire-days "999" sitea/bucket
sleep 5s
./mc ilm edit --id "${id}" --expire-days "888" siteb/bucket # when ilm expiry re-enabled, this should win
## Check re-enabling of ILM expiry rules replication
./mc admin replicate update sitea --enable-ilm-expiry-replication
flag=$(./mc admin replicate info sitea --json | jq '.sites[] | select (.name=="sitea") | ."replicate-ilm-expiry"')
if [ "$flag" != "true" ]; then
echo "BUG: ILM expiry replication not enabled for 'sitea'"
exit 1
fi
flag=$(./mc admin replicate info siteb --json | jq '.sites[] | select (.name=="sitea") | ."replicate-ilm-expiry"')
if [ "$flag" != "true" ]; then
echo "BUG: ILM expiry replication not enabled for 'siteb'"
exit 1
fi
## Check if latest updated rules get replicated to all sites post re-enable of ILM expiry rules replication
sleep 30s
count1=$(./mc ilm rule list sitea/bucket --json | jq '.config.Rules[0].Expiration.Days')
count2=$(./mc ilm rule list siteb/bucket --json | jq '.config.Rules[0].Expiration.Days')
if [ $count1 -ne 888 ]; then
echo "BUG: Latest expiration days not updated on 'sitea'"
exit 1
fi
if [ $count2 -ne 888 ]; then
echo "BUG: Latest expiration days not updated on 'siteb'"
exit 1
fi
## Check to make sure sitea transition rule is not overwritten
transDays=$(./mc ilm rule list sitea/bucket --json | jq '.config.Rules[0].Transition.Days')
if [ $transDays -ne 0 ] || [ "${transDays}" == "null" ]; then
echo "BUG: Transition rule on sitea seems to be overwritten"
exit 1
fi
## Check replication of edit of prefix, tags and status of ILM Expiry Rules
./mc ilm rule edit --id "${id}" --prefix "newprefix" --tags "ntag1=nval1&ntag2=nval2" --disable sitea/bucket
sleep 30s
nprefix=$(./mc ilm rule list siteb/bucket --json | jq '.config.Rules[0].Filter.And.Prefix' | sed 's/"//g')
ntagName1=$(./mc ilm rule list siteb/bucket --json | jq '.config.Rules[0].Filter.And.Tags[0].Key' | sed 's/"//g')
ntagVal1=$(./mc ilm rule list siteb/bucket --json | jq '.config.Rules[0].Filter.And.Tags[0].Value' | sed 's/"//g')
ntagName2=$(./mc ilm rule list siteb/bucket --json | jq '.config.Rules[0].Filter.And.Tags[1].Key' | sed 's/"//g')
ntagVal2=$(./mc ilm rule list siteb/bucket --json | jq '.config.Rules[0].Filter.And.Tags[1].Value' | sed 's/"//g')
st=$(./mc ilm rule list siteb/bucket --json | jq '.config.Rules[0].Status' | sed 's/"//g')
if [ "${nprefix}" != "newprefix" ]; then
echo "BUG: ILM expiry rules prefix not replicated to 'siteb'"
exit 1
fi
if [ "${ntagName1}" != "ntag1" ] || [ "${ntagVal1}" != "nval1" ] || [ "${ntagName2}" != "ntag2" ] || [ "${ntagVal2}" != "nval2" ]; then
echo "BUG: ILM expiry rules tags not replicated to 'siteb'"
exit 1
fi
if [ "${st}" != "Disabled" ]; then
echo "BUG: ILM expiry rules status not replicated to 'siteb'"
exit 1
fi
## Check replication of deleted ILM expiry rules
./mc ilm rule remove --id "${id}" sitea/bucket
sleep 30s
# should error as rule doesn't exist
error=$(./mc ilm rule list siteb/bucket --json | jq '.error.cause.message' | sed 's/"//g')
if [ "$error" != "The lifecycle configuration does not exist" ]; then
echo "BUG: Removed ILM expiry rule not replicated to 'siteb'"
exit 1
fi
## Check addition of new replication site to existing site replication setup
# Add rules again as previous tests removed all
./mc ilm add sitea/bucket --transition-days 0 --transition-tier WARM-TIER --transition-days 0 --noncurrent-expire-days 2 --expire-days 3 --prefix "myprefix" --tags "tag1=val1&tag2=val2"
./mc admin replicate add sitea siteb sited
sleep 30s
# Check site replication info and status for new site
sitesCount=$(mc admin replicate info sited --json | jq '.sites | length')
if [ ${sitesCount} -ne 3 ]; then
echo "BUG: New site 'sited' not appearing in site replication info"
exit 1
fi
flag3=$(./mc admin replicate info sited --json | jq '.sites[2]."replicate-ilm-expiry"')
if [ "${flag3}" != "true" ]; then
echo "BUG: ILM expiry replication not enabled for 'sited'"
exit 1
fi
rulesCount=$(./mc ilm rule list sited/bucket --json | jq '.config.Rules | length')
if [ ${rulesCount} -ne 1 ]; then
echo "BUG: ILM expiry rules not replicated to 'sited'"
exit 1
fi
prefix=$(./mc ilm rule list sited/bucket --json | jq '.config.Rules[0].Filter.And.Prefix' | sed 's/"//g')
tagName1=$(./mc ilm rule list sited/bucket --json | jq '.config.Rules[0].Filter.And.Tags[0].Key' | sed 's/"//g')
tagVal1=$(./mc ilm rule list sited/bucket --json | jq '.config.Rules[0].Filter.And.Tags[0].Value' | sed 's/"//g')
tagName2=$(./mc ilm rule list sited/bucket --json | jq '.config.Rules[0].Filter.And.Tags[1].Key' | sed 's/"//g')
tagVal2=$(./mc ilm rule list sited/bucket --json | jq '.config.Rules[0].Filter.And.Tags[1].Value' | sed 's/"//g')
if [ "${prefix}" != "myprefix" ]; then
echo "BUG: ILM expiry rules prefix not replicated to 'sited'"
exit 1
fi
if [ "${tagName1}" != "tag1" ] || [ "${tagVal1}" != "val1" ] || [ "${tagName2}" != "tag2" ] || [ "${tagVal2}" != "val2" ]; then
echo "BUG: ILM expiry rules tags not replicated to 'sited'"
exit 1
fi
## Check replication of deleted ILM expiry rules when target has transition part as well
## Only the expiry part of rules should get removed as part if replication of removal from
## other site
id=$(./mc ilm rule list siteb/bucket --json | jq '.config.Rules[] | select(.Expiration.Days==3) | .ID' | sed 's/"//g')
# Remove rule from siteb
./mc ilm rule remove --id "${id}" siteb/bucket
sleep 30s # allow to replicate
# sitea should still contain the transition portion of rule
transitionRuleDays=$(./mc ilm rule list sitea/bucket --json | jq '.config.Rules[0].Transition.Days')
expirationRuleDet=$(./mc ilm rule list sitea/bucket --json | jq '.config.Rules[0].Expiration')
if [ ${transitionRuleDays} -ne 0 ]; then
echo "BUG: Transition rules not retained as part of replication of deleted ILM expiry rules on 'sitea'"
exit 1
fi
if [ ${expirationRuleDet} != null ]; then
echo "BUG: removed ILM expiry rule not replicated to 'sitea'"
exit 1
fi
catch

101
docs/bucket/replication/setup_replication.sh Executable file
View File

@@ -0,0 +1,101 @@
#!/bin/sh
# Create buckets with versioning and object locking enabled.
mc mb -l source/bucket
mc mb -l dest/bucket
#### Create a replication admin on source alias
# create a replication admin user : repladmin
mc admin user add source repladmin repladmin123
# create a replication policy for repladmin
cat >repladmin-policy-source.json <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"admin:SetBucketTarget",
"admin:GetBucketTarget"
],
"Effect": "Allow",
"Sid": ""
},
{
"Effect": "Allow",
"Action": [
"s3:GetReplicationConfiguration",
"s3:PutReplicationConfiguration",
"s3:ListBucket",
"s3:ListBucketMultipartUploads",
"s3:GetBucketLocation",
"s3:GetBucketVersioning"
],
"Resource": [
"arn:aws:s3:::bucket"
]
}
]
}
EOF
mc admin policy create source repladmin-policy ./repladmin-policy-source.json
cat ./repladmin-policy-source.json
#assign this replication policy to repladmin
mc admin policy attach source repladmin-policy --user=repladmin
### on dest alias
# Create a replication user : repluser on dest alias
mc admin user add dest repluser repluser123
# create a replication policy for repluser
# Remove "s3:GetBucketObjectLockConfiguration" if object locking is not enabled, i.e. bucket was not created with `mc mb --with-lock` option
# Remove "s3:ReplicateDelete" if delete marker replication is not required
cat >replpolicy.json <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetReplicationConfiguration",
"s3:ListBucket",
"s3:ListBucketMultipartUploads",
"s3:GetBucketLocation",
"s3:GetBucketVersioning",
"s3:GetBucketObjectLockConfiguration"
],
"Resource": [
"arn:aws:s3:::bucket"
]
},
{
"Effect": "Allow",
"Action": [
"s3:GetReplicationConfiguration",
"s3:ReplicateTags",
"s3:AbortMultipartUpload",
"s3:GetObject",
"s3:GetObjectVersion",
"s3:GetObjectVersionTagging",
"s3:PutObject",
"s3:DeleteObject",
"s3:ReplicateObject",
"s3:ReplicateDelete"
],
"Resource": [
"arn:aws:s3:::bucket/*"
]
}
]
}
EOF
mc admin policy create dest replpolicy ./replpolicy.json
cat ./replpolicy.json
# assign this replication policy to repluser
mc admin policy attach dest replpolicy --user=repluser
# configure replication config to remote bucket at http://localhost:9000
mc replicate add source/bucket --priority 1 --remote-bucket http://repluser:repluser123@localhost:9000/bucket \
--replicate existing-objects,delete,delete-marker,replica-metadata-sync

68
docs/bucket/replication/sio-error.sh Executable file
View File

@@ -0,0 +1,68 @@
#!/bin/bash
echo "Running $0"
set -e
set -x
export CI=1
make || exit 255
killall -9 minio || true
rm -rf /tmp/xl/
mkdir -p /tmp/xl/1/ /tmp/xl/2/
export MINIO_KMS_SECRET_KEY="my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw="
NODES=4
args1=()
args2=()
for i in $(seq 1 $NODES); do
args1+=("http://localhost:$((9000 + i))/tmp/xl/1/$i ")
args2+=("http://localhost:$((9100 + i))/tmp/xl/2/$i ")
done
for i in $(seq 1 $NODES); do
./minio server --address "127.0.0.1:$((9000 + i))" ${args1[@]} & # | tee /tmp/minio/node.$i &
./minio server --address "127.0.0.1:$((9100 + i))" ${args2[@]} & # | tee /tmp/minio/node.$i &
done
sleep 10
./mc alias set myminio1 http://localhost:9001 minioadmin minioadmin
./mc alias set myminio2 http://localhost:9101 minioadmin minioadmin
./mc ready myminio1
./mc ready myminio2
sleep 1
./mc mb myminio1/testbucket/ --with-lock
./mc mb myminio2/testbucket/ --with-lock
./mc encrypt set sse-s3 my-minio-key myminio1/testbucket/
./mc encrypt set sse-s3 my-minio-key myminio2/testbucket/
./mc replicate add myminio1/testbucket --remote-bucket http://minioadmin:minioadmin@localhost:9101/testbucket --priority 1
./mc replicate add myminio2/testbucket --remote-bucket http://minioadmin:minioadmin@localhost:9001/testbucket --priority 1
sleep 1
cp README.md internal.tar
./mc cp internal.tar myminio1/testbucket/dir/1.tar
./mc cp internal.tar myminio2/testbucket/dir/2.tar
sleep 1
./mc ls -r --versions myminio1/testbucket/dir/ >/tmp/dir_1.txt
./mc ls -r --versions myminio2/testbucket/dir/ >/tmp/dir_2.txt
out=$(diff -qpruN /tmp/dir_1.txt /tmp/dir_2.txt)
ret=$?
if [ $ret -ne 0 ]; then
echo "BUG: expected no 'diff' after replication: $out"
exit 1
fi

81
docs/bucket/replication/test_del_marker_proxying.sh Executable file
View File

@@ -0,0 +1,81 @@
#!/usr/bin/env bash
# shellcheck disable=SC2120
exit_1() {
cleanup
for site in sitea siteb; do
echo "$site server logs ========="
cat "/tmp/${site}_1.log"
echo "==========================="
cat "/tmp/${site}_2.log"
done
exit 1
}
cleanup() {
echo -n "Cleaning up instances of MinIO ..."
pkill -9 minio || sudo pkill -9 minio
rm -rf /tmp/sitea
rm -rf /tmp/siteb
echo "done"
}
cleanup
export MINIO_CI_CD=1
export MINIO_BROWSER=off
make install-race
# Start MinIO instances
echo -n "Starting MinIO instances ..."
minio server --address 127.0.0.1:9001 --console-address ":10000" "http://127.0.0.1:9001/tmp/sitea/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9002/tmp/sitea/data/disterasure/xl{5...8}" >/tmp/sitea_1.log 2>&1 &
minio server --address 127.0.0.1:9002 "http://127.0.0.1:9001/tmp/sitea/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9002/tmp/sitea/data/disterasure/xl{5...8}" >/tmp/sitea_2.log 2>&1 &
minio server --address 127.0.0.1:9003 --console-address ":10001" "http://127.0.0.1:9003/tmp/siteb/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9004/tmp/siteb/data/disterasure/xl{5...8}" >/tmp/siteb_1.log 2>&1 &
minio server --address 127.0.0.1:9004 "http://127.0.0.1:9003/tmp/siteb/data/disterasure/xl{1...4}" \
"http://127.0.0.1:9004/tmp/siteb/data/disterasure/xl{5...8}" >/tmp/siteb_2.log 2>&1 &
echo "done"
if [ ! -f ./mc ]; then
wget --quiet -O mc https://dl.minio.io/client/mc/release/linux-amd64/mc &&
chmod +x mc
fi
export MC_HOST_sitea=http://minioadmin:minioadmin@127.0.0.1:9001
export MC_HOST_siteb=http://minioadmin:minioadmin@127.0.0.1:9004
./mc ready sitea
./mc ready siteb
./mc mb sitea/bucket
./mc version enable sitea/bucket
./mc mb siteb/bucket
./mc version enable siteb/bucket
# Set bucket replication
./mc replicate add sitea/bucket --remote-bucket siteb/bucket
# Run the test to make sure proxying of DEL marker doesn't happen
loop_count=0
while true; do
if [ $loop_count -eq 1000 ]; then
break
fi
echo "Hello World" | ./mc pipe sitea/bucket/obj$loop_count
./mc rm sitea/bucket/obj$loop_count
RESULT=$({ ./mc stat --no-list sitea/bucket/obj$loop_count; } 2>&1)
if [[ ${RESULT} != *"Object does not exist"* ]]; then
echo "BUG: stat should fail. succeeded."
exit_1
fi
loop_count=$((loop_count + 1))
done
cleanup

59
docs/bucket/retention/README.md Normal file
View File

@@ -0,0 +1,59 @@
# Object Lock and Immutability Guide [![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io)
MinIO server allows WORM for specific objects or by configuring a bucket with default object lock configuration that applies default retention mode and retention duration to all objects. This makes objects in the bucket immutable i.e. delete of the version are not allowed until an expiry specified in the bucket's object lock configuration or object retention.
Object locking requires locking to be enabled on a bucket at the time of bucket creation refer to `mc mb --with-lock`, object locking enables versioning on the bucket and cannot be disabled.
A default retention period and retention mode can be configured on a bucket to be applied to objects created in that bucket. Independent of retention, an object can also be under legal hold. This effectively disallows all deletes of an object under legal hold until the legal hold is removed by an API call.
## Get Started
### 1. Prerequisites
- Install MinIO - [MinIO Quickstart Guide](https://docs.min.io/community/minio-object-store/operations/deployments/baremetal-deploy-minio-on-redhat-linux.html)
- Install `awscli` - [Installing AWS Command Line Interface](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html)
### 2. Set bucket WORM configuration
WORM on a bucket is enabled by setting object lock configuration. This configuration is applied to all the objects in the bucket. Below is an example to set `Governance` mode and one day retention time on `mybucket`.
```sh
awscli s3api put-object-lock-configuration --bucket mybucket --object-lock-configuration 'ObjectLockEnabled=\"Enabled\",Rule={DefaultRetention={Mode=\"GOVERNANCE\",Days=1}}'
```
### Set object lock
PutObject API allows setting per object retention mode and retention duration using `x-amz-object-lock-mode` and `x-amz-object-lock-retain-until-date` headers. This takes precedence over any bucket object lock configuration w.r.t retention.
```sh
aws s3api put-object --bucket testbucket --key lockme --object-lock-mode GOVERNANCE --object-lock-retain-until-date "2019-11-20" --body /etc/issue
```
See <https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock-overview.html> for AWS S3 spec on object locking and permissions required for object retention and governance bypass overrides.
### Set legal hold on an object
PutObject API allows setting legal hold using `x-amz-object-lock-legal-hold` header.
```sh
aws s3api put-object --bucket testbucket --key legalhold --object-lock-legal-hold-status ON --body /etc/issue
```
See <https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock-overview.html> for AWS S3 spec on object locking and permissions required for specifying legal hold.
## Concepts
- If an object is under legal hold, it cannot be deleted unless the legal hold is explicitly removed for the respective version id. DeleteObjectVersion() would fail otherwise.
- In `Compliance` mode, objects cannot be deleted by anyone until retention period has expired for the given version id. If user has governance bypass permissions, an object's retention date can be extended in `Compliance` mode.
- Once object lock configuration is set to a bucket
- New objects inherit the retention settings of the bucket object lock configuration automatically
- Retention headers can be optionally set while uploading objects
- Once objects are uploaded PutObjectRetention API can be called to change retention settings
- *MINIO_NTP_SERVER* environment variable can be set to remote NTP server endpoint if system time is not desired for setting retention dates.
## Explore Further
- [Use `mc` with MinIO Server](https://docs.min.io/community/minio-object-store/reference/minio-mc.html#quickstart)
- [Use `aws-cli` with MinIO Server](https://docs.min.io/community/minio-object-store/integrations/aws-cli-with-minio.html)
- [Use `minio-go` SDK with MinIO Server](https://docs.min.io/community/minio-object-store/developers/go/minio-go.html)
- [The MinIO documentation website](https://docs.min.io/community/minio-object-store/index.html)

138
docs/bucket/versioning/DESIGN.md Normal file
View File

@@ -0,0 +1,138 @@
# Bucket Versioning Design Guide [![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io) [![Docker Pulls](https://img.shields.io/docker/pulls/minio/minio.svg?maxAge=604800)](https://hub.docker.com/r/minio/minio/)
## Description of `xl.meta`
`xl.meta` is a new self describing backend format used by MinIO to support AWS S3 compatible versioning.
This file is the source of truth for each `version` at rest. `xl.meta` is a msgpack file serialized from a
well defined data structure. To understand `xl.meta` here are the few things to start with
`xl.meta` carries first 8 bytes an XL header which describes the current format and the format version,
allowing the unmarshaller's to automatically use the right data structures to parse the subsequent content in the stream.
### v1.0
| Entry | Encoding | Content
| ----------|-------------|----------------------------------------
| xlHeader | [4]byte | `'X', 'L', '2', ' '`
| xlVersion | [4]byte | `'1', ' ', ' ', ' '`
| xlMetaV2 | msgp object | All versions as single messagepack object
| [EOF] | |
### v1.1+
Version 1.1 added inline data, which will be placed after the metadata.
Therefore, the metadata is wrapped as a binary array for easy skipping.
| Entry | Encoding | Content
| ---------------|----------------|----------------------------------------
| xlHeader | [4]byte | `'X', 'L', '2', ' '`
| xlVersionMajor | uint16 | Major xl-meta version.
| xlVersionMinor | uint16 | Minor xl-meta version.
| xlMetaV2 | msgp bin array | Bin array with serialized metadata
| crc | msgp uint | Lower 32 bits of 64 bit xxhash of previous array contents (v1.2+ only)
| inline data | binary | Inline data if any, see Inline Data section for encoding.
| [EOF] | |
## v1.0-v1.2 Versions
`xl.meta` carries three types of object entries which designate the type of version object stored.
- ObjectType (default)
- LegacyObjectType (preserves existing deployments and older xl.json format)
- DeleteMarker (a versionId to capture the DELETE sequences implemented primarily for AWS spec compatibility)
A sample msgpack-JSON `xl.meta`, you can debug the content inside `xl.meta` using [xl-meta.go](https://github.com/minio/minio/tree/master/docs/debugging#decoding-metadata) program.
```json
{
"Versions": [
{
"Type": 1,
"V2Obj": {
"ID": "KWUs8S+8RZq4Vp5TWy6KFg==",
"DDir": "X3pDAFu8Rjyft7QD6t7W5g==",
"EcAlgo": 1,
"EcM": 2,
"EcN": 2,
"EcBSize": 10485760,
"EcIndex": 3,
"EcDist": [3, 4, 1, 2],
"CSumAlgo": 1,
"PartNums": [1],
"PartETags": [""],
"PartSizes": [314],
"PartASizes": [282],
"Size": 314,
"MTime": 1591820730,
"MetaSys": {
"X-Minio-Internal-Server-Side-Encryption-S3-Kms-Key-Id": "bXktbWluaW8ta2V5",
"X-Minio-Internal-Server-Side-Encryption-S3-Kms-Sealed-Key": "ZXlKaFpXRmtJam9pUVVWVExUSTFOaTFIUTAwdFNFMUJReTFUU0VFdE1qVTJJaXdpYVhZaU9pSkJMMVZzZFVnelZYVjZSR2N6UkhGWUwycEViRmRCUFQwaUxDSnViMjVqWlNJNklpdE9lbkJXVWtseFlWSlNVa2t2UVhNaUxDSmllWFJsY3lJNklrNDBabVZsZG5WU1NWVnRLMFoyUWpBMVlYTk9aMU41YVhoU1RrNUpkMDlhTkdKa2RuaGpLMjFuVDNnMFFYbFJhbE15V0hkU1pEZzNRMk54ZUN0SFFuSWlmUT09",
"X-Minio-Internal-Server-Side-Encryption-Seal-Algorithm": "REFSRXYyLUhNQUMtU0hBMjU2",
"X-Minio-Internal-Server-Side-Encryption-Iv": "bW5YRDhRUGczMVhkc2pJT1V1UVlnbWJBcndIQVhpTUN1dnVBS0QwNUVpaz0=",
"X-Minio-Internal-Server-Side-Encryption-S3-Sealed-Key": "SUFBZkFPeUo5ZHVVSEkxYXFLU0NSRkJTTnM0QkVJNk9JWU1QcFVTSXFhK2dHVThXeE9oSHJCZWwwdnRvTldUNE8zS1BtcWluR0cydmlNNFRWa0N0Mmc9PQ=="
},
"MetaUsr": {
"content-type": "application/octet-stream",
"etag": "20000f00f58c508b40720270929bd90e9f07b9bd78fb605e5432a67635fc34722e4fc53b1d5fab9ff8400eb9ded4fba2"
}
}
}
]
}
```
### v1.3+ versions
Version 1.3 introduces changes to help with [faster metadata reads and updates](https://blog.min.io/minio-versioning-metadata-deep-dive/)
| Entry | Encoding | Content
| ----------------|-----------------------------|----------------------------------------
| xlHeaderVersion | msgp uint | header version identifier
| xlMetaVersion | msgp uint | metadata version identifier
| versions | msgp int | Number of versions following
| header_1 | msgp bin array | Header of version 1
| metadata_1 | msgp bin array | Metadata of version 1
| ...header_n | msgp bin array | Header of last version
| ...metadata_n | msgp bin array | Metadata of last version
Each header contains a mspg array (tuple) encoded object:
xlHeaderVersion version == 1:
```
//msgp:tuple xlMetaV2VersionHeader
type xlMetaV2VersionHeader struct {
VersionID [16]byte // Version UUID, raw.
ModTime int64 // Unix nanoseconds.
Signature [4]byte // Signature of metadata.
Type uint8 // Type if the version
Flags uint8
}
```
The following flags are defined:
```
const (
FreeVersion = 1 << 0
UsesDataDir = 1 << 1
InlineData = 1 << 2
)
```
The "Metadata" section contains a single version, encoded in similar fashion as each version in the `Versions` array
of the previous version.
## Inline Data
Inline data is optional. If no inline data is present, it is encoded as 0 bytes.
| Entry | Encoding | Content
| --------------------|-----------------------------|----------------------------------------
| xlMetaInlineDataVer | byte | version identifier
| id -> data | msgp `map[string][]byte` | Map of string id -> byte content
Currently only xlMetaInlineDataVer == 1 exists.
The ID is the string encoded Version ID of which the data corresponds.

217
docs/bucket/versioning/README.md Normal file
View File

@@ -0,0 +1,217 @@
# Bucket Versioning Guide [![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io) [![Docker Pulls](https://img.shields.io/docker/pulls/minio/minio.svg?maxAge=604800)](https://hub.docker.com/r/minio/minio/)
MinIO versioning is designed to keep multiple versions of an object in one bucket. For example, you could store `spark.csv` (version `ede336f2`) and `spark.csv` (version `fae684da`) in a single bucket. Versioning protects you from unintended overwrites, deletions, protect objects with retention policies.
To control data retention and storage usage, use object versioning with [object lifecycle management](https://github.com/minio/minio/blob/master/docs/bucket/lifecycle/README.md). If you have an object expiration lifecycle policy in your non-versioned bucket and you want to maintain the same permanent delete behavior when on versioning-enabled bucket, you must add a noncurrent expiration policy. The noncurrent expiration lifecycle policy will manage the deletes of the noncurrent object versions in the versioning-enabled bucket. (A version-enabled bucket maintains one current and zero or more noncurrent object versions.)
Versioning must be explicitly enabled on a bucket, versioning is not enabled by default. Object locking enabled buckets have versioning enabled automatically. Enabling and suspending versioning is done at the bucket level.
Only MinIO generates version IDs, and they can't be edited. Version IDs are simply of `DCE 1.1 v4 UUID 4` (random data based), UUIDs are 128 bit numbers which are intended to have a high likelihood of uniqueness over space and time and are computationally difficult to guess. They are globally unique identifiers which can be locally generated without contacting a global registration authority. UUIDs are intended as unique identifiers for both mass tagging objects with an extremely short lifetime and to reliably identifying very persistent objects across a network.
When you PUT an object in a versioning-enabled bucket, the noncurrent version is not overwritten. The following figure shows that when a new version of `spark.csv` is PUT into a bucket that already contains an object with the same name, the original object (ID = `ede336f2`) remains in the bucket, MinIO generates a new version (ID = `fae684da`), and adds the newer version to the bucket.
![put](https://raw.githubusercontent.com/minio/minio/master/docs/bucket/versioning/versioning_PUT_versionEnabled.png)
This protects against accidental overwrites or deletes of objects, allows previous versions to be retrieved.
When you DELETE an object, all versions remain in the bucket and MinIO adds a delete marker, as shown below:
![delete](https://raw.githubusercontent.com/minio/minio/master/docs/bucket/versioning/versioning_DELETE_versionEnabled.png)
Now the delete marker becomes the current version of the object. GET requests by default always retrieve the latest stored version. So performing a simple GET object request when the current version is a delete marker would return `404` `The specified key does not exist` as shown below:
![get](https://raw.githubusercontent.com/minio/minio/master/docs/bucket/versioning/versioning_GET_versionEnabled.png)
GET requests by specifying a version ID as shown below, you can retrieve the specific object version `fae684da`.
![get_version_id](https://raw.githubusercontent.com/minio/minio/master/docs/bucket/versioning/versioning_GET_versionEnabled_id.png)
To permanently delete an object you need to specify the version you want to delete, only the user with appropriate permissions can permanently delete a version. As shown below DELETE request called with a specific version id permanently deletes an object from a bucket. Delete marker is not added for DELETE requests with version id.
![delete_version_id](https://raw.githubusercontent.com/minio/minio/master/docs/bucket/versioning/versioning_DELETE_versionEnabled_id.png)
## Concepts
- All Buckets on MinIO are always in one of the following states: unversioned (the default) and all other existing deployments, versioning-enabled, or versioning-suspended.
- Versioning state applies to all of the objects in the versioning enabled bucket. The first time you enable a bucket for versioning, objects in the bucket are thereafter always versioned and given a unique version ID.
- Existing or newer buckets can be created with versioning enabled and eventually can be suspended as well. Existing versions of objects stay as is and can still be accessed using the version ID.
- All versions, including delete-markers should be deleted before deleting a bucket.
- **Versioning feature is only available in erasure coded and distributed erasure coded setups**.
## How to configure versioning on a bucket
Each bucket created has a versioning configuration associated with it. By default bucket is unversioned as shown below
```
<VersioningConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
</VersioningConfiguration>
```
To enable versioning, you send a request to MinIO with a versioning configuration with Status set to `Enabled`.
```
<VersioningConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Status>Enabled</Status>
</VersioningConfiguration>
```
Similarly to suspend versioning set the configuration with Status set to `Suspended`.
```
<VersioningConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Status>Suspended</Status>
</VersioningConfiguration>
```
## MinIO extension to Bucket Versioning
### Idempotent versions on directory objects
All directory objects such as objects that end with `/`, will only have one versionId (i.e `null`). A delete marker will never be created on these directory objects, instead a DELETE will delete the directory objects. This is done to ensure that directory objects even with multiple overwrites - do not ever need multiple versions in the first place. All overwrite calls on these directory objects are idempotent.
> NOTE: Server side replication is supported for idempotent versions on directory objects.
### Idempotent versions on delete markers
Duplicate delete markers are not created on MinIO buckets with versioning, if an application performs a soft delete on an object repeatedly - that object will only ever have a single DELETE marker for all such successive attempts. This is done to ensure that repeated soft deletes do not ever need multiple versions in the first place.
> NOTE: Server side replication is supported for idempotent versions on delete marked objects.
### Motivation
**PLEASE READ: This feature is meant for advanced use-cases only where the setup is using bucket versioning or with replicated buckets, use this feature to optimize versioning behavior for some specific applications. MinIO experts will evaluate and guide on the benefits for your application, please reach out to us on <https://subnet.min.io>.**
Spark/Hadoop workloads which use Hadoop MR Committer v1/v2 algorithm upload objects to a temporary prefix in a bucket. These objects are 'renamed' to a different prefix on Job commit. Object storage admins are forced to configure separate ILM policies to expire these objects and their versions to reclaim space.
### Solution
To exclude objects under a list of prefix (glob) patterns from being versioned, you can send the following versioning configuration with Status set to `Enabled`.
```
<VersioningConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Status>Enabled</Status>
<ExcludedPrefixes>
<Prefix>*/_temporary</Prefix>
</ExcludedPrefixes>
<ExcludedPrefixes>
<Prefix>*/__magic</Prefix>
</ExcludedPrefixes>
<ExcludedPrefixes>
<Prefix>*/_staging</Prefix>
</ExcludedPrefixes>
<!-- .. up to 10 prefixes in all -->
</VersioningConfiguration>
```
### Features
- Objects matching these prefixes will behave as though versioning were suspended. These objects **will not** be replicated if bucket has replication configured.
- Objects matching these prefixes will also not leave `null` delete markers, dramatically reduces namespace pollution while keeping the benefits of replication.
- Users with explicit permissions or the root credential can configure the versioning state of any bucket.
## Examples of enabling bucket versioning using MinIO Java SDK
### EnableVersioning() API
```
import io.minio.EnableVersioningArgs;
import io.minio.MinioClient;
import io.minio.errors.MinioException;
import java.io.IOException;
import java.security.InvalidKeyException;
import java.security.NoSuchAlgorithmException;
public class EnableVersioning {
/** MinioClient.enableVersioning() example. */
public static void main(String[] args)
throws IOException, NoSuchAlgorithmException, InvalidKeyException {
try {
/* play.min.io for test and development. */
MinioClient minioClient =
MinioClient.builder()
.endpoint("https://play.min.io")
.credentials("Q3AM3UQ867SPQQA43P2F", "zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG")
.build();
/* Amazon S3: */
// MinioClient minioClient =
// MinioClient.builder()
// .endpoint("https://s3.amazonaws.com")
// .credentials("YOUR-ACCESSKEY", "YOUR-SECRETACCESSKEY")
// .build();
// Enable versioning on 'my-bucketname'.
minioClient.enableVersioning(EnableVersioningArgs.builder().bucket("my-bucketname").build());
System.out.println("Bucket versioning is enabled successfully");
} catch (MinioException e) {
System.out.println("Error occurred: " + e);
}
}
}
```
### isVersioningEnabled() API
```
public class IsVersioningEnabled {
/** MinioClient.isVersioningEnabled() example. */
public static void main(String[] args)
throws IOException, NoSuchAlgorithmException, InvalidKeyException {
try {
/* play.min.io for test and development. */
MinioClient minioClient =
MinioClient.builder()
.endpoint("https://play.min.io")
.credentials("Q3AM3UQ867SPQQA43P2F", "zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG")
.build();
/* Amazon S3: */
// MinioClient minioClient =
// MinioClient.builder()
// .endpoint("https://s3.amazonaws.com")
// .credentials("YOUR-ACCESSKEY", "YOUR-SECRETACCESSKEY")
// .build();
// Create bucket 'my-bucketname' if it doesn`t exist.
if (!minioClient.bucketExists(BucketExistsArgs.builder().bucket("my-bucketname").build())) {
minioClient.makeBucket(MakeBucketArgs.builder().bucket("my-bucketname").build());
System.out.println("my-bucketname is created successfully");
}
boolean isVersioningEnabled =
minioClient.isVersioningEnabled(
IsVersioningEnabledArgs.builder().bucket("my-bucketname").build());
if (isVersioningEnabled) {
System.out.println("Bucket versioning is enabled");
} else {
System.out.println("Bucket versioning is disabled");
}
// Enable versioning on 'my-bucketname'.
minioClient.enableVersioning(EnableVersioningArgs.builder().bucket("my-bucketname").build());
System.out.println("Bucket versioning is enabled successfully");
isVersioningEnabled =
minioClient.isVersioningEnabled(
IsVersioningEnabledArgs.builder().bucket("my-bucketname").build());
if (isVersioningEnabled) {
System.out.println("Bucket versioning is enabled");
} else {
System.out.println("Bucket versioning is disabled");
}
} catch (MinioException e) {
System.out.println("Error occurred: " + e);
}
}
}
```
## Explore Further
- [Use `minio-java` SDK with MinIO Server](https://docs.min.io/community/minio-object-store/developers/java/minio-java.html)
- [Object Lock and Immutability Guide](https://docs.min.io/community/minio-object-store/administration/object-management/object-retention.html)
- [MinIO Admin Complete Guide](https://docs.min.io/community/minio-object-store/reference/minio-mc-admin.html)
- [The MinIO documentation website](https://docs.min.io/community/minio-object-store/index.html)

99
docs/bucket/versioning/versioning-tests.sh Executable file
View File

@@ -0,0 +1,99 @@
#!/usr/bin/env bash
if [ -n "$TEST_DEBUG" ]; then
set -x
fi
trap 'catch $LINENO' ERR
# shellcheck disable=SC2120
catch() {
if [ $# -ne 0 ]; then
echo "error on line $1"
echo "server logs ========="
cat "/tmp/sitea_1.log"
echo "==========================="
cat "/tmp/sitea_2.log"
fi
echo "Cleaning up instances of MinIO"
pkill minio
pkill -9 minio
rm -rf /tmp/multisitea
if [ $# -ne 0 ]; then
exit $#
fi
}
catch
set -e
export MINIO_CI_CD=1
export MINIO_BROWSER=off
export MINIO_KMS_AUTO_ENCRYPTION=off
export MINIO_PROMETHEUS_AUTH_TYPE=public
export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
unset MINIO_KMS_KES_CERT_FILE
unset MINIO_KMS_KES_KEY_FILE
unset MINIO_KMS_KES_ENDPOINT
unset MINIO_KMS_KES_KEY_NAME
if [ ! -f ./mc ]; then
wget -O mc https://dl.minio.io/client/mc/release/linux-amd64/mc &&
chmod +x mc
fi
minio server -S /tmp/no-certs --address ":9001" "http://localhost:9001/tmp/multisitea/data/disterasure/xl{1...4}" \
"http://localhost:9002/tmp/multisitea/data/disterasure/xl{5...8}" >/tmp/sitea_1.log 2>&1 &
minio server -S /tmp/no-certs --address ":9002" "http://localhost:9001/tmp/multisitea/data/disterasure/xl{1...4}" \
"http://localhost:9002/tmp/multisitea/data/disterasure/xl{5...8}" >/tmp/sitea_2.log 2>&1 &
export MC_HOST_sitea=http://minioadmin:minioadmin@localhost:9002
./mc ready sitea
./mc mb sitea/delissue --insecure
./mc version enable sitea/delissue --insecure
echo hello | ./mc pipe sitea/delissue/hello --insecure
./mc version suspend sitea/delissue --insecure
./mc rm sitea/delissue/hello --insecure
./mc version enable sitea/delissue --insecure
echo hello | ./mc pipe sitea/delissue/hello --insecure
./mc version suspend sitea/delissue --insecure
./mc rm sitea/delissue/hello --insecure
count=$(./mc ls --versions sitea/delissue --insecure | wc -l)
if [ ${count} -ne 3 ]; then
echo "BUG: expected number of versions to be '3' found ${count}"
echo "===== DEBUG ====="
./mc ls --versions sitea/delissue
fi
./mc mb sitea/testbucket
./mc version enable sitea/testbucket
./mc put --quiet README.md sitea/testbucket/file
etag1=$(./mc cat sitea/testbucket/file | md5sum --tag | awk {'print $4'})
./mc cp --quiet --storage-class "STANDARD" sitea/testbucket/file sitea/testbucket/file
etag2=$(./mc cat sitea/testbucket/file | md5sum --tag | awk {'print $4'})
if [ $etag1 != $etag2 ]; then
echo "expected $etag1, got $etag2"
exit 1
fi
echo "SUCCESS:"
./mc ls --versions sitea/delissue --insecure
catch

BIN
docs/bucket/versioning/versioning_DELETE_versionEnabled.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 144 KiB

BIN
docs/bucket/versioning/versioning_DELETE_versionEnabled_id.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 123 KiB

BIN
docs/bucket/versioning/versioning_GET_versionEnabled.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 134 KiB

BIN
docs/bucket/versioning/versioning_GET_versionEnabled_id.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 169 KiB

BIN
docs/bucket/versioning/versioning_PUT_versionEnabled.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 103 KiB

58
docs/chroot/README.md Normal file
View File

@@ -0,0 +1,58 @@
# Deploy MinIO on Chrooted Environment [![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io) [![Docker Pulls](https://img.shields.io/docker/pulls/minio/minio.svg?maxAge=604800)](https://hub.docker.com/r/minio/minio/)
Chroot allows user based namespace isolation on many standard Linux deployments.
## 1. Prerequisites
- Familiarity with [chroot](http://man7.org/linux/man-pages/man2/chroot.2.html)
- Chroot installed on your machine.
## 2. Install MinIO in Chroot
> **Note:** MinIO community edition is now distributed as source code only. Pre-compiled binaries are no longer provided for new releases.
Build MinIO from source and install it in the chroot directory:
```sh
# Build MinIO from source
go install github.com/minio/minio@latest
# Create the bin directory in your chroot
mkdir -p /mnt/export/${USER}/bin
# Copy the built binary to the chroot directory
cp $(go env GOPATH)/bin/minio /mnt/export/${USER}/bin/minio
chmod +x /mnt/export/${USER}/bin/minio
```
Alternatively, if you have an existing legacy binary, you can still use it, but note that it will not receive updates.
Bind your `proc` mount to the target chroot directory
```
sudo mount --bind /proc /mnt/export/${USER}/proc
```
## 3. Run Standalone MinIO in Chroot
### GNU/Linux
```sh
sudo chroot --userspec username:group /mnt/export/${USER} /bin/minio --config-dir=/.minio server /data
Endpoint: http://192.168.1.92:9000 http://65.19.167.92:9000
AccessKey: MVPSPBW4NP2CMV1W3TXD
SecretKey: X3RKxEeFOI8InuNWoPsbG+XEVoaJVCqbvxe+PTOa
...
...
```
Instance is now accessible on the host at port 9000, proceed to access the Web browser at <http://127.0.0.1:9000/>
## Explore Further
- [MinIO Erasure Code Overview](https://docs.min.io/community/minio-object-store/operations/concepts/erasure-coding.html)
- [Use `mc` with MinIO Server](https://docs.min.io/community/minio-object-store/reference/minio-mc.html)
- [Use `aws-cli` with MinIO Server](https://docs.min.io/community/minio-object-store/integrations/aws-cli-with-minio.html)
- [Use `minio-go` SDK with MinIO Server](https://docs.min.io/community/minio-object-store/developers/go/minio-go.html)
- [The MinIO documentation website](https://docs.min.io/community/minio-object-store/index.html)

137
docs/compression/README.md Normal file
View File

@@ -0,0 +1,137 @@
# Compression Guide [![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io)
MinIO server allows streaming compression to ensure efficient disk space usage.
Compression happens inflight, i.e objects are compressed before being written to disk(s).
MinIO uses [`klauspost/compress/s2`](https://github.com/klauspost/compress/tree/master/s2)
streaming compression due to its stability and performance.
This algorithm is specifically optimized for machine generated content.
Write throughput is typically at least 500MB/s per CPU core,
and scales with the number of available CPU cores.
Decompression speed is typically at least 1GB/s.
This means that in cases where raw IO is below these numbers
compression will not only reduce disk usage but also help increase system throughput.
Typically, enabling compression on spinning disk systems
will increase speed when the content can be compressed.
## Get Started
### 1. Prerequisites
Install MinIO - [MinIO Quickstart Guide](https://docs.min.io/community/minio-object-store/operations/deployments/baremetal-deploy-minio-on-redhat-linux.html).
### 2. Run MinIO with compression
Compression can be enabled by updating the `compress` config settings for MinIO server config.
Config `compress` settings take extensions and mime-types to be compressed.
```bash
~ mc admin config get myminio compression
compression extensions=".txt,.log,.csv,.json,.tar,.xml,.bin" mime_types="text/*,application/json,application/xml"
```
Default config includes most common highly compressible content extensions and mime-types.
```bash
~ mc admin config set myminio compression extensions=".pdf" mime_types="application/pdf"
```
To show help on setting compression config values.
```bash
~ mc admin config set myminio compression
```
To enable compression for all content, no matter the extension and content type
(except for the default excluded types) set BOTH extensions and mime types to empty.
```bash
~ mc admin config set myminio compression enable="on" extensions="" mime_types=""
```
The compression settings may also be set through environment variables.
When set, environment variables override the defined `compress` config settings in the server config.
```bash
export MINIO_COMPRESSION_ENABLE="on"
export MINIO_COMPRESSION_EXTENSIONS=".txt,.log,.csv,.json,.tar,.xml,.bin"
export MINIO_COMPRESSION_MIME_TYPES="text/*,application/json,application/xml"
```
> [!NOTE]
> To enable compression for all content when using environment variables, set either or both of the extensions and MIME types to `*` instead of an empty string:
> ```bash
> export MINIO_COMPRESSION_ENABLE="on"
> export MINIO_COMPRESSION_EXTENSIONS="*"
> export MINIO_COMPRESSION_MIME_TYPES="*"
> ```
### 3. Compression + Encryption
Combining encryption and compression is not safe in all setups.
This is particularly so if the compression ratio of your content reveals information about it.
See [CRIME TLS](https://en.wikipedia.org/wiki/CRIME) as an example of this.
Therefore, compression is disabled when encrypting by default, and must be enabled separately.
Consult our security experts on [SUBNET](https://min.io/pricing) to help you evaluate if
your setup can use this feature combination safely.
To enable compression+encryption use:
```bash
~ mc admin config set myminio compression allow_encryption=on
```
Or alternatively through the environment variable `MINIO_COMPRESSION_ALLOW_ENCRYPTION=on`.
### 4. Excluded Types
- Already compressed objects are not fit for compression since they do not have compressible patterns.
Such objects do not produce efficient [`LZ compression`](https://en.wikipedia.org/wiki/LZ77_and_LZ78)
which is a fitness factor for a lossless data compression.
Pre-compressed input typically compresses in excess of 2GiB/s per core,
so performance impact should be minimal even if precompressed data is re-compressed.
Decompressing incompressible data has no significant performance impact.
Below is a list of common files and content-types which are typically not suitable for compression.
- Extensions
| `gz` | (GZIP) |
| `bz2` | (BZIP2) |
| `rar` | (WinRAR) |
| `zip` | (ZIP) |
| `7z` | (7-Zip) |
| `xz` | (LZMA) |
| `mp4` | (MP4) |
| `mkv` | (MKV media) |
| `mov` | (MOV) |
- Content-Types
| `video/*` |
| `audio/*` |
| `application/zip` |
| `application/x-gzip` |
| `application/zip` |
| `application/x-bz2` |
| `application/x-compress` |
| `application/x-xz` |
All files with these extensions and mime types are excluded from compression,
even if compression is enabled for all types.
## To test the setup
To test this setup, practice put calls to the server using `mc` and use `mc ls` on
the data directory to view the size of the object.
## Explore Further
- [Use `mc` with MinIO Server](https://docs.min.io/community/minio-object-store/reference/minio-mc.html)
- [Use `aws-cli` with MinIO Server](https://docs.min.io/community/minio-object-store/integrations/aws-cli-with-minio.html)
- [Use `minio-go` SDK with MinIO Server](https://docs.min.io/community/minio-object-store/developers/go/minio-go.html)
- [The MinIO documentation website](https://docs.min.io/community/minio-object-store/index.html)

340
docs/config/README.md Normal file
View File

@@ -0,0 +1,340 @@
# MinIO Server Config Guide [![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io) [![Docker Pulls](https://img.shields.io/docker/pulls/minio/minio.svg?maxAge=604800)](https://hub.docker.com/r/minio/minio/)
## Configuration Directory
MinIO stores all its config as part of the server deployment, config is erasure coded on MinIO. On a fresh deployment MinIO automatically generates a new `config` and this config is available to be configured via `mc admin config` command. MinIO also encrypts all the config, IAM and policies content if KMS is configured. Please refer to how to encrypt your config and IAM credentials [here](https://github.com/minio/minio/blob/master/docs/kms/IAM.md).
### Certificate Directory
TLS certificates by default are expected to be stored under ``${HOME}/.minio/certs`` directory. You need to place certificates here to enable `HTTPS` based access. Read more about [How to secure access to MinIO server with TLS](https://docs.min.io/community/minio-object-store/operations/network-encryption.html).
Following is a sample directory structure for MinIO server with TLS certificates.
```sh
$ mc tree --files ~/.minio
/home/user1/.minio
└─ certs
├─ CAs
├─ private.key
└─ public.crt
```
You can provide a custom certs directory using `--certs-dir` command line option.
#### Credentials
On MinIO admin credentials or root credentials are only allowed to be changed using ENVs namely `MINIO_ROOT_USER` and `MINIO_ROOT_PASSWORD`.
```sh
export MINIO_ROOT_USER=minio
export MINIO_ROOT_PASSWORD=minio13
minio server /data
```
#### Site
```
KEY:
site label the server and its location
ARGS:
name (string) name for the site e.g. "cal-rack0"
region (string) name of the location of the server e.g. "us-west-1"
comment (sentence) optionally add a comment to this setting
```
or environment variables
```
KEY:
site label the server and its location
ARGS:
MINIO_SITE_NAME (string) name for the site e.g. "cal-rack0"
MINIO_SITE_REGION (string) name of the location of the server e.g. "us-west-1"
MINIO_SITE_COMMENT (sentence) optionally add a comment to this setting
```
Example:
```sh
export MINIO_SITE_REGION="us-west-0"
export MINIO_SITE_NAME="sfo-rack-1"
minio server /data
```
### Storage Class
By default, parity for objects with standard storage class is set to `N/2`, and parity for objects with reduced redundancy storage class objects is set to `2`. Read more about storage class support in MinIO server [here](https://github.com/minio/minio/blob/master/docs/erasure/storage-class/README.md).
```
KEY:
storage_class define object level redundancy
ARGS:
standard (string) set the parity count for default standard storage class e.g. "EC:4"
rrs (string) set the parity count for reduced redundancy storage class e.g. "EC:2"
comment (sentence) optionally add a comment to this setting
```
or environment variables
```
KEY:
storage_class define object level redundancy
ARGS:
MINIO_STORAGE_CLASS_STANDARD (string) set the parity count for default standard storage class e.g. "EC:4"
MINIO_STORAGE_CLASS_RRS (string) set the parity count for reduced redundancy storage class e.g. "EC:2"
MINIO_STORAGE_CLASS_COMMENT (sentence) optionally add a comment to this setting
```
#### Etcd
MinIO supports storing encrypted IAM assets in etcd, if KMS is configured. Please refer to how to encrypt your config and IAM credentials [here](https://github.com/minio/minio/blob/master/docs/kms/IAM.md).
> NOTE: if *path_prefix* is set then MinIO will not federate your buckets, namespaced IAM assets are assumed as isolated tenants, only buckets are considered globally unique but performing a lookup with a *bucket* which belongs to a different tenant will fail unlike federated setups where MinIO would port-forward and route the request to relevant cluster accordingly. This is a special feature, federated deployments should not need to set *path_prefix*.
```
KEY:
etcd federate multiple clusters for IAM and Bucket DNS
ARGS:
endpoints* (csv) comma separated list of etcd endpoints e.g. "http://localhost:2379"
path_prefix (path) namespace prefix to isolate tenants e.g. "customer1/"
coredns_path (path) shared bucket DNS records, default is "/skydns"
client_cert (path) client cert for mTLS authentication
client_cert_key (path) client cert key for mTLS authentication
comment (sentence) optionally add a comment to this setting
```
or environment variables
```
KEY:
etcd federate multiple clusters for IAM and Bucket DNS
ARGS:
MINIO_ETCD_ENDPOINTS* (csv) comma separated list of etcd endpoints e.g. "http://localhost:2379"
MINIO_ETCD_PATH_PREFIX (path) namespace prefix to isolate tenants e.g. "customer1/"
MINIO_ETCD_COREDNS_PATH (path) shared bucket DNS records, default is "/skydns"
MINIO_ETCD_CLIENT_CERT (path) client cert for mTLS authentication
MINIO_ETCD_CLIENT_CERT_KEY (path) client cert key for mTLS authentication
MINIO_ETCD_COMMENT (sentence) optionally add a comment to this setting
```
### API
By default, there is no limitation on the number of concurrent requests that a server/cluster processes at the same time. However, it is possible to impose such limitation using the API subsystem. Read more about throttling limitation in MinIO server [here](https://github.com/minio/minio/blob/master/docs/throttle/README.md).
```
KEY:
api manage global HTTP API call specific features, such as throttling, authentication types, etc.
ARGS:
requests_max (number) set the maximum number of concurrent requests (default: 'auto')
cluster_deadline (duration) set the deadline for cluster readiness check (default: '10s')
cors_allow_origin (csv) set comma separated list of origins allowed for CORS requests (default: '*')
remote_transport_deadline (duration) set the deadline for API requests on remote transports while proxying between federated instances e.g. "2h" (default: '2h')
list_quorum (string) set the acceptable quorum expected for list operations e.g. "optimal", "reduced", "disk", "strict", "auto" (default: 'strict')
replication_priority (string) set replication priority (default: 'auto')
replication_max_workers (number) set the maximum number of replication workers (default: '500')
replication_max_lrg_workers (number) set the maximum number of replication workers MinIO uses to replicate large objects between sites. (default: '10')
transition_workers (number) set the number of transition workers (default: '100')
stale_uploads_expiry (duration) set to expire stale multipart uploads older than this values (default: '24h')
stale_uploads_cleanup_interval (duration) set to change intervals when stale multipart uploads are expired (default: '6h')
delete_cleanup_interval (duration) set to change intervals when deleted objects are permanently deleted from ".trash" folder (default: '5m')
odirect (boolean) set to enable or disable O_DIRECT for writes under special conditions. NOTE: do not disable O_DIRECT without prior testing (default: 'on')
root_access (boolean) turn 'off' root credential access for all API calls including s3, admin operations (default: 'on')
sync_events (boolean) set to enable synchronous bucket notifications (default: 'off')
object_max_versions (number) set max allowed number of versions per object (default: '9223372036854775807')
```
or environment variables
```
MINIO_API_REQUESTS_MAX (number) set the maximum number of concurrent requests (default: 'auto')
MINIO_API_CLUSTER_DEADLINE (duration) set the deadline for cluster readiness check (default: '10s')
MINIO_API_CORS_ALLOW_ORIGIN (csv) set comma separated list of origins allowed for CORS requests (default: '*')
MINIO_API_REMOTE_TRANSPORT_DEADLINE (duration) set the deadline for API requests on remote transports while proxying between federated instances e.g. "2h" (default: '2h')
MINIO_API_LIST_QUORUM (string) set the acceptable quorum expected for list operations e.g. "optimal", "reduced", "disk", "strict", "auto" (default: 'strict')
MINIO_API_REPLICATION_PRIORITY (string) set replication priority (default: 'auto')
MINIO_API_REPLICATION_MAX_WORKERS (number) set the maximum number of replication workers (default: '500')
MINIO_API_TRANSITION_WORKERS (number) set the number of transition workers (default: '100')
MINIO_API_STALE_UPLOADS_EXPIRY (duration) set to expire stale multipart uploads older than this values (default: '24h')
MINIO_API_STALE_UPLOADS_CLEANUP_INTERVAL (duration) set to change intervals when stale multipart uploads are expired (default: '6h')
MINIO_API_DELETE_CLEANUP_INTERVAL (duration) set to change intervals when deleted objects are permanently deleted from ".trash" folder (default: '5m')
MINIO_API_ODIRECT (boolean) set to enable or disable O_DIRECT for writes under special conditions. NOTE: do not disable O_DIRECT without prior testing (default: 'on')
MINIO_API_ROOT_ACCESS (boolean) turn 'off' root credential access for all API calls including s3, admin operations (default: 'on')
MINIO_API_SYNC_EVENTS (boolean) set to enable synchronous bucket notifications (default: 'off')
MINIO_API_OBJECT_MAX_VERSIONS (number) set max allowed number of versions per object (default: '9223372036854775807')
```
#### Notifications
Notification targets supported by MinIO are in the following list. To configure individual targets please refer to more detailed documentation [here](https://docs.min.io/community/minio-object-store/administration/monitoring.html#bucket-notifications).
```
notify_webhook publish bucket notifications to webhook endpoints
notify_amqp publish bucket notifications to AMQP endpoints
notify_kafka publish bucket notifications to Kafka endpoints
notify_mqtt publish bucket notifications to MQTT endpoints
notify_nats publish bucket notifications to NATS endpoints
notify_nsq publish bucket notifications to NSQ endpoints
notify_mysql publish bucket notifications to MySQL databases
notify_postgres publish bucket notifications to Postgres databases
notify_elasticsearch publish bucket notifications to Elasticsearch endpoints
notify_redis publish bucket notifications to Redis datastores
```
### Accessing configuration
All configuration changes can be made using [`mc admin config` get/set/reset/export/import commands](https://github.com/minio/mc/blob/master/docs/minio-admin-complete-guide.md).
#### List all config keys available
```
~ mc admin config set myminio/
```
#### Obtain help for each key
```
~ mc admin config set myminio/ <key>
```
e.g: `mc admin config set myminio/ etcd` returns available `etcd` config args
```
~ mc admin config set play/ etcd
KEY:
etcd federate multiple clusters for IAM and Bucket DNS
ARGS:
endpoints* (csv) comma separated list of etcd endpoints e.g. "http://localhost:2379"
path_prefix (path) namespace prefix to isolate tenants e.g. "customer1/"
coredns_path (path) shared bucket DNS records, default is "/skydns"
client_cert (path) client cert for mTLS authentication
client_cert_key (path) client cert key for mTLS authentication
comment (sentence) optionally add a comment to this setting
```
To get ENV equivalent for each config args use `--env` flag
```
~ mc admin config set play/ etcd --env
KEY:
etcd federate multiple clusters for IAM and Bucket DNS
ARGS:
MINIO_ETCD_ENDPOINTS* (csv) comma separated list of etcd endpoints e.g. "http://localhost:2379"
MINIO_ETCD_PATH_PREFIX (path) namespace prefix to isolate tenants e.g. "customer1/"
MINIO_ETCD_COREDNS_PATH (path) shared bucket DNS records, default is "/skydns"
MINIO_ETCD_CLIENT_CERT (path) client cert for mTLS authentication
MINIO_ETCD_CLIENT_CERT_KEY (path) client cert key for mTLS authentication
MINIO_ETCD_COMMENT (sentence) optionally add a comment to this setting
```
This behavior is consistent across all keys; each key self-documents itself with valid examples.
## Dynamic systems without restarting server
The following sub-systems are dynamic i.e., configuration parameters for each sub-systems can be changed while the server is running without any restarts.
```
api manage global HTTP API call specific features, such as throttling, authentication types, etc.
heal manage object healing frequency and bitrot verification checks
scanner manage namespace scanning for usage calculation, lifecycle, healing and more
```
> NOTE: if you set any of the following sub-system configuration using ENVs, dynamic behavior is not supported.
### Usage scanner
Data usage scanner is enabled by default. The following configuration settings allow for more staggered delay in terms of usage calculation. The scanner adapts to the system speed and completely pauses when the system is under load. It is possible to adjust the speed of the scanner and thereby the latency of updates being reflected. The delays between each operation of the scanner can be adjusted by the `mc admin config set alias/ delay=15.0`. By default the value is `10.0`. This means the scanner will sleep *10x* the time each operation takes.
In most setups this will keep the scanner slow enough to not impact overall system performance. Setting the `delay` key to a *lower* value will make the scanner faster and setting it to 0 will make the scanner run at full speed (not recommended in production). Setting it to a higher value will make the scanner slower, consuming less resources with the trade off of not collecting metrics for operations like healing and disk usage as fast.
```
~ mc admin config set alias/ scanner
KEY:
scanner manage namespace scanning for usage calculation, lifecycle, healing and more
ARGS:
delay (float) scanner delay multiplier, defaults to '10.0'
max_wait (duration) maximum wait time between operations, defaults to '15s'
cycle (duration) time duration between scanner cycles
```
Example: the following setting will decrease the scanner speed by a factor of 3, reducing the system resource use, but increasing the latency of updates being reflected.
```sh
~ mc admin config set alias/ scanner delay=30.0
```
Once set the scanner settings are automatically applied without the need for server restarts.
### Healing
Healing is enabled by default. The following configuration settings allow for more staggered delay in terms of healing. The healing system by default adapts to the system speed and pauses up to '250ms' per object when the system has `max_io` number of concurrent requests. It is possible to adjust the `max_sleep` and `max_io` values thereby increasing the healing speed. The delays between each operation of the healer can be adjusted by the `mc admin config set alias/ heal max_sleep=1s` and maximum concurrent requests allowed before we start slowing things down can be configured with `mc admin config set alias/ heal max_io=30` . By default the wait delay is `250ms` beyond 100 concurrent operations. This means the healer will sleep *250 milliseconds* at max for each heal operation if there are more than *100* concurrent client requests.
In most setups this is sufficient to heal the content after drive replacements. Setting `max_sleep` to a *lower* value and setting `max_io` to a *higher* value would make heal go faster.
Each node is responsible of healing its local drives; Each drive will have multiple heal workers which is the quarter of the number of CPU cores of the node or the quarter of the configured nr_requests of the drive (https://www.kernel.org/doc/Documentation/block/queue-sysfs.txt). It is also possible to provide a custom number of workers by using this command: `mc admin config set alias/ heal drive_workers=100` .
```
~ mc admin config set alias/ heal
KEY:
heal manage object healing frequency and bitrot verification checks
ARGS:
bitrotscan (on|off) perform bitrot scan on drives when checking objects during scanner
max_sleep (duration) maximum sleep duration between objects to slow down heal operation. eg. 2s
max_io (int) maximum IO requests allowed between objects to slow down heal operation. eg. 3
drive_workers (int) the number of workers per drive to heal a new disk replacement.
```
Example: The following settings will increase the heal operation speed by allowing healing operation to run without delay up to `100` concurrent requests, and the maximum delay between each heal operation is set to `300ms`.
```sh
~ mc admin config set alias/ heal max_sleep=300ms max_io=100
```
Once set the healer settings are automatically applied without the need for server restarts.
## Environment only settings (not in config)
### Browser
Enable or disable access to console web UI. By default it is set to `on`. You may override this field with `MINIO_BROWSER` environment variable.
Example:
```sh
export MINIO_BROWSER=off
minio server /data
```
### Domain
By default, MinIO supports path-style requests that are of the format <http://mydomain.com/bucket/object>. `MINIO_DOMAIN` environment variable is used to enable virtual-host-style requests. If the request `Host` header matches with `(.+).mydomain.com` then the matched pattern `$1` is used as bucket and the path is used as object. Read more about path-style and virtual-host-style [here](http://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAPI.html).
Example:
```sh
export MINIO_DOMAIN=mydomain.com
minio server /data
```
For advanced use cases `MINIO_DOMAIN` environment variable supports multiple-domains with comma separated values.
```sh
export MINIO_DOMAIN=sub1.mydomain.com,sub2.mydomain.com
minio server /data
```
## Explore Further
* [MinIO Quickstart Guide](https://docs.min.io/community/minio-object-store/operations/deployments/baremetal-deploy-minio-on-redhat-linux.html)
* [Configure MinIO Server with TLS](https://docs.min.io/community/minio-object-store/operations/network-encryption.html)

138
docs/debugging/README.md Normal file
View File

@@ -0,0 +1,138 @@
# MinIO Server Debugging Guide [![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io) [![Docker Pulls](https://img.shields.io/docker/pulls/minio/minio.svg?maxAge=604800)](https://hub.docker.com/r/minio/minio/)
## HTTP Trace
HTTP tracing can be enabled by using [`mc admin trace`](https://docs.min.io/community/minio-object-store/reference/minio-mc-admin/mc-admin-trace.html) command.
Example:
```sh
minio server /data
```
Default trace is succinct only to indicate the API operations being called and the HTTP response status.
```sh
mc admin trace myminio
```
To trace entire HTTP request
```sh
mc admin trace --verbose myminio
```
To trace entire HTTP request and also internode communication
```sh
mc admin trace --all --verbose myminio
```
## Subnet Health
Subnet Health diagnostics help ensure that the underlying infrastructure that runs MinIO is configured correctly, and is functioning properly. This test is one-shot long running one, that is recommended to be run as soon as the cluster is first provisioned, and each time a failure scenario is encountered. Note that the test incurs majority of the available resources on the system. Care must be taken when using this to debug failure scenario, so as to prevent larger outages. Health tests can be triggered using `mc support diagnostics` command.
Example:
```sh
minio server /data{1...4}
```
The command takes no flags
```sh
mc support diagnostics myminio/
```
The output printed will be of the form
```sh
● Admin Info ... ✔
● CPU ... ✔
● Disk Hardware ... ✔
● Os Info ... ✔
● Mem Info ... ✔
● Process Info ... ✔
● Config ... ✔
● Drive ... ✔
● Net ... ✔
*********************************************************************************
WARNING!!
** THIS FILE MAY CONTAIN SENSITIVE INFORMATION ABOUT YOUR ENVIRONMENT **
** PLEASE INSPECT CONTENTS BEFORE SHARING IT ON ANY PUBLIC FORUM **
*********************************************************************************
mc: Health data saved to dc-11-health_20200321053323.json.gz
```
The gzipped output contains debugging information for your system
## Decoding Metadata
Metadata is stored in `xl.meta` files for erasure coded objects. Each disk in the set containing the object has this file. The file format is a binary format and therefore requires tools to view values.
### Installing xl-meta
To install, [Go](https://golang.org/dl/) must be installed. Once installed, execute this to install the binary:
```bash
go install github.com/minio/minio/docs/debugging/xl-meta@latest
```
### Using xl-meta
Executing `xl-meta` will look for an `xl.meta` in the current folder and decode it to JSON. It is also possible to specify multiple files or wildcards, for example `xl-meta ./**/xl.meta` will output decoded metadata recursively. It is possible to view what inline data is stored inline in the metadata using `--data` parameter `xl-meta -data xl.json` will display an id -> data size. To export inline data to a file use the `--export` option.
### Remotely Inspecting backend data
`mc support inspect` allows collecting files based on *path* from all backend drives. Matching files will be collected in a zip file with their respective host+drive+path. A MinIO host from October 2021 or later is required for full functionality. Syntax is `mc support inspect ALIAS/path/to/files`. This can for example be used to collect `xl.meta` from objects that are misbehaving. To collect `xl.meta` from a specific object, for example placed at `ALIAS/bucket/path/to/file.txt` append `/xl.meta`, for instance `mc support inspect ALIAS/bucket/path/to/file.txt/xl.meta`. All files can be collected, so this can also be used to retrieve `part.*` files, etc.
Wildcards can be used, for example `mc support inspect ALIAS/bucket/path/**/xl.meta` will collect all `xl.meta` recursively. `mc support inspect ALIAS/bucket/path/to/file.txt/*/part.*` will collect parts for all versions for the object located at `bucket/path/to/file.txt`.
`xl-meta` accepts zip files as input and will output all `xl.meta` files found within the archive. For example:
```
$ mc support inspect play/test123/test*/xl.meta
mc: File data successfully downloaded as inspect.6f96b336.zip
$ xl-meta inspect.6f96b336.zip
{
"bf6178f9-4014-4008-9699-86f2fac62226/test123/testw3c.pdf/xl.meta": {"Versions":[{"Type":1,"V2Obj":{"ID":"aGEA/ZUOR4ueRIZsAgfDqA==","DDir":"9MMwM47bS+K6KvQqN3hlDw==","EcAlgo":1,"EcM":2,"EcN":2,"EcBSize":1048576,"EcIndex":4,"EcDist":[4,1,2,3],"CSumAlgo":1,"PartNums":[1],"PartETags":[""],"PartSizes":[101974],"PartASizes":[176837],"Size":101974,"MTime":1634106631319256439,"MetaSys":{"X-Minio-Internal-compression":"a2xhdXNwb3N0L2NvbXByZXNzL3My","X-Minio-Internal-actual-size":"MTc2ODM3","x-minio-internal-objectlock-legalhold-timestamp":"MjAyMS0xMC0xOVQyMjozNTo0Ni4zNTE4MDU3NTda"},"MetaUsr":{"x-amz-object-lock-mode":"COMPLIANCE","x-amz-object-lock-retain-until-date":"2022-10-13T06:30:31.319Z","etag":"67ed8f49b7137cb957858ce468f2e79e","content-type":"application/pdf","x-amz-object-lock-legal-hold":"OFF"}}}]},
"fe012443-6ba9-4ef2-bb94-b729d2060c78/test123/testw3c.pdf/xl.meta": {"Versions":[{"Type":1,"V2Obj":{"ID":"aGEA/ZUOR4ueRIZsAgfDqA==","DDir":"9MMwM47bS+K6KvQqN3hlDw==","EcAlgo":1,"EcM":2,"EcN":2,"EcBSize":1048576,"EcIndex":1,"EcDist":[4,1,2,3],"CSumAlgo":1,"PartNums":[1],"PartETags":[""],"PartSizes":[101974],"PartASizes":[176837],"Size":101974,"MTime":1634106631319256439,"MetaSys":{"X-Minio-Internal-compression":"a2xhdXNwb3N0L2NvbXByZXNzL3My","X-Minio-Internal-actual-size":"MTc2ODM3","x-minio-internal-objectlock-legalhold-timestamp":"MjAyMS0xMC0xOVQyMjozNTo0Ni4zNTE4MDU3NTda"},"MetaUsr":{"content-type":"application/pdf","x-amz-object-lock-legal-hold":"OFF","x-amz-object-lock-mode":"COMPLIANCE","x-amz-object-lock-retain-until-date":"2022-10-13T06:30:31.319Z","etag":"67ed8f49b7137cb957858ce468f2e79e"}}}]},
"5dcb9f38-08ea-4728-bb64-5cecc7102436/test123/testw3c.pdf/xl.meta": {"Versions":[{"Type":1,"V2Obj":{"ID":"aGEA/ZUOR4ueRIZsAgfDqA==","DDir":"9MMwM47bS+K6KvQqN3hlDw==","EcAlgo":1,"EcM":2,"EcN":2,"EcBSize":1048576,"EcIndex":2,"EcDist":[4,1,2,3],"CSumAlgo":1,"PartNums":[1],"PartETags":[""],"PartSizes":[101974],"PartASizes":[176837],"Size":101974,"MTime":1634106631319256439,"MetaSys":{"X-Minio-Internal-compression":"a2xhdXNwb3N0L2NvbXByZXNzL3My","X-Minio-Internal-actual-size":"MTc2ODM3","x-minio-internal-objectlock-legalhold-timestamp":"MjAyMS0xMC0xOVQyMjozNTo0Ni4zNTE4MDU3NTda"},"MetaUsr":{"content-type":"application/pdf","x-amz-object-lock-legal-hold":"OFF","x-amz-object-lock-mode":"COMPLIANCE","x-amz-object-lock-retain-until-date":"2022-10-13T06:30:31.319Z","etag":"67ed8f49b7137cb957858ce468f2e79e"}}}]},
"48beacc7-4be0-4660-9026-4eceaf147504/test123/testw3c.pdf/xl.meta": {"Versions":[{"Type":1,"V2Obj":{"ID":"aGEA/ZUOR4ueRIZsAgfDqA==","DDir":"9MMwM47bS+K6KvQqN3hlDw==","EcAlgo":1,"EcM":2,"EcN":2,"EcBSize":1048576,"EcIndex":3,"EcDist":[4,1,2,3],"CSumAlgo":1,"PartNums":[1],"PartETags":[""],"PartSizes":[101974],"PartASizes":[176837],"Size":101974,"MTime":1634106631319256439,"MetaSys":{"X-Minio-Internal-compression":"a2xhdXNwb3N0L2NvbXByZXNzL3My","X-Minio-Internal-actual-size":"MTc2ODM3","x-minio-internal-objectlock-legalhold-timestamp":"MjAyMS0xMC0xOVQyMjozNTo0Ni4zNTE4MDU3NTda"},"MetaUsr":{"x-amz-object-lock-retain-until-date":"2022-10-13T06:30:31.319Z","x-amz-object-lock-legal-hold":"OFF","etag":"67ed8f49b7137cb957858ce468f2e79e","content-type":"application/pdf","x-amz-object-lock-mode":"COMPLIANCE"}}}]}
}
```
Optionally `--encrypt` can be specified. This will output an encrypted file and a decryption key:
```
$ mc support inspect --encrypt play/test123/test*/*/part.*
mc: Encrypted file data successfully downloaded as inspect.ad2b43d8.enc
mc: Decryption key: ad2b43d847fdb14e54c5836200177f7158b3f745433525f5d23c0e0208e50c9948540b54
mc: The decryption key will ONLY be shown here. It cannot be recovered.
mc: The encrypted file can safely be shared without the decryption key.
mc: Even with the decryption key, data stored with encryption cannot be accessed.
```
This file can be decrypted using the decryption tool below:
### Installing decryption tool
To install, [Go](https://golang.org/dl/) must be installed.
Once installed, execute this to install the binary:
```bash
go install github.com/minio/minio/docs/debugging/inspect@latest
```
### Usage
To decrypt the file above:
```
$ inspect -key=ad2b43d847fdb14e54c5836200177f7158b3f745433525f5d23c0e0208e50c9948540b54 inspect.ad2b43d8.enc
Output decrypted to inspect.ad2b43d8.zip
```
If `--key` is not specified an interactive prompt will ask for it. The file name will contain the beginning of the key. This can be used to verify that the key is for the encrypted file.

7
docs/debugging/build.sh Executable file
View File

@@ -0,0 +1,7 @@
#!/bin/bash
export CGO_ENABLED=0
for dir in docs/debugging/*/; do
bin=$(basename ${dir})
go build -C ${dir} -o ${PWD}/${bin}
done

91
docs/distributed/CONFIG.md Normal file
View File

@@ -0,0 +1,91 @@
## MinIO configuration YAML
MinIO now supports starting the server arguments and configuration via a YAML configuration file. This YAML configuration describes everything that can be configured in a MinIO setup, such as '--address', '--console-address' and command line arguments for the MinIO server.
Historically everything to MinIO was provided via command arguments for the hostnames and the drives via an ellipses syntax such as `minio server http://host{1...4}/disk{1...4}` this requirement added an additional burden to have sequential hostnames for us to make sure that we can provide horizontal distribution, however we have come across situations where sometimes this is not feasible and there are no easier alternatives without modifying /etc/hosts on the host system as root user. Many times in airgapped deployments this is not allowed or requires audits and approvals.
MinIO server configuration file allows users to provide topology that allows for heterogeneous hostnames, allowing MinIO to deployed in pre-existing environments without any further OS level configurations.
### Usage
```
minio server --config config.yaml
```
Lets you start MinIO server with all inputs to start MinIO server provided via this configuration file, once the configuration file is provided all other pre-existing values on disk for configuration are overridden by the new values set in this configuration file.
Following is an example YAML configuration structure.
```yaml
version: v2
address: ":9000"
rootUser: "minioadmin"
rootPassword: "minioadmin"
console-address: ":9001"
certs-dir: "/home/user/.minio/certs/"
pools: # Specify the nodes and drives with pools
- args:
- "https://server-example-pool1:9000/mnt/disk{1...4}/"
- "https://server{1...2}-pool1:9000/mnt/disk{1...4}/"
- "https://server3-pool1:9000/mnt/disk{1...4}/"
- "https://server4-pool1:9000/mnt/disk{1...4}/"
- args:
- "https://server-example-pool2:9000/mnt/disk{1...4}/"
- "https://server{1...2}-pool2:9000/mnt/disk{1...4}/"
- "https://server3-pool2:9000/mnt/disk{1...4}/"
- "https://server4-pool2:9000/mnt/disk{1...4}/"
# more args
options:
ftp: # settings for MinIO to act as an ftp server
address: ":8021"
passive-port-range: "30000-40000"
sftp: # settings for MinIO to act as an sftp server
address: ":8022"
ssh-private-key: "/home/user/.ssh/id_rsa"
```
If you are using the config `v1` YAML you should migrate your `pools:` field values to the following format
`v1` format
```yaml
pools: # Specify the nodes and drives with pools
-
- "https://server-example-pool1:9000/mnt/disk{1...4}/"
- "https://server{1...2}-pool1:9000/mnt/disk{1...4}/"
- "https://server3-pool1:9000/mnt/disk{1...4}/"
- "https://server4-pool1:9000/mnt/disk{1...4}/"
```
to `v2` format
```yaml
pools:
- args:
- "https://server-example-pool1:9000/mnt/disk{1...4}/"
- "https://server{1...2}-pool1:9000/mnt/disk{1...4}/"
- "https://server3-pool1:9000/mnt/disk{1...4}/"
- "https://server4-pool1:9000/mnt/disk{1...4}/"
set-drive-count: 4 # Advanced option, must be used under guidance from MinIO team.
```
### Things to know
- Fields such as `version` and `pools` are mandatory, however all other fields are optional.
- Each pool expects a minimum of 2 nodes per pool, and unique non-repeating hosts for each argument.
- Each pool expects each host in this pool has the same number of drives specified as any other host.
- Mixing `local-path` and `distributed-path` is not allowed, doing so would cause MinIO to refuse starting the server.
- Ellipses and bracket notation (e.g. `{1...10}`) are allowed.
> NOTE: MinIO environmental variables still take precedence over the `config.yaml` file, however `config.yaml` is preferred over MinIO internal config KV settings via `mc admin config set alias/ <sub-system>`.
### TODO
In subsequent releases we are planning to extend this to provide things like
- Reload() of MinIO server arguments without fully restarting the process.
- Expanding 1 node at a time by automating the process of creating a new pool
and decommissioning to provide a functionality that smaller deployments
care about.
- Fully allow bracket notation (e.g. `{a,c,f}`) to have multiple entries on one line.

122
docs/distributed/DECOMMISSION.md Normal file
View File

@@ -0,0 +1,122 @@
# Decommissioning
Decommissiong is a mechanism in MinIO to drain older pools (usually with old hardware) and migrate the content from such pools to a newer pools (usually better hardware). Decommissioning spreads the data across all pools - for example, if you decommission `pool1`, all the data from `pool1` spreads across `pool2` and `pool3`.
## Features
- A pool in decommission still allows READ access to all its contents, newer WRITEs will automatically be scheduled to only pools not in decommission status.
- All versioned buckets maintain the same order for "versions" for each object after being decommissioned to the other pools.
- A pool interrupted during the decommission process, such as for a cluster restart, resumes from where it left off.
## How to decommission a pool
```
λ mc admin decommission start alias/ http://minio{1...2}/data{1...4}
```
## Status decommissioning a pool
### Decommissioning without args lists all pools
```
λ mc admin decommission status alias/
┌─────┬─────────────────────────────────┬──────────────────────────────────┬────────┐
│ ID │ Pools │ Capacity │ Status │
│ 1st │ http://minio{1...2}/data{1...4} │ 439 GiB (used) / 561 GiB (total) │ Active │
│ 2nd │ http://minio{3...4}/data{1...4} │ 329 GiB (used) / 421 GiB (total) │ Active │
└─────┴─────────────────────────────────┴──────────────────────────────────┴────────┘
```
### Decommissioning status
```
λ mc admin decommission status alias/ http://minio{1...2}/data{1...4}
Decommissioning rate at 36 MiB/sec [4 TiB/50 TiB]
Started: 1 minute ago
```
Once it is **Complete**
```
λ mc admin decommission status alias/ http://minio{1...2}/data{1...4}
Decommission of pool http://minio{1...2}/data{1...4} is complete, you may now remove it from server command line
```
### A pool not under decommissioning will throw an error
```
λ mc admin decommission status alias/ http://minio{1...2}/data{1...4}
ERROR: This pool is not scheduled for decommissioning currently.
```
## Canceling a decommission
Stop an on-going decommission in progress, mainly used in situations when the load may be too high and you may want to schedule the decommission at a later point in time.
`mc admin decommission cancel` without an argument, lists out any on-going decommission in progress.
```
λ mc admin decommission cancel alias/
┌─────┬─────────────────────────────────┬──────────────────────────────────┬──────────┐
│ ID │ Pools │ Capacity │ Status │
│ 1st │ http://minio{1...2}/data{1...4} │ 439 GiB (used) / 561 GiB (total) │ Draining │
└─────┴─────────────────────────────────┴──────────────────────────────────┴──────────┘
```
> NOTE: Canceled decommission will not make the pool active again, since we might have potentially partial namespace on the other pools, to avoid this scenario be absolutely sure to make decommissioning a planned well thought activity. This is not to be run on a daily basis.
```
λ mc admin decommission cancel alias/ http://minio{1...2}/data{1...4}
┌─────┬─────────────────────────────────┬──────────────────────────────────┬────────────────────┐
│ ID │ Pools │ Capacity │ Status │
│ 1st │ http://minio{1...2}/data{1...4} │ 439 GiB (used) / 561 GiB (total) │ Draining(Canceled) │
└─────┴─────────────────────────────────┴──────────────────────────────────┴────────────────────┘
```
If the decommission process fails for any reason, the status indicates failed.
```
λ mc admin decommission status alias/
┌─────┬─────────────────────────────────┬──────────────────────────────────┬──────────────────┐
│ ID │ Pools │ Capacity │ Status │
│ 1st │ http://minio{1...2}/data{1...4} │ 439 GiB (used) / 561 GiB (total) │ Draining(Failed) │
│ 2nd │ http://minio{3...4}/data{1...4} │ 329 GiB (used) / 421 GiB (total) │ Active │
└─────┴─────────────────────────────────┴──────────────────────────────────┴──────────────────┘
```
## Restart a canceled or failed decommission
```
λ mc admin decommission start alias/ http://minio{1...2}/data{1...4}
```
## When decommission is 'Complete'
Once decommission is complete, it will be indicated with *Complete* status. *Complete* means that now you can now safely remove the first pool argument from the MinIO command line.
```
λ mc admin decommission status alias/
┌─────┬─────────────────────────────────┬──────────────────────────────────┬──────────┐
│ ID │ Pools │ Capacity │ Status │
│ 1st │ http://minio{1...2}/data{1...4} │ 439 GiB (used) / 561 GiB (total) │ Complete │
│ 2nd │ http://minio{3...4}/data{1...4} │ 329 GiB (used) / 421 GiB (total) │ Active │
└─────┴─────────────────────────────────┴──────────────────────────────────┴──────────┘
```
- On baremetal setups, if you have `MINIO_VOLUMES="http://minio{1...2}/data{1...4} http://minio{3...4}/data{1...4}"`, you can remove the first argument `http://minio{1...2}/data{1...4}` to update your `MINIO_VOLUMES` setting, then restart all the servers in the setup in parallel using `systemctl restart minio`.
- On Kubernetes setups, the statefulset specification needs to be modified by changing the command line input for the MinIO container. Once the relevant changes are done, proceed to execute `kubectl apply -f statefulset.yaml`.
- On Operator based MinIO deployments, you need to modify the `tenant.yaml` specification and modify the `pools:` section from two entries to a single entry. After making relevant changes, proceed to execute `kubectl apply -f tenant.yaml`.
> Without a 'Complete' status any 'Active' or 'Draining' pool(s) are not allowed to be removed once configured.
## NOTE
- Empty delete markers (such as for objects with no other successor versions) do not transition to the new pool to avoid creating empty metadata on the other pool(s). If you believe transitioning empty delete markers is required, open a GitHub issue.
## TODO
- Richer progress UI is not present at the moment, this will be addressed in subsequent releases. Currently however a RATE of data transfer and usage increase is displayed via `mc`.
- Transitioned Hot Tier's as pooled setups are not currently supported, attempting to decommission buckets with ILM Transition will be rejected by the server. This will be supported in future releases.
- Embedded Console UI does not support Decommissioning through the UI yet. This will be supported in future releases.

166
docs/distributed/DESIGN.md Normal file
View File

@@ -0,0 +1,166 @@
# Distributed Server Design Guide [![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io)
This document explains the design, architecture and advanced use cases of the MinIO distributed server.
## Command-line
```
NAME:
minio server - start object storage server
USAGE:
minio server [FLAGS] DIR1 [DIR2..]
minio server [FLAGS] DIR{1...64}
minio server [FLAGS] DIR{1...64} DIR{65...128}
DIR:
DIR points to a directory on a filesystem. When you want to combine
multiple drives into a single large system, pass one directory per
filesystem separated by space. You may also use a '...' convention
to abbreviate the directory arguments. Remote directories in a
distributed setup are encoded as HTTP(s) URIs.
```
## Common usage
Standalone erasure coded configuration with 4 sets with 16 drives each.
```
minio server dir{1...64}
```
Distributed erasure coded configuration with 64 sets with 16 drives each.
```
minio server http://host{1...16}/export{1...64}
```
## Architecture
Expansion of ellipses and choice of erasure sets based on this expansion is an automated process in MinIO. Here are some of the details of our underlying erasure coding behavior.
- Erasure coding used by MinIO is [Reed-Solomon](https://github.com/klauspost/reedsolomon) erasure coding scheme, which has a total shard maximum of 256 i.e 128 data and 128 parity. MinIO design goes beyond this limitation by doing some practical architecture choices.
- Erasure set is a single erasure coding unit within a MinIO deployment. An object is sharded within an erasure set. Erasure set size is automatically calculated based on the number of drives. MinIO supports unlimited number of drives but each erasure set can be up to 16 drives and a minimum of 2 drives.
- We limited the number of drives to 16 for erasure set because, erasure code shards more than 16 can become chatty and do not have any performance advantages. Additionally since 16 drive erasure set gives you tolerance of 8 drives per object by default which is plenty in any practical scenario.
- Choice of erasure set size is automatic based on the number of drives available, let's say for example if there are 32 servers and 32 drives which is a total of 1024 drives. In this scenario 16 becomes the erasure set size. This is decided based on the greatest common divisor (GCD) of acceptable erasure set sizes ranging from *4 to 16*.
- *If total drives has many common divisors the algorithm chooses the minimum amounts of erasure sets possible for a erasure set size of any N*. In the example with 1024 drives - 4, 8, 16 are GCD factors. With 16 drives we get a total of 64 possible sets, with 8 drives we get a total of 128 possible sets, with 4 drives we get a total of 256 possible sets. So algorithm automatically chooses 64 sets, which is *16* 64 = 1024* drives in total.
- *If total number of nodes are of odd number then GCD algorithm provides affinity towards odd number erasure sets to provide for uniform distribution across nodes*. This is to ensure that same number of drives are pariticipating in any erasure set. For example if you have 2 nodes with 180 drives then GCD is 15 but this would lead to uneven distribution, one of the nodes would participate more drives. To avoid this the affinity is given towards nodes which leads to next best GCD factor of 12 which provides uniform distribution.
- In this algorithm, we also make sure that we spread the drives out evenly. MinIO server expands ellipses passed as arguments. Here is a sample expansion to demonstrate the process.
```
minio server http://host{1...2}/export{1...8}
```
Expected expansion
```
> http://host1/export1
> http://host2/export1
> http://host1/export2
> http://host2/export2
> http://host1/export3
> http://host2/export3
> http://host1/export4
> http://host2/export4
> http://host1/export5
> http://host2/export5
> http://host1/export6
> http://host2/export6
> http://host1/export7
> http://host2/export7
> http://host1/export8
> http://host2/export8
```
*A noticeable trait of this expansion is that it chooses unique hosts such the setup provides maximum protection and availability.*
- Choosing an erasure set for the object is decided during `PutObject()`, object names are used to find the right erasure set using the following pseudo code.
```go
// hashes the key returning an integer.
func sipHashMod(key string, cardinality int, id [16]byte) int {
if cardinality <= 0 {
return -1
}
sip := siphash.New(id[:])
sip.Write([]byte(key))
return int(sip.Sum64() % uint64(cardinality))
}
```
Input for the key is the object name specified in `PutObject()`, returns a unique index. This index is one of the erasure sets where the object will reside. This function is a consistent hash for a given object name i.e for a given object name the index returned is always the same.
- Write and Read quorum are required to be satisfied only across the erasure set for an object. Healing is also done per object within the erasure set which contains the object.
- MinIO does erasure coding at the object level not at the volume level, unlike other object storage vendors. This allows applications to choose different storage class by setting `x-amz-storage-class=STANDARD/REDUCED_REDUNDANCY` for each object uploads so effectively utilizing the capacity of the cluster. Additionally these can also be enforced using IAM policies to make sure the client uploads with correct HTTP headers.
- MinIO also supports expansion of existing clusters in server pools. Each pool is a self contained entity with same SLA's (read/write quorum) for each object as original cluster. By using the existing namespace for lookup validation MinIO ensures conflicting objects are not created. When no such object exists then MinIO simply uses the least used pool to place new objects.
### There are no limits on how many server pools can be combined
```
minio server http://host{1...32}/export{1...32} http://host{1...12}/export{1...12}
```
In above example there are two server pools
- 32 * 32 = 1024 drives pool1
- 12 * 12 = 144 drives pool2
> Notice the requirement of common SLA here original cluster had 1024 drives with 16 drives per erasure set with default parity of '4', second pool is expected to have a minimum of 8 drives per erasure set to match the original cluster SLA (parity count) of '4'. '12' drives stripe per erasure set in the second pool satisfies the original pool's parity count.
Refer to the sizing guide with details on the default parity count chosen for different erasure stripe sizes [here](https://github.com/minio/minio/blob/master/docs/distributed/SIZING.md)
MinIO places new objects in server pools based on proportionate free space, per pool. Following pseudo code demonstrates this behavior.
```go
func getAvailablePoolIdx(ctx context.Context) int {
serverPools := z.getServerPoolsAvailableSpace(ctx)
total := serverPools.TotalAvailable()
// choose when we reach this many
choose := rand.Uint64() % total
atTotal := uint64(0)
for _, pool := range serverPools {
atTotal += pool.Available
if atTotal > choose && pool.Available > 0 {
return pool.Index
}
}
// Should not happen, but print values just in case.
panic(fmt.Errorf("reached end of serverPools (total: %v, atTotal: %v, choose: %v)", total, atTotal, choose))
}
```
## Other usages
### Advanced use cases with multiple ellipses
Standalone erasure coded configuration with 4 sets with 16 drives each, which spawns drives across controllers.
```
minio server /mnt/controller{1...4}/data{1...16}
```
Standalone erasure coded configuration with 16 sets, 16 drives per set, across mounts and controllers.
```
minio server /mnt{1...4}/controller{1...4}/data{1...16}
```
Distributed erasure coded configuration with 2 sets, 16 drives per set across hosts.
```
minio server http://host{1...32}/disk1
```
Distributed erasure coded configuration with rack level redundancy 32 sets in total, 16 drives per set.
```
minio server http://rack{1...4}-host{1...8}.example.net/export{1...16}
```

109
docs/distributed/README.md Normal file
View File

@@ -0,0 +1,109 @@
# Distributed MinIO Quickstart Guide [![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io) [![Docker Pulls](https://img.shields.io/docker/pulls/minio/minio.svg?maxAge=604800)](https://hub.docker.com/r/minio/minio/)
MinIO in distributed mode lets you pool multiple drives (even on different machines) into a single object storage server. As drives are distributed across several nodes, distributed MinIO can withstand multiple node failures and yet ensure full data protection.
## Why distributed MinIO?
MinIO in distributed mode can help you setup a highly-available storage system with a single object storage deployment. With distributed MinIO, you can optimally use storage devices, irrespective of their location in a network.
### Data protection
Distributed MinIO provides protection against multiple node/drive failures and [bit rot](https://github.com/minio/minio/blob/master/docs/erasure/README.md#what-is-bit-rot-protection) using [erasure code](https://docs.min.io/community/minio-object-store/operations/concepts/erasure-coding.html). As the minimum drives required for distributed MinIO is 2 (same as minimum drives required for erasure coding), erasure code automatically kicks in as you launch distributed MinIO.
If one or more drives are offline at the start of a PutObject or NewMultipartUpload operation the object will have additional data protection bits added automatically to provide additional safety for these objects.
### High availability
A stand-alone MinIO server would go down if the server hosting the drives goes offline. In contrast, a distributed MinIO setup with _m_ servers and _n_ drives will have your data safe as long as _m/2_ servers or _m*n_/2 or more drives are online.
For example, an 16-server distributed setup with 200 drives per node would continue serving files, up to 4 servers can be offline in default configuration i.e around 800 drives down MinIO would continue to read and write objects.
Refer to sizing guide for more understanding on default values chosen depending on your erasure stripe size [here](https://github.com/minio/minio/blob/master/docs/distributed/SIZING.md). Parity settings can be changed using [storage classes](https://github.com/minio/minio/tree/master/docs/erasure/storage-class).
### Consistency Guarantees
MinIO follows strict **read-after-write** and **list-after-write** consistency model for all i/o operations both in distributed and standalone modes. This consistency model is only guaranteed if you use disk filesystems such as xfs, zfs or btrfs etc.. for distributed setup.
**In our tests we also found ext4 does not honor POSIX O_DIRECT/Fdatasync semantics, ext4 trades performance for consistency guarantees. Please avoid ext4 in your setup.**
**If MinIO distributed setup is using NFS volumes underneath it is not guaranteed MinIO will provide these consistency guarantees since NFS is not strictly consistent (If you must use NFS we recommend that you at least use NFSv4 instead of NFSv3 for relatively better outcomes).**
## Get started
If you're aware of stand-alone MinIO set up, the process remains largely the same. MinIO server automatically switches to stand-alone or distributed mode, depending on the command line parameters.
### 1. Prerequisites
Install MinIO either on Kubernetes or Distributed Linux.
Install MinIO on Kubernetes:
- [MinIO Quickstart Guide for Kubernetes](https://docs.min.io/community/minio-object-store/operations/deployments/kubernetes.html).
- [Deploy a Tenant from the MinIO Operator](https://docs.min.io/community/minio-object-store/operations/deployments/k8s-deploy-minio-tenant-on-kubernetes.html)
Install Distributed MinIO on Linux:
- [Deploy Distributed MinIO on Linux](https://docs.min.io/community/minio-object-store/operations/install-deploy-manage/deploy-minio-multi-node-multi-drive.html#deploy-distributed-minio)
### 2. Run distributed MinIO
To start a distributed MinIO instance, you just need to pass drive locations as parameters to the minio server command. Then, you’ll need to run the same command on all the participating nodes.
**NOTE:**
- All the nodes running distributed MinIO should share a common root credentials, for the nodes to connect and trust each other. To achieve this, it is **recommended** to export root user and root password as environment variables, `MINIO_ROOT_USER` and `MINIO_ROOT_PASSWORD`, on all the nodes before executing MinIO server command. If not exported, default `minioadmin/minioadmin` credentials shall be used.
- **MinIO creates erasure-coding sets of _2_ to _16_ drives per set. The number of drives you provide in total must be a multiple of one of those numbers.**
- **MinIO chooses the largest EC set size which divides into the total number of drives or total number of nodes given - making sure to keep the uniform distribution i.e each node participates equal number of drives per set**.
- **Each object is written to a single EC set, and therefore is spread over no more than 16 drives.**
- **All the nodes running distributed MinIO setup are recommended to be homogeneous, i.e. same operating system, same number of drives and same network interconnects.**
- MinIO distributed mode requires **fresh directories**. If required, the drives can be shared with other applications. You can do this by using a sub-directory exclusive to MinIO. For example, if you have mounted your volume under `/export`, pass `/export/data` as arguments to MinIO server.
- The IP addresses and drive paths below are for demonstration purposes only, you need to replace these with the actual IP addresses and drive paths/folders.
- Servers running distributed MinIO instances should be less than 15 minutes apart. You can enable [NTP](http://www.ntp.org/) service as a best practice to ensure same times across servers.
- `MINIO_DOMAIN` environment variable should be defined and exported for bucket DNS style support.
- Running Distributed MinIO on **Windows** operating system is considered **experimental**. Please proceed with caution.
Example 1: Start distributed MinIO instance on n nodes with m drives each mounted at `/export1` to `/exportm` (pictured below), by running this command on all the n nodes:
![Distributed MinIO, n nodes with m drives each](https://github.com/minio/minio/blob/master/docs/screenshots/Architecture-diagram_distributed_nm.png?raw=true)
### GNU/Linux and macOS
```sh
export MINIO_ROOT_USER=<ACCESS_KEY>
export MINIO_ROOT_PASSWORD=<SECRET_KEY>
minio server http://host{1...n}/export{1...m}
```
> **NOTE:** In above example `n` and `m` represent positive integers, _do not copy paste and expect it work make the changes according to local deployment and setup_.
> **NOTE:** `{1...n}` shown have 3 dots! Using only 2 dots `{1..n}` will be interpreted by your shell and won't be passed to MinIO server, affecting the erasure coding order, which would impact performance and high availability. **Always use ellipses syntax `{1...n}` (3 dots!) for optimal erasure-code distribution**
### Expanding existing distributed setup
MinIO supports expanding distributed erasure coded clusters by specifying new set of clusters on the command-line as shown below:
```sh
export MINIO_ROOT_USER=<ACCESS_KEY>
export MINIO_ROOT_PASSWORD=<SECRET_KEY>
minio server http://host{1...n}/export{1...m} http://host{o...z}/export{1...m}
```
For example:
```
minio server http://host{1...4}/export{1...16} http://host{5...12}/export{1...16}
```
Now the server has expanded total storage by _(newly_added_servers\*m)_ more drives, taking the total count to _(existing_servers\*m)+(newly_added_servers\*m)_ drives. New object upload requests automatically start using the least used cluster. This expansion strategy works endlessly, so you can perpetually expand your clusters as needed. When you restart, it is immediate and non-disruptive to the applications. Each group of servers in the command-line is called a pool. There are 2 server pools in this example. New objects are placed in server pools in proportion to the amount of free space in each pool. Within each pool, the location of the erasure-set of drives is determined based on a deterministic hashing algorithm.
> **NOTE:** **Each pool you add must have the same erasure coding parity configuration as the original pool, so the same data redundancy SLA is maintained.**
## 3. Test your setup
To test this setup, access the MinIO server via browser or [`mc`](https://docs.min.io/community/minio-object-store/reference/minio-mc.html#quickstart).
## Explore Further
- [MinIO Erasure Code QuickStart Guide](https://docs.min.io/community/minio-object-store/operations/concepts/erasure-coding.html)
- [Use `mc` with MinIO Server](https://docs.min.io/community/minio-object-store/reference/minio-mc.html)
- [Use `aws-cli` with MinIO Server](https://docs.min.io/community/minio-object-store/integrations/aws-cli-with-minio.html)
- [Use `minio-go` SDK with MinIO Server](https://docs.min.io/community/minio-object-store/developers/go/minio-go.html)
- [The MinIO documentation website](https://docs.min.io/community/minio-object-store/index.html)

39
docs/distributed/SIZING.md Normal file
View File

@@ -0,0 +1,39 @@
# Erasure code sizing guide
## Toy Setups
Capacity constrained environments, MinIO will work but not recommended for production.
| servers | drives (per node) | stripe_size | parity chosen (default) | tolerance for reads (servers) | tolerance for writes (servers) |
|--------:|------------------:|------------:|------------------------:|------------------------------:|-------------------------------:|
| 1 | 1 | 1 | 0 | 0 | 0 |
| 1 | 4 | 4 | 2 | 0 | 0 |
| 4 | 1 | 4 | 2 | 2 | 1 |
| 5 | 1 | 5 | 2 | 2 | 2 |
| 6 | 1 | 6 | 3 | 3 | 2 |
| 7 | 1 | 7 | 3 | 3 | 3 |
## Minimum System Configuration for Production
| servers | drives (per node) | stripe_size | parity chosen (default) | tolerance for reads (servers) | tolerance for writes (servers) |
|--------:|------------------:|------------:|------------------------:|------------------------------:|-------------------------------:|
| 4 | 2 | 8 | 4 | 2 | 1 |
| 5 | 2 | 10 | 4 | 2 | 2 |
| 6 | 2 | 12 | 4 | 2 | 2 |
| 7 | 2 | 14 | 4 | 2 | 2 |
| 8 | 1 | 8 | 4 | 4 | 3 |
| 8 | 2 | 16 | 4 | 2 | 2 |
| 9 | 2 | 9 | 4 | 4 | 4 |
| 10 | 2 | 10 | 4 | 4 | 4 |
| 11 | 2 | 11 | 4 | 4 | 4 |
| 12 | 2 | 12 | 4 | 4 | 4 |
| 13 | 2 | 13 | 4 | 4 | 4 |
| 14 | 2 | 14 | 4 | 4 | 4 |
| 15 | 2 | 15 | 4 | 4 | 4 |
| 16 | 2 | 16 | 4 | 4 | 4 |
If one or more drives are offline at the start of a PutObject or NewMultipartUpload operation the object will have additional data
protection bits added automatically to provide the regular safety for these objects up to 50% of the number of drives.
This will allow normal write operations to take place on systems that exceed the write tolerance.
This means that in the examples above the system will always write 4 parity shards at the expense of slightly higher disk usage.

153
docs/distributed/decom-compressed-sse-s3.sh Executable file
View File

@@ -0,0 +1,153 @@
#!/bin/bash
if [ -n "$TEST_DEBUG" ]; then
set -x
fi
pkill minio
rm -rf /tmp/xl
if [ ! -f ./mc ]; then
wget --quiet -O mc https://dl.minio.io/client/mc/release/linux-amd64/mc &&
chmod +x mc
fi
export CI=true
export MINIO_COMPRESSION_ENABLE="on"
export MINIO_COMPRESSION_EXTENSIONS=".go"
export MINIO_COMPRESSION_MIME_TYPES="application/*"
export MINIO_COMPRESSION_ALLOW_ENCRYPTION="on"
export MINIO_KMS_AUTO_ENCRYPTION=on
export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
export MC_HOST_myminio="http://minioadmin:minioadmin@localhost:9000/"
(minio server http://localhost:9000/tmp/xl/{1...10}/disk{0...1} 2>&1 >/dev/null) &
pid=$!
./mc ready myminio
./mc admin user add myminio/ minio123 minio123
./mc admin user add myminio/ minio12345 minio12345
./mc admin policy create myminio/ rw ./docs/distributed/rw.json
./mc admin policy create myminio/ lake ./docs/distributed/rw.json
./mc admin policy attach myminio/ rw --user=minio123
./mc admin policy attach myminio/ lake --user=minio12345
./mc mb -l myminio/versioned
./mc mirror internal myminio/versioned/ --quiet >/dev/null
## Soft delete (creates delete markers)
./mc rm -r --force myminio/versioned >/dev/null
## mirror again to create another set of version on top
./mc mirror internal myminio/versioned/ --quiet >/dev/null
expected_checksum=$(./mc cat internal/dsync/drwmutex.go | md5sum)
user_count=$(./mc admin user list myminio/ | wc -l)
policy_count=$(./mc admin policy list myminio/ | wc -l)
kill $pid
(minio server http://localhost:9000/tmp/xl/{1...10}/disk{0...1} http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/expanded_1.log) &
pid_1=$!
(minio server --address ":9001" http://localhost:9000/tmp/xl/{1...10}/disk{0...1} http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/expanded_2.log) &
pid_2=$!
sleep 30
./mc ready myminio
expanded_user_count=$(./mc admin user list myminio/ | wc -l)
expanded_policy_count=$(./mc admin policy list myminio/ | wc -l)
if [ $user_count -ne $expanded_user_count ]; then
echo "BUG: original user count differs from expanded setup"
exit 1
fi
if [ $policy_count -ne $expanded_policy_count ]; then
echo "BUG: original policy count differs from expanded setup"
exit 1
fi
./mc version info myminio/versioned | grep -q "versioning is enabled"
ret=$?
if [ $ret -ne 0 ]; then
echo "expected versioning enabled after expansion"
exit 1
fi
./mc mirror cmd myminio/versioned/ --quiet >/dev/null
./mc ls -r myminio/versioned/ >expanded_ns.txt
./mc ls -r --versions myminio/versioned/ >expanded_ns_versions.txt
./mc admin decom start myminio/ http://localhost:9000/tmp/xl/{1...10}/disk{0...1}
until $(./mc admin decom status myminio/ | grep -q Complete); do
echo "waiting for decom to finish..."
sleep 1
done
kill $pid_1
kill $pid_2
sleep 5
(minio server --address ":9001" http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/removed.log) &
pid=$!
export MC_HOST_myminio="http://minioadmin:minioadmin@localhost:9001/"
./mc ready myminio
decom_user_count=$(./mc admin user list myminio/ | wc -l)
decom_policy_count=$(./mc admin policy list myminio/ | wc -l)
if [ $user_count -ne $decom_user_count ]; then
echo "BUG: original user count differs after decommission"
exit 1
fi
if [ $policy_count -ne $decom_policy_count ]; then
echo "BUG: original policy count differs after decommission"
exit 1
fi
./mc version info myminio/versioned | grep -q "versioning is enabled"
ret=$?
if [ $ret -ne 0 ]; then
echo "BUG: expected versioning enabled after decommission"
exit 1
fi
got_checksum=$(./mc cat myminio/versioned/dsync/drwmutex.go | md5sum)
if [ "${expected_checksum}" != "${got_checksum}" ]; then
echo "BUG: decommission failed on encrypted objects: expected ${expected_checksum} got ${got_checksum}"
exit 1
fi
./mc ls -r myminio/versioned >decommissioned_ns.txt
./mc ls -r --versions myminio/versioned >decommissioned_ns_versions.txt
out=$(diff -qpruN expanded_ns.txt decommissioned_ns.txt)
ret=$?
if [ $ret -ne 0 ]; then
echo "BUG: expected no missing entries after decommission: $out"
exit 1
fi
out=$(diff -qpruN expanded_ns_versions.txt decommissioned_ns_versions.txt)
ret=$?
if [ $ret -ne 0 ]; then
echo "BUG: expected no missing entries after decommission: $out"
exit 1
fi
./s3-check-md5 -versions -access-key minioadmin -secret-key minioadmin -endpoint http://127.0.0.1:9001/ -bucket versioned
kill $pid

245
docs/distributed/decom-encrypted-kes.sh Executable file
View File

@@ -0,0 +1,245 @@
#!/bin/bash
if [ -n "$TEST_DEBUG" ]; then
set -x
fi
pkill minio
pkill kes
rm -rf /tmp/xl
if [ ! -f ./mc ]; then
wget --quiet -O mc https://dl.minio.io/client/mc/release/linux-amd64/mc &&
chmod +x mc
fi
if [ ! -f ./kes ]; then
wget --quiet -O kes https://github.com/minio/kes/releases/latest/download/kes-linux-amd64 &&
chmod +x kes
fi
if ! openssl version &>/dev/null; then
apt install openssl || sudo apt install opensssl
fi
# Start KES Server
(./kes server --dev 2>&1 >kes-server.log) &
kes_pid=$!
sleep 5s
API_KEY=$(grep "API Key" <kes-server.log | awk -F" " '{print $3}')
(openssl s_client -connect 127.0.0.1:7373 2>/dev/null 1>public.crt)
export CI=true
export MINIO_KMS_KES_ENDPOINT=https://127.0.0.1:7373
export MINIO_KMS_KES_API_KEY="${API_KEY}"
export MINIO_KMS_KES_KEY_NAME=minio-default-key
export MINIO_KMS_KES_CAPATH=public.crt
export MC_HOST_myminio="http://minioadmin:minioadmin@localhost:9000/"
(minio server http://localhost:9000/tmp/xl/{1...10}/disk{0...1} 2>&1 >/dev/null) &
pid=$!
./mc ready myminio
./mc admin user add myminio/ minio123 minio123
./mc admin user add myminio/ minio12345 minio12345
./mc admin policy create myminio/ rw ./docs/distributed/rw.json
./mc admin policy create myminio/ lake ./docs/distributed/rw.json
./mc admin policy attach myminio/ rw --user=minio123
./mc admin policy attach myminio/ lake --user=minio12345
./mc mb -l myminio/versioned
./mc mb -l myminio/versioned-1
./mc encrypt set sse-s3 myminio/versioned
./mc encrypt set sse-kms minio-default-key myminio/versioned-1
./mc mirror internal myminio/versioned/ --quiet >/dev/null
./mc mirror internal myminio/versioned-1/ --quiet >/dev/null
## Soft delete (creates delete markers)
./mc rm -r --force myminio/versioned >/dev/null
./mc rm -r --force myminio/versioned-1 >/dev/null
## mirror again to create another set of version on top
./mc mirror internal myminio/versioned/ --quiet >/dev/null
./mc mirror internal myminio/versioned-1/ --quiet >/dev/null
expected_checksum=$(./mc cat internal/dsync/drwmutex.go | md5sum)
user_count=$(./mc admin user list myminio/ | wc -l)
policy_count=$(./mc admin policy list myminio/ | wc -l)
kill $pid
(minio server http://localhost:9000/tmp/xl/{1...10}/disk{0...1} http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/expanded_1.log) &
pid_1=$!
(minio server --address ":9001" http://localhost:9000/tmp/xl/{1...10}/disk{0...1} http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/expanded_2.log) &
pid_2=$!
./mc ready myminio
expanded_user_count=$(./mc admin user list myminio/ | wc -l)
expanded_policy_count=$(./mc admin policy list myminio/ | wc -l)
if [ "$user_count" -ne "$expanded_user_count" ]; then
echo "BUG: original user count differs from expanded setup"
exit 1
fi
if [ "$policy_count" -ne "$expanded_policy_count" ]; then
echo "BUG: original policy count differs from expanded setup"
exit 1
fi
./mc version info myminio/versioned | grep -q "versioning is enabled"
ret=$?
if [ $ret -ne 0 ]; then
echo "expected versioning enabled after expansion"
exit 1
fi
./mc encrypt info myminio/versioned | grep -q "Auto encryption 'sse-s3' is enabled"
ret=$?
if [ $ret -ne 0 ]; then
echo "expected encryption enabled after expansion"
exit 1
fi
./mc version info myminio/versioned-1 | grep -q "versioning is enabled"
ret=$?
if [ $ret -ne 0 ]; then
echo "expected versioning enabled after expansion"
exit 1
fi
./mc encrypt info myminio/versioned-1 | grep -q "Auto encryption 'sse-kms' is enabled"
ret=$?
if [ $ret -ne 0 ]; then
echo "expected encryption enabled after expansion"
exit 1
fi
./mc mirror cmd myminio/versioned/ --quiet >/dev/null
./mc mirror cmd myminio/versioned-1/ --quiet >/dev/null
./mc ls -r myminio/versioned/ >expanded_ns.txt
./mc ls -r --versions myminio/versioned/ >expanded_ns_versions.txt
./mc ls -r myminio/versioned-1/ >expanded_ns_1.txt
./mc ls -r --versions myminio/versioned-1/ >expanded_ns_versions_1.txt
./mc admin decom start myminio/ http://localhost:9000/tmp/xl/{1...10}/disk{0...1}
until $(./mc admin decom status myminio/ | grep -q Complete); do
echo "waiting for decom to finish..."
sleep 1s
done
kill $pid_1
kill $pid_2
sleep 5s
(minio server --address ":9001" http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/removed.log) &
pid=$!
sleep 30s
export MC_HOST_myminio="http://minioadmin:minioadmin@localhost:9001/"
./mc ready myminio
decom_user_count=$(./mc admin user list myminio/ | wc -l)
decom_policy_count=$(./mc admin policy list myminio/ | wc -l)
if [ "$user_count" -ne "$decom_user_count" ]; then
echo "BUG: original user count differs after decommission"
exit 1
fi
if [ "$policy_count" -ne "$decom_policy_count" ]; then
echo "BUG: original policy count differs after decommission"
exit 1
fi
./mc version info myminio/versioned | grep -q "versioning is enabled"
ret=$?
if [ $ret -ne 0 ]; then
echo "BUG: expected versioning enabled after decommission"
exit 1
fi
./mc encrypt info myminio/versioned | grep -q "Auto encryption 'sse-s3' is enabled"
ret=$?
if [ $ret -ne 0 ]; then
echo "BUG: expected encryption enabled after expansion"
exit 1
fi
./mc version info myminio/versioned-1 | grep -q "versioning is enabled"
ret=$?
if [ $ret -ne 0 ]; then
echo "BUG: expected versioning enabled after decommission"
exit 1
fi
./mc encrypt info myminio/versioned-1 | grep -q "Auto encryption 'sse-kms' is enabled"
ret=$?
if [ $ret -ne 0 ]; then
echo "BUG: expected encryption enabled after expansion"
exit 1
fi
got_checksum=$(./mc cat myminio/versioned/dsync/drwmutex.go | md5sum)
if [ "${expected_checksum}" != "${got_checksum}" ]; then
echo "BUG: decommission failed on encrypted objects: expected ${expected_checksum} got ${got_checksum}"
exit 1
fi
got_checksum_1=$(./mc cat myminio/versioned-1/dsync/drwmutex.go | md5sum)
if [ "${expected_checksum}" != "${got_checksum_1}" ]; then
echo "BUG: decommission failed on encrypted objects: expected ${expected_checksum} got ${got_checksum_1}"
exit 1
fi
./mc ls -r myminio/versioned >decommissioned_ns.txt
./mc ls -r --versions myminio/versioned >decommissioned_ns_versions.txt
./mc ls -r myminio/versioned-1 >decommissioned_ns_1.txt
./mc ls -r --versions myminio/versioned-1 >decommissioned_ns_versions_1.txt
out=$(diff -qpruN expanded_ns.txt decommissioned_ns.txt)
ret=$?
if [ $ret -ne 0 ]; then
echo "BUG: expected no missing entries after decommission: $out"
exit 1
fi
out=$(diff -qpruN expanded_ns_versions.txt decommissioned_ns_versions.txt)
ret=$?
if [ $ret -ne 0 ]; then
echo "BUG: expected no missing entries after decommission: $out"
exit 1
fi
out1=$(diff -qpruN expanded_ns_1.txt decommissioned_ns_1.txt)
ret=$?
if [ $ret -ne 0 ]; then
echo "BUG: expected no missing entries after decommission: $out1"
exit 1
fi
out1=$(diff -qpruN expanded_ns_versions_1.txt decommissioned_ns_versions_1.txt)
ret=$?
if [ $ret -ne 0 ]; then
echo "BUG: expected no missing entries after decommission: $out1"
exit 1
fi
./s3-check-md5 -versions -access-key minioadmin -secret-key minioadmin -endpoint http://127.0.0.1:9001/ -bucket versioned
./s3-check-md5 -versions -access-key minioadmin -secret-key minioadmin -endpoint http://127.0.0.1:9001/ -bucket versioned-1
kill $pid
kill $kes_pid

163
docs/distributed/decom-encrypted-sse-s3.sh Executable file
View File

@@ -0,0 +1,163 @@
#!/bin/bash
if [ -n "$TEST_DEBUG" ]; then
set -x
fi
pkill minio
rm -rf /tmp/xl
if [ ! -f ./mc ]; then
wget --quiet -O mc https://dl.minio.io/client/mc/release/linux-amd64/mc &&
chmod +x mc
fi
export CI=true
export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
export MC_HOST_myminio="http://minioadmin:minioadmin@localhost:9000/"
(minio server http://localhost:9000/tmp/xl/{1...10}/disk{0...1} 2>&1 >/dev/null) &
pid=$!
./mc ready myminio
./mc admin user add myminio/ minio123 minio123
./mc admin user add myminio/ minio12345 minio12345
./mc admin policy create myminio/ rw ./docs/distributed/rw.json
./mc admin policy create myminio/ lake ./docs/distributed/rw.json
./mc admin policy attach myminio/ rw --user=minio123
./mc admin policy attach myminio/ lake --user=minio12345
./mc mb -l myminio/versioned
./mc encrypt set sse-s3 myminio/versioned
./mc mirror internal myminio/versioned/ --quiet >/dev/null
## Soft delete (creates delete markers)
./mc rm -r --force myminio/versioned >/dev/null
## mirror again to create another set of version on top
./mc mirror internal myminio/versioned/ --quiet >/dev/null
expected_checksum=$(./mc cat internal/dsync/drwmutex.go | md5sum)
user_count=$(./mc admin user list myminio/ | wc -l)
policy_count=$(./mc admin policy list myminio/ | wc -l)
kill $pid
(minio server http://localhost:9000/tmp/xl/{1...10}/disk{0...1} http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/expanded_1.log) &
pid_1=$!
(minio server --address ":9001" http://localhost:9000/tmp/xl/{1...10}/disk{0...1} http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/expanded_2.log) &
pid_2=$!
./mc ready myminio
expanded_user_count=$(./mc admin user list myminio/ | wc -l)
expanded_policy_count=$(./mc admin policy list myminio/ | wc -l)
if [ $user_count -ne $expanded_user_count ]; then
echo "BUG: original user count differs from expanded setup"
exit 1
fi
if [ $policy_count -ne $expanded_policy_count ]; then
echo "BUG: original policy count differs from expanded setup"
exit 1
fi
./mc version info myminio/versioned | grep -q "versioning is enabled"
ret=$?
if [ $ret -ne 0 ]; then
echo "expected versioning enabled after expansion"
exit 1
fi
./mc encrypt info myminio/versioned | grep -q "Auto encryption 'sse-s3' is enabled"
ret=$?
if [ $ret -ne 0 ]; then
echo "expected encryption enabled after expansion"
exit 1
fi
./mc mirror cmd myminio/versioned/ --quiet >/dev/null
./mc ls -r myminio/versioned/ >expanded_ns.txt
./mc ls -r --versions myminio/versioned/ >expanded_ns_versions.txt
./mc admin decom start myminio/ http://localhost:9000/tmp/xl/{1...10}/disk{0...1}
until $(./mc admin decom status myminio/ | grep -q Complete); do
echo "waiting for decom to finish..."
sleep 1
done
kill $pid_1
kill $pid_2
sleep 5
(minio server --address ":9001" http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/removed.log) &
pid=$!
export MC_HOST_myminio="http://minioadmin:minioadmin@localhost:9001/"
./mc ready myminio
decom_user_count=$(./mc admin user list myminio/ | wc -l)
decom_policy_count=$(./mc admin policy list myminio/ | wc -l)
if [ $user_count -ne $decom_user_count ]; then
echo "BUG: original user count differs after decommission"
exit 1
fi
if [ $policy_count -ne $decom_policy_count ]; then
echo "BUG: original policy count differs after decommission"
exit 1
fi
./mc version info myminio/versioned | grep -q "versioning is enabled"
ret=$?
if [ $ret -ne 0 ]; then
echo "BUG: expected versioning enabled after decommission"
exit 1
fi
./mc encrypt info myminio/versioned | grep -q "Auto encryption 'sse-s3' is enabled"
ret=$?
if [ $ret -ne 0 ]; then
echo "BUG: expected encryption enabled after expansion"
exit 1
fi
got_checksum=$(./mc cat myminio/versioned/dsync/drwmutex.go | md5sum)
if [ "${expected_checksum}" != "${got_checksum}" ]; then
echo "BUG: decommission failed on encrypted objects: expected ${expected_checksum} got ${got_checksum}"
exit 1
fi
./mc ls -r myminio/versioned >decommissioned_ns.txt
./mc ls -r --versions myminio/versioned >decommissioned_ns_versions.txt
out=$(diff -qpruN expanded_ns.txt decommissioned_ns.txt)
ret=$?
if [ $ret -ne 0 ]; then
echo "BUG: expected no missing entries after decommission: $out"
exit 1
fi
out=$(diff -qpruN expanded_ns_versions.txt decommissioned_ns_versions.txt)
ret=$?
if [ $ret -ne 0 ]; then
echo "BUG: expected no missing entries after decommission: $out"
exit 1
fi
./s3-check-md5 -versions -access-key minioadmin -secret-key minioadmin -endpoint http://127.0.0.1:9001/ -bucket versioned
kill $pid

149
docs/distributed/decom-encrypted.sh Executable file
View File

@@ -0,0 +1,149 @@
#!/bin/bash
if [ -n "$TEST_DEBUG" ]; then
set -x
fi
pkill minio
rm -rf /tmp/xl
if [ ! -f ./mc ]; then
wget --quiet -O mc https://dl.minio.io/client/mc/release/linux-amd64/mc &&
chmod +x mc
fi
export CI=true
export MINIO_KMS_AUTO_ENCRYPTION=on
export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
(minio server http://localhost:9000/tmp/xl/{1...10}/disk{0...1} 2>&1 >/dev/null) &
pid=$!
export MC_HOST_myminio="http://minioadmin:minioadmin@localhost:9000/"
./mc ready myminio
./mc admin user add myminio/ minio123 minio123
./mc admin user add myminio/ minio12345 minio12345
./mc admin policy create myminio/ rw ./docs/distributed/rw.json
./mc admin policy create myminio/ lake ./docs/distributed/rw.json
./mc admin policy attach myminio/ rw --user=minio123
./mc admin policy attach myminio/ lake --user=minio12345
./mc mb -l myminio/versioned
./mc mirror internal myminio/versioned/ --quiet >/dev/null
## Soft delete (creates delete markers)
./mc rm -r --force myminio/versioned >/dev/null
## mirror again to create another set of version on top
./mc mirror internal myminio/versioned/ --quiet >/dev/null
expected_checksum=$(./mc cat internal/dsync/drwmutex.go | md5sum)
user_count=$(./mc admin user list myminio/ | wc -l)
policy_count=$(./mc admin policy list myminio/ | wc -l)
kill $pid
(minio server http://localhost:9000/tmp/xl/{1...10}/disk{0...1} http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/expanded_1.log) &
pid_1=$!
(minio server --address ":9001" http://localhost:9000/tmp/xl/{1...10}/disk{0...1} http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/expanded_2.log) &
pid_2=$!
./mc ready myminio
expanded_user_count=$(./mc admin user list myminio/ | wc -l)
expanded_policy_count=$(./mc admin policy list myminio/ | wc -l)
if [ $user_count -ne $expanded_user_count ]; then
echo "BUG: original user count differs from expanded setup"
exit 1
fi
if [ $policy_count -ne $expanded_policy_count ]; then
echo "BUG: original policy count differs from expanded setup"
exit 1
fi
./mc version info myminio/versioned | grep -q "versioning is enabled"
ret=$?
if [ $ret -ne 0 ]; then
echo "expected versioning enabled after expansion"
exit 1
fi
./mc mirror cmd myminio/versioned/ --quiet >/dev/null
./mc ls -r myminio/versioned/ >expanded_ns.txt
./mc ls -r --versions myminio/versioned/ >expanded_ns_versions.txt
./mc admin decom start myminio/ http://localhost:9000/tmp/xl/{1...10}/disk{0...1}
until $(./mc admin decom status myminio/ | grep -q Complete); do
echo "waiting for decom to finish..."
sleep 1
done
kill $pid_1
kill $pid_2
sleep 5
(minio server --address ":9001" http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/removed.log) &
pid=$!
export MC_HOST_myminio="http://minioadmin:minioadmin@localhost:9001/"
./mc ready myminio
decom_user_count=$(./mc admin user list myminio/ | wc -l)
decom_policy_count=$(./mc admin policy list myminio/ | wc -l)
if [ $user_count -ne $decom_user_count ]; then
echo "BUG: original user count differs after decommission"
exit 1
fi
if [ $policy_count -ne $decom_policy_count ]; then
echo "BUG: original policy count differs after decommission"
exit 1
fi
./mc version info myminio/versioned | grep -q "versioning is enabled"
ret=$?
if [ $ret -ne 0 ]; then
echo "BUG: expected versioning enabled after decommission"
exit 1
fi
./mc ls -r myminio/versioned >decommissioned_ns.txt
./mc ls -r --versions myminio/versioned >decommissioned_ns_versions.txt
out=$(diff -qpruN expanded_ns.txt decommissioned_ns.txt)
ret=$?
if [ $ret -ne 0 ]; then
echo "BUG: expected no missing entries after decommission: $out"
exit 1
fi
out=$(diff -qpruN expanded_ns_versions.txt decommissioned_ns_versions.txt)
ret=$?
if [ $ret -ne 0 ]; then
echo "BUG: expected no missing entries after decommission: $out"
exit 1
fi
got_checksum=$(./mc cat myminio/versioned/dsync/drwmutex.go | md5sum)
if [ "${expected_checksum}" != "${got_checksum}" ]; then
echo "BUG: decommission failed on encrypted objects: expected ${expected_checksum} got ${got_checksum}"
exit 1
fi
./s3-check-md5 -versions -access-key minioadmin -secret-key minioadmin -endpoint http://127.0.0.1:9001/ -bucket versioned
kill $pid

218
docs/distributed/decom.sh Executable file
View File

@@ -0,0 +1,218 @@
#!/bin/bash
if [ -n "$TEST_DEBUG" ]; then
set -x
fi
pkill minio
rm -rf /tmp/xl
rm -rf /tmp/xltier
if [ ! -f ./mc ]; then
wget --quiet -O mc https://dl.minio.io/client/mc/release/linux-amd64/mc &&
chmod +x mc
fi
export CI=true
export MINIO_SCANNER_SPEED=fastest
(minio server http://localhost:9000/tmp/xl/{1...10}/disk{0...1} 2>&1 >/tmp/decom.log) &
pid=$!
export MC_HOST_myminio="http://minioadmin:minioadmin@localhost:9000/"
./mc ready myminio
./mc admin user add myminio/ minio123 minio123
./mc admin user add myminio/ minio12345 minio12345
./mc admin policy create myminio/ rw ./docs/distributed/rw.json
./mc admin policy create myminio/ lake ./docs/distributed/rw.json
./mc admin policy attach myminio/ rw --user=minio123
./mc admin policy attach myminio/ lake --user=minio12345
./mc mb -l myminio/versioned
./mc mirror internal myminio/versioned/ --quiet >/dev/null
## Soft delete (creates delete markers)
./mc rm -r --force myminio/versioned >/dev/null
## mirror again to create another set of version on top
./mc mirror internal myminio/versioned/ --quiet >/dev/null
expected_checksum=$(./mc cat internal/dsync/drwmutex.go | md5sum)
user_count=$(./mc admin user list myminio/ | wc -l)
policy_count=$(./mc admin policy list myminio/ | wc -l)
## create a warm tier instance
(minio server /tmp/xltier/{1...4}/disk{0...1} --address :9002 2>&1 >/dev/null) &
export MC_HOST_mytier="http://minioadmin:minioadmin@localhost:9002/"
./mc ready myminio
./mc mb -l myminio/bucket2
./mc mb -l mytier/tiered
## create a tier and set up ilm policy to tier immediately
./mc admin tier add minio myminio TIER1 --endpoint http://localhost:9002 --access-key minioadmin --secret-key minioadmin --bucket tiered --prefix prefix5/
./mc ilm add myminio/bucket2 --transition-days 0 --transition-tier TIER1 --transition-days 0
## mirror some content to bucket2 and capture versions tiered
./mc mirror internal myminio/bucket2/ --quiet >/dev/null
./mc ls -r myminio/bucket2/ >bucket2_ns.txt
./mc ls -r --versions myminio/bucket2/ >bucket2_ns_versions.txt
sleep 30
./mc ls -r --versions mytier/tiered/ >tiered_ns_versions.txt
kill $pid
(minio server http://localhost:9000/tmp/xl/{1...10}/disk{0...1} http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/expanded_1.log) &
pid_1=$!
(minio server --address ":9001" http://localhost:9000/tmp/xl/{1...10}/disk{0...1} http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/expanded_2.log) &
pid_2=$!
./mc ready myminio
expanded_user_count=$(./mc admin user list myminio/ | wc -l)
expanded_policy_count=$(./mc admin policy list myminio/ | wc -l)
if [ $user_count -ne $expanded_user_count ]; then
echo "BUG: original user count differs from expanded setup"
exit 1
fi
if [ $policy_count -ne $expanded_policy_count ]; then
echo "BUG: original policy count differs from expanded setup"
exit 1
fi
./mc version info myminio/versioned | grep -q "versioning is enabled"
ret=$?
if [ $ret -ne 0 ]; then
echo "expected versioning enabled after expansion"
exit 1
fi
./mc mirror cmd myminio/versioned/ --quiet >/dev/null
./mc ls -r myminio/versioned/ >expanded_ns.txt
./mc ls -r --versions myminio/versioned/ >expanded_ns_versions.txt
./mc admin decom start myminio/ http://localhost:9000/tmp/xl/{1...10}/disk{0...1}
count=0
until $(./mc admin decom status myminio/ | grep -q Complete); do
echo "waiting for decom to finish..."
count=$((count + 1))
if [ ${count} -eq 120 ]; then
./mc cat /tmp/expanded_*.log
fi
sleep 1
done
kill $pid_1
kill $pid_2
sleep 5
(minio server --address ":9001" http://localhost:9001/tmp/xl/{11...30}/disk{0...3} 2>&1 >/tmp/removed.log) &
pid=$!
sleep 5
export MC_HOST_myminio="http://minioadmin:minioadmin@localhost:9001/"
./mc ready myminio
decom_user_count=$(./mc admin user list myminio/ | wc -l)
decom_policy_count=$(./mc admin policy list myminio/ | wc -l)
if [ $user_count -ne $decom_user_count ]; then
echo "BUG: original user count differs after decommission"
exit 1
fi
if [ $policy_count -ne $decom_policy_count ]; then
echo "BUG: original policy count differs after decommission"
exit 1
fi
./mc version info myminio/versioned | grep -q "versioning is enabled"
ret=$?
if [ $ret -ne 0 ]; then
echo "BUG: expected versioning enabled after decommission"
exit 1
fi
./mc ls -r myminio/versioned >decommissioned_ns.txt
./mc ls -r --versions myminio/versioned >decommissioned_ns_versions.txt
out=$(diff -qpruN expanded_ns.txt decommissioned_ns.txt)
ret=$?
if [ $ret -ne 0 ]; then
echo "BUG: expected no missing entries after decommission: $out"
exit 1
fi
out=$(diff -qpruN expanded_ns_versions.txt decommissioned_ns_versions.txt)
ret=$?
if [ $ret -ne 0 ]; then
echo "BUG: expected no missing entries after decommission: $out"
exit 1
fi
got_checksum=$(./mc cat myminio/versioned/dsync/drwmutex.go | md5sum)
if [ "${expected_checksum}" != "${got_checksum}" ]; then
echo "BUG: decommission failed on encrypted objects: expected ${expected_checksum} got ${got_checksum}"
exit 1
fi
# after decommissioning, compare listings in bucket2 and tiered
./mc version info myminio/bucket2 | grep -q "versioning is enabled"
ret=$?
if [ $ret -ne 0 ]; then
echo "BUG: expected versioning enabled after decommission on bucket2"
exit 1
fi
./mc ls -r myminio/bucket2 >decommissioned_bucket2_ns.txt
./mc ls -r --versions myminio/bucket2 >decommissioned_bucket2_ns_versions.txt
./mc ls -r --versions mytier/tiered/ >tiered_ns_versions2.txt
out=$(diff -qpruN bucket2_ns.txt decommissioned_bucket2_ns.txt)
ret=$?
if [ $ret -ne 0 ]; then
echo "BUG: expected no missing entries after decommission in bucket2: $out"
exit 1
fi
out=$(diff -qpruN bucket2_ns_versions.txt decommissioned_bucket2_ns_versions.txt)
ret=$?
if [ $ret -ne 0 ]; then
echo "BUG: expected no missing entries after decommission in bucket2x: $out"
exit 1
fi
out=$(diff -qpruN tiered_ns_versions.txt tiered_ns_versions2.txt)
ret=$?
if [ $ret -ne 0 ]; then
echo "BUG: expected no missing entries after decommission in warm tier: $out"
exit 1
fi
got_checksum=$(./mc cat myminio/bucket2/dsync/drwmutex.go | md5sum)
if [ "${expected_checksum}" != "${got_checksum}" ]; then
echo "BUG: decommission failed on encrypted objects with tiering: expected ${expected_checksum} got ${got_checksum}"
exit 1
fi
s3-check-md5 -versions -access-key minioadmin -secret-key minioadmin -endpoint http://127.0.0.1:9001/ -bucket bucket2
s3-check-md5 -versions -access-key minioadmin -secret-key minioadmin -endpoint http://127.0.0.1:9001/ -bucket versioned
kill $pid

109
docs/distributed/distributed-from-config-file.sh Executable file
View File

@@ -0,0 +1,109 @@
#!/usr/bin/env bash
set -e
cleanup() {
echo "Cleaning up instances of MinIO"
pkill minio || true
pkill -9 minio || true
rm -rf /tmp/xl/ || true
rm -rf /tmp/minio.configfile.{1,2,3,4} || true
}
cleanup
unset MINIO_KMS_KES_CERT_FILE
unset MINIO_KMS_KES_KEY_FILE
unset MINIO_KMS_KES_ENDPOINT
unset MINIO_KMS_KES_KEY_NAME
export MINIO_CI_CD=1
if [ ! -f ./mc ]; then
os="$(uname -s)"
arch="$(uname -m)"
case "${arch}" in
"x86_64")
arch="amd64"
;;
esac
wget -O mc https://dl.minio.io/client/mc/release/${os,,}-${arch,,}/mc &&
chmod +x mc
fi
for i in $(seq 1 4); do
s3Port="$((9000 + i))"
consolePort="$((s3Port + 1000))"
cat <<EOF >/tmp/minio.configfile.$i
version: v1
address: ':${s3Port}'
console-address: ':${consolePort}'
rootUser: 'minr0otUS2r'
rootPassword: 'pBU94AGAY85e'
pools: # Specify the nodes and drives with pools
-
- 'http://localhost:9001/tmp/xl/node9001/mnt/disk{1...4}/'
- 'http://localhost:9002/tmp/xl/node9002/mnt/disk{1,2,3,4}/'
-
- 'http://localhost:9003/tmp/xl/node9003/mnt/disk{1...4}/'
- 'http://localhost:9004/tmp/xl/node9004/mnt/disk1/'
- 'http://localhost:9004/tmp/xl/node9004/mnt/disk2/'
- 'http://localhost:9004/tmp/xl/node9004/mnt/disk3/'
- 'http://localhost:9004/tmp/xl/node9004/mnt/disk4/'
EOF
done
minio server --config /tmp/minio.configfile.1 >/tmp/minio1_1.log 2>&1 &
site1_pid=$!
minio server --config /tmp/minio.configfile.2 >/tmp/minio2_1.log 2>&1 &
site2_pid=$!
minio server --config /tmp/minio.configfile.3 >/tmp/minio3_1.log 2>&1 &
site3_pid=$!
minio server --config /tmp/minio.configfile.4 >/tmp/minio4_1.log 2>&1 &
site4_pid=$!
export MC_HOST_minio1=http://minr0otUS2r:pBU94AGAY85e@localhost:9001
export MC_HOST_minio3=http://minr0otUS2r:pBU94AGAY85e@localhost:9003
./mc ready minio1
./mc ready minio3
./mc mb minio1/testbucket
# copy large upload to newbucket on minio1
truncate -s 17M lrgfile
expected_checksum=$(cat ./lrgfile | md5sum)
./mc cp ./lrgfile minio1/testbucket
actual_checksum=$(./mc cat minio3/testbucket/lrgfile | md5sum)
if [ "${expected_checksum}" != "${actual_checksum}" ]; then
echo "unexpected object checksum, expected: ${expected_checksum} got: ${actual_checksum}"
exit
fi
# Compare the difference of the list of disks and their location, with the below expected output
diff <(./mc admin info minio1 --json | jq -r '.info.servers[].drives[] | "\(.pool_index),\(.set_index),\(.disk_index) \(.endpoint)"' | sort) <(
cat <<EOF
0,0,0 http://localhost:9001/tmp/xl/node9001/mnt/disk1
0,0,1 http://localhost:9002/tmp/xl/node9002/mnt/disk1
0,0,2 http://localhost:9001/tmp/xl/node9001/mnt/disk2
0,0,3 http://localhost:9002/tmp/xl/node9002/mnt/disk2
0,0,4 http://localhost:9001/tmp/xl/node9001/mnt/disk3
0,0,5 http://localhost:9002/tmp/xl/node9002/mnt/disk3
0,0,6 http://localhost:9001/tmp/xl/node9001/mnt/disk4
0,0,7 http://localhost:9002/tmp/xl/node9002/mnt/disk4
1,0,0 http://localhost:9003/tmp/xl/node9003/mnt/disk1
1,0,1 http://localhost:9004/tmp/xl/node9004/mnt/disk1
1,0,2 http://localhost:9003/tmp/xl/node9003/mnt/disk2
1,0,3 http://localhost:9004/tmp/xl/node9004/mnt/disk2
1,0,4 http://localhost:9003/tmp/xl/node9003/mnt/disk3
1,0,5 http://localhost:9004/tmp/xl/node9004/mnt/disk3
1,0,6 http://localhost:9003/tmp/xl/node9003/mnt/disk4
1,0,7 http://localhost:9004/tmp/xl/node9004/mnt/disk4
EOF
)
cleanup

115
docs/distributed/iam-import-with-missing-entities.sh Executable file
View File

@@ -0,0 +1,115 @@
#!/bin/bash
if [ -n "$TEST_DEBUG" ]; then
set -x
fi
pkill minio
docker rm -f $(docker ps -aq)
rm -rf /tmp/ldap{1..4}
rm -rf /tmp/ldap1{1..4}
if [ ! -f ./mc ]; then
wget --quiet -O mc https://dl.minio.io/client/mc/release/linux-amd64/mc &&
chmod +x mc
fi
mc -v
# Start LDAP server
echo "Copying docs/distributed/samples/bootstrap-complete.ldif => minio-iam-testing/ldap/50-bootstrap.ldif"
cp docs/distributed/samples/bootstrap-complete.ldif minio-iam-testing/ldap/50-bootstrap.ldif || exit 1
cd ./minio-iam-testing
make docker-images
make docker-run
cd -
export MC_HOST_myminio="http://minioadmin:minioadmin@localhost:22000"
export MC_HOST_myminio1="http://minioadmin:minioadmin@localhost:24000"
# Start MinIO instance
export CI=true
(minio server --address :22000 --console-address :10000 http://localhost:22000/tmp/ldap{1...4} 2>&1 >/dev/null) &
sleep 30
./mc ready myminio
./mc idp ldap add myminio server_addr=localhost:389 server_insecure=on \
lookup_bind_dn=cn=admin,dc=min,dc=io lookup_bind_password=admin \
user_dn_search_base_dn=dc=min,dc=io user_dn_search_filter="(uid=%s)" \
group_search_base_dn=ou=swengg,dc=min,dc=io group_search_filter="(&(objectclass=groupOfNames)(member=%d))"
./mc admin service restart myminio --json
./mc ready myminio
./mc admin cluster iam import myminio docs/distributed/samples/myminio-iam-info.zip
sleep 10
# Verify the list of users and service accounts from the import
./mc admin user list myminio
USER_COUNT=$(./mc admin user list myminio | wc -l)
if [ "${USER_COUNT}" -ne 2 ]; then
echo "BUG: Expected no of users: 2 Found: ${USER_COUNT}"
exit 1
fi
./mc admin user svcacct list myminio "uid=bobfisher,ou=people,ou=hwengg,dc=min,dc=io" --json
SVCACCT_COUNT_1=$(./mc admin user svcacct list myminio "uid=bobfisher,ou=people,ou=hwengg,dc=min,dc=io" --json | jq '.accessKey' | wc -l)
if [ "${SVCACCT_COUNT_1}" -ne 2 ]; then
echo "BUG: Expected svcacct count for 'uid=bobfisher,ou=people,ou=hwengg,dc=min,dc=io': 2. Found: ${SVCACCT_COUNT_1}"
exit 1
fi
./mc admin user svcacct list myminio "uid=dillon,ou=people,ou=swengg,dc=min,dc=io" --json
SVCACCT_COUNT_2=$(./mc admin user svcacct list myminio "uid=dillon,ou=people,ou=swengg,dc=min,dc=io" --json | jq '.accessKey' | wc -l)
if [ "${SVCACCT_COUNT_2}" -ne 2 ]; then
echo "BUG: Expected svcacct count for 'uid=dillon,ou=people,ou=swengg,dc=min,dc=io': 2. Found: ${SVCACCT_COUNT_2}"
exit 1
fi
# Kill MinIO and LDAP to start afresh with missing groups/DN
pkill minio
docker rm -f $(docker ps -aq)
rm -rf /tmp/ldap{1..4}
# Deploy the LDAP config witg missing groups/DN
echo "Copying docs/distributed/samples/bootstrap-partial.ldif => minio-iam-testing/ldap/50-bootstrap.ldif"
cp docs/distributed/samples/bootstrap-partial.ldif minio-iam-testing/ldap/50-bootstrap.ldif || exit 1
cd ./minio-iam-testing
make docker-images
make docker-run
cd -
(minio server --address ":24000" --console-address :10000 http://localhost:24000/tmp/ldap1{1...4} 2>&1 >/dev/null) &
sleep 30
./mc ready myminio1
./mc idp ldap add myminio1 server_addr=localhost:389 server_insecure=on \
lookup_bind_dn=cn=admin,dc=min,dc=io lookup_bind_password=admin \
user_dn_search_base_dn=dc=min,dc=io user_dn_search_filter="(uid=%s)" \
group_search_base_dn=ou=hwengg,dc=min,dc=io group_search_filter="(&(objectclass=groupOfNames)(member=%d))"
./mc admin service restart myminio1 --json
./mc ready myminio1
./mc admin cluster iam import myminio1 docs/distributed/samples/myminio-iam-info.zip
sleep 10
# Verify the list of users and service accounts from the import
./mc admin user list myminio1
USER_COUNT=$(./mc admin user list myminio1 | wc -l)
if [ "${USER_COUNT}" -ne 1 ]; then
echo "BUG: Expected no of users: 1 Found: ${USER_COUNT}"
exit 1
fi
./mc admin user svcacct list myminio1 "uid=bobfisher,ou=people,ou=hwengg,dc=min,dc=io" --json
SVCACCT_COUNT_1=$(./mc admin user svcacct list myminio1 "uid=bobfisher,ou=people,ou=hwengg,dc=min,dc=io" --json | jq '.accessKey' | wc -l)
if [ "${SVCACCT_COUNT_1}" -ne 2 ]; then
echo "BUG: Expected svcacct count for 'uid=bobfisher,ou=people,ou=hwengg,dc=min,dc=io': 2. Found: ${SVCACCT_COUNT_1}"
exit 1
fi
./mc admin user svcacct list myminio1 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io" --json
SVCACCT_COUNT_2=$(./mc admin user svcacct list myminio1 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io" --json | jq '.accessKey' | wc -l)
if [ "${SVCACCT_COUNT_2}" -ne 0 ]; then
echo "BUG: Expected svcacct count for 'uid=dillon,ou=people,ou=swengg,dc=min,dc=io': 0. Found: ${SVCACCT_COUNT_2}"
exit 1
fi
# Finally kill running processes
pkill minio
docker rm -f $(docker ps -aq)

82
docs/distributed/iam-import-with-openid.sh Executable file
View File

@@ -0,0 +1,82 @@
#!/bin/bash
if [ -n "$TEST_DEBUG" ]; then
set -x
fi
pkill minio
docker rm -f $(docker ps -aq)
rm -rf /tmp/openid{1..4}
export MC_HOST_myminio="http://minioadmin:minioadmin@localhost:22000"
# The service account used below is already present in iam configuration getting imported
export MC_HOST_myminio1="http://dillon-service-2:dillon-service-2@localhost:22000"
# Start MinIO instance
export CI=true
if [ ! -f ./mc ]; then
wget --quiet -O mc https://dl.minio.io/client/mc/release/linux-amd64/mc &&
chmod +x mc
fi
mc -v
# Start openid server
(
cd ./minio-iam-testing
make docker-images
make docker-run
cd -
)
(minio server --address :22000 --console-address :10000 http://localhost:22000/tmp/openid{1...4} 2>&1 >/tmp/server.log) &
./mc ready myminio
./mc mb myminio/test-bucket
./mc cp /etc/hosts myminio/test-bucket
./mc idp openid add myminio \
config_url="http://localhost:5556/dex/.well-known/openid-configuration" \
client_id="minio-client-app" \
client_secret="minio-client-app-secret" \
scopes="openid,groups,email,profile" \
redirect_uri="http://127.0.0.1:10000/oauth_callback" \
display_name="Login via dex1" \
role_policy="consoleAdmin"
./mc admin service restart myminio --json
./mc ready myminio
./mc admin cluster iam import myminio docs/distributed/samples/myminio-iam-info-openid.zip
# Verify if buckets / objects accessible using service account
echo "Verifying buckets and objects access for the imported service account"
./mc ls myminio1/ --json
BKT_COUNT=$(./mc ls myminio1/ --json | jq '.key' | wc -l)
if [ "${BKT_COUNT}" -ne 1 ]; then
echo "BUG: Expected no of bucket: 1, Found: ${BKT_COUNT}"
exit 1
fi
BKT_NAME=$(./mc ls myminio1/ --json | jq '.key' | sed 's/"//g' | sed 's\/\\g')
if [[ ${BKT_NAME} != "test-bucket" ]]; then
echo "BUG: Expected bucket: test-bucket, Found: ${BKT_NAME}"
exit 1
fi
./mc ls myminio1/test-bucket
OBJ_COUNT=$(./mc ls myminio1/test-bucket --json | jq '.key' | wc -l)
if [ "${OBJ_COUNT}" -ne 1 ]; then
echo "BUG: Expected no of objects: 1, Found: ${OBJ_COUNT}"
exit 1
fi
OBJ_NAME=$(./mc ls myminio1/test-bucket --json | jq '.key' | sed 's/"//g')
if [[ ${OBJ_NAME} != "hosts" ]]; then
echo "BUG: Expected object: hosts, Found: ${BKT_NAME}"
exit 1
fi
# Finally kill running processes
pkill minio
docker rm -f $(docker ps -aq)

14
docs/distributed/rw.json Normal file
View File

@@ -0,0 +1,14 @@
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:*"
],
"Resource": [
"arn:aws:s3:::*"
]
}
]
}

123
docs/distributed/samples/bootstrap-complete.ldif Normal file
View File

@@ -0,0 +1,123 @@
# Create hardware engg org unit
dn: ou=hwengg,dc=min,dc=io
objectClass: organizationalUnit
ou: hwengg
# Create people sub-org
dn: ou=people,ou=hwengg,dc=min,dc=io
objectClass: organizationalUnit
ou: people
# Create Alice, Bob and Cody in hwengg
dn: uid=alice1,ou=people,ou=hwengg,dc=min,dc=io
objectClass: inetOrgPerson
cn: Alice Smith
sn: Smith
uid: alice1
mail: alice@example.io
userPassword: {SSHA}Yeh2/IV/q/HjG2yzN3YdE9CAF3EJFCLu
dn: uid=bobfisher,ou=people,ou=hwengg,dc=min,dc=io
objectClass: inetOrgPerson
cn: Robert Fisher
sn: Fisher
uid: bobfisher
mail: bob@example.io
userPassword: {SSHA}LktfbhK5oXSdDWCNzauJ9JA+Poxinl3y
dn: uid=cody3,ou=people,ou=hwengg,dc=min,dc=io
objectClass: inetOrgPerson
cn: Cody Thomas
sn: Thomas
uid: cody3
mail: cody@example.io
userPassword: {SSHA}H8B0gaOd4bRklK3fXj9ltHvJXWQFXW5Q
# Create groups ou for hwengg
dn: ou=groups,ou=hwengg,dc=min,dc=io
objectclass: organizationalUnit
ou: groups
description: groups branch
# Create project groups
dn: cn=projectx,ou=groups,ou=hwengg,dc=min,dc=io
objectclass: groupofnames
cn: projectx
description: Project X group members
member: uid=alice1,ou=people,ou=hwengg,dc=min,dc=io
member: uid=bobfisher,ou=people,ou=hwengg,dc=min,dc=io
dn: cn=projecty,ou=groups,ou=hwengg,dc=min,dc=io
objectclass: groupofnames
cn: projecty
description: Project Y group members
member: uid=bobfisher,ou=people,ou=hwengg,dc=min,dc=io
member: uid=cody3,ou=people,ou=hwengg,dc=min,dc=io
# Create software engg org unit
dn: ou=swengg,dc=min,dc=io
objectClass: organizationalUnit
ou: swengg
# Create people sub-org
dn: ou=people,ou=swengg,dc=min,dc=io
objectClass: organizationalUnit
ou: people
# Create Dillon, Elizabeth and Fahim in swengg
dn: uid=dillon,ou=people,ou=swengg,dc=min,dc=io
objectClass: inetOrgPerson
cn: Dillon Harper
sn: Harper
uid: dillon
mail: dillon@example.io
userPassword: {SSHA}UH+LmoEhWWW6s9rjgdpqHPI0qCMouY8+
dn: uid=liza,ou=people,ou=swengg,dc=min,dc=io
objectClass: inetOrgPerson
cn: Elizabeth Jones
sn: Jones
uid: liza
mail: ejones@example.io
userPassword: {SSHA}feVkKkafHtsu2Io7n0tQP4Cnh8/Oy1PK
dn: uid=fahim,ou=people,ou=swengg,dc=min,dc=io
objectClass: inetOrgPerson
cn: Fahim Ahmed
sn: Ahmed
uid: fahim
mail: fahmed@example.io
userPassword: {SSHA}lRNH+PHooRaruiEb+CBEA21EZLMkAmcc
# Add a user with special chars. The password = example here.
dn: uid=Пользователь,OU=people,OU=swengg,DC=min,DC=io
objectClass: inetOrgPerson
cn: Special Charsman
sn: Charsman
uid: Пользователь
mail: scharsman@example.io
userPassword: {SSHA}XQSZqLPvYgm30wR7pk67a1GW+q+DDvSj
# Creates groups ou for swengg
dn: ou=groups,ou=swengg,dc=min,dc=io
objectclass: organizationalUnit
ou: groups
description: groups branch
# Create project groups
dn: cn=projecta,ou=groups,ou=swengg,dc=min,dc=io
objectclass: groupofnames
cn: projecta
description: Project A group members
member: uid=dillon,ou=people,ou=swengg,dc=min,dc=io
dn: cn=projectb,ou=groups,ou=swengg,dc=min,dc=io
objectclass: groupofnames
cn: projectb
description: Project B group members
member: uid=dillon,ou=people,ou=swengg,dc=min,dc=io
member: uid=liza,ou=people,ou=swengg,dc=min,dc=io
member: uid=fahim,ou=people,ou=swengg,dc=min,dc=io
member: uid=Пользователь,OU=people,OU=swengg,DC=min,DC=io

56
docs/distributed/samples/bootstrap-partial.ldif Normal file
View File

@@ -0,0 +1,56 @@
# Create hardware engg org unit
dn: ou=hwengg,dc=min,dc=io
objectClass: organizationalUnit
ou: hwengg
# Create people sub-org
dn: ou=people,ou=hwengg,dc=min,dc=io
objectClass: organizationalUnit
ou: people
# Create Alice, Bob and Cody in hwengg
dn: uid=alice1,ou=people,ou=hwengg,dc=min,dc=io
objectClass: inetOrgPerson
cn: Alice Smith
sn: Smith
uid: alice1
mail: alice@example.io
userPassword: {SSHA}Yeh2/IV/q/HjG2yzN3YdE9CAF3EJFCLu
dn: uid=bobfisher,ou=people,ou=hwengg,dc=min,dc=io
objectClass: inetOrgPerson
cn: Robert Fisher
sn: Fisher
uid: bobfisher
mail: bob@example.io
userPassword: {SSHA}LktfbhK5oXSdDWCNzauJ9JA+Poxinl3y
dn: uid=cody3,ou=people,ou=hwengg,dc=min,dc=io
objectClass: inetOrgPerson
cn: Cody Thomas
sn: Thomas
uid: cody3
mail: cody@example.io
userPassword: {SSHA}H8B0gaOd4bRklK3fXj9ltHvJXWQFXW5Q
# Create groups ou for hwengg
dn: ou=groups,ou=hwengg,dc=min,dc=io
objectclass: organizationalUnit
ou: groups
description: groups branch
# Create project groups
dn: cn=projectx,ou=groups,ou=hwengg,dc=min,dc=io
objectclass: groupofnames
cn: projectx
description: Project X group members
member: uid=alice1,ou=people,ou=hwengg,dc=min,dc=io
member: uid=bobfisher,ou=people,ou=hwengg,dc=min,dc=io
dn: cn=projecty,ou=groups,ou=hwengg,dc=min,dc=io
objectclass: groupofnames
cn: projecty
description: Project Y group members
member: uid=bobfisher,ou=people,ou=hwengg,dc=min,dc=io
member: uid=cody3,ou=people,ou=hwengg,dc=min,dc=io

BIN
docs/distributed/samples/myminio-iam-info-openid.zip Normal file
View File

Binary file not shown.

BIN
docs/distributed/samples/myminio-iam-info.zip Normal file
View File

Binary file not shown.

215
docs/docker/README.md Normal file
View File

@@ -0,0 +1,215 @@
# MinIO Docker Quickstart Guide [![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io) [![Docker Pulls](https://img.shields.io/docker/pulls/minio/minio.svg?maxAge=604800)](https://hub.docker.com/r/minio/minio/)
See our web documentation on [Deploying MinIO in Standalone Mode](Deploy Standalone MinIO in a Container) for a more structured tutorial on deploying MinIO in a container.
## Prerequisites
Docker installed on your machine. Download the relevant installer from [here](https://www.docker.com/community-edition#/download).
## Run Standalone MinIO on Docker
*Note*: Standalone MinIO is intended for early development and evaluation. For production clusters, deploy a [Distributed](https://docs.min.io/community/minio-object-store/operations/deployments/baremetal-deploy-minio-as-a-container.html) MinIO deployment.
MinIO needs a persistent volume to store configuration and application data. For testing purposes, you can launch MinIO by simply passing a directory (`/data` in the example below). This directory gets created in the container filesystem at the time of container start. But all the data is lost after container exits.
```sh
docker run \
-p 9000:9000 \
-p 9001:9001 \
-e "MINIO_ROOT_USER=AKIAIOSFODNN7EXAMPLE" \
-e "MINIO_ROOT_PASSWORD=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" \
quay.io/minio/minio server /data --console-address ":9001"
```
To create a MinIO container with persistent storage, you need to map local persistent directories from the host OS to virtual config. To do this, run the below commands
### GNU/Linux and macOS
```sh
mkdir -p ~/minio/data
docker run \
-p 9000:9000 \
-p 9001:9001 \
--name minio1 \
-v ~/minio/data:/data \
-e "MINIO_ROOT_USER=AKIAIOSFODNN7EXAMPLE" \
-e "MINIO_ROOT_PASSWORD=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" \
quay.io/minio/minio server /data --console-address ":9001"
```
The command creates a new local directory `~/minio/data` in your user home directory. It then starts the MinIO container with the `-v` argument to map the local path (`~/minio/data`) to the specified virtual container directory (`/data`). When MinIO writes data to `/data`, that data is actually written to the local path `~/minio/data` where it can persist between container restarts.
### Windows
```sh
docker run \
-p 9000:9000 \
-p 9001:9001 \
--name minio1 \
-v D:\data:/data \
-e "MINIO_ROOT_USER=AKIAIOSFODNN7EXAMPLE" \
-e "MINIO_ROOT_PASSWORD=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" \
quay.io/minio/minio server /data --console-address ":9001"
```
## Run Distributed MinIO on Containers
We recommend kubernetes based deployment for production level deployment <https://github.com/minio/operator>.
See the [Kubernetes documentation](https://docs.min.io/community/minio-object-store/operations/deployments/kubernetes.html) for more information.
## MinIO Docker Tips
### MinIO Custom Access and Secret Keys
To override MinIO's auto-generated keys, you may pass secret and access keys explicitly as environment variables. MinIO server also allows regular strings as access and secret keys.
#### GNU/Linux and macOS (custom access and secret keys)
```sh
docker run \
-p 9000:9000 \
-p 9001:9001 \
--name minio1 \
-e "MINIO_ROOT_USER=AKIAIOSFODNN7EXAMPLE" \
-e "MINIO_ROOT_PASSWORD=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" \
-v /mnt/data:/data \
quay.io/minio/minio server /data --console-address ":9001"
```
#### Windows (custom access and secret keys)
```powershell
docker run \
-p 9000:9000 \
-p 9001:9001 \
--name minio1 \
-e "MINIO_ROOT_USER=AKIAIOSFODNN7EXAMPLE" \
-e "MINIO_ROOT_PASSWORD=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" \
-v D:\data:/data \
quay.io/minio/minio server /data --console-address ":9001"
```
### Run MinIO Docker as a regular user
Docker provides standardized mechanisms to run docker containers as non-root users.
#### GNU/Linux and macOS (regular user)
On Linux and macOS you can use `--user` to run the container as regular user.
> NOTE: make sure --user has write permission to *${HOME}/data* prior to using `--user`.
```sh
mkdir -p ${HOME}/data
docker run \
-p 9000:9000 \
-p 9001:9001 \
--user $(id -u):$(id -g) \
--name minio1 \
-e "MINIO_ROOT_USER=AKIAIOSFODNN7EXAMPLE" \
-e "MINIO_ROOT_PASSWORD=wJalrXUtnFEMIK7MDENGbPxRfiCYEXAMPLEKEY" \
-v ${HOME}/data:/data \
quay.io/minio/minio server /data --console-address ":9001"
```
#### Windows (regular user)
On windows you would need to use [Docker integrated windows authentication](https://success.docker.com/article/modernizing-traditional-dot-net-applications#integratedwindowsauthentication) and [Create a container with Active Directory Support](https://blogs.msdn.microsoft.com/containerstuff/2017/01/30/create-a-container-with-active-directory-support/)
> NOTE: make sure your AD/Windows user has write permissions to *D:\data* prior to using `credentialspec=`.
```powershell
docker run \
-p 9000:9000 \
-p 9001:9001 \
--name minio1 \
--security-opt "credentialspec=file://myuser.json"
-e "MINIO_ROOT_USER=AKIAIOSFODNN7EXAMPLE" \
-e "MINIO_ROOT_PASSWORD=wJalrXUtnFEMIK7MDENGbPxRfiCYEXAMPLEKEY" \
-v D:\data:/data \
quay.io/minio/minio server /data --console-address ":9001"
```
### MinIO Custom Access and Secret Keys using Docker secrets
To override MinIO's auto-generated keys, you may pass secret and access keys explicitly by creating access and secret keys as [Docker secrets](https://docs.docker.com/engine/swarm/secrets/). MinIO server also allows regular strings as access and secret keys.
```
echo "AKIAIOSFODNN7EXAMPLE" | docker secret create access_key -
echo "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" | docker secret create secret_key -
```
Create a MinIO service using `docker service` to read from Docker secrets.
```
docker service create --name="minio-service" --secret="access_key" --secret="secret_key" quay.io/minio/minio server /data
```
Read more about `docker service` [here](https://docs.docker.com/engine/swarm/how-swarm-mode-works/services/)
#### MinIO Custom Access and Secret Key files
To use other secret names follow the instructions above and replace `access_key` and `secret_key` with your custom names (e.g. `my_secret_key`,`my_custom_key`). Run your service with
```
docker service create --name="minio-service" \
--secret="my_access_key" \
--secret="my_secret_key" \
--env="MINIO_ROOT_USER_FILE=my_access_key" \
--env="MINIO_ROOT_PASSWORD_FILE=my_secret_key" \
quay.io/minio/minio server /data
```
`MINIO_ROOT_USER_FILE` and `MINIO_ROOT_PASSWORD_FILE` also support custom absolute paths, in case Docker secrets are mounted to custom locations or other tools are used to mount secrets into the container. For example, HashiCorp Vault injects secrets to `/vault/secrets`. With the custom names above, set the environment variables to
```
MINIO_ROOT_USER_FILE=/vault/secrets/my_access_key
MINIO_ROOT_PASSWORD_FILE=/vault/secrets/my_secret_key
```
### Retrieving Container ID
To use Docker commands on a specific container, you need to know the `Container ID` for that container. To get the `Container ID`, run
```sh
docker ps -a
```
`-a` flag makes sure you get all the containers (Created, Running, Exited). Then identify the `Container ID` from the output.
### Starting and Stopping Containers
To start a stopped container, you can use the [`docker start`](https://docs.docker.com/engine/reference/commandline/start/) command.
```sh
docker start <container_id>
```
To stop a running container, you can use the [`docker stop`](https://docs.docker.com/engine/reference/commandline/stop/) command.
```sh
docker stop <container_id>
```
### MinIO container logs
To access MinIO logs, you can use the [`docker logs`](https://docs.docker.com/engine/reference/commandline/logs/) command.
```sh
docker logs <container_id>
```
### Monitor MinIO Docker Container
To monitor the resources used by MinIO container, you can use the [`docker stats`](https://docs.docker.com/engine/reference/commandline/stats/) command.
```sh
docker stats <container_id>
```
## Explore Further
* [MinIO in a Container Installation Guide](https://docs.min.io/community/minio-object-store/operations/deployments/baremetal-deploy-minio-as-a-container.html)
* [MinIO Erasure Code QuickStart Guide](https://docs.min.io/community/minio-object-store/operations/concepts/erasure-coding.html)

67
docs/erasure/README.md Normal file
View File

@@ -0,0 +1,67 @@
# MinIO Erasure Code Quickstart Guide [![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io)
MinIO protects data against hardware failures and silent data corruption using erasure code and checksums. With the highest level of redundancy, you may lose up to half (N/2) of the total drives and still be able to recover the data.
## What is Erasure Code?
Erasure code is a mathematical algorithm to reconstruct missing or corrupted data. MinIO uses Reed-Solomon code to shard objects into variable data and parity blocks. For example, in a 12 drive setup, an object can be sharded to a variable number of data and parity blocks across all the drives - ranging from six data and six parity blocks to ten data and two parity blocks.
By default, MinIO shards the objects across N/2 data and N/2 parity drives. Though, you can use [storage classes](https://github.com/minio/minio/tree/master/docs/erasure/storage-class) to use a custom configuration. We recommend N/2 data and parity blocks, as it ensures the best protection from drive failures.
In 12 drive example above, with MinIO server running in the default configuration, you can lose any of the six drives and still reconstruct the data reliably from the remaining drives.
## Why is Erasure Code useful?
Erasure code protects data from multiple drives failure, unlike RAID or replication. For example, RAID6 can protect against two drive failure whereas in MinIO erasure code you can lose as many as half of drives and still the data remains safe. Further, MinIO's erasure code is at the object level and can heal one object at a time. For RAID, healing can be done only at the volume level which translates into high downtime. As MinIO encodes each object individually, it can heal objects incrementally. Storage servers once deployed should not require drive replacement or healing for the lifetime of the server. MinIO's erasure coded backend is designed for operational efficiency and takes full advantage of hardware acceleration whenever available.
![Erasure](https://github.com/minio/minio/blob/master/docs/screenshots/erasure-code.jpg?raw=true)
## What is Bit Rot protection?
Bit Rot, also known as data rot or silent data corruption is a data loss issue faced by disk drives today. Data on the drive may silently get corrupted without signaling an error has occurred, making bit rot more dangerous than a permanent hard drive failure.
MinIO's erasure coded backend uses high speed [HighwayHash](https://github.com/minio/highwayhash) checksums to protect against Bit Rot.
## How are drives used for Erasure Code?
MinIO divides the drives you provide into erasure-coding sets of *2 to 16* drives. Therefore, the number of drives you present must be a multiple of one of these numbers. Each object is written to a single erasure-coding set.
Minio uses the largest possible EC set size which divides into the number of drives given. For example, *18 drives* are configured as *2 sets of 9 drives*, and *24 drives* are configured as *2 sets of 12 drives*. This is true for scenarios when running MinIO as a standalone erasure coded deployment. In [distributed setup however node (affinity) based](https://docs.min.io/community/minio-object-store/operations/install-deploy-manage/deploy-minio-multi-node-multi-drive.html) erasure stripe sizes are chosen.
The drives should all be of approximately the same size.
## Get Started with MinIO in Erasure Code
### 1. Prerequisites
Install MinIO - [MinIO Quickstart Guide](https://docs.min.io/community/minio-object-store/operations/deployments/baremetal-deploy-minio-on-redhat-linux.html)
### 2. Run MinIO Server with Erasure Code
Example: Start MinIO server in a 12 drives setup, using MinIO binary.
```sh
minio server /data{1...12}
```
Example: Start MinIO server in a 8 drives setup, using MinIO Docker image.
```sh
podman run \
-p 9000:9000 \
-p 9001:9001 \
--name minio \
-v /mnt/data1:/data1 \
-v /mnt/data2:/data2 \
-v /mnt/data3:/data3 \
-v /mnt/data4:/data4 \
-v /mnt/data5:/data5 \
-v /mnt/data6:/data6 \
-v /mnt/data7:/data7 \
-v /mnt/data8:/data8 \
quay.io/minio/minio server /data{1...8} --console-address ":9001"
```
### 3. Test your setup
You may unplug drives randomly and continue to perform I/O on the system.

119
docs/erasure/storage-class/README.md Normal file
View File

@@ -0,0 +1,119 @@
# MinIO Storage Class Quickstart Guide [![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io)
MinIO server supports storage class in erasure coding mode. This allows configurable data and parity drives per object.
This page is intended as a summary of MinIO Erasure Coding. For a more complete explanation, see <https://docs.min.io/community/minio-object-store/operations/concepts/erasure-coding.html>.
## Overview
MinIO supports two storage classes, Reduced Redundancy class and Standard class. These classes can be defined using environment variables
set before starting MinIO server. After the data and parity drives for each storage class are defined using environment variables,
you can set the storage class of an object via request metadata field `x-amz-storage-class`. MinIO server then honors the storage class by
saving the object in specific number of data and parity drives.
## Storage usage
The selection of varying data and parity drives has a direct impact on the drive space usage. With storage class, you can optimize for high
redundancy or better drive space utilization.
To get an idea of how various combinations of data and parity drives affect the storage usage, let’s take an example of a 100 MiB file stored
on 16 drive MinIO deployment. If you use eight data and eight parity drives, the file space usage will be approximately twice, i.e. 100 MiB
file will take 200 MiB space. But, if you use ten data and six parity drives, same 100 MiB file takes around 160 MiB. If you use 14 data and
two parity drives, 100 MiB file takes only approximately 114 MiB.
Below is a list of data/parity drives and corresponding _approximate_ storage space usage on a 16 drive MinIO deployment. The field _storage
usage ratio_ is simply the drive space used by the file after erasure-encoding, divided by actual file size.
| Total Drives (N) | Data Drives (D) | Parity Drives (P) | Storage Usage Ratio |
|------------------|-----------------|-------------------|---------------------|
| 16 | 8 | 8 | 2.00 |
| 16 | 9 | 7 | 1.79 |
| 16 | 10 | 6 | 1.60 |
| 16 | 11 | 5 | 1.45 |
| 16 | 12 | 4 | 1.34 |
| 16 | 13 | 3 | 1.23 |
| 16 | 14 | 2 | 1.14 |
You can calculate _approximate_ storage usage ratio using the formula - total drives (N) / data drives (D).
### Allowed values for STANDARD storage class
`STANDARD` storage class implies more parity than `REDUCED_REDUNDANCY` class. So, `STANDARD` parity drives should be
- Greater than or equal to 2, if `REDUCED_REDUNDANCY` parity is not set.
- Greater than `REDUCED_REDUNDANCY` parity, if it is set.
Parity blocks can not be higher than data blocks, so `STANDARD` storage class parity can not be higher than N/2. (N being total number of drives)
The default value for the `STANDARD` storage class depends on the number of volumes in the erasure set:
| Erasure Set Size | Default Parity (EC:N) |
|------------------|-----------------------|
| 5 or fewer | EC:2 |
| 6-7 | EC:3 |
| 8 or more | EC:4 |
For more complete documentation on Erasure Set sizing, see the [MinIO Documentation on Erasure Sets](https://docs.min.io/community/minio-object-store/operations/concepts/erasure-coding.html#erasure-sets).
### Allowed values for REDUCED_REDUNDANCY storage class
`REDUCED_REDUNDANCY` implies lesser parity than `STANDARD` class. So,`REDUCED_REDUNDANCY` parity drives should be
- Less than N/2, if `STANDARD` parity is not set.
- Less than `STANDARD` Parity, if it is set.
Default value for `REDUCED_REDUNDANCY` storage class is `1`.
## Get started with Storage Class
### Set storage class
The format to set storage class environment variables is as follows
`MINIO_STORAGE_CLASS_STANDARD=EC:parity`
`MINIO_STORAGE_CLASS_RRS=EC:parity`
For example, set `MINIO_STORAGE_CLASS_RRS` parity 2 and `MINIO_STORAGE_CLASS_STANDARD` parity 3
```sh
export MINIO_STORAGE_CLASS_STANDARD=EC:3
export MINIO_STORAGE_CLASS_RRS=EC:2
```
Storage class can also be set via `mc admin config` get/set commands to update the configuration. Refer [storage class](https://github.com/minio/minio/tree/master/docs/config#storage-class) for
more details.
#### Note
- If `STANDARD` storage class is set via environment variables or `mc admin config` get/set commands, and `x-amz-storage-class` is not present in request metadata, MinIO server will
apply `STANDARD` storage class to the object. This means the data and parity drives will be used as set in `STANDARD` storage class.
- If storage class is not defined before starting MinIO server, and subsequent PutObject metadata field has `x-amz-storage-class` present
with values `REDUCED_REDUNDANCY` or `STANDARD`, MinIO server uses default parity values.
### Set metadata
In below example `minio-go` is used to set the storage class to `REDUCED_REDUNDANCY`. This means this object will be split across 6 data drives and 2 parity drives (as per the storage class set in previous step).
```go
s3Client, err := minio.New("localhost:9000", "YOUR-ACCESSKEYID", "YOUR-SECRETACCESSKEY", true)
if err != nil {
log.Fatalln(err)
}
object, err := os.Open("my-testfile")
if err != nil {
log.Fatalln(err)
}
defer object.Close()
objectStat, err := object.Stat()
if err != nil {
log.Fatalln(err)
}
n, err := s3Client.PutObject("my-bucketname", "my-objectname", object, objectStat.Size(), minio.PutObjectOptions{ContentType: "application/octet-stream", StorageClass: "REDUCED_REDUNDANCY"})
if err != nil {
log.Fatalln(err)
}
log.Println("Uploaded", "my-objectname", " of size: ", n, "Successfully.")
```

18
docs/extensions/fan-out/README.md Normal file
View File

@@ -0,0 +1,18 @@
# Fan-Out Uploads [![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io) [![Docker Pulls](https://img.shields.io/docker/pulls/minio/minio.svg?maxAge=604800)](https://hub.docker.com/r/minio/minio/)
## Overview
MinIO implements an S3 extension to perform multiple concurrent fan-out upload operations. A perfect use case scenario for performing fan-out operations of incoming TSB (Time Shift Buffer's). TSBs are a method of facilitating time-shifted playback of television signaling, and media content.
MinIO implements an S3 extension to the [PostUpload](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPOST.html) where in a special fan-out list is sent along with the TSB's for MinIO make multiple uploads from a single source stream. Optionally supports custom metadata, tags and other retention settings. All objects are also readable independently once upload is completed via the regular S3 [GetObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html) API.
## How to enable Fan-Out Uploads ?
Fan-Out uploads are automatically enabled if `x-minio-fanout-list` form-field is provided with the PostUpload API, to keep things simple higher level APIs are provided in our SDKs for example in `minio-go` SDK:
```
PutObjectFanOut(ctx context.Context, bucket string, fanOutContent io.Reader, fanOutReq minio.PutObjectFanOutRequest) ([]minio.PutObjectFanOutResponse, error)
```

45
docs/extensions/s3zip/README.md Normal file
View File

@@ -0,0 +1,45 @@
# Perform S3 operations in a ZIP content[![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io) [![Docker Pulls](https://img.shields.io/docker/pulls/minio/minio.svg?maxAge=604800)](https://hub.docker.com/r/minio/minio/)
## Overview
MinIO implements an S3 extension to list, stat and download files inside a ZIP file stored in any bucket. A perfect use case scenario is when you have a lot of small files archived in multiple ZIP files. Uploading them is faster than uploading small files individually. Besides, your S3 applications will be able to access to the data with little performance overhead.
The main limitation is that to update or delete content of a file inside a ZIP file the entire ZIP file must be replaced.
## How to enable S3 ZIP behavior ?
Ensure to set the following header `x-minio-extract` to `true` in your S3 requests.
## How to access to files inside a ZIP archive
Accessing to contents inside an archive can be done using regular S3 API with a modified request path. You just need to append the path of the content inside the archive to the path of the archive itself.
e.g.:
To download `2021/taxes.csv` archived in `financial.zip` and stored under a bucket named `company-data`, you can issue a GET request using the following path 'company-data/financial.zip/2021/taxes.csv`
## Contents properties
All properties except the file size are tied to the zip file. This means that modification date, headers, tags, etc. can only be set for the zip file as a whole. In similar fashion, replication will replicate the zip file as a whole and not individual files.
## Code Examples
[Using minio-go library](https://github.com/minio/minio/blob/master/docs/extensions/s3zip/examples/minio-go/main.go)
[Using AWS JS SDK v2](https://github.com/minio/minio/blob/master/docs/extensions/s3zip/examples/aws-js/main.js)
[Using boto3](https://github.com/minio/minio/blob/master/docs/extensions/s3zip/examples/boto3/main.py)
## Requirements and limits
- ListObjectsV2 can only list the most recent ZIP archive version of your object, applicable only for versioned buckets.
- ListObjectsV2 API calls must be used to list zip file content.
- Range requests for GetObject/HeadObject for individual files from zip is not supported.
- Names inside ZIP files are kept unmodified, but some may lead to invalid paths. See [Object key naming guidelines](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html) on safe names.
- This API behavior is limited for following **read** operations on files inside a zip archive:
- `HeadObject`
- `GetObject`
- `ListObjectsV2`
- If the ZIP file directory isn't located within the last 100MB the file will not be parsed.
- A maximum of 100M inside a single zip is allowed. However, a reasonable limit of 100,000 files inside a single ZIP archive is recommended for best performance and memory usage trade-off.
## Content-Type
The Content-Type of the response will be determined by the extension and the following: https://pkg.go.dev/mime#TypeByExtension

31
docs/extensions/s3zip/examples/aws-js/main.js Normal file
View File

@@ -0,0 +1,31 @@
var AWS = require('aws-sdk');
var s3 = new AWS.S3({
accessKeyId: 'YOUR-ACCESSKEYID' ,
secretAccessKey: 'YOUR-SECRETACCESSKEY' ,
endpoint: 'http://127.0.0.1:9000' ,
s3ForcePathStyle: true,
signatureVersion: 'v4'
});
// List all contents stored in the zip archive
s3.listObjectsV2({Bucket : 'your-bucket', Prefix: 'path/to/file.zip/'}).
on('build', function(req) { req.httpRequest.headers['X-Minio-Extract'] = 'true'; }).
send(function(err, data) {
if (err) {
console.log("Error", err);
} else {
console.log("Success", data);
}
});
// Download a file in the archive and store it in /tmp/data.csv
var file = require('fs').createWriteStream('/tmp/data.csv');
s3.getObject({Bucket: 'your-bucket', Key: 'path/to/file.zip/data.csv'}).
on('build', function(req) { req.httpRequest.headers['X-Minio-Extract'] = 'true'; }).
on('httpData', function(chunk) { file.write(chunk); }).
on('httpDone', function() { file.end(); }).
send();

8
docs/extensions/s3zip/examples/aws-js/package.json Normal file
View File

@@ -0,0 +1,8 @@
{
"name": "s3-zip-example",
"version": "1.0.0",
"main": "main.js",
"dependencies": {
"aws-sdk": "^2.924.0"
}
}

25
docs/extensions/s3zip/examples/boto3/main.py Normal file
View File

@@ -0,0 +1,25 @@
#!/usr/bin/env/python
import boto3
from botocore.client import Config
s3 = boto3.client('s3',
endpoint_url='http://localhost:9000',
aws_access_key_id='YOUR-ACCESSKEYID',
aws_secret_access_key='YOUR-SECRETACCESSKEY',
config=Config(signature_version='s3v4'),
region_name='us-east-1')
def _add_header(request, **kwargs):
request.headers.add_header('x-minio-extract', 'true')
event_system = s3.meta.events
event_system.register_first('before-sign.s3.*', _add_header)
# List zip contents
response = s3.list_objects_v2(Bucket="your-bucket", Prefix="path/to/file.zip/")
print(response)
# Download data.csv stored in the zip file
s3.download_file(Bucket='your-bucket', Key='path/to/file.zip/data.csv', Filename='/tmp/data.csv')

35
docs/extensions/s3zip/examples/minio-go/main.go Normal file
View File

@@ -0,0 +1,35 @@
package main
import (
"context"
"io"
"log"
"os"
"github.com/minio/minio-go/v7"
"github.com/minio/minio-go/v7/pkg/credentials"
)
func main() {
s3Client, err := minio.New("minio-server-address:9000", &minio.Options{
Creds: credentials.NewStaticV4("access-key", "secret-key", ""),
})
if err != nil {
log.Fatalln(err)
}
var opts minio.GetObjectOptions
// Add extract header to request:
opts.Set("x-minio-extract", "true")
// Download API.md from the archive
rd, err := s3Client.GetObject(context.Background(), "your-bucket", "path/to/file.zip/data.csv", opts)
if err != nil {
log.Fatalln(err)
}
_, err = io.Copy(os.Stdout, rd)
if err != nil {
log.Fatalln(err)
}
}

11
docs/federation/lookup/Corefile.example Normal file
View File

@@ -0,0 +1,11 @@
. {
etcd churchofminio.com {
endpoint http://localhost:2379 http://localhost:4001
upstream /etc/resolv.conf
}
debug
prometheus
cache 160 mydomain.com
loadbalance
forward . /etc/resolv.conf
}

86
docs/federation/lookup/README.md Normal file
View File

@@ -0,0 +1,86 @@
# Federation Quickstart Guide [![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io) *Federation feature is deprecated and should be avoided for future deployments*
This document explains how to configure MinIO with `Bucket lookup from DNS` style federation.
## Get started
### 1. Prerequisites
Install MinIO - [MinIO Quickstart Guide](https://docs.min.io/community/minio-object-store/operations/deployments/baremetal-deploy-minio-on-redhat-linux.html).
### 2. Run MinIO in federated mode
Bucket lookup from DNS federation requires two dependencies
- etcd (for bucket DNS service records)
- CoreDNS (for DNS management based on populated bucket DNS service records, optional)
## Architecture
![bucket-lookup](https://github.com/minio/minio/blob/master/docs/federation/lookup/bucket-lookup.png?raw=true)
### Environment variables
#### MINIO_ETCD_ENDPOINTS
This is comma separated list of etcd servers that you want to use as the MinIO federation back-end. This should
be same across the federated deployment, i.e. all the MinIO instances within a federated deployment should use same
etcd back-end.
#### MINIO_DOMAIN
This is the top level domain name used for the federated setup. This domain name should ideally resolve to a load-balancer
running in front of all the federated MinIO instances. The domain name is used to create sub domain entries to etcd. For
example, if the domain is set to `domain.com`, the buckets `bucket1`, `bucket2` will be accessible as `bucket1.domain.com`
and `bucket2.domain.com`.
#### MINIO_PUBLIC_IPS
This is comma separated list of IP addresses to which buckets created on this MinIO instance will resolve to. For example,
a bucket `bucket1` created on current MinIO instance will be accessible as `bucket1.domain.com`, and the DNS entry for
`bucket1.domain.com` will point to IP address set in `MINIO_PUBLIC_IPS`.
- This field is mandatory for standalone and erasure code MinIO server deployments, to enable federated mode.
- This field is optional for distributed deployments. If you don't set this field in a federated setup, we use the IP addresses of
hosts passed to the MinIO server startup and use them for DNS entries.
### Run Multiple Clusters
> cluster1
```sh
export MINIO_ETCD_ENDPOINTS="http://remote-etcd1:2379,http://remote-etcd2:4001"
export MINIO_DOMAIN=domain.com
export MINIO_PUBLIC_IPS=44.35.2.1,44.35.2.2,44.35.2.3,44.35.2.4
minio server http://rack{1...4}.host{1...4}.domain.com/mnt/export{1...32}
```
> cluster2
```sh
export MINIO_ETCD_ENDPOINTS="http://remote-etcd1:2379,http://remote-etcd2:4001"
export MINIO_DOMAIN=domain.com
export MINIO_PUBLIC_IPS=44.35.1.1,44.35.1.2,44.35.1.3,44.35.1.4
minio server http://rack{5...8}.host{5...8}.domain.com/mnt/export{1...32}
```
In this configuration you can see `MINIO_ETCD_ENDPOINTS` points to the etcd backend which manages MinIO's
`config.json` and bucket DNS SRV records. `MINIO_DOMAIN` indicates the domain suffix for the bucket which
will be used to resolve bucket through DNS. For example if you have a bucket such as `mybucket`, the
client can use now `mybucket.domain.com` to directly resolve itself to the right cluster. `MINIO_PUBLIC_IPS`
points to the public IP address where each cluster might be accessible, this is unique for each cluster.
NOTE: `mybucket` only exists on one cluster either `cluster1` or `cluster2` this is random and
is decided by how `domain.com` gets resolved, if there is a round-robin DNS on `domain.com` then
it is randomized which cluster might provision the bucket.
### 3. Test your setup
To test this setup, access the MinIO server via browser or [`mc`](https://docs.min.io/community/minio-object-store/reference/minio-mc.html#quickstart). You’ll see the uploaded files are accessible from the all the MinIO endpoints.
## Explore Further
- [Use `mc` with MinIO Server](https://docs.min.io/community/minio-object-store/reference/minio-mc.html)
- [Use `aws-cli` with MinIO Server](https://docs.min.io/community/minio-object-store/integrations/aws-cli-with-minio.html)
- [Use `minio-go` SDK with MinIO Server](https://docs.min.io/community/minio-object-store/developers/go/minio-go.html)
- [The MinIO documentation website](https://docs.min.io/community/minio-object-store/index.html)

BIN
docs/federation/lookup/bucket-lookup.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 36 KiB

257
docs/ftp/README.md Normal file
View File

@@ -0,0 +1,257 @@
# MinIO FTP/SFTP Server
MinIO natively supports FTP/SFTP protocol, this allows any ftp/sftp client to upload and download files.
Currently supported `FTP/SFTP` operations are as follows:
| ftp-client commands | supported |
|:-------------------:|:----------|
| get | yes |
| put | yes |
| ls | yes |
| mkdir | yes |
| rmdir | yes |
| delete | yes |
| append | no |
| rename | no |
MinIO supports following FTP/SFTP based protocols to access and manage data.
- Secure File Transfer Protocol (SFTP) – Defined by the Internet Engineering Task Force (IETF) as an
extended version of SSH 2.0, allowing file transfer over SSH and for use with Transport Layer
Security (TLS) and VPN applications.
- File Transfer Protocol over SSL/TLS (FTPS) – Encrypted FTP communication via TLS certificates.
- File Transfer Protocol (FTP) – Defined by RFC114 originally, and replaced by RFC765 and RFC959
unencrypted FTP communication (Not-recommended)
## Scope
- All IAM Credentials are allowed access excluding rotating credentials, rotating credentials
are not allowed to login via FTP/SFTP ports, you must use S3 API port for if you are using
rotating credentials.
- Access to bucket(s) and object(s) are governed via IAM policies associated with the incoming
login credentials.
- Allows authentication and access for all
- Built-in IDP users and their respective service accounts
- LDAP/AD users and their respective service accounts
- OpenID/OIDC service accounts
- On versioned buckets, FTP/SFTP only operates on latest objects, if you need to retrieve
an older version you must use an `S3 API client` such as [`mc`](https://github.com/minio/mc).
- All features currently used by your buckets will work as is without any changes
- SSE (Server Side Encryption)
- Replication (Server Side Replication)
## Prerequisites
- It is assumed you have users created and configured with relevant access policies, to start with
use basic "readwrite" canned policy to test all the operations before you finalize on what level
of restrictions are needed for a user.
- No "admin:*" operations are needed for FTP/SFTP access to the bucket(s) and object(s), so you may
skip them for restrictions.
## Usage
Start MinIO in a distributed setup, with 'ftp/sftp' enabled.
```
minio server http://server{1...4}/disk{1...4}
--ftp="address=:8021" --ftp="passive-port-range=30000-40000" \
--sftp="address=:8022" --sftp="ssh-private-key=/home/miniouser/.ssh/id_rsa"
...
...
```
Following example shows connecting via ftp client using `minioadmin` credentials, and list a bucket named `runner`:
```
ftp localhost -P 8021
Connected to localhost.
220 Welcome to MinIO FTP Server
Name (localhost:user): minioadmin
331 User name ok, password required
Password:
230 Password ok, continue
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> ls runner/
229 Entering Extended Passive Mode (|||39155|)
150 Opening ASCII mode data connection for file list
drwxrwxrwx 1 nobody nobody 0 Jan 1 00:00 chunkdocs/
drwxrwxrwx 1 nobody nobody 0 Jan 1 00:00 testdir/
...
```
Following example shows how to list an object and download it locally via `ftp` client:
```
ftp> ls runner/chunkdocs/metadata
229 Entering Extended Passive Mode (|||44269|)
150 Opening ASCII mode data connection for file list
-rwxrwxrwx 1 nobody nobody 45 Apr 1 06:13 chunkdocs/metadata
226 Closing data connection, sent 75 bytes
ftp> get
(remote-file) runner/chunkdocs/metadata
(local-file) test
local: test remote: runner/chunkdocs/metadata
229 Entering Extended Passive Mode (|||37785|)
150 Data transfer starting 45 bytes
45 3.58 KiB/s
226 Closing data connection, sent 45 bytes
45 bytes received in 00:00 (3.55 KiB/s)
...
```
Following example shows connecting via sftp client using `minioadmin` credentials, and list a bucket named `runner`:
```
sftp -P 8022 minioadmin@localhost
minioadmin@localhost's password:
Connected to localhost.
sftp> ls runner/
chunkdocs testdir
```
Following example shows how to download an object locally via `sftp` client:
```
sftp> get runner/chunkdocs/metadata metadata
Fetching /runner/chunkdocs/metadata to metadata
metadata 100% 226 16.6KB/s 00:00
sftp>
```
## Advanced options
### Change default FTP port
Default port '8021' can be changed via
```
--ftp="address=:3021"
```
### Change FTP passive port range
By default FTP requests OS to give a free port automatically, however you may want to restrict
this to specific ports in certain restricted environments via
```
--ftp="passive-port-range=30000-40000"
```
### Change default SFTP port
Default port '8022' can be changed via
```
--sftp="address=:3022"
```
### TLS (FTP)
Unlike SFTP server, FTP server is insecure by default. To operate under TLS mode, you need to provide certificates via
```
--ftp="tls-private-key=path/to/private.key" --ftp="tls-public-cert=path/to/public.crt"
```
> NOTE: if MinIO distributed setup is already configured to run under TLS, FTP will automatically use the relevant
> certs from the server certificate chain, this is mainly to add simplicity of setup. However if you wish to terminate
> TLS certificates via a different domain for your FTP servers you may choose the above command line options.
### Custom Algorithms (SFTP)
Custom algorithms can be specified via command line parameters.
Algorithms are comma separated.
Note that valid values does not in all cases represent default values.
`--sftp=pub-key-algos=...` specifies the supported client public key
authentication algorithms. Note that this doesn't include certificate types
since those use the underlying algorithm. This list is sent to the client if
it supports the server-sig-algs extension. Order is irrelevant.
Valid values
```
ssh-ed25519
sk-ssh-ed25519@openssh.com
sk-ecdsa-sha2-nistp256@openssh.com
ecdsa-sha2-nistp256
ecdsa-sha2-nistp384
ecdsa-sha2-nistp521
rsa-sha2-256
rsa-sha2-512
ssh-rsa
ssh-dss
```
`--sftp=kex-algos=...` specifies the supported key-exchange algorithms in preference order.
Valid values:
```
curve25519-sha256
curve25519-sha256@libssh.org
ecdh-sha2-nistp256
ecdh-sha2-nistp384
ecdh-sha2-nistp521
diffie-hellman-group14-sha256
diffie-hellman-group16-sha512
diffie-hellman-group14-sha1
diffie-hellman-group1-sha1
```
`--sftp=cipher-algos=...` specifies the allowed cipher algorithms.
If unspecified then a sensible default is used.
Valid values:
```
aes128-ctr
aes192-ctr
aes256-ctr
aes128-gcm@openssh.com
aes256-gcm@openssh.com
chacha20-poly1305@openssh.com
arcfour256
arcfour128
arcfour
aes128-cbc
3des-cbc
```
`--sftp=mac-algos=...` specifies a default set of MAC algorithms in preference order.
This is based on RFC 4253, section 6.4, but with hmac-md5 variants removed because they have
reached the end of their useful life.
Valid values:
```
hmac-sha2-256-etm@openssh.com
hmac-sha2-512-etm@openssh.com
hmac-sha2-256
hmac-sha2-512
hmac-sha1
hmac-sha1-96
```
### Certificate-based authentication
`--sftp=trusted-user-ca-key=...` specifies a file containing public key of certificate authority that is trusted
to sign user certificates for authentication.
Implementation is identical with "TrustedUserCAKeys" setting in OpenSSH server with exception that only one CA
key can be defined.
If a certificate is presented for authentication and has its signing CA key is in this file, then it may be
used for authentication for any user listed in the certificate's principals list.
Note that certificates that lack a list of principals will not be permitted for authentication using trusted-user-ca-key.
For more details on certificates, see the CERTIFICATES section in ssh-keygen(1).

161
docs/iam/access-management-plugin.md Normal file
View File

@@ -0,0 +1,161 @@
# Access Management Plugin Guide [![Slack](https://slack.minio.io/slack?type=svg)](https://slack.minio.io)
MinIO now includes support for using an Access Management Plugin. This is to allow object storage access control to be managed externally via a webhook.
When configured, MinIO sends request and credential details for every API call to an external HTTP(S) endpoint and expects an allow/deny response. MinIO is thus able to delegate access management to an external system, and users are able to use a custom solution instead of S3 standard IAM policies.
Latency sensitive applications may notice an increased latency due to a request to the external plugin upon every authenticated request to MinIO. User are advised to provision their infrastructure such that latency and performance is acceptable.
## Quickstart
To easily try out the feature, run the included demo Access Management Plugin program in this directory:
```sh
go run access-manager-plugin.go
```
This program, lets the admin user perform any action and prevents all other users from performing `s3:Put*` operations.
In another terminal start MinIO:
```sh
export MINIO_CI_CD=1
export MINIO_ROOT_USER=minio
export MINIO_ROOT_PASSWORD=minio123
export MINIO_POLICY_PLUGIN_URL=http://localhost:8080/
minio server /tmp/disk{1...4}
```
Now, let's test it out with `mc`:
```sh
mc alias set myminio http://localhost:9000 minio minio123
mc ls myminio
mc mb myminio/test
mc cp /etc/issue myminio/test
mc admin user add myminio foo foobar123
export MC_HOST_foo=http://foo:foobar123@localhost:9000
mc ls foo
mc cp /etc/issue myminio/test/issue2
```
Only the last operation would fail with a permissions error.
## Configuration
Access Management Plugin can be configured with environment variables:
```sh
$ mc admin config set myminio policy_plugin --env
KEY:
policy_plugin enable Access Management Plugin for policy enforcement
ARGS:
MINIO_POLICY_PLUGIN_URL* (url) plugin hook endpoint (HTTP(S)) e.g. "http://localhost:8181/v1/data/httpapi/authz/allow"
MINIO_POLICY_PLUGIN_AUTH_TOKEN (string) authorization header for plugin hook endpoint
MINIO_POLICY_PLUGIN_ENABLE_HTTP2 (bool) Enable experimental HTTP2 support to connect to plugin service (default: 'off')
MINIO_POLICY_PLUGIN_COMMENT (sentence) optionally add a comment to this setting
```
By default this plugin uses HTTP 1.x. To enable HTTP2 use the `MINIO_POLICY_PLUGIN_ENABLE_HTTP2` environment variable.
## Request and Response
MinIO will make a `POST` request with a JSON body to the given plugin URL. If the auth token parameter is set, it will be sent as an authorization header.
The JSON body structure can be seen from this sample:
<details><summary>Request Body Sample</summary>
```json
{
"input": {
"account": "minio",
"groups": null,
"action": "s3:ListBucket",
"bucket": "test",
"conditions": {
"Authorization": [
"AWS4-HMAC-SHA256 Credential=minio/20220507/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=62012db6c47d697620cf6c68f0f45f6e34894589a53ab1faf6dc94338468c78a"
],
"CurrentTime": [
"2022-05-07T18:31:41Z"
],
"Delimiter": [
"/"
],
"EpochTime": [
"1651948301"
],
"Prefix": [
""
],
"Referer": [
""
],
"SecureTransport": [
"false"
],
"SourceIp": [
"127.0.0.1"
],
"User-Agent": [
"MinIO (linux; amd64) minio-go/v7.0.24 mc/DEVELOPMENT.2022-04-20T23-07-53Z"
],
"UserAgent": [
"MinIO (linux; amd64) minio-go/v7.0.24 mc/DEVELOPMENT.2022-04-20T23-07-53Z"
],
"X-Amz-Content-Sha256": [
"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
],
"X-Amz-Date": [
"20220507T183141Z"
],
"authType": [
"REST-HEADER"
],
"principaltype": [
"Account"
],
"signatureversion": [
"AWS4-HMAC-SHA256"
],
"userid": [
"minio"
],
"username": [
"minio"
],
"versionid": [
""
]
},
"owner": true,
"object": "",
"claims": {},
"denyOnly": false
}
}
```
</details>
The response expected by MinIO, is a JSON body with a boolean:
```json
{
"result": true
}
```
The following structure is also accepted:
```json
{
"result": {
"allow": true
}
}
```
Any unmentioned JSON object keys in the above are ignored.

112
docs/iam/access-manager-plugin.go Normal file
View File

@@ -0,0 +1,112 @@
//go:build ignore
// +build ignore
// Copyright (c) 2015-2022 MinIO, Inc.
//
// This file is part of MinIO Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package main
import (
"bytes"
"encoding/json"
"flag"
"fmt"
"io"
"log"
"net/http"
"strings"
)
var (
keyFile string
certFile string
)
func init() {
flag.StringVar(&keyFile, "key-file", "", "Path to TLS cert key file")
flag.StringVar(&certFile, "cert-file", "", "Path to TLS cert file")
}
func writeErrorResponse(w http.ResponseWriter, err error) {
w.WriteHeader(http.StatusBadRequest)
json.NewEncoder(w).Encode(map[string]string{
"error": fmt.Sprintf("%v", err),
})
}
type Result struct {
Result bool `json:"result"`
}
func mainHandler(w http.ResponseWriter, r *http.Request) {
body, err := io.ReadAll(r.Body)
if err != nil {
writeErrorResponse(w, err)
return
}
var out bytes.Buffer
json.Indent(&out, body, "", " ")
fmt.Printf("Received JSON payload:\n%s\n", out.String())
reqMap := make(map[string]interface{})
err = json.Unmarshal(body, &reqMap)
if err != nil {
writeErrorResponse(w, err)
return
}
m := reqMap["input"].(map[string]interface{})
accountValue := m["account"].(string)
actionValue := m["action"].(string)
// Allow user `minio` to perform any action.
var res Result
if accountValue == "minio" {
res.Result = true
} else {
// All other users may not perform any `s3:Put*` operations.
res.Result = true
if strings.HasPrefix(actionValue, "s3:Put") {
res.Result = false
}
}
fmt.Printf("account: %v | action: %v | allowed: %v\n", accountValue, actionValue, res.Result)
json.NewEncoder(w).Encode(res)
return
}
func main() {
flag.Parse()
serveFunc := func() error {
return http.ListenAndServe(":8080", nil)
}
if certFile != "" || keyFile != "" {
if certFile == "" || keyFile == "" {
log.Fatal("Please provide both a key file and a cert file to enable TLS.")
}
serveFunc = func() error {
return http.ListenAndServeTLS(":8080", certFile, keyFile, nil)
}
}
http.HandleFunc("/", mainHandler)
log.Print("Listening on :8080")
log.Fatal(serveFunc())
}

76
docs/iam/identity-management-plugin.md Normal file
View File

@@ -0,0 +1,76 @@
# Identity Management Plugin Guide [![Slack](https://slack.minio.io/slack?type=svg)](https://slack.minio.io)
## Introduction
To enable the integration of custom authentication methods, MinIO can be configured with an Identity Management Plugin webhook. When configured, this plugin enables the `AssumeRoleWithCustomToken` STS API extension. A user or application can now present a token to the `AssumeRoleWithCustomToken` API, and MinIO verifies this token by sending it to the Identity Management Plugin webhook. This plugin responds with some information and MinIO is able to generate temporary STS credentials to interact with object storage.
The authentication flow is similar to that of OpenID, however the token is "opaque" to MinIO - it is simply sent to the plugin for verification. CAVEAT: There is no console UI integration for this method of authentication and it is intended primarily for machine authentication.
It can be configured via MinIO's standard configuration API (i.e. using `mc admin config set/get`), or equivalently with environment variables. For brevity we show only environment variables here:
```sh
$ mc admin config set myminio identity_plugin --env
KEY:
identity_plugin enable Identity Plugin via external hook
ARGS:
MINIO_IDENTITY_PLUGIN_URL* (url) plugin hook endpoint (HTTP(S)) e.g. "http://localhost:8181/path/to/endpoint"
MINIO_IDENTITY_PLUGIN_AUTH_TOKEN (string) authorization token for plugin hook endpoint
MINIO_IDENTITY_PLUGIN_ROLE_POLICY* (string) policies to apply for plugin authorized users
MINIO_IDENTITY_PLUGIN_ROLE_ID (string) unique ID to generate the ARN
MINIO_IDENTITY_PLUGIN_COMMENT (sentence) optionally add a comment to this setting
```
If provided, the auth token parameter is sent as an authorization header.
`MINIO_IDENTITY_PLUGIN_ROLE_POLICY` is a required parameter and can be list of comma separated policy names.
On setting up the plugin, the MinIO server prints the Role ARN to its log. The Role ARN is generated by default based on the given plugin URL. To avoid this and use a configurable value set a unique role ID via `MINIO_IDENTITY_PLUGIN_ROLE_ID`.
## REST API call to plugin
To verify the custom token presented in the `AssumeRoleWithCustomToken` API, MinIO makes a POST request to the configured identity management plugin endpoint and expects a response with some details as shown below:
### Request `POST` to plugin endpoint
Query parameters:
| Parameter Name | Value Type | Purpose |
|----------------|------------|-------------------------------------------------------------------------|
| token | string | Token from the AssumeRoleWithCustomToken call for external verification |
### Response
If the token is valid and access is approved, the plugin must return a `200` (OK) HTTP status code.
A `200 OK` Response should have `application/json` content-type and body with the following structure:
```json
{
"user": <string>,
"maxValiditySeconds": <integer>,
"claims": <key-value-pairs>
}
```
| Parameter Name | Value Type | Purpose |
|--------------------|-----------------------------------------|--------------------------------------------------------|
| user | string | Identifier for owner of requested credentials |
| maxValiditySeconds | integer (>= 900 seconds and < 365 days) | Maximum allowed expiry duration for the credentials |
| claims | key-value pairs | Claims to be associated with the requested credentials |
The keys "exp", "parent" and "sub" in the `claims` object are reserved and if present are ignored by MinIO.
If the token is not valid or access is not approved, the plugin must return a `403` (forbidden) HTTP status code. The body must have an `application/json` content-type with the following structure:
```json
{
"reason": <string>
}
```
The reason message is returned to the client.
## Example Plugin Implementation
A toy example for the Identity Management Plugin is given [here](./identity-manager-plugin.go).

86
docs/iam/identity-manager-plugin.go Normal file
View File

@@ -0,0 +1,86 @@
//go:build ignore
// +build ignore
// Copyright (c) 2015-2022 MinIO, Inc.
//
// This file is part of MinIO Object Storage stack
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package main
import (
"encoding/json"
"errors"
"fmt"
"log"
"net/http"
)
func writeErrorResponse(w http.ResponseWriter, err error) {
w.WriteHeader(http.StatusBadRequest)
json.NewEncoder(w).Encode(map[string]string{
"reason": fmt.Sprintf("%v", err),
})
}
type Resp struct {
User string `json:"user"`
MaxValiditySeconds int `json:"maxValiditySeconds"`
Claims map[string]interface{} `json:"claims"`
}
var tokens map[string]Resp = map[string]Resp{
"aaa": {
User: "Alice",
MaxValiditySeconds: 3600,
Claims: map[string]interface{}{
"groups": []string{"data-science"},
},
},
"bbb": {
User: "Bart",
MaxValiditySeconds: 3600,
Claims: map[string]interface{}{
"groups": []string{"databases"},
},
},
}
func mainHandler(w http.ResponseWriter, r *http.Request) {
token := r.FormValue("token")
if token == "" {
writeErrorResponse(w, errors.New("token parameter not given"))
return
}
rsp, ok := tokens[token]
if !ok {
w.WriteHeader(http.StatusForbidden)
return
}
fmt.Printf("Allowed for token: %s user: %s\n", token, rsp.User)
w.WriteHeader(http.StatusOK)
json.NewEncoder(w).Encode(rsp)
return
}
func main() {
http.HandleFunc("/", mainHandler)
log.Print("Listening on :8081")
log.Fatal(http.ListenAndServe(":8081", nil))
}

83
docs/iam/opa.md Normal file
View File

@@ -0,0 +1,83 @@
# OPA Quickstart Guide [![Slack](https://slack.minio.io/slack?type=svg)](https://slack.minio.io)
OPA is a lightweight general-purpose policy engine that can be co-located with MinIO server, in this document we talk about how to use OPA HTTP API to authorize requests. It can be used with any type of credentials (STS based like OpenID or LDAP, regular IAM users or service accounts).
OPA is enabled through MinIO's Access Management Plugin feature.
## Get started
### 1. Start OPA in a container
```sh
podman run -it \
--name opa \
--publish 8181:8181 \
docker.io/openpolicyagent/opa:0.40.0-rootless \
run --server \
--log-format=json-pretty \
--log-level=debug \
--set=decision_logs.console=true
```
### 2. Create a sample OPA Policy
In another terminal, create a policy that allows root user all access and for all other users denies `PutObject`:
```sh
cat > example.rego <<EOF
package httpapi.authz
import input
default allow = false
# Allow the root user to perform any action.
allow {
input.owner == true
}
# All other users may do anything other than call PutObject
allow {
input.action != "s3:PutObject"
input.owner == false
}
EOF
```
Then load the policy via OPA's REST API.
```
curl -X PUT --data-binary @example.rego \
localhost:8181/v1/policies/putobject
```
### 4. Setup MinIO with OPA
Set the `MINIO_POLICY_PLUGIN_URL` as the endpoint that MinIO should send authorization requests to. Then start the server.
```sh
export MINIO_POLICY_PLUGIN_URL=http://localhost:8181/v1/data/httpapi/authz/allow
export MINIO_CI_CD=1
export MINIO_ROOT_USER=minio
export MINIO_ROOT_PASSWORD=minio123
minio server /mnt/data
```
### 5. Test with a regular IAM user
Ensure that `mc` is installed and the configured with the above server with the alias `myminio`.
```sh
# 1. Create a bucket and a user, and upload a file. These operations will succeed.
mc mb myminio/test
mc admin user add myminio foo foobar123
mc cp /etc/issue myminio/test/
# 2. Now access the server as user `foo`. These operations will also succeed.
export MC_HOST_foo=http://foo:foobar123@localhost:9000
mc ls foo/test
mc cat foo/test/issue
# 3. Attempt to upload an object as user `foo` - this will fail with a permissions error.
mc cp /etc/issue myminio/test/issue2
```

17
docs/iam/policies/deny-non-sse-kms-objects.json Normal file
View File

@@ -0,0 +1,17 @@
{
"Version":"2012-10-17",
"Id":"PutObjectPolicy",
"Statement":[{
"Sid":"DenyObjectsThatAreNotSSEKMS",
"Effect":"Deny",
"Principal":"*",
"Action":"s3:PutObject",
"Resource":"arn:aws:s3:::multi-key-poc/*",
"Condition":{
"Null":{
"s3:x-amz-server-side-encryption-aws-kms-key-id":"true"
}
}
}
]
}

17
docs/iam/policies/deny-objects-with-invalid-sse-kms-key-id.json Normal file
View File

@@ -0,0 +1,17 @@
{
"Version":"2012-10-17",
"Id":"PutObjectPolicy1",
"Statement":[{
"Sid":"DenyObjectsWithInvalidSSEKMS",
"Effect":"Deny",
"Principal":"*",
"Action":"s3:PutObject",
"Resource":"arn:aws:s3:::multi-key-poc/*",
"Condition":{
"StringNotEquals":{
"s3:x-amz-server-side-encryption-aws-kms-key-id":"minio-default-key"
}
}
}
]
}

85
docs/iam/policies/pbac-tests.sh Executable file
View File

@@ -0,0 +1,85 @@
#!/bin/bash
if [ -n "$TEST_DEBUG" ]; then
set -x
fi
pkill minio
pkill kes
rm -rf /tmp/xl
go install -v github.com/minio/mc@master
cp -a $(go env GOPATH)/bin/mc ./mc
if [ ! -f ./kes ]; then
wget --quiet -O kes https://github.com/minio/kes/releases/latest/download/kes-linux-amd64 &&
chmod +x kes
fi
if ! openssl version &>/dev/null; then
apt install openssl || sudo apt install opensssl
fi
# Start KES Server
(./kes server --dev 2>&1 >kes-server.log) &
kes_pid=$!
sleep 5s
API_KEY=$(grep "API Key" <kes-server.log | awk -F" " '{print $3}')
(openssl s_client -connect 127.0.0.1:7373 2>/dev/null 1>public.crt)
export CI=true
export MINIO_KMS_KES_ENDPOINT=https://127.0.0.1:7373
export MINIO_KMS_KES_API_KEY="${API_KEY}"
export MINIO_KMS_KES_KEY_NAME=minio-default-key
export MINIO_KMS_KES_CAPATH=public.crt
export MC_HOST_myminio="http://minioadmin:minioadmin@localhost:9000/"
(minio server http://localhost:9000/tmp/xl/{1...10}/disk{0...1} 2>&1 >/dev/null) &
pid=$!
mc ready myminio
mc admin user add myminio/ minio123 minio123
mc admin policy create myminio/ deny-non-sse-kms-pol ./docs/iam/policies/deny-non-sse-kms-objects.json
mc admin policy create myminio/ deny-invalid-sse-kms-pol ./docs/iam/policies/deny-objects-with-invalid-sse-kms-key-id.json
mc admin policy attach myminio deny-non-sse-kms-pol --user minio123
mc admin policy attach myminio deny-invalid-sse-kms-pol --user minio123
mc admin policy attach myminio consoleAdmin --user minio123
mc mb -l myminio/test-bucket
mc mb -l myminio/multi-key-poc
export MC_HOST_myminio1="http://minio123:minio123@localhost:9000/"
mc cp /etc/issue myminio1/test-bucket
ret=$?
if [ $ret -ne 0 ]; then
echo "BUG: PutObject to bucket: test-bucket should succeed. Failed"
exit 1
fi
mc cp /etc/issue myminio1/multi-key-poc | grep -q "Insufficient permissions to access this path"
ret=$?
if [ $ret -eq 0 ]; then
echo "BUG: PutObject to bucket: multi-key-poc without sse-kms should fail. Succedded"
exit 1
fi
mc cp /etc/hosts myminio1/multi-key-poc/hosts --enc-kms "myminio1/multi-key-poc/hosts=minio-default-key"
ret=$?
if [ $ret -ne 0 ]; then
echo "BUG: PutObject to bucket: multi-key-poc with valid sse-kms should succeed. Failed"
exit 1
fi
mc cp /etc/issue myminio1/multi-key-poc/issue --enc-kms "myminio1/multi-key-poc/issue=minio-default-key-xxx" | grep "Insufficient permissions to access this path"
ret=$?
if [ $ret -eq 0 ]; then
echo "BUG: PutObject to bucket: multi-key-poc with invalid sse-kms should fail. Succeeded"
exit 1
fi
kill $pid
kill $kes_pid

93
docs/integrations/veeam/README.md Normal file
View File

@@ -0,0 +1,93 @@
# Using MinIO with Veeam
When using Veeam Backup and Replication, you can use S3 compatible object storage such as MinIO as a capacity tier for backups. This disaggregates storage for the Veeam infrastructure and allows you to retain control of your data. With the ease of use of setup and administration of MinIO, it allows a Veeam backup admin to easily deploy their own object store for capacity tiering.
## Prerequisites
- One or both of Veeam Backup and Replication with support for S3 compatible object store (e.g. 9.5.4) and Veeam Backup for Office365 (VBO)
- MinIO object storage set up per <https://docs.min.io/community/minio-object-store/index.html>
- Veeam requires TLS connections to the object storage. This can be configured per <https://docs.min.io/community/minio-object-store/operations/network-encryption.html>
- The S3 bucket, Access Key and Secret Key have to be created before and outside of Veeam.
- Configure the minio client for the Veeam MinIO endpoint - <https://docs.min.io/community/minio-object-store/operations/deployments/baremetal-deploy-minio-on-redhat-linux.html>
## Setting up an S3 compatible object store for Veeam Backup and Replication
### Create a bucket for Veeam backups
Create a bucket for Veeam Backup, e.g.,
```
mc mb myminio/veeambackup
```
> NOTE: For Veeam Backup with Immutability, create the bucket with object lock enabled, e.g.,
```
mc mb -l myminio/veeambackup
```
> Object locking requires erasure coding enabled on the minio server. For more information see <https://docs.min.io/community/minio-object-store/operations/concepts/erasure-coding.html>.
### Add MinIO as an object store for Veeam
Follow the steps from the Veeam documentation for adding MinIO as an object store - <https://helpcenter.veeam.com/docs/backup/vsphere/adding_s3c_object_storage.html?ver=100>
For Veeam Backup with Immutability, choose the amount of days you want to make backups immutable for
![Choose Immutability Days for Object Store](https://raw.githubusercontent.com/minio/minio/master/docs/integrations/veeam/screenshots/object_store_immutable_days.png)
### Creating the Scale-out Backup Repository
- Under the Backup Infrastructure view, click on Scale-out Repositories and click the Add Scale-out Repository button on the ribbon.
- Follow the on screen wizard
- On the Capacity Tier screen, check the box to Extend scale-out backup repository capacity with object storage checkbox and select the object storage. If you want to be able to test backup data immediately after a job is run, under the object storage selection, check the "Copy" box and uncheck the "Move" box.
### Create a backup job
#### Backup Virtual Machines with Veeam Backup and Replication
- Under Home > Jobs > Backup in Navigation Pane, click on Backup Job button in the ribbon and choose Virtual Machine. Follow the on screen wizard.
- On the Storage screen, choose the Scale-out Backup Repository that was configured previously.
- Continue with the backup job creation. On the Summary screen, check the Run the Job when I click Finish checkbox and click the Finish button. The backup job will start immediately. This will create an Active Full backup of the VMs within the backup job.
- Since we selected Copy mode when creating the SOBR, the backup will be copied to the capacity tier as soon as it is created on the performance tier.
- For Veeam Backup with Immutability, you can choose a number of restore points or days to make backups immutable.
![Choose Immutability Options for Backups](https://raw.githubusercontent.com/minio/minio/master/docs/integrations/veeam/screenshots/backup_job_immutable_days.png)
#### Backup Office 365 with VBO
- Create a new bucket for VBO backups
```
mc mb -l myminio/vbo
```
- Under Backup Infrastructure, right click on Object Storage Repositories and choose "Add object storage"
![Adding Object Storage to VBO Step 1](https://raw.githubusercontent.com/minio/minio/master/docs/integrations/veeam/screenshots/1_add_object_store.png)
- Follow through the wizard as above for Veeam Backup and Replication as the steps are the same between both products
- Under Backup Infrastructure -> Backup Repositories, right click and "Add Backup Repository"
- Follow the wizard. Under the "Object Storage Backup Repository" section, choose the MinIO object storage you created above
![Adding Object Storage to VBO Backup Repository](https://raw.githubusercontent.com/minio/minio/master/docs/integrations/veeam/screenshots/6_add_sobr_with_object_store.png)
- When you create your backup job, choose the backup repository you created above.
## Test the setup
The next time the backup job runs, you can use the `mc admin trace myminio` command and verify traffic is flowing to the MinIO nodes. For Veeam Backup and Replication you will need to wait for the backup to complete to the performance tier before it migrates data to the capacity tier (i.e., MinIO).
```
20:09:10.216 [200 OK] s3.GetObject veeam-minio01:9000/vbo/Veeam/Backup365/vbotest/Organizations/6571606ecbc4455dbfe23b83f6f45597/Webs/ca2d0986229b4ec88e3a217ef8f04a1d/Items/efaa67764b304e77badb213d131beab6/f4f0cf600f494c3eb702d8eafe0fabcc.aac07493e6cd4c71845d2495a4e1e19b 139.178.68.158 9.789ms ↑ 90 B ↓ 8.5 KiB
20:09:10.244 [200 OK] s3.GetObject veeam-minio01:9000/vbo/Veeam/Backup365/vbotest/RepositoryLock/cad99aceb50c49ecb9e07246c3b9fadc_bfd985e5deec4cebaf481847f2c34797 139.178.68.158 16.21ms ↑ 90 B ↓ 402 B
20:09:10.283 [200 OK] s3.PutObject veeam-minio01:9000/vbo/Veeam/Backup365/vbotest/CommonInfo/WebRestorePoints/18f1aba8f55f4ac6b805c4de653eb781 139.178.68.158 29.787ms ↑ 1005 B ↓ 296 B
```

BIN
docs/integrations/veeam/screenshots/1_add_object_store.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 43 KiB

BIN
docs/integrations/veeam/screenshots/6_add_sobr_with_object_store.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 32 KiB

BIN
docs/integrations/veeam/screenshots/backup_job_immutable_days.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 18 KiB

BIN
docs/integrations/veeam/screenshots/object_store_immutable_days.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 13 KiB

109
docs/kms/IAM.md Normal file
View File

@@ -0,0 +1,109 @@
# KMS IAM/Config Encryption
MinIO supports encrypting config, IAM assets with KMS provided keys. If the KMS is not enabled, MinIO will store the config, IAM data as plain text erasure coded in its backend.
## MinIO KMS Quick Start
MinIO supports two ways of encrypting IAM and configuration data.
You can either use KES - together with an external KMS - or, much simpler,
set the env. variable `MINIO_KMS_SECRET_KEY` and start/restart the MinIO server. For more details about KES and how
to set it up refer to our [KMS Guide](https://github.com/minio/minio/blob/master/docs/kms/README.md).
Instead of configuring an external KMS you can start with a single key by
setting the env. variable `MINIO_KMS_SECRET_KEY`. It expects the following
format:
```sh
MINIO_KMS_SECRET_KEY=<key-name>:<base64-value>
```
First generate a 256 bit random key via:
```sh
$ cat /dev/urandom | head -c 32 | base64 -
OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
```
Now, you can set `MINIO_KMS_SECRET_KEY` like this:
```sh
export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
```
> You can choose an arbitrary name for the key - instead of `my-minio-key`.
> Please note that losing the `MINIO_KMS_SECRET_KEY` will cause data loss
> since you will not be able to decrypt the IAM/configuration data anymore.
For distributed MinIO deployments, specify the *same* `MINIO_KMS_SECRET_KEY` for each MinIO server process.
At any point in time you can switch from `MINIO_KMS_SECRET_KEY` to a full KMS
deployment. You just need to import the generated key into KES - for example via
the KES CLI once you have successfully setup KES:
```sh
kes key create my-minio-key OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
```
- For instructions on setting up KES, see the [KES Getting Started guide](https://github.com/minio/kes/wiki/Getting-Started)
- For instructions on using KES for encrypting the MinIO backend, follow the [KMS Quick Start](https://github.com/minio/minio/tree/master/docs/kms). The SSE-S3 configuration setup also supports MinIO KMS backend encryption.
## FAQ
> Why is this change needed?
Before, there were two separate mechanisms - S3 objects got encrypted using a KMS,
if present, and the IAM / configuration data got encrypted with the root credentials.
Now, MinIO encrypts IAM / configuration and S3 objects with a KMS, if present. This
change unified the key-management aspect within MinIO.
The unified KMS-based approach has several advantages:
- Key management is now centralized. There is one way to change or rotate encryption keys.
There used to be two different mechanisms - one for regular S3 objects and one for IAM data.
- Reduced server startup time. For IAM encryption with the root credentials, MinIO had
to use a memory-hard function (Argon2) that (on purpose) consumes a lot of memory and CPU.
The new KMS-based approach can use a key derivation function that is orders of magnitudes
cheaper w.r.t. memory and CPU.
- Root credentials can now be changed easily. Before, a two-step process was required to
change the cluster root credentials since they were used to en/decrypt the IAM data.
So, both - the old and new credentials - had to be present at the same time during a rotation
and the old credentials had to be removed once the rotation completed. This process is now gone.
The root credentials can now be changed easily.
> Does this mean I need an enterprise KMS setup to run MinIO (securely)?
No, MinIO does not depend on any third-party KMS provider. You have three options here:
- Run MinIO without a KMS. In this case all IAM data will be stored in plain-text.
- Run MinIO with a single secret key. MinIO supports a static cryptographic key
that can act as minimal KMS. With this method all IAM data will be stored
encrypted. The encryption key has to be passed as environment variable.
- Run MinIO with KES (minio/kes) in combination with any supported KMS as
secure key store. For example, you can run MinIO + KES + Hashicorp Vault.
> What about an exiting MinIO deployment? Can I just upgrade my cluster?
Yes, MinIO will try to transparently migrate any existing IAM data and either stores
it in plaintext (no KMS) or re-encrypts using the KMS.
> Is this change backward compatible? Will it break my setup?
This change is not backward compatible for all setups. In particular, the native
Hashicorp Vault integration - which has been deprecated already - won't be
supported anymore. KES is now mandatory if a third-party KMS should be used.
Further, since the configuration data is encrypted with the KMS, the KMS
configuration itself can no longer be stored in the MinIO config file and
instead must be provided via environment variables. If you have set your KMS
configuration using e.g. the `mc admin config` commands you will need to adjust
your deployment.
Even though this change is backward compatible we do not expect that it affects
the vast majority of deployments in any negative way.
> Will an upgrade of an existing MinIO cluster impact the SLA of the cluster or will it even cause downtime?
No, an upgrade should not cause any downtime. However, on the first startup -
since MinIO will attempt to migrate any existing IAM data - the boot process may
take slightly longer, but may not be visibly noticeable. Once the migration has
completed, any subsequent restart should be as fast as before or even faster.

143
docs/kms/README.md Normal file
View File

@@ -0,0 +1,143 @@
# KMS Guide [![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io)
MinIO uses a key-management-system (KMS) to support SSE-S3. If a client requests SSE-S3, or auto-encryption is enabled, the MinIO server encrypts each object with a unique object key which is protected by a master key managed by the KMS.
## Quick Start
MinIO supports multiple KMS implementations via our [KES](https://github.com/minio/kes#kes) project. We run a KES instance at `https://play.min.io:7373` for you to experiment and quickly get started. To run MinIO with a KMS just fetch the root identity, set the following environment variables and then start your MinIO server. If you haven't installed MinIO, yet, then follow the MinIO [install instructions](https://docs.min.io/community/minio-object-store/operations/deployments/baremetal-deploy-minio-on-redhat-linux.html) first.
### 1. Fetch the root identity
As the initial step, fetch the private key and certificate of the root identity:
```sh
curl -sSL --tlsv1.2 \
-O 'https://raw.githubusercontent.com/minio/kes/master/root.key' \
-O 'https://raw.githubusercontent.com/minio/kes/master/root.cert'
```
### 2. Set the MinIO-KES configuration
```sh
export MINIO_KMS_KES_ENDPOINT=https://play.min.io:7373
export MINIO_KMS_KES_KEY_FILE=root.key
export MINIO_KMS_KES_CERT_FILE=root.cert
export MINIO_KMS_KES_KEY_NAME=my-minio-key
```
### 3. Start the MinIO Server
```sh
export MINIO_ROOT_USER=minio
export MINIO_ROOT_PASSWORD=minio123
minio server ~/export
```
> The KES instance at `https://play.min.io:7373` is meant to experiment and provides a way to get started quickly.
> Note that anyone can access or delete master keys at `https://play.min.io:7373`. You should run your own KES
> instance in production.
## Configuration Guides
A typical MinIO deployment that uses a KMS for SSE-S3 looks like this:
```
┌────────────┐
│ ┌──────────┴─┬─────╮ ┌────────────┐
└─┤ ┌──────────┴─┬───┴──────────┤ ┌──────────┴─┬─────────────────╮
└─┤ ┌──────────┴─┬─────┬──────┴─┤ KES Server ├─────────────────┤
└─┤ MinIO ├─────╯ └────────────┘ ┌────┴────┐
└────────────┘ │ KMS │
└─────────┘
```
In a given setup, there are `n` MinIO instances talking to `m` KES servers but only `1` central KMS. The most simple setup consists of `1` MinIO server or cluster talking to `1` KMS via `1` KES server.
The main difference between various MinIO-KMS deployments is the KMS implementation. The following table helps you select the right option for your use case:
| KMS | Purpose |
|:---------------------------------------------------------------------------------------------|:------------------------------------------------------------------|
| [Hashicorp Vault](https://github.com/minio/kes/wiki/Hashicorp-Vault-Keystore) | Local KMS. MinIO and KMS on-prem (**Recommended**) |
| [AWS-KMS + SecretsManager](https://github.com/minio/kes/wiki/AWS-SecretsManager) | Cloud KMS. MinIO in combination with a managed KMS installation |
| [Gemalto KeySecure /Thales CipherTrust](https://github.com/minio/kes/wiki/Gemalto-KeySecure) | Local KMS. MinIO and KMS On-Premises. |
| [Google Cloud Platform SecretManager](https://github.com/minio/kes/wiki/GCP-SecretManager) | Cloud KMS. MinIO in combination with a managed KMS installation |
| [FS](https://github.com/minio/kes/wiki/Filesystem-Keystore) | Local testing or development (**Not recommended for production**) |
The MinIO-KES configuration is always the same - regardless of the underlying KMS implementation. Checkout the MinIO-KES [configuration example](https://github.com/minio/kes/wiki/MinIO-Object-Storage).
### Further references
- [Run MinIO with TLS / HTTPS](https://docs.min.io/community/minio-object-store/operations/network-encryption.html)
- [Tweak the KES server configuration](https://github.com/minio/kes/wiki/Configuration)
- [Run a load balancer in front of KES](https://github.com/minio/kes/wiki/TLS-Proxy)
- [Understand the KES server concepts](https://github.com/minio/kes/wiki/Concepts)
## Auto Encryption
Auto-Encryption is useful when MinIO administrator wants to ensure that all data stored on MinIO is encrypted at rest.
### Using `mc encrypt` (recommended)
MinIO automatically encrypts all objects on buckets if KMS is successfully configured and bucket encryption configuration is enabled for each bucket as shown below:
```
mc encrypt set sse-s3 myminio/bucket/
```
Verify if MinIO has `sse-s3` enabled
```
mc encrypt info myminio/bucket/
Auto encryption 'sse-s3' is enabled
```
### Using environment (not-recommended)
MinIO automatically encrypts all objects on buckets if KMS is successfully configured and following ENV is enabled:
```
export MINIO_KMS_AUTO_ENCRYPTION=on
```
### Verify auto-encryption
> Note that auto-encryption only affects requests without S3 encryption headers. So, if a S3 client sends
> e.g. SSE-C headers, MinIO will encrypt the object with the key sent by the client and won't reach out to
> the configured KMS.
To verify auto-encryption, use the following `mc` command:
```
mc cp test.file myminio/bucket/
test.file: 5 B / 5 B ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ 100.00% 337 B/s 0s
```
```
mc stat myminio/bucket/test.file
Name : test.file
...
Encrypted :
X-Amz-Server-Side-Encryption: AES256
```
## Encrypted Private Key
MinIO supports encrypted KES client private keys. Therefore, you can use
an password-protected private keys for `MINIO_KMS_KES_KEY_FILE`.
When using password-protected private keys for accessing KES you need to
provide the password via:
```
export MINIO_KMS_KES_KEY_PASSWORD=<your-password>
```
Note that MinIO only supports encrypted private keys - not encrypted certificates.
Certificates are no secrets and sent in plaintext as part of the TLS handshake.
## Explore Further
- [Use `mc` with MinIO Server](https://docs.min.io/community/minio-object-store/reference/minio-mc.html)
- [Use `aws-cli` with MinIO Server](https://docs.min.io/community/minio-object-store/integrations/aws-cli-with-minio.html)
- [Use `minio-go` SDK with MinIO Server](https://docs.min.io/community/minio-object-store/developers/go/minio-go.html)
- [The MinIO documentation website](https://docs.min.io/community/minio-object-store/index.html)

186
docs/lambda/README.md Normal file
View File

@@ -0,0 +1,186 @@
# Object Lambda
MinIO's Object Lambda implementation allows for transforming your data to serve unique data format requirements for each application. For example, a dataset created by an ecommerce application might include personally identifiable information (PII). When the same data is processed for analytics, PII should be redacted. However, if the same dataset is used for a marketing campaign, you might need to enrich the data with additional details, such as information from the customer loyalty database.
MinIO's Object Lambda, enables application developers to process data retrieved from MinIO before returning it to an application. You can register a Lambda Function target on MinIO, once successfully registered it can be used to transform the data for application GET requests on demand.
This document focuses on showing a working example on how to use Object Lambda with MinIO, you must have [MinIO deployed in your environment](https://docs.min.io/community/minio-object-store/operations/installation.html) before you can start using external lambda functions. You also must install Python version 3.8 or later for the lambda handlers to work.
## Example Lambda handler
Install the necessary dependencies.
```sh
pip install flask requests
```
Following is an example lambda handler.
```py
from flask import Flask, request, abort, make_response
import requests
app = Flask(__name__)
@app.route('/', methods=['POST'])
def get_webhook():
if request.method == 'POST':
# obtain the request event from the 'POST' call
event = request.json
object_context = event["getObjectContext"]
# Get the presigned URL to fetch the requested
# original object from MinIO
s3_url = object_context["inputS3Url"]
# Extract the route and request token from the input context
request_route = object_context["outputRoute"]
request_token = object_context["outputToken"]
# Get the original S3 object using the presigned URL
r = requests.get(s3_url)
original_object = r.content.decode('utf-8')
# Transform all text in the original object to uppercase
# You can replace it with your custom code based on your use case
transformed_object = original_object.upper()
# Write object back to S3 Object Lambda
# response sends the transformed data
# back to MinIO and then to the user
resp = make_response(transformed_object, 200)
resp.headers['x-amz-request-route'] = request_route
resp.headers['x-amz-request-token'] = request_token
return resp
else:
abort(400)
if __name__ == '__main__':
app.run()
```
When you're writing a Lambda function for use with MinIO, the function is based on event context that MinIO provides to the Lambda function. The event context provides information about the request being made. It contains the parameters with relevant context. The fields used to create the Lambda function are as follows:
The field of `getObjectContext` means the input and output details for connections to MinIO. It has the following fields:
- `inputS3Url` – A presigned URL that the Lambda function can use to download the original object. By using a presigned URL, the Lambda function doesn't need to have MinIO credentials to retrieve the original object. This allows Lambda function to focus on transformation of the object instead of securing the credentials.
- `outputRoute` – A routing token that is added to the response headers when the Lambda function returns the transformed object. This is used by MinIO to further verify the incoming response validity.
- `outputToken` – A token added to the response headers when the Lambda function returns the transformed object. This is used by MinIO to verify the incoming response validity.
Lets start the lambda handler.
```
python lambda_handler.py
* Serving Flask app 'webhook'
* Debug mode: off
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
* Running on http://127.0.0.1:5000
Press CTRL+C to quit
```
## Start MinIO with Lambda target
Register MinIO with a Lambda function, we are calling our target name as `function`, but you may call it any other friendly name of your choice.
```
MINIO_LAMBDA_WEBHOOK_ENABLE_function=on MINIO_LAMBDA_WEBHOOK_ENDPOINT_function=http://localhost:5000 minio server /data &
...
...
MinIO Object Storage Server
Copyright: 2015-2023 MinIO, Inc.
License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
Version: DEVELOPMENT.2023-02-05T05-17-27Z (go1.19.4 linux/amd64)
...
...
Object Lambda ARNs: arn:minio:s3-object-lambda::function:webhook
```
### Lambda Target with Auth Token
If your lambda target expects an authorization token then you can enable it per function target as follows
```
MINIO_LAMBDA_WEBHOOK_ENABLE_function=on MINIO_LAMBDA_WEBHOOK_ENDPOINT_function=http://localhost:5000 MINIO_LAMBDA_WEBHOOK_AUTH_TOKEN="mytoken" minio server /data &
```
### Lambda Target with mTLS authentication
If your lambda target expects mTLS client you can enable it per function target as follows
```
MINIO_LAMBDA_WEBHOOK_ENABLE_function=on MINIO_LAMBDA_WEBHOOK_ENDPOINT_function=http://localhost:5000 MINIO_LAMBDA_WEBHOOK_CLIENT_CERT=client.crt MINIO_LAMBDA_WEBHOOK_CLIENT_KEY=client.key minio server /data &
```
## Create a bucket and upload some data
Create a bucket named `functionbucket`
```
mc alias set myminio/ http://localhost:9000 minioadmin minioadmin
mc mb myminio/functionbucket
```
Create a file `testobject` with some test data that will be transformed
```
cat > testobject << EOF
MinIO is a High Performance Object Storage released under GNU Affero General Public License v3.0. It is API compatible with Amazon S3 cloud storage service. Use MinIO to build high performance infrastructure for machine learning, analytics and application data workloads.
EOF
```
Upload this object to the bucket via `mc cp`
```
mc cp testobject myminio/functionbucket/
```
## Invoke Lambda transformation via PresignedGET
Following example shows how you can use [`minio-go` PresignedGetObject](https://docs.min.io/community/minio-object-store/developers/go/API.html#presignedgetobject-ctx-context-context-bucketname-objectname-string-expiry-time-duration-reqparams-url-values-url-url-error)
```go
package main
import (
"context"
"log"
"net/url"
"time"
"fmt"
"github.com/minio/minio-go/v7"
"github.com/minio/minio-go/v7/pkg/credentials"
)
func main() {
s3Client, err := minio.New("localhost:9000", &minio.Options{
Creds: credentials.NewStaticV4("minioadmin", "minioadmin", ""),
Secure: false,
})
if err != nil {
log.Fatalln(err)
}
// Set lambda function target via `lambdaArn`
reqParams := make(url.Values)
reqParams.Set("lambdaArn", "arn:minio:s3-object-lambda::function:webhook")
// Generate presigned GET url with lambda function
presignedURL, err := s3Client.PresignedGetObject(context.Background(), "functionbucket", "testobject", time.Duration(1000)*time.Second, reqParams)
if err != nil {
log.Fatalln(err)
}
fmt.Println(presignedURL)
}
```
Use the Presigned URL via `curl` to receive the transformed object.
```
curl -v $(go run presigned.go)
...
...
> GET /functionbucket/testobject?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minioadmin%2F20230205%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230205T173023Z&X-Amz-Expires=1000&X-Amz-SignedHeaders=host&lambdaArn=arn%3Aminio%3As3-object-lambda%3A%3Atoupper%3Awebhook&X-Amz-Signature=d7e343f0da9d4fa2bc822c12ad2f54300ff16796a1edaa6d31f1313c8e94d5b2 HTTP/1.1
> Host: localhost:9000
> User-Agent: curl/7.81.0
> Accept: */*
>
MINIO IS A HIGH PERFORMANCE OBJECT STORAGE RELEASED UNDER GNU AFFERO GENERAL PUBLIC LICENSE V3.0. IT IS API COMPATIBLE WITH AMAZON S3 CLOUD STORAGE SERVICE. USE MINIO TO BUILD HIGH PERFORMANCE INFRASTRUCTURE FOR MACHINE LEARNING, ANALYTICS AND APPLICATION DATA WORKLOADS.
```

228
docs/logging/README.md Normal file
View File

@@ -0,0 +1,228 @@
# MinIO Logging Quickstart Guide [![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io)
This document explains how to configure MinIO server to log to different logging targets.
## Log Targets
MinIO supports currently two target types
- console
- http
### Logging Console Target
Console target is on always and cannot be disabled.
### Logging HTTP Target
HTTP target logs to a generic HTTP endpoint in JSON format and is not enabled by default. To enable HTTP target logging you would have to update your MinIO server configuration using `mc admin config set` command.
Assuming `mc` is already [configured](https://docs.min.io/community/minio-object-store/reference/minio-mc.html#quickstart)
```
mc admin config get myminio/ logger_webhook
logger_webhook:name1 auth_token="" endpoint=""
```
```
mc admin config set myminio logger_webhook:name1 auth_token="" endpoint="http://endpoint:port/path"
mc admin service restart myminio
```
NOTE: `http://endpoint:port/path` is a placeholder value to indicate the URL format, please change this accordingly as per your configuration.
MinIO also honors environment variable for HTTP target logging as shown below, this setting will override the endpoint settings in the MinIO server config.
```
export MINIO_LOGGER_WEBHOOK_ENABLE_target1="on"
export MINIO_LOGGER_WEBHOOK_AUTH_TOKEN_target1="token"
export MINIO_LOGGER_WEBHOOK_ENDPOINT_target1=http://localhost:8080/minio/logs
minio server /mnt/data
```
## Audit Targets
Assuming `mc` is already [configured](https://docs.min.io/community/minio-object-store/reference/minio-mc.html#quickstart)
### Audit HTTP Target
```
mc admin config get myminio/ audit_webhook
audit_webhook:name1 enable=off endpoint= auth_token= client_cert= client_key=
```
```
mc admin config set myminio audit_webhook:name1 auth_token="" endpoint="http://endpoint:port/path"
mc admin service restart myminio
```
NOTE: `http://endpoint:port/path` is a placeholder value to indicate the URL format, please change this accordingly as per your configuration.
MinIO also honors environment variable for HTTP target Audit logging as shown below, this setting will override the endpoint settings in the MinIO server config.
```
export MINIO_AUDIT_WEBHOOK_ENABLE_target1="on"
export MINIO_AUDIT_WEBHOOK_AUTH_TOKEN_target1="token"
export MINIO_AUDIT_WEBHOOK_ENDPOINT_target1=http://localhost:8080/minio/logs
export MINIO_AUDIT_WEBHOOK_CLIENT_CERT="/tmp/cert.pem"
export MINIO_AUDIT_WEBHOOK_CLIENT_KEY=="/tmp/key.pem"
minio server /mnt/data
```
Setting this environment variable automatically enables audit logging to the HTTP target. The audit logging is in JSON format as described below.
NOTE:
- `timeToFirstByte` and `timeToResponse` will be expressed in Nanoseconds.
- Additionally in the case of the erasure coded setup `tags.objectLocation` provides per object details about
- Pool number the object operation was performed on.
- Set number the object operation was performed on.
- The list of drives participating in this operation belong to the set.
```json
{
"version": "1",
"deploymentid": "90e81272-45d9-4fe8-9c45-c9a7322bf4b5",
"time": "2024-05-09T07:38:10.449688982Z",
"event": "",
"trigger": "incoming",
"api": {
"name": "PutObject",
"bucket": "testbucket",
"object": "hosts",
"status": "OK",
"statusCode": 200,
"rx": 401,
"tx": 0,
"timeToResponse": "13309747ns",
"timeToResponseInNS": "13309747"
},
"remotehost": "127.0.0.1",
"requestID": "17CDC1F4D7E69123",
"userAgent": "MinIO (linux; amd64) minio-go/v7.0.70 mc/RELEASE.2024-04-30T17-44-48Z",
"requestPath": "/testbucket/hosts",
"requestHost": "localhost:9000",
"requestHeader": {
"Accept-Encoding": "zstd,gzip",
"Authorization": "AWS4-HMAC-SHA256 Credential=minioadmin/20240509/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length,Signature=d4d6862e6cc61011a61fa801da71048ece4f32a0562cad6bb88bdda50d7fcb95",
"Content-Length": "401",
"Content-Type": "application/octet-stream",
"User-Agent": "MinIO (linux; amd64) minio-go/v7.0.70 mc/RELEASE.2024-04-30T17-44-48Z",
"X-Amz-Content-Sha256": "STREAMING-AWS4-HMAC-SHA256-PAYLOAD",
"X-Amz-Date": "20240509T073810Z",
"X-Amz-Decoded-Content-Length": "228"
},
"responseHeader": {
"Accept-Ranges": "bytes",
"Content-Length": "0",
"ETag": "9fe7a344ef4227d3e53751e9d88ce41e",
"Server": "MinIO",
"Strict-Transport-Security": "max-age=31536000; includeSubDomains",
"Vary": "Origin,Accept-Encoding",
"X-Amz-Id-2": "dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8",
"X-Amz-Request-Id": "17CDC1F4D7E69123",
"X-Content-Type-Options": "nosniff",
"X-Xss-Protection": "1; mode=block"
},
"tags": {
"objectLocation": {
"name": "hosts",
"poolId": 1,
"setId": 1,
"drives": [
"/mnt/data1",
"/mnt/data2",
"/mnt/data3",
"/mnt/data4"
]
}
},
"accessKey": "minioadmin"
}
```
### Kafka Target
Assuming that you already have Apache Kafka configured and running.
```
mc admin config set myminio/ audit_kafka
KEY:
audit_kafka[:name] send audit logs to kafka endpoints
ARGS:
brokers* (csv) comma separated list of Kafka broker addresses
topic (string) Kafka topic used for bucket notifications
sasl_username (string) username for SASL/PLAIN or SASL/SCRAM authentication
sasl_password (string) password for SASL/PLAIN or SASL/SCRAM authentication
sasl_mechanism (string) sasl authentication mechanism, default 'plain'
tls_client_auth (string) clientAuth determines the Kafka server's policy for TLS client auth
sasl (on|off) set to 'on' to enable SASL authentication
tls (on|off) set to 'on' to enable TLS
tls_skip_verify (on|off) trust server TLS without verification, defaults to "on" (verify)
client_tls_cert (path) path to client certificate for mTLS auth
client_tls_key (path) path to client key for mTLS auth
version (string) specify the version of the Kafka cluster
comment (sentence) optionally add a comment to this setting
```
Configure MinIO to send audit logs to locally running Kafka brokers
```
mc admin config set myminio/ audit_kafka:target1 brokers=localhost:29092 topic=auditlog
mc admin service restart myminio/
```
On another terminal assuming you have `kafkacat` installed
```
kafkacat -b localhost:29092 -t auditlog -C
{"version":"1","deploymentid":"90e81272-45d9-4fe8-9c45-c9a7322bf4b5","time":"2024-05-09T07:38:10.449688982Z","event":"","trigger":"incoming","api":{"name":"PutObject","bucket":"testbucket","object":"hosts","status":"OK","statusCode":200,"rx":401,"tx":0,"timeToResponse":"13309747ns","timeToResponseInNS":"13309747"},"remotehost":"127.0.0.1","requestID":"17CDC1F4D7E69123","userAgent":"MinIO (linux; amd64) minio-go/v7.0.70 mc/RELEASE.2024-04-30T17-44-48Z","requestPath":"/testbucket/hosts","requestHost":"localhost:9000","requestHeader":{"Accept-Encoding":"zstd,gzip","Authorization":"AWS4-HMAC-SHA256 Credential=minioadmin/20240509/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length,Signature=d4d6862e6cc61011a61fa801da71048ece4f32a0562cad6bb88bdda50d7fcb95","Content-Length":"401","Content-Type":"application/octet-stream","User-Agent":"MinIO (linux; amd64) minio-go/v7.0.70 mc/RELEASE.2024-04-30T17-44-48Z","X-Amz-Content-Sha256":"STREAMING-AWS4-HMAC-SHA256-PAYLOAD","X-Amz-Date":"20240509T073810Z","X-Amz-Decoded-Content-Length":"228"},"responseHeader":{"Accept-Ranges":"bytes","Content-Length":"0","ETag":"9fe7a344ef4227d3e53751e9d88ce41e","Server":"MinIO","Strict-Transport-Security":"max-age=31536000; includeSubDomains","Vary":"Origin,Accept-Encoding","X-Amz-Id-2":"dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8","X-Amz-Request-Id":"17CDC1F4D7E69123","X-Content-Type-Options":"nosniff","X-Xss-Protection":"1; mode=block"},"tags":{"objectLocation":{"name":"hosts","poolId":1,"setId":1,"drives":["/mnt/data1","/mnt/data2","/mnt/data3","/mnt/data4"]}},"accessKey":"minioadmin"}
```
MinIO also honors environment variable for Kafka target Audit logging as shown below, this setting will override the endpoint settings in the MinIO server config.
```
mc admin config set myminio/ audit_kafka --env
KEY:
audit_kafka[:name] send audit logs to kafka endpoints
ARGS:
MINIO_AUDIT_KAFKA_ENABLE* (on|off) enable audit_kafka target, default is 'off'
MINIO_AUDIT_KAFKA_BROKERS* (csv) comma separated list of Kafka broker addresses
MINIO_AUDIT_KAFKA_TOPIC (string) Kafka topic used for bucket notifications
MINIO_AUDIT_KAFKA_SASL_USERNAME (string) username for SASL/PLAIN or SASL/SCRAM authentication
MINIO_AUDIT_KAFKA_SASL_PASSWORD (string) password for SASL/PLAIN or SASL/SCRAM authentication
MINIO_AUDIT_KAFKA_SASL_MECHANISM (string) sasl authentication mechanism, default 'plain'
MINIO_AUDIT_KAFKA_TLS_CLIENT_AUTH (string) clientAuth determines the Kafka server's policy for TLS client auth
MINIO_AUDIT_KAFKA_SASL (on|off) set to 'on' to enable SASL authentication
MINIO_AUDIT_KAFKA_TLS (on|off) set to 'on' to enable TLS
MINIO_AUDIT_KAFKA_TLS_SKIP_VERIFY (on|off) trust server TLS without verification, defaults to "on" (verify)
MINIO_AUDIT_KAFKA_CLIENT_TLS_CERT (path) path to client certificate for mTLS auth
MINIO_AUDIT_KAFKA_CLIENT_TLS_KEY (path) path to client key for mTLS auth
MINIO_AUDIT_KAFKA_VERSION (string) specify the version of the Kafka cluster
MINIO_AUDIT_KAFKA_COMMENT (sentence) optionally add a comment to this setting
```
```
export MINIO_AUDIT_KAFKA_ENABLE_target1="on"
export MINIO_AUDIT_KAFKA_BROKERS_target1="localhost:29092"
export MINIO_AUDIT_KAFKA_TOPIC_target1="auditlog"
minio server /mnt/data
```
Setting this environment variable automatically enables audit logging to the Kafka target. The audit logging is in JSON format as described below.
NOTE:
- `timeToFirstByte` and `timeToResponse` will be expressed in Nanoseconds.
- Additionally in the case of the erasure coded setup `tags.objectLocation` provides per object details about
- Pool number the object operation was performed on.
- Set number the object operation was performed on.
- The list of drives participating in this operation belong to the set.
## Explore Further
- [MinIO Quickstart Guide](https://docs.min.io/community/minio-object-store/operations/deployments/baremetal-deploy-minio-on-redhat-linux.html)
- [Configure MinIO Server with TLS](https://docs.min.io/community/minio-object-store/operations/network-encryption.html)

32
docs/metrics/README.md Normal file
View File

@@ -0,0 +1,32 @@
# MinIO Monitoring Guide
MinIO server exposes monitoring data over endpoints. Monitoring tools can pick the data from these endpoints. This document lists the monitoring endpoints and relevant documentation.
## Healthcheck Probe
MinIO server has two healthcheck related un-authenticated endpoints, a liveness probe to indicate if server is responding, cluster probe to check if server can be taken down for maintenance.
- Liveness probe available at `/minio/health/live`
- Cluster probe available at `/minio/health/cluster`
Read more on how to use these endpoints in [MinIO healthcheck guide](https://github.com/minio/minio/blob/master/docs/metrics/healthcheck/README.md).
## Prometheus Probe
MinIO allows reading metrics for the entire cluster from any single node. This allows for metrics collection for a MinIO instance across all servers. Thus, metrics collection for instances behind a load balancer can be done without any knowledge of the individual node addresses. The cluster wide metrics can be read at
`<Address for MinIO Service>/minio/v2/metrics/cluster`.
The additional node specific metrics which include additional go metrics or process metrics are exposed at
`<Address for MinIO Node>/minio/v2/metrics/node`.
The additional bucket specific metrics which include additional go metrics or process metrics are exposed at
`<Address for MinIO Node>/minio/v2/metrics/bucket`.
The additional resource specific metrics which include additional go metrics or process metrics are exposed at
`<Address for MinIO Node>/minio/v2/metrics/resource`.
To use this endpoint, setup Prometheus to scrape data from this endpoint. Read more on how to configure and use Prometheus to monitor MinIO server in [How to monitor MinIO server with Prometheus](https://github.com/minio/minio/blob/master/docs/metrics/prometheus/README.md).
### **Deprecated metrics monitoring**
- Prometheus' data available at `/minio/prometheus/metrics` is deprecated

Some files were not shown because too many files have changed in this diff Show More