OP#792 initial commit after fork
This commit is contained in:
63
docs/site-replication/README.md
Normal file
63
docs/site-replication/README.md
Normal file
@@ -0,0 +1,63 @@
|
||||
# Automatic Site Replication
|
||||
|
||||
This feature allows multiple independent MinIO sites (or clusters) that are using the same external IDentity Provider (IDP) to be configured as replicas. In this situation the set of replica sites are referred to as peer sites or just sites. When site-replication is enabled on a set of sites, the following changes are replicated to all other sites:
|
||||
|
||||
- Creation and deletion of buckets and objects
|
||||
- Creation and deletion of all IAM users, groups, policies and their mappings to users or groups
|
||||
- Creation of STS credentials
|
||||
- Creation and deletion of service accounts (except those owned by the root user)
|
||||
- Changes to Bucket features such as:
|
||||
- Bucket Policies
|
||||
- Bucket Tags
|
||||
- Bucket Object-Lock configurations (including retention and legal hold configuration)
|
||||
- Bucket Encryption configuration
|
||||
|
||||
> NOTE: Bucket versioning is automatically enabled for all new and existing buckets on all replicated sites.
|
||||
|
||||
The following Bucket features will **not be replicated**, is designed to differ between sites:
|
||||
|
||||
- Bucket notification configuration
|
||||
- Bucket lifecycle (ILM) configuration
|
||||
|
||||
## Pre-requisites
|
||||
|
||||
- Initially, only **one** of the sites added for replication may have data. After site-replication is successfully configured, this data is replicated to the other (initially empty) sites. Subsequently, objects may be written to any of the sites, and they will be replicated to all other sites.
|
||||
|
||||
- **Removing a site** is not allowed from a set of replicated sites once configured.
|
||||
- All sites must be using the **same** external IDP(s) if any.
|
||||
- For [SSE-S3 or SSE-KMS encryption via KMS](https://docs.min.io/community/minio-object-store/operations/server-side-encryption.html "MinIO KMS Guide"), all sites **must** have access to a central KMS deployment. This can be achieved via a central KES server or multiple KES servers (say one per site) connected via a central KMS (Vault) server.
|
||||
|
||||
## Configuring Site Replication
|
||||
|
||||
- Configure an alias in `mc` for each of the sites. For example if you have three MinIO sites, you may run:
|
||||
|
||||
```sh
|
||||
mc alias set minio1 https://minio1.example.com:9000 adminuser adminpassword
|
||||
mc alias set minio2 https://minio2.example.com:9000 adminuser adminpassword
|
||||
mc alias set minio3 https://minio3.example.com:9000 adminuser adminpassword
|
||||
```
|
||||
|
||||
or
|
||||
|
||||
```sh
|
||||
export MC_HOST_minio1=https://adminuser:adminpassword@minio1.example.com
|
||||
export MC_HOST_minio2=https://adminuser:adminpassword@minio2.example.com
|
||||
export MC_HOST_minio3=https://adminuser:adminpassword@minio3.example.com
|
||||
```
|
||||
|
||||
- Add site replication configuration with:
|
||||
|
||||
```sh
|
||||
mc admin replicate add minio1 minio2 minio3
|
||||
```
|
||||
|
||||
- Once the above command returns success, you may query site replication configuration with:
|
||||
|
||||
```sh
|
||||
mc admin replicate info minio1
|
||||
```
|
||||
|
||||
** Note **
|
||||
Previously, site replication required the root credentials of peer sites to be identical. This is no longer necessary because STS tokens are now signed with the site replicator service account credentials, thus allowing flexibility in the independent management of root accounts across sites and the ability to disable root accounts eventually.
|
||||
|
||||
However, this means that STS tokens signed previously by root credentials will no longer be valid upon upgrading to the latest version with this change. Please re-generate them as you usually do. Additionally, if site replication is ever removed - the STS tokens will become invalid, regenerate them as you usually do.
|
||||
80
docs/site-replication/gen-oidc-sts-cred.go
Normal file
80
docs/site-replication/gen-oidc-sts-cred.go
Normal file
@@ -0,0 +1,80 @@
|
||||
//go:build ignore
|
||||
// +build ignore
|
||||
|
||||
// Copyright (c) 2015-2022 MinIO, Inc.
|
||||
//
|
||||
// This file is part of MinIO Object Storage stack
|
||||
//
|
||||
// This program is free software: you can redistribute it and/or modify
|
||||
// it under the terms of the GNU Affero General Public License as published by
|
||||
// the Free Software Foundation, either version 3 of the License, or
|
||||
// (at your option) any later version.
|
||||
//
|
||||
// This program is distributed in the hope that it will be useful
|
||||
// but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
// GNU Affero General Public License for more details.
|
||||
//
|
||||
// You should have received a copy of the GNU Affero General Public License
|
||||
// along with this program. If not, see <http://www.gnu.org/licenses/>.
|
||||
|
||||
package main
|
||||
|
||||
// This programs mocks user interaction against Dex IDP and generates STS
|
||||
// credentials. It is for MinIO testing purposes only.
|
||||
//
|
||||
// Run like:
|
||||
//
|
||||
// $ MINIO_ENDPOINT=http://localhost:9000 go run gen-oidc-sts-cred.go
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log"
|
||||
"net/http"
|
||||
"os"
|
||||
|
||||
cr "github.com/minio/minio-go/v7/pkg/credentials"
|
||||
cmd "github.com/minio/minio/cmd"
|
||||
)
|
||||
|
||||
func main() {
|
||||
ctx := context.Background()
|
||||
|
||||
endpoint := os.Getenv("MINIO_ENDPOINT")
|
||||
if endpoint == "" {
|
||||
log.Fatalf("Please specify a MinIO server endpoint environment variable like:\n\n\texport MINIO_ENDPOINT=http://localhost:9000")
|
||||
}
|
||||
|
||||
appParams := cmd.OpenIDClientAppParams{
|
||||
ClientID: "minio-client-app",
|
||||
ClientSecret: "minio-client-app-secret",
|
||||
ProviderURL: "http://127.0.0.1:5556/dex",
|
||||
RedirectURL: "http://127.0.0.1:10000/oauth_callback",
|
||||
}
|
||||
|
||||
oidcToken, err := cmd.MockOpenIDTestUserInteraction(ctx, appParams, "dillon@example.io", "dillon")
|
||||
if err != nil {
|
||||
log.Fatalf("Failed to generate OIDC token: %v", err)
|
||||
}
|
||||
|
||||
roleARN := os.Getenv("ROLE_ARN")
|
||||
webID := cr.STSWebIdentity{
|
||||
Client: &http.Client{},
|
||||
STSEndpoint: endpoint,
|
||||
GetWebIDTokenExpiry: func() (*cr.WebIdentityToken, error) {
|
||||
return &cr.WebIdentityToken{
|
||||
Token: oidcToken,
|
||||
}, nil
|
||||
},
|
||||
RoleARN: roleARN,
|
||||
}
|
||||
|
||||
value, err := webID.Retrieve()
|
||||
if err != nil {
|
||||
log.Fatalf("Expected to generate credentials: %v", err)
|
||||
}
|
||||
|
||||
// Print credentials separated by colons:
|
||||
fmt.Printf("%s:%s:%s\n", value.AccessKeyID, value.SecretAccessKey, value.SessionToken)
|
||||
}
|
||||
14
docs/site-replication/ldap.yaml
Normal file
14
docs/site-replication/ldap.yaml
Normal file
@@ -0,0 +1,14 @@
|
||||
# To run locally an OpenLDAP instance using Docker
|
||||
# $ docker-compose -f ldap.yaml up -d
|
||||
version: '3.7'
|
||||
|
||||
services:
|
||||
openldap:
|
||||
image: quay.io/minio/openldap
|
||||
ports:
|
||||
- "389:389"
|
||||
- "636:636"
|
||||
environment:
|
||||
LDAP_ORGANIZATION: "MinIO Inc"
|
||||
LDAP_DOMAIN: "min.io"
|
||||
LDAP_ADMIN_PASSWORD: "admin"
|
||||
391
docs/site-replication/run-multi-site-ldap.sh
Executable file
391
docs/site-replication/run-multi-site-ldap.sh
Executable file
@@ -0,0 +1,391 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
# shellcheck disable=SC2120
|
||||
exit_1() {
|
||||
cleanup
|
||||
|
||||
echo "minio1 ============"
|
||||
cat /tmp/minio1_1.log
|
||||
cat /tmp/minio1_2.log
|
||||
echo "minio2 ============"
|
||||
cat /tmp/minio2_1.log
|
||||
cat /tmp/minio2_2.log
|
||||
echo "minio3 ============"
|
||||
cat /tmp/minio3_1.log
|
||||
cat /tmp/minio3_2.log
|
||||
|
||||
exit 1
|
||||
}
|
||||
|
||||
cleanup() {
|
||||
echo "Cleaning up instances of MinIO"
|
||||
pkill minio
|
||||
pkill -9 minio
|
||||
rm -rf /tmp/minio-ldap-idp{1,2,3}
|
||||
}
|
||||
|
||||
cleanup
|
||||
|
||||
unset MINIO_KMS_KES_CERT_FILE
|
||||
unset MINIO_KMS_KES_KEY_FILE
|
||||
unset MINIO_KMS_KES_ENDPOINT
|
||||
unset MINIO_KMS_KES_KEY_NAME
|
||||
|
||||
export MINIO_CI_CD=1
|
||||
export MINIO_BROWSER=off
|
||||
export MINIO_ROOT_USER="minio"
|
||||
export MINIO_ROOT_PASSWORD="minio123"
|
||||
export MINIO_KMS_AUTO_ENCRYPTION=off
|
||||
export MINIO_PROMETHEUS_AUTH_TYPE=public
|
||||
export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
|
||||
export MINIO_IDENTITY_LDAP_SERVER_ADDR="localhost:389"
|
||||
export MINIO_IDENTITY_LDAP_SERVER_INSECURE="on"
|
||||
export MINIO_IDENTITY_LDAP_LOOKUP_BIND_DN="cn=admin,dc=min,dc=io"
|
||||
export MINIO_IDENTITY_LDAP_LOOKUP_BIND_PASSWORD="admin"
|
||||
export MINIO_IDENTITY_LDAP_USER_DN_SEARCH_BASE_DN="dc=min,dc=io"
|
||||
export MINIO_IDENTITY_LDAP_USER_DN_SEARCH_FILTER="(uid=%s)"
|
||||
export MINIO_IDENTITY_LDAP_GROUP_SEARCH_BASE_DN="ou=swengg,dc=min,dc=io"
|
||||
export MINIO_IDENTITY_LDAP_GROUP_SEARCH_FILTER="(&(objectclass=groupOfNames)(member=%d))"
|
||||
|
||||
if [ ! -f ./mc ]; then
|
||||
wget -O mc https://dl.minio.io/client/mc/release/linux-amd64/mc &&
|
||||
chmod +x mc
|
||||
fi
|
||||
|
||||
minio server --config-dir /tmp/minio-ldap --address ":9001" /tmp/minio-ldap-idp1/{1...4} >/tmp/minio1_1.log 2>&1 &
|
||||
site1_pid=$!
|
||||
minio server --config-dir /tmp/minio-ldap --address ":9002" /tmp/minio-ldap-idp2/{1...4} >/tmp/minio2_1.log 2>&1 &
|
||||
site2_pid=$!
|
||||
minio server --config-dir /tmp/minio-ldap --address ":9003" /tmp/minio-ldap-idp3/{1...4} >/tmp/minio3_1.log 2>&1 &
|
||||
site3_pid=$!
|
||||
|
||||
export MC_HOST_minio1=http://minio:minio123@localhost:9001
|
||||
export MC_HOST_minio2=http://minio:minio123@localhost:9002
|
||||
export MC_HOST_minio3=http://minio:minio123@localhost:9003
|
||||
|
||||
./mc ready minio1
|
||||
./mc ready minio2
|
||||
./mc ready minio3
|
||||
|
||||
./mc admin replicate add minio1 minio2 minio3
|
||||
|
||||
./mc idp ldap policy attach minio1 consoleAdmin --user="uid=dillon,ou=people,ou=swengg,dc=min,dc=io"
|
||||
sleep 5
|
||||
|
||||
./mc admin user info minio2 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io"
|
||||
./mc admin user info minio3 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io"
|
||||
./mc admin policy create minio1 rw ./docs/site-replication/rw.json
|
||||
|
||||
sleep 5
|
||||
./mc admin policy info minio2 rw >/dev/null 2>&1
|
||||
./mc admin policy info minio3 rw >/dev/null 2>&1
|
||||
|
||||
./mc admin policy remove minio3 rw
|
||||
|
||||
sleep 10
|
||||
./mc admin policy info minio1 rw
|
||||
if [ $? -eq 0 ]; then
|
||||
echo "expecting the command to fail, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc admin policy info minio2 rw
|
||||
if [ $? -eq 0 ]; then
|
||||
echo "expecting the command to fail, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc admin user info minio1 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io"
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "policy mapping missing, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc admin user info minio2 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io"
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "policy mapping missing, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc admin user info minio3 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io"
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "policy mapping missing, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
# LDAP simple user
|
||||
./mc admin user svcacct add minio2 dillon --access-key testsvc --secret-key testsvc123
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "adding svc account failed, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
sleep 10
|
||||
|
||||
./mc idp ldap policy entities minio1
|
||||
./mc idp ldap policy entities minio2
|
||||
./mc idp ldap policy entities minio3
|
||||
|
||||
./mc admin service restart minio1
|
||||
./mc admin service restart minio2
|
||||
./mc admin service restart minio3
|
||||
|
||||
sleep 10
|
||||
|
||||
./mc idp ldap policy entities minio1
|
||||
./mc idp ldap policy entities minio2
|
||||
./mc idp ldap policy entities minio3
|
||||
|
||||
./mc admin user svcacct info minio1 testsvc
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "svc account not mirrored, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc admin user svcacct info minio2 testsvc
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "svc account not mirrored, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc admin user svcacct info minio3 testsvc
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "svc account not mirrored, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
MC_HOST_svc1=http://testsvc:testsvc123@localhost:9001 ./mc ls svc1
|
||||
MC_HOST_svc2=http://testsvc:testsvc123@localhost:9002 ./mc ls svc2
|
||||
MC_HOST_svc3=http://testsvc:testsvc123@localhost:9003 ./mc ls svc3
|
||||
|
||||
./mc admin user svcacct rm minio1 testsvc
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "removing svc account failed, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc admin user info minio1 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io"
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "policy mapping missing, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc admin user info minio2 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io"
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "policy mapping missing, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc admin user info minio3 "uid=dillon,ou=people,ou=swengg,dc=min,dc=io"
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "policy mapping missing, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
sleep 10
|
||||
|
||||
./mc admin user svcacct info minio2 testsvc
|
||||
if [ $? -eq 0 ]; then
|
||||
echo "svc account found after delete, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc admin user svcacct info minio3 testsvc
|
||||
if [ $? -eq 0 ]; then
|
||||
echo "svc account found after delete, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc mb minio1/newbucket
|
||||
# copy large upload to newbucket on minio1
|
||||
truncate -s 17M lrgfile
|
||||
expected_checksum=$(cat ./lrgfile | md5sum)
|
||||
|
||||
./mc cp ./lrgfile minio1/newbucket
|
||||
|
||||
# create a bucket bucket2 on minio1.
|
||||
./mc mb minio1/bucket2
|
||||
|
||||
sleep 5
|
||||
./mc stat --no-list minio2/newbucket
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "expecting bucket to be present. exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc stat --no-list minio3/newbucket
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "expecting bucket to be present. exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc cp README.md minio2/newbucket/
|
||||
|
||||
sleep 5
|
||||
./mc stat --no-list minio1/newbucket/README.md
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "expecting object to be present. exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc stat --no-list minio3/newbucket/README.md
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "expecting object to be present. exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
sleep 10
|
||||
./mc stat --no-list minio3/newbucket/lrgfile
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "expected object to be present, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
actual_checksum=$(./mc cat minio3/newbucket/lrgfile | md5sum)
|
||||
if [ "${expected_checksum}" != "${actual_checksum}" ]; then
|
||||
echo "replication failed on multipart objects expected ${expected_checksum} got ${actual_checksum}"
|
||||
exit
|
||||
fi
|
||||
rm ./lrgfile
|
||||
|
||||
./mc rm -r --versions --force minio1/newbucket/lrgfile
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "expected object to be present, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
sleep 5
|
||||
./mc stat --no-list minio1/newbucket/lrgfile
|
||||
if [ $? -eq 0 ]; then
|
||||
echo "expected object to be deleted permanently after replication, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
vID=$(./mc stat --no-list minio2/newbucket/README.md --json | jq .versionID)
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "expecting object to be present. exiting.."
|
||||
exit_1
|
||||
fi
|
||||
./mc tag set --version-id "${vID}" minio2/newbucket/README.md "key=val"
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "expecting tag set to be successful. exiting.."
|
||||
exit_1
|
||||
fi
|
||||
sleep 5
|
||||
|
||||
./mc tag remove --version-id "${vID}" minio2/newbucket/README.md
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "expecting tag removal to be successful. exiting.."
|
||||
exit_1
|
||||
fi
|
||||
sleep 5
|
||||
|
||||
replStatus_minio2=$(./mc stat --no-list minio2/newbucket/README.md --json | jq -r .replicationStatus)
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "expecting object to be present. exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
if [ ${replStatus_minio2} != "COMPLETED" ]; then
|
||||
echo "expected tag removal to have replicated, exiting..."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc rm minio3/newbucket/README.md
|
||||
sleep 5
|
||||
|
||||
./mc stat --no-list minio2/newbucket/README.md
|
||||
if [ $? -eq 0 ]; then
|
||||
echo "expected file to be deleted, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc stat --no-list minio1/newbucket/README.md
|
||||
if [ $? -eq 0 ]; then
|
||||
echo "expected file to be deleted, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc mb --with-lock minio3/newbucket-olock
|
||||
sleep 5
|
||||
|
||||
enabled_minio2=$(./mc stat --json minio2/newbucket-olock | jq -r .ObjectLock.enabled)
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "expected bucket to be mirrored with object-lock but not present, exiting..."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
if [ "${enabled_minio2}" != "Enabled" ]; then
|
||||
echo "expected bucket to be mirrored with object-lock enabled, exiting..."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
enabled_minio1=$(./mc stat --json minio1/newbucket-olock | jq -r .ObjectLock.enabled)
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "expected bucket to be mirrored with object-lock but not present, exiting..."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
if [ "${enabled_minio1}" != "Enabled" ]; then
|
||||
echo "expected bucket to be mirrored with object-lock enabled, exiting..."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
# "Test if most recent tag update is replicated"
|
||||
./mc tag set minio2/newbucket "key=val1"
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "expecting tag set to be successful. exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
sleep 10
|
||||
|
||||
val=$(./mc tag list minio1/newbucket --json | jq -r .tagset | jq -r .key)
|
||||
if [ "${val}" != "val1" ]; then
|
||||
echo "expected bucket tag to have replicated, exiting..."
|
||||
exit_1
|
||||
fi
|
||||
# stop minio1
|
||||
kill -9 ${site1_pid}
|
||||
# Update tag on minio2/newbucket when minio1 is down
|
||||
./mc tag set minio2/newbucket "key=val2"
|
||||
# create a new bucket on minio2. This should replicate to minio1 after it comes online.
|
||||
./mc mb minio2/newbucket2
|
||||
# delete bucket2 on minio2. This should replicate to minio1 after it comes online.
|
||||
./mc rb minio2/bucket2
|
||||
|
||||
# Restart minio1 instance
|
||||
minio server --config-dir /tmp/minio-ldap --address ":9001" /tmp/minio-ldap-idp1/{1...4} >/tmp/minio1_1.log 2>&1 &
|
||||
sleep 200
|
||||
|
||||
# Test whether most recent tag update on minio2 is replicated to minio1
|
||||
val=$(./mc tag list minio1/newbucket --json | jq -r .tagset | jq -r .key)
|
||||
if [ "${val}" != "val2" ]; then
|
||||
echo "expected bucket tag to have replicated, exiting..."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
# Test if bucket created/deleted when minio1 is down healed
|
||||
diff -q <(./mc ls minio1) <(./mc ls minio2) 1>/dev/null
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "expected 'bucket2' delete and 'newbucket2' creation to have replicated, exiting..."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
# force a resync after removing all site replication
|
||||
./mc admin replicate rm --all --force minio1
|
||||
./mc rb minio2 --force --dangerous
|
||||
./mc admin replicate add minio1 minio2
|
||||
./mc admin replicate resync start minio1 minio2
|
||||
sleep 30
|
||||
|
||||
./mc ls -r --versions minio1/newbucket >/tmp/minio1.txt
|
||||
./mc ls -r --versions minio2/newbucket >/tmp/minio2.txt
|
||||
|
||||
out=$(diff -qpruN /tmp/minio1.txt /tmp/minio2.txt)
|
||||
ret=$?
|
||||
if [ $ret -ne 0 ]; then
|
||||
echo "BUG: expected no missing entries after replication resync: $out"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
cleanup
|
||||
458
docs/site-replication/run-multi-site-minio-idp.sh
Executable file
458
docs/site-replication/run-multi-site-minio-idp.sh
Executable file
@@ -0,0 +1,458 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
# shellcheck disable=SC2120
|
||||
exit_1() {
|
||||
cleanup
|
||||
|
||||
echo "minio1 ============"
|
||||
cat /tmp/minio1_1.log
|
||||
cat /tmp/minio1_2.log
|
||||
echo "minio2 ============"
|
||||
cat /tmp/minio2_1.log
|
||||
cat /tmp/minio2_2.log
|
||||
echo "minio3 ============"
|
||||
cat /tmp/minio3_1.log
|
||||
cat /tmp/minio3_2.log
|
||||
|
||||
exit 1
|
||||
}
|
||||
|
||||
cleanup() {
|
||||
echo "Cleaning up instances of MinIO"
|
||||
pkill minio
|
||||
pkill -9 minio
|
||||
rm -rf /tmp/minio-internal-idp{1,2,3}
|
||||
}
|
||||
|
||||
cleanup
|
||||
|
||||
unset MINIO_KMS_KES_CERT_FILE
|
||||
unset MINIO_KMS_KES_KEY_FILE
|
||||
unset MINIO_KMS_KES_ENDPOINT
|
||||
unset MINIO_KMS_KES_KEY_NAME
|
||||
|
||||
export MINIO_CI_CD=1
|
||||
export MINIO_BROWSER=off
|
||||
export MINIO_ROOT_USER="minio"
|
||||
export MINIO_ROOT_PASSWORD="minio123"
|
||||
export MINIO_KMS_AUTO_ENCRYPTION=off
|
||||
export MINIO_PROMETHEUS_AUTH_TYPE=public
|
||||
export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
|
||||
|
||||
if [ ! -f ./mc ]; then
|
||||
wget -O mc https://dl.minio.io/client/mc/release/linux-amd64/mc &&
|
||||
chmod +x mc
|
||||
fi
|
||||
|
||||
minio server --config-dir /tmp/minio-internal --address ":9001" http://localhost:9001/tmp/minio-internal-idp1/{1...4} http://localhost:9010/tmp/minio-internal-idp1/{5...8} >/tmp/minio1_1.log 2>&1 &
|
||||
site1_pid1=$!
|
||||
minio server --config-dir /tmp/minio-internal --address ":9010" http://localhost:9001/tmp/minio-internal-idp1/{1...4} http://localhost:9010/tmp/minio-internal-idp1/{5...8} >/tmp/minio1_2.log 2>&1 &
|
||||
site1_pid2=$!
|
||||
|
||||
minio server --config-dir /tmp/minio-internal --address ":9002" http://localhost:9002/tmp/minio-internal-idp2/{1...4} http://localhost:9020/tmp/minio-internal-idp2/{5...8} >/tmp/minio2_1.log 2>&1 &
|
||||
site2_pid1=$!
|
||||
minio server --config-dir /tmp/minio-internal --address ":9020" http://localhost:9002/tmp/minio-internal-idp2/{1...4} http://localhost:9020/tmp/minio-internal-idp2/{5...8} >/tmp/minio2_2.log 2>&1 &
|
||||
site2_pid2=$!
|
||||
|
||||
minio server --config-dir /tmp/minio-internal --address ":9003" http://localhost:9003/tmp/minio-internal-idp3/{1...4} http://localhost:9030/tmp/minio-internal-idp3/{5...8} >/tmp/minio3_1.log 2>&1 &
|
||||
site3_pid1=$!
|
||||
minio server --config-dir /tmp/minio-internal --address ":9030" http://localhost:9003/tmp/minio-internal-idp3/{1...4} http://localhost:9030/tmp/minio-internal-idp3/{5...8} >/tmp/minio3_2.log 2>&1 &
|
||||
site3_pid2=$!
|
||||
|
||||
export MC_HOST_minio1=http://minio:minio123@localhost:9001
|
||||
export MC_HOST_minio2=http://minio:minio123@localhost:9002
|
||||
export MC_HOST_minio3=http://minio:minio123@localhost:9003
|
||||
|
||||
export MC_HOST_minio10=http://minio:minio123@localhost:9010
|
||||
export MC_HOST_minio20=http://minio:minio123@localhost:9020
|
||||
export MC_HOST_minio30=http://minio:minio123@localhost:9030
|
||||
|
||||
./mc ready minio1
|
||||
./mc ready minio2
|
||||
./mc ready minio3
|
||||
./mc ready minio10
|
||||
./mc ready minio20
|
||||
./mc ready minio30
|
||||
|
||||
./mc admin replicate add minio1 minio2
|
||||
|
||||
site_enabled=$(./mc admin replicate info minio1)
|
||||
site_enabled_peer=$(./mc admin replicate info minio10)
|
||||
|
||||
[[ $site_enabled =~ "is not enabled" ]] && {
|
||||
echo "expected both peers to have same information"
|
||||
exit_1
|
||||
}
|
||||
|
||||
[[ $site_enabled_peer =~ "is not enabled" ]] && {
|
||||
echo "expected both peers to have same information"
|
||||
exit_1
|
||||
}
|
||||
|
||||
./mc admin user add minio1 foobar foo12345
|
||||
|
||||
## add foobar-g group with foobar
|
||||
./mc admin group add minio2 foobar-g foobar
|
||||
|
||||
./mc admin policy attach minio1 consoleAdmin --user=foobar
|
||||
sleep 5
|
||||
|
||||
./mc admin user info minio2 foobar
|
||||
|
||||
./mc admin group info minio1 foobar-g
|
||||
|
||||
./mc admin policy create minio1 rw ./docs/site-replication/rw.json
|
||||
|
||||
sleep 5
|
||||
./mc admin policy info minio2 rw >/dev/null 2>&1
|
||||
|
||||
./mc admin replicate status minio1
|
||||
|
||||
## Add a new empty site
|
||||
./mc admin replicate add minio1 minio2 minio3
|
||||
|
||||
sleep 10
|
||||
|
||||
./mc admin policy info minio3 rw >/dev/null 2>&1
|
||||
|
||||
./mc admin policy remove minio3 rw
|
||||
|
||||
./mc admin replicate status minio3
|
||||
|
||||
sleep 10
|
||||
|
||||
./mc admin policy info minio1 rw
|
||||
if [ $? -eq 0 ]; then
|
||||
echo "expecting the command to fail, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc admin policy info minio2 rw
|
||||
if [ $? -eq 0 ]; then
|
||||
echo "expecting the command to fail, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc admin policy info minio3 rw
|
||||
if [ $? -eq 0 ]; then
|
||||
echo "expecting the command to fail, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc admin user info minio1 foobar
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "policy mapping missing on 'minio1', exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc admin user info minio2 foobar
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "policy mapping missing on 'minio2', exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc admin user info minio3 foobar
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "policy mapping missing on 'minio3', exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc admin group info minio3 foobar-g
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "group mapping missing on 'minio3', exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc admin user svcacct add minio2 foobar --access-key testsvc --secret-key testsvc123
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "adding svc account failed, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc admin user svcacct add minio2 minio --access-key testsvc2 --secret-key testsvc123
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "adding root svc account testsvc2 failed, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
sleep 10
|
||||
|
||||
export MC_HOST_rootsvc=http://testsvc2:testsvc123@localhost:9002
|
||||
./mc ls rootsvc
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "root service account not inherited root permissions, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc admin user svcacct info minio1 testsvc
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "svc account not mirrored, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc admin user svcacct info minio2 testsvc
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "svc account not mirrored, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc admin user svcacct rm minio1 testsvc
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "removing svc account failed, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
sleep 10
|
||||
./mc admin user svcacct info minio2 testsvc
|
||||
if [ $? -eq 0 ]; then
|
||||
echo "svc account found after delete, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc admin user svcacct info minio3 testsvc
|
||||
if [ $? -eq 0 ]; then
|
||||
echo "svc account found after delete, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc mb minio1/newbucket
|
||||
# copy large upload to newbucket on minio1
|
||||
truncate -s 17M lrgfile
|
||||
expected_checksum=$(cat ./lrgfile | md5sum)
|
||||
|
||||
./mc cp ./lrgfile minio1/newbucket
|
||||
|
||||
sleep 5
|
||||
./mc stat --no-list minio2/newbucket
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "expecting bucket to be present. exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc stat --no-list minio3/newbucket
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "expecting bucket to be present. exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
err_minio2=$(./mc stat --no-list minio2/newbucket/xxx --json | jq -r .error.cause.message)
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "expecting object to be missing. exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
if [ "${err_minio2}" != "Object does not exist" ]; then
|
||||
echo "expected to see Object does not exist error, exiting..."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc cp README.md minio2/newbucket/
|
||||
|
||||
sleep 5
|
||||
./mc stat --no-list minio1/newbucket/README.md
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "expecting object to be present. exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc stat --no-list minio3/newbucket/README.md
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "expecting object to be present. exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
sleep 10
|
||||
./mc stat --no-list minio3/newbucket/lrgfile
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "expected object to be present, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
actual_checksum=$(./mc cat minio3/newbucket/lrgfile | md5sum)
|
||||
if [ "${expected_checksum}" != "${actual_checksum}" ]; then
|
||||
echo "replication failed on multipart objects expected ${expected_checksum} got ${actual_checksum}"
|
||||
exit
|
||||
fi
|
||||
rm ./lrgfile
|
||||
|
||||
./mc rm -r --versions --force minio1/newbucket/lrgfile
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "expected object to be present, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
sleep 5
|
||||
./mc stat --no-list minio1/newbucket/lrgfile
|
||||
if [ $? -eq 0 ]; then
|
||||
echo "expected object to be deleted permanently after replication, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
vID=$(./mc stat --no-list minio2/newbucket/README.md --json | jq .versionID)
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "expecting object to be present. exiting.."
|
||||
exit_1
|
||||
fi
|
||||
./mc tag set --version-id "${vID}" minio2/newbucket/README.md "key=val"
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "expecting tag set to be successful. exiting.."
|
||||
exit_1
|
||||
fi
|
||||
sleep 5
|
||||
val=$(./mc tag list minio1/newbucket/README.md --version-id "${vID}" --json | jq -r .tagset.key)
|
||||
if [ "${val}" != "val" ]; then
|
||||
echo "expected bucket tag to have replicated, exiting..."
|
||||
exit_1
|
||||
fi
|
||||
./mc tag remove --version-id "${vID}" minio2/newbucket/README.md
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "expecting tag removal to be successful. exiting.."
|
||||
exit_1
|
||||
fi
|
||||
sleep 5
|
||||
|
||||
replStatus_minio2=$(./mc stat --no-list minio2/newbucket/README.md --json | jq -r .replicationStatus)
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "expecting object to be present. exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
if [ ${replStatus_minio2} != "COMPLETED" ]; then
|
||||
echo "expected tag removal to have replicated, exiting..."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc rm minio3/newbucket/README.md
|
||||
sleep 5
|
||||
|
||||
./mc stat --no-list minio2/newbucket/README.md
|
||||
if [ $? -eq 0 ]; then
|
||||
echo "expected file to be deleted, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc stat --no-list minio1/newbucket/README.md
|
||||
if [ $? -eq 0 ]; then
|
||||
echo "expected file to be deleted, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc mb --with-lock minio3/newbucket-olock
|
||||
sleep 5
|
||||
|
||||
set -x
|
||||
|
||||
enabled_minio2=$(./mc stat --json minio2/newbucket-olock | jq -r .ObjectLock.enabled)
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "expected bucket to be mirrored with object-lock but not present, exiting..."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
if [ "${enabled_minio2}" != "Enabled" ]; then
|
||||
echo "expected bucket to be mirrored with object-lock enabled, exiting..."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
enabled_minio1=$(./mc stat --json minio1/newbucket-olock | jq -r .ObjectLock.enabled)
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "expected bucket to be mirrored with object-lock but not present, exiting..."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
if [ "${enabled_minio1}" != "Enabled" ]; then
|
||||
echo "expected bucket to be mirrored with object-lock enabled, exiting..."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
set +x
|
||||
|
||||
# "Test if most recent tag update is replicated"
|
||||
./mc tag set minio2/newbucket "key=val1"
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "expecting tag set to be successful. exiting.."
|
||||
exit_1
|
||||
fi
|
||||
sleep 5
|
||||
|
||||
val=$(./mc tag list minio1/newbucket --json | jq -r .tagset | jq -r .key)
|
||||
if [ "${val}" != "val1" ]; then
|
||||
echo "expected bucket tag to have replicated, exiting..."
|
||||
exit_1
|
||||
fi
|
||||
# Create user with policy consoleAdmin on minio1
|
||||
./mc admin user add minio1 foobarx foobar123
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "adding user failed, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
./mc admin policy attach minio1 consoleAdmin --user=foobarx
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "adding policy mapping failed, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
sleep 10
|
||||
|
||||
# unset policy for foobarx in minio2
|
||||
./mc admin policy detach minio2 consoleAdmin --user=foobarx
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "unset policy mapping failed, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
# create a bucket bucket2 on minio1.
|
||||
./mc mb minio1/bucket2
|
||||
|
||||
sleep 10
|
||||
|
||||
# Test whether policy detach replicated to minio1
|
||||
policy=$(./mc admin user info minio1 foobarx --json | jq -r .policyName)
|
||||
if [ "${policy}" != "null" ]; then
|
||||
echo "expected policy detach to have replicated, exiting..."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
kill -9 ${site1_pid1} ${site1_pid2}
|
||||
|
||||
# Update tag on minio2/newbucket when minio1 is down
|
||||
./mc tag set minio2/newbucket "key=val2"
|
||||
# create a new bucket on minio2. This should replicate to minio1 after it comes online.
|
||||
./mc mb minio2/newbucket2
|
||||
|
||||
# delete bucket2 on minio2. This should replicate to minio1 after it comes online.
|
||||
./mc rb minio2/bucket2
|
||||
|
||||
# Restart minio1 instance
|
||||
minio server --config-dir /tmp/minio-internal --address ":9001" http://localhost:9001/tmp/minio-internal-idp1/{1...4} http://localhost:9010/tmp/minio-internal-idp1/{5...8} >/tmp/minio1_1.log 2>&1 &
|
||||
minio server --config-dir /tmp/minio-internal --address ":9010" http://localhost:9001/tmp/minio-internal-idp1/{1...4} http://localhost:9010/tmp/minio-internal-idp1/{5...8} >/tmp/minio1_2.log 2>&1 &
|
||||
sleep 200
|
||||
|
||||
# Test whether most recent tag update on minio2 is replicated to minio1
|
||||
val=$(./mc tag list minio1/newbucket --json | jq -r .tagset | jq -r .key)
|
||||
if [ "${val}" != "val2" ]; then
|
||||
echo "expected bucket tag to have replicated, exiting..."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
# Test if bucket created/deleted when minio1 is down healed
|
||||
diff -q <(./mc ls minio1) <(./mc ls minio2) 1>/dev/null
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "expected 'bucket2' delete and 'newbucket2' creation to have replicated, exiting..."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
# force a resync after removing all site replication
|
||||
./mc admin replicate rm --all --force minio1
|
||||
./mc rb minio2 --force --dangerous
|
||||
./mc admin replicate add minio1 minio2
|
||||
./mc admin replicate resync start minio1 minio2
|
||||
sleep 30
|
||||
|
||||
./mc ls -r --versions minio1/newbucket >/tmp/minio1.txt
|
||||
./mc ls -r --versions minio2/newbucket >/tmp/minio2.txt
|
||||
|
||||
out=$(diff -qpruN /tmp/minio1.txt /tmp/minio2.txt)
|
||||
ret=$?
|
||||
if [ $ret -ne 0 ]; then
|
||||
echo "BUG: expected no missing entries after replication resync: $out"
|
||||
exit 1
|
||||
fi
|
||||
327
docs/site-replication/run-multi-site-oidc.sh
Executable file
327
docs/site-replication/run-multi-site-oidc.sh
Executable file
@@ -0,0 +1,327 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
# shellcheck disable=SC2120
|
||||
exit_1() {
|
||||
cleanup
|
||||
|
||||
echo "minio1 ============"
|
||||
cat /tmp/minio1_1.log
|
||||
echo "minio2 ============"
|
||||
cat /tmp/minio2_1.log
|
||||
echo "minio3 ============"
|
||||
cat /tmp/minio3_1.log
|
||||
|
||||
exit 1
|
||||
}
|
||||
|
||||
cleanup() {
|
||||
echo "Cleaning up instances of MinIO"
|
||||
pkill minio
|
||||
pkill -9 minio
|
||||
rm -rf /tmp/minio{1,2,3}
|
||||
}
|
||||
|
||||
cleanup
|
||||
|
||||
unset MINIO_KMS_KES_CERT_FILE
|
||||
unset MINIO_KMS_KES_KEY_FILE
|
||||
unset MINIO_KMS_KES_ENDPOINT
|
||||
unset MINIO_KMS_KES_KEY_NAME
|
||||
|
||||
export MINIO_CI_CD=1
|
||||
export MINIO_BROWSER=off
|
||||
export MINIO_ROOT_USER="minio"
|
||||
export MINIO_ROOT_PASSWORD="minio123"
|
||||
export MINIO_KMS_AUTO_ENCRYPTION=off
|
||||
export MINIO_PROMETHEUS_AUTH_TYPE=public
|
||||
export MINIO_KMS_SECRET_KEY=my-minio-key:OSMM+vkKUTCvQs9YL/CVMIMt43HFhkUpqJxTmGl6rYw=
|
||||
export MINIO_IDENTITY_OPENID_CONFIG_URL="http://localhost:5556/dex/.well-known/openid-configuration"
|
||||
export MINIO_IDENTITY_OPENID_CLIENT_ID="minio-client-app"
|
||||
export MINIO_IDENTITY_OPENID_CLIENT_SECRET="minio-client-app-secret"
|
||||
export MINIO_IDENTITY_OPENID_CLAIM_NAME="groups"
|
||||
export MINIO_IDENTITY_OPENID_SCOPES="openid,groups"
|
||||
|
||||
export MINIO_IDENTITY_OPENID_REDIRECT_URI="http://127.0.0.1:10000/oauth_callback"
|
||||
minio server --address ":9001" --console-address ":10000" /tmp/minio1/{1...4} >/tmp/minio1_1.log 2>&1 &
|
||||
site1_pid=$!
|
||||
export MINIO_IDENTITY_OPENID_REDIRECT_URI="http://127.0.0.1:11000/oauth_callback"
|
||||
minio server --address ":9002" --console-address ":11000" /tmp/minio2/{1...4} >/tmp/minio2_1.log 2>&1 &
|
||||
site2_pid=$!
|
||||
|
||||
export MINIO_IDENTITY_OPENID_REDIRECT_URI="http://127.0.0.1:12000/oauth_callback"
|
||||
minio server --address ":9003" --console-address ":12000" /tmp/minio3/{1...4} >/tmp/minio3_1.log 2>&1 &
|
||||
site3_pid=$!
|
||||
|
||||
if [ ! -f ./mc ]; then
|
||||
wget -O mc https://dl.minio.io/client/mc/release/linux-amd64/mc &&
|
||||
chmod +x mc
|
||||
fi
|
||||
|
||||
export MC_HOST_minio1=http://minio:minio123@localhost:9001
|
||||
export MC_HOST_minio2=http://minio:minio123@localhost:9002
|
||||
export MC_HOST_minio3=http://minio:minio123@localhost:9003
|
||||
|
||||
./mc ready minio1
|
||||
./mc ready minio2
|
||||
./mc ready minio3
|
||||
|
||||
./mc admin replicate add minio1 minio2 minio3
|
||||
|
||||
./mc admin policy create minio1 projecta ./docs/site-replication/rw.json
|
||||
sleep 5
|
||||
|
||||
./mc admin policy info minio2 projecta >/dev/null 2>&1
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "expecting the command to succeed, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
./mc admin policy info minio3 projecta >/dev/null 2>&1
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "expecting the command to succeed, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc admin policy remove minio3 projecta
|
||||
|
||||
sleep 10
|
||||
./mc admin policy info minio1 projecta
|
||||
if [ $? -eq 0 ]; then
|
||||
echo "expecting the command to fail, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc admin policy info minio2 projecta
|
||||
if [ $? -eq 0 ]; then
|
||||
echo "expecting the command to fail, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc admin policy create minio1 projecta ./docs/site-replication/rw.json
|
||||
sleep 5
|
||||
|
||||
# Generate STS credential with STS call to minio1
|
||||
STS_CRED=$(MINIO_ENDPOINT=http://localhost:9001 go run ./docs/site-replication/gen-oidc-sts-cred.go)
|
||||
|
||||
MC_HOST_foo=http://${STS_CRED}@localhost:9001 ./mc ls foo
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "Expected sts credential to work, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
sleep 2
|
||||
|
||||
# Check that the STS credential works on minio2 and minio3.
|
||||
MC_HOST_foo=http://${STS_CRED}@localhost:9002 ./mc ls foo
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "Expected sts credential to work, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
MC_HOST_foo=http://${STS_CRED}@localhost:9003 ./mc ls foo
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "Expected sts credential to work, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
STS_ACCESS_KEY=$(echo ${STS_CRED} | cut -d ':' -f 1)
|
||||
|
||||
# Create service account for STS user
|
||||
./mc admin user svcacct add minio2 $STS_ACCESS_KEY --access-key testsvc --secret-key testsvc123
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "adding svc account failed, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
sleep 10
|
||||
|
||||
./mc admin user svcacct info minio1 testsvc
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "svc account not mirrored, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc admin user svcacct info minio2 testsvc
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "svc account not mirrored, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc admin user svcacct rm minio1 testsvc
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "removing svc account failed, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
sleep 10
|
||||
./mc admin user svcacct info minio2 testsvc
|
||||
if [ $? -eq 0 ]; then
|
||||
echo "svc account found after delete, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc admin user svcacct info minio3 testsvc
|
||||
if [ $? -eq 0 ]; then
|
||||
echo "svc account found after delete, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
# create a bucket bucket2 on minio1.
|
||||
./mc mb minio1/bucket2
|
||||
|
||||
./mc mb minio1/newbucket
|
||||
|
||||
# copy large upload to newbucket on minio1
|
||||
truncate -s 17M lrgfile
|
||||
expected_checksum=$(cat ./lrgfile | md5sum)
|
||||
|
||||
./mc cp ./lrgfile minio1/newbucket
|
||||
sleep 5
|
||||
./mc stat --no-list minio2/newbucket
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "expecting bucket to be present. exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc stat --no-list minio3/newbucket
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "expecting bucket to be present. exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc cp README.md minio2/newbucket/
|
||||
|
||||
sleep 5
|
||||
./mc stat --no-list minio1/newbucket/README.md
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "expecting object to be present. exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc stat --no-list minio3/newbucket/README.md
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "expecting object to be present. exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc rm minio3/newbucket/README.md
|
||||
sleep 5
|
||||
|
||||
./mc stat --no-list minio2/newbucket/README.md
|
||||
if [ $? -eq 0 ]; then
|
||||
echo "expected file to be deleted, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc stat --no-list minio1/newbucket/README.md
|
||||
if [ $? -eq 0 ]; then
|
||||
echo "expected file to be deleted, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
sleep 10
|
||||
./mc stat --no-list minio3/newbucket/lrgfile
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "expected object to be present, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
actual_checksum=$(./mc cat minio3/newbucket/lrgfile | md5sum)
|
||||
if [ "${expected_checksum}" != "${actual_checksum}" ]; then
|
||||
echo "replication failed on multipart objects expected ${expected_checksum} got ${actual_checksum}"
|
||||
exit
|
||||
fi
|
||||
rm ./lrgfile
|
||||
|
||||
./mc rm -r --versions --force minio1/newbucket/lrgfile
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "expected object to be present, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
sleep 5
|
||||
./mc stat --no-list minio1/newbucket/lrgfile
|
||||
if [ $? -eq 0 ]; then
|
||||
echo "expected object to be deleted permanently after replication, exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
./mc mb --with-lock minio3/newbucket-olock
|
||||
sleep 5
|
||||
|
||||
enabled_minio2=$(./mc stat --json minio2/newbucket-olock | jq -r .ObjectLock.enabled)
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "expected bucket to be mirrored with object-lock but not present, exiting..."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
if [ "${enabled_minio2}" != "Enabled" ]; then
|
||||
echo "expected bucket to be mirrored with object-lock enabled, exiting..."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
enabled_minio1=$(./mc stat --json minio1/newbucket-olock | jq -r .ObjectLock.enabled)
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "expected bucket to be mirrored with object-lock but not present, exiting..."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
if [ "${enabled_minio1}" != "Enabled" ]; then
|
||||
echo "expected bucket to be mirrored with object-lock enabled, exiting..."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
# "Test if most recent tag update is replicated"
|
||||
./mc tag set minio2/newbucket "key=val1"
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "expecting tag set to be successful. exiting.."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
sleep 10
|
||||
val=$(./mc tag list minio1/newbucket --json | jq -r .tagset | jq -r .key)
|
||||
if [ "${val}" != "val1" ]; then
|
||||
echo "expected bucket tag to have replicated, exiting..."
|
||||
exit_1
|
||||
fi
|
||||
# stop minio1 instance
|
||||
kill -9 ${site1_pid}
|
||||
# Update tag on minio2/newbucket when minio1 is down
|
||||
./mc tag set minio2/newbucket "key=val2"
|
||||
# create a new bucket on minio2. This should replicate to minio1 after it comes online.
|
||||
./mc mb minio2/newbucket2
|
||||
# delete bucket2 on minio2. This should replicate to minio1 after it comes online.
|
||||
./mc rb minio2/bucket2
|
||||
|
||||
# Restart minio1 instance
|
||||
minio server --address ":9001" --console-address ":10000" /tmp/minio1/{1...4} >/tmp/minio1_1.log 2>&1 &
|
||||
sleep 200
|
||||
|
||||
# Test whether most recent tag update on minio2 is replicated to minio1
|
||||
val=$(./mc tag list minio1/newbucket --json | jq -r .tagset | jq -r .key)
|
||||
if [ "${val}" != "val2" ]; then
|
||||
echo "expected bucket tag to have replicated, exiting..."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
# Test if bucket created/deleted when minio1 is down healed
|
||||
diff -q <(./mc ls minio1) <(./mc ls minio2) 1>/dev/null
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "expected 'bucket2' delete and 'newbucket2' creation to have replicated, exiting..."
|
||||
exit_1
|
||||
fi
|
||||
|
||||
# force a resync after removing all site replication
|
||||
./mc admin replicate rm --all --force minio1
|
||||
./mc rb minio2 --force --dangerous
|
||||
./mc admin replicate add minio1 minio2
|
||||
./mc admin replicate resync start minio1 minio2
|
||||
sleep 30
|
||||
|
||||
./mc ls -r --versions minio1/newbucket >/tmp/minio1.txt
|
||||
./mc ls -r --versions minio2/newbucket >/tmp/minio2.txt
|
||||
|
||||
out=$(diff -qpruN /tmp/minio1.txt /tmp/minio2.txt)
|
||||
ret=$?
|
||||
if [ $ret -ne 0 ]; then
|
||||
echo "BUG: expected no missing entries after replication resync: $out"
|
||||
exit 1
|
||||
fi
|
||||
262
docs/site-replication/run-replication-with-checksum-header.sh
Executable file
262
docs/site-replication/run-replication-with-checksum-header.sh
Executable file
@@ -0,0 +1,262 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
# shellcheck disable=SC2120
|
||||
exit_1() {
|
||||
cleanup
|
||||
|
||||
echo "minio1 ============"
|
||||
cat /tmp/minio1_1.log
|
||||
echo "minio2 ============"
|
||||
cat /tmp/minio2_1.log
|
||||
|
||||
exit 1
|
||||
}
|
||||
|
||||
cleanup() {
|
||||
echo -n "Cleaning up instances of MinIO ..."
|
||||
pkill -9 minio || sudo pkill -9 minio
|
||||
rm -rf /tmp/minio{1,2}
|
||||
echo "done"
|
||||
}
|
||||
|
||||
# Function to convert number to corresponding alphabet
|
||||
num_to_alpha() {
|
||||
local num=$1
|
||||
# ASCII value of 'a' is 97, so we add (num - 1) to 97 to get the corresponding alphabet
|
||||
local ascii_value=$((96 + num))
|
||||
# Convert the ASCII value to the character using printf
|
||||
printf "\\$(printf '%03o' "$ascii_value")"
|
||||
}
|
||||
|
||||
cleanup
|
||||
|
||||
export MINIO_CI_CD=1
|
||||
export MINIO_BROWSER=off
|
||||
export MINIO_ROOT_USER="minio"
|
||||
export MINIO_ROOT_PASSWORD="minio123"
|
||||
|
||||
# Download AWS CLI
|
||||
echo -n "Download and install AWS CLI"
|
||||
rm -rf /usr/local/aws-cli || sudo rm -rf /usr/local/aws-cli
|
||||
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
|
||||
unzip -qq awscliv2.zip
|
||||
./aws/install || sudo ./aws/install
|
||||
echo "done"
|
||||
|
||||
# Add credentials to ~/.aws/credentials
|
||||
if ! [ -d ~/.aws ]; then
|
||||
mkdir -p ~/.aws
|
||||
fi
|
||||
cat >~/.aws/credentials <<EOF
|
||||
[enterprise]
|
||||
region = us-east-1
|
||||
aws_access_key_id = minio
|
||||
aws_secret_access_key = minio123
|
||||
EOF
|
||||
|
||||
# Create certificates for TLS enabled MinIO
|
||||
echo -n "Setup certs for MinIO instances ..."
|
||||
wget -O certgen https://github.com/minio/certgen/releases/latest/download/certgen-linux-amd64 && chmod +x certgen
|
||||
./certgen --host localhost
|
||||
mkdir -p /tmp/certs
|
||||
mv public.crt /tmp/certs || sudo mv public.crt /tmp/certs
|
||||
mv private.key /tmp/certs || sudo mv private.key /tmp/certs
|
||||
echo "done"
|
||||
|
||||
# Start MinIO instances
|
||||
echo -n "Starting MinIO instances ..."
|
||||
CI=on MINIO_KMS_SECRET_KEY=minio-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=minio123 minio server --certs-dir /tmp/certs --address ":9001" --console-address ":10000" /tmp/minio1/{1...4}/disk{1...4} /tmp/minio1/{5...8}/disk{1...4} >/tmp/minio1_1.log 2>&1 &
|
||||
CI=on MINIO_KMS_SECRET_KEY=minio-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=minio123 minio server --certs-dir /tmp/certs --address ":9002" --console-address ":11000" /tmp/minio2/{1...4}/disk{1...4} /tmp/minio2/{5...8}/disk{1...4} >/tmp/minio2_1.log 2>&1 &
|
||||
echo "done"
|
||||
|
||||
if [ ! -f ./mc ]; then
|
||||
echo -n "Downloading MinIO client ..."
|
||||
wget -O mc https://dl.min.io/client/mc/release/linux-amd64/mc &&
|
||||
chmod +x mc
|
||||
echo "done"
|
||||
fi
|
||||
|
||||
export MC_HOST_minio1=https://minio:minio123@localhost:9001
|
||||
export MC_HOST_minio2=https://minio:minio123@localhost:9002
|
||||
|
||||
./mc ready minio1 --insecure
|
||||
./mc ready minio2 --insecure
|
||||
|
||||
# Prepare data for tests
|
||||
echo -n "Preparing test data ..."
|
||||
mkdir -p /tmp/data
|
||||
echo "Hello World" >/tmp/data/obj
|
||||
touch /tmp/data/mpartobj
|
||||
shred -s 500M /tmp/data/mpartobj
|
||||
echo "done"
|
||||
|
||||
# Add replication site
|
||||
./mc admin replicate add minio1 minio2 --insecure
|
||||
# sleep for replication to complete
|
||||
sleep 30
|
||||
|
||||
# Create bucket in source cluster
|
||||
echo "Create bucket in source MinIO instance"
|
||||
./mc mb minio1/test-bucket --insecure
|
||||
|
||||
# Load objects to source site with checksum header
|
||||
echo "Loading objects to source MinIO instance"
|
||||
OBJ_CHKSUM=$(openssl dgst -sha256 -binary </tmp/data/obj | base64)
|
||||
aws s3api --endpoint-url=https://localhost:9001 put-object --checksum-algorithm SHA256 --checksum-sha256 "${OBJ_CHKSUM}" --bucket test-bucket --key obj --body /tmp/data/obj --no-verify-ssl --profile enterprise
|
||||
|
||||
split -n 10 /tmp/data/mpartobj
|
||||
CREATE_MPART_OUT=$(aws s3api --endpoint-url=https://localhost:9001 create-multipart-upload --bucket test-bucket --key mpartobj --checksum-algorithm SHA256 --no-verify-ssl --profile enterprise)
|
||||
UPLOAD_ID=$(echo "${CREATE_MPART_OUT}" | jq '.UploadId' | sed 's/"//g')
|
||||
|
||||
PARTS=""
|
||||
for idx in {1..10}; do
|
||||
F_SUFFIX=$(num_to_alpha "$idx")
|
||||
PART_CHKSUM=$(openssl dgst -sha256 -binary <"xa${F_SUFFIX}" | base64)
|
||||
UPLOAD_PART_OUT=$(aws s3api --endpoint-url=https://localhost:9001 upload-part --checksum-algorithm SHA256 --checksum-sha256 "${PART_CHKSUM}" --bucket test-bucket --key mpartobj --part-number "${idx}" --body "xa${F_SUFFIX}" --upload-id "${UPLOAD_ID}" --no-verify-ssl --profile enterprise)
|
||||
PART_ETAG=$(echo "${UPLOAD_PART_OUT}" | jq '.ETag')
|
||||
if [ "${idx}" == 10 ]; then
|
||||
PARTS="${PARTS}{\"ETag\": ${PART_ETAG}, \"ChecksumSHA256\": \"${PART_CHKSUM}\", \"PartNumber\": ${idx}}"
|
||||
else
|
||||
PARTS="${PARTS}{\"ETag\": ${PART_ETAG}, \"ChecksumSHA256\": \"${PART_CHKSUM}\", \"PartNumber\": ${idx}},"
|
||||
fi
|
||||
done
|
||||
|
||||
echo "{\"Parts\":[${PARTS}]}" >fileparts.json
|
||||
jq <fileparts.json
|
||||
aws s3api --endpoint-url=https://localhost:9001 complete-multipart-upload --multipart-upload file://fileparts.json --bucket test-bucket --key mpartobj --upload-id "${UPLOAD_ID}" --no-verify-ssl --profile enterprise
|
||||
sleep 120
|
||||
|
||||
# List the objects from replicated site
|
||||
echo "Objects from replicated site"
|
||||
./mc ls minio2/test-bucket --insecure
|
||||
count1=$(./mc ls minio2/test-bucket/obj --insecure | wc -l)
|
||||
if [ "${count1}" -ne 1 ]; then
|
||||
echo "BUG: object minio1/test-bucket/obj not replicated"
|
||||
exit_1
|
||||
fi
|
||||
count2=$(./mc ls minio2/test-bucket/mpartobj --insecure | wc -l)
|
||||
if [ "${count2}" -ne 1 ]; then
|
||||
echo "BUG: object minio1/test-bucket/mpartobj not replicated"
|
||||
exit_1
|
||||
fi
|
||||
|
||||
# Stat the objects from source site
|
||||
echo "Stat minio1/test-bucket/obj"
|
||||
SRC_OUT_1=$(aws s3api --endpoint-url https://localhost:9001 head-object --bucket test-bucket --key obj --checksum-mode ENABLED --no-verify-ssl --profile enterprise)
|
||||
SRC_OUT_2=$(aws s3api --endpoint-url https://localhost:9001 head-object --bucket test-bucket --key mpartobj --checksum-mode ENABLED --no-verify-ssl --profile enterprise)
|
||||
SRC_OBJ_1_CHKSUM=$(echo "${SRC_OUT_1}" | jq '.ChecksumSHA256')
|
||||
SRC_OBJ_2_CHKSUM=$(echo "${SRC_OUT_2}" | jq '.ChecksumSHA256')
|
||||
SRC_OBJ_1_ETAG=$(echo "${SRC_OUT_1}" | jq '.ETag')
|
||||
SRC_OBJ_2_ETAG=$(echo "${SRC_OUT_2}" | jq '.ETag')
|
||||
|
||||
# Stat the objects from replicated site
|
||||
echo "Stat minio2/test-bucket/obj"
|
||||
DEST_OUT_1=$(aws s3api --endpoint-url https://localhost:9002 head-object --bucket test-bucket --key obj --checksum-mode ENABLED --no-verify-ssl --profile enterprise)
|
||||
DEST_OUT_2=$(aws s3api --endpoint-url https://localhost:9002 head-object --bucket test-bucket --key mpartobj --checksum-mode ENABLED --no-verify-ssl --profile enterprise)
|
||||
DEST_OBJ_1_CHKSUM=$(echo "${DEST_OUT_1}" | jq '.ChecksumSHA256')
|
||||
DEST_OBJ_2_CHKSUM=$(echo "${DEST_OUT_2}" | jq '.ChecksumSHA256')
|
||||
DEST_OBJ_1_ETAG=$(echo "${DEST_OUT_1}" | jq '.ETag')
|
||||
DEST_OBJ_2_ETAG=$(echo "${DEST_OUT_2}" | jq '.ETag')
|
||||
|
||||
# Check the replication of checksums and etags
|
||||
if [[ ${SRC_OBJ_1_CHKSUM} != "${DEST_OBJ_1_CHKSUM}" && ${DEST_OBJ_1_CHKSUM} != "" ]]; then
|
||||
echo "BUG: Checksums dont match for 'obj'. Source: ${SRC_OBJ_1_CHKSUM}, Destination: ${DEST_OBJ_1_CHKSUM}"
|
||||
exit_1
|
||||
fi
|
||||
if [[ ${SRC_OBJ_2_CHKSUM} != "${DEST_OBJ_2_CHKSUM}" && ${DEST_OBJ_2_CHKSUM} != "" ]]; then
|
||||
echo "BUG: Checksums dont match for 'mpartobj'. Source: ${SRC_OBJ_2_CHKSUM}, Destination: ${DEST_OBJ_2_CHKSUM}"
|
||||
exit_1
|
||||
fi
|
||||
if [ "${SRC_OBJ_1_ETAG}" != "${DEST_OBJ_1_ETAG}" ]; then
|
||||
echo "BUG: Etags dont match for 'obj'. Source: ${SRC_OBJ_1_ETAG}, Destination: ${DEST_OBJ_1_ETAG}"
|
||||
exit_1
|
||||
fi
|
||||
if [ "${SRC_OBJ_2_ETAG}" != "${DEST_OBJ_2_ETAG}" ]; then
|
||||
echo "BUG: Etags dont match for 'mpartobj'. Source: ${SRC_OBJ_2_ETAG}, Destination: ${DEST_OBJ_2_ETAG}"
|
||||
exit_1
|
||||
fi
|
||||
|
||||
echo "Set default encryption on "
|
||||
|
||||
# test if checksum header is replicated for encrypted objects
|
||||
# Enable SSE KMS for test-bucket bucket
|
||||
./mc encrypt set sse-kms minio-default-key minio1/test-bucket --insecure
|
||||
|
||||
# Load objects to source site with checksum header
|
||||
echo "Loading objects to source MinIO instance"
|
||||
OBJ_CHKSUM=$(openssl dgst -sha256 -binary </tmp/data/obj | base64)
|
||||
aws s3api --endpoint-url=https://localhost:9001 put-object --checksum-algorithm SHA256 --checksum-sha256 "${OBJ_CHKSUM}" --bucket test-bucket --key obj2 --body /tmp/data/obj --no-verify-ssl --profile enterprise
|
||||
|
||||
split -n 10 /tmp/data/mpartobj
|
||||
CREATE_MPART_OUT=$(aws s3api --endpoint-url=https://localhost:9001 create-multipart-upload --bucket test-bucket --key mpartobj2 --checksum-algorithm SHA256 --no-verify-ssl --profile enterprise)
|
||||
UPLOAD_ID=$(echo "${CREATE_MPART_OUT}" | jq '.UploadId' | sed 's/"//g')
|
||||
|
||||
PARTS=""
|
||||
for idx in {1..10}; do
|
||||
F_SUFFIX=$(num_to_alpha "$idx")
|
||||
PART_CHKSUM=$(openssl dgst -sha256 -binary <"xa${F_SUFFIX}" | base64)
|
||||
UPLOAD_PART_OUT=$(aws s3api --endpoint-url=https://localhost:9001 upload-part --checksum-algorithm SHA256 --checksum-sha256 "${PART_CHKSUM}" --bucket test-bucket --key mpartobj2 --part-number "${idx}" --body "xa${F_SUFFIX}" --upload-id "${UPLOAD_ID}" --no-verify-ssl --profile enterprise)
|
||||
PART_ETAG=$(echo "${UPLOAD_PART_OUT}" | jq '.ETag')
|
||||
if [ "${idx}" == 10 ]; then
|
||||
PARTS="${PARTS}{\"ETag\": ${PART_ETAG}, \"ChecksumSHA256\": \"${PART_CHKSUM}\", \"PartNumber\": ${idx}}"
|
||||
else
|
||||
PARTS="${PARTS}{\"ETag\": ${PART_ETAG}, \"ChecksumSHA256\": \"${PART_CHKSUM}\", \"PartNumber\": ${idx}},"
|
||||
fi
|
||||
done
|
||||
|
||||
echo "{\"Parts\":[${PARTS}]}" >fileparts.json
|
||||
jq <fileparts.json
|
||||
aws s3api --endpoint-url=https://localhost:9001 complete-multipart-upload --multipart-upload file://fileparts.json --bucket test-bucket --key mpartobj2 --upload-id "${UPLOAD_ID}" --no-verify-ssl --profile enterprise
|
||||
sleep 120
|
||||
|
||||
# List the objects from replicated site
|
||||
echo "Objects from replicated site"
|
||||
./mc ls minio2/test-bucket --insecure
|
||||
count1=$(./mc ls minio2/test-bucket/obj2 --insecure | wc -l)
|
||||
if [ "${count1}" -ne 1 ]; then
|
||||
echo "BUG: object minio1/test-bucket/obj2 not replicated"
|
||||
exit_1
|
||||
fi
|
||||
count2=$(./mc ls minio2/test-bucket/mpartobj2 --insecure | wc -l)
|
||||
if [ "${count2}" -ne 1 ]; then
|
||||
echo "BUG: object minio1/test-bucket/mpartobj2 not replicated"
|
||||
exit_1
|
||||
fi
|
||||
|
||||
# Stat the objects from source site
|
||||
echo "Stat minio1/test-bucket/obj2"
|
||||
SRC_OUT_1=$(aws s3api --endpoint-url https://localhost:9001 head-object --bucket test-bucket --key obj2 --checksum-mode ENABLED --no-verify-ssl --profile enterprise)
|
||||
SRC_OUT_2=$(aws s3api --endpoint-url https://localhost:9001 head-object --bucket test-bucket --key mpartobj2 --checksum-mode ENABLED --no-verify-ssl --profile enterprise)
|
||||
SRC_OBJ_1_CHKSUM=$(echo "${SRC_OUT_1}" | jq '.ChecksumSHA256')
|
||||
SRC_OBJ_2_CHKSUM=$(echo "${SRC_OUT_2}" | jq '.ChecksumSHA256')
|
||||
SRC_OBJ_1_ETAG=$(echo "${SRC_OUT_1}" | jq '.ETag')
|
||||
SRC_OBJ_2_ETAG=$(echo "${SRC_OUT_2}" | jq '.ETag')
|
||||
|
||||
# Stat the objects from replicated site
|
||||
echo "Stat minio2/test-bucket/obj2"
|
||||
DEST_OUT_1=$(aws s3api --endpoint-url https://localhost:9002 head-object --bucket test-bucket --key obj2 --checksum-mode ENABLED --no-verify-ssl --profile enterprise)
|
||||
DEST_OUT_2=$(aws s3api --endpoint-url https://localhost:9002 head-object --bucket test-bucket --key mpartobj2 --checksum-mode ENABLED --no-verify-ssl --profile enterprise)
|
||||
DEST_OBJ_1_CHKSUM=$(echo "${DEST_OUT_1}" | jq '.ChecksumSHA256')
|
||||
DEST_OBJ_2_CHKSUM=$(echo "${DEST_OUT_2}" | jq '.ChecksumSHA256')
|
||||
DEST_OBJ_1_ETAG=$(echo "${DEST_OUT_1}" | jq '.ETag')
|
||||
DEST_OBJ_2_ETAG=$(echo "${DEST_OUT_2}" | jq '.ETag')
|
||||
|
||||
# Check the replication of checksums and etags
|
||||
if [[ ${SRC_OBJ_1_CHKSUM} != "${DEST_OBJ_1_CHKSUM}" && ${DEST_OBJ_1_CHKSUM} != "" ]]; then
|
||||
echo "BUG: Checksums dont match for 'obj2'. Source: ${SRC_OBJ_1_CHKSUM}, Destination: ${DEST_OBJ_1_CHKSUM}"
|
||||
exit_1
|
||||
fi
|
||||
if [[ ${SRC_OBJ_2_CHKSUM} != "${DEST_OBJ_2_CHKSUM}" && ${DEST_OBJ_2_CHKSUM} != "" ]]; then
|
||||
echo "BUG: Checksums dont match for 'mpartobj2'. Source: ${SRC_OBJ_2_CHKSUM}, Destination: ${DEST_OBJ_2_CHKSUM}"
|
||||
exit_1
|
||||
fi
|
||||
if [ "${SRC_OBJ_1_ETAG}" != "${DEST_OBJ_1_ETAG}" ]; then
|
||||
echo "BUG: Etags dont match for 'obj2'. Source: ${SRC_OBJ_1_ETAG}, Destination: ${DEST_OBJ_1_ETAG}"
|
||||
exit_1
|
||||
fi
|
||||
if [ "${SRC_OBJ_2_ETAG}" != "${DEST_OBJ_2_ETAG}" ]; then
|
||||
echo "BUG: Etags dont match for 'mpartobj2'. Source: ${SRC_OBJ_2_ETAG}, Destination: ${DEST_OBJ_2_ETAG}"
|
||||
exit_1
|
||||
fi
|
||||
|
||||
cleanup
|
||||
256
docs/site-replication/run-sse-kms-object-replication.sh
Executable file
256
docs/site-replication/run-sse-kms-object-replication.sh
Executable file
@@ -0,0 +1,256 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
# shellcheck disable=SC2120
|
||||
exit_1() {
|
||||
cleanup
|
||||
|
||||
echo "minio1 ============"
|
||||
cat /tmp/minio1_1.log
|
||||
echo "minio2 ============"
|
||||
cat /tmp/minio2_1.log
|
||||
echo "minio3 ============"
|
||||
cat /tmp/minio3_1.log
|
||||
echo "minio4 ============"
|
||||
cat /tmp/minio4_1.log
|
||||
exit 1
|
||||
}
|
||||
|
||||
cleanup() {
|
||||
echo -n "Cleaning up instances of MinIO ..."
|
||||
pkill -9 minio || sudo pkill -9 minio
|
||||
pkill -9 kes || sudo pkill -9 kes
|
||||
rm -rf ${PWD}/keys
|
||||
rm -rf /tmp/minio{1,2,3,4}
|
||||
echo "done"
|
||||
}
|
||||
|
||||
cleanup
|
||||
|
||||
export MINIO_CI_CD=1
|
||||
export MINIO_BROWSER=off
|
||||
export MINIO_ROOT_USER="minio"
|
||||
export MINIO_ROOT_PASSWORD="minio123"
|
||||
TEST_MINIO_ENC_KEY="MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTIzNDU2Nzg5MDA"
|
||||
|
||||
# Create certificates for TLS enabled MinIO
|
||||
echo -n "Setup certs for MinIO instances ..."
|
||||
wget -O certgen https://github.com/minio/certgen/releases/latest/download/certgen-linux-amd64 && chmod +x certgen
|
||||
./certgen --host localhost
|
||||
mkdir -p /tmp/certs
|
||||
mv public.crt /tmp/certs || sudo mv public.crt /tmp/certs
|
||||
mv private.key /tmp/certs || sudo mv private.key /tmp/certs
|
||||
echo "done"
|
||||
|
||||
# Start MinIO instances
|
||||
echo -n "Starting MinIO instances ..."
|
||||
CI=on MINIO_KMS_SECRET_KEY=minio-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=minio123 minio server --certs-dir /tmp/certs --address ":9001" --console-address ":10000" /tmp/minio1/{1...4}/disk{1...4} /tmp/minio1/{5...8}/disk{1...4} >/tmp/minio1_1.log 2>&1 &
|
||||
CI=on MINIO_KMS_SECRET_KEY=minio-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=minio123 minio server --certs-dir /tmp/certs --address ":9002" --console-address ":11000" /tmp/minio2/{1...4}/disk{1...4} /tmp/minio2/{5...8}/disk{1...4} >/tmp/minio2_1.log 2>&1 &
|
||||
echo "done"
|
||||
|
||||
if [ ! -f ./mc ]; then
|
||||
echo -n "Downloading MinIO client ..."
|
||||
wget -O mc https://dl.min.io/client/mc/release/linux-amd64/mc &&
|
||||
chmod +x mc
|
||||
echo "done"
|
||||
fi
|
||||
|
||||
export MC_HOST_minio1=https://minio:minio123@localhost:9001
|
||||
export MC_HOST_minio2=https://minio:minio123@localhost:9002
|
||||
|
||||
./mc ready minio1 --insecure
|
||||
./mc ready minio2 --insecure
|
||||
|
||||
# Prepare data for tests
|
||||
echo -n "Preparing test data ..."
|
||||
mkdir -p /tmp/data
|
||||
echo "Hello from encrypted world" >/tmp/data/encrypted
|
||||
touch /tmp/data/mpartobj
|
||||
shred -s 500M /tmp/data/mpartobj
|
||||
touch /tmp/data/defpartsize
|
||||
shred -s 500M /tmp/data/defpartsize
|
||||
touch /tmp/data/custpartsize
|
||||
shred -s 500M /tmp/data/custpartsize
|
||||
echo "done"
|
||||
|
||||
# Add replication site
|
||||
./mc admin replicate add minio1 minio2 --insecure
|
||||
# sleep for replication to complete
|
||||
sleep 30
|
||||
|
||||
# Create bucket in source cluster
|
||||
echo "Create bucket in source MinIO instance"
|
||||
./mc mb minio1/test-bucket --insecure
|
||||
|
||||
# Enable SSE KMS for the bucket
|
||||
./mc encrypt set sse-kms minio-default-key minio1/test-bucket --insecure
|
||||
|
||||
# Load objects to source site
|
||||
echo "Loading objects to source MinIO instance"
|
||||
./mc cp /tmp/data/encrypted minio1/test-bucket --insecure
|
||||
./mc cp /tmp/data/mpartobj minio1/test-bucket/mpartobj --enc-c "minio1/test-bucket/mpartobj=${TEST_MINIO_ENC_KEY}" --insecure
|
||||
./mc cp /tmp/data/defpartsize minio1/test-bucket --insecure
|
||||
./mc put /tmp/data/custpartsize minio1/test-bucket --insecure --part-size 50MiB
|
||||
sleep 120
|
||||
|
||||
# List the objects from source site
|
||||
echo "Objects from source instance"
|
||||
./mc ls minio1/test-bucket --insecure
|
||||
count1=$(./mc ls minio1/test-bucket/encrypted --insecure | wc -l)
|
||||
if [ "${count1}" -ne 1 ]; then
|
||||
echo "BUG: object minio1/test-bucket/encrypted not found"
|
||||
exit_1
|
||||
fi
|
||||
count2=$(./mc ls minio1/test-bucket/mpartobj --insecure | wc -l)
|
||||
if [ "${count2}" -ne 1 ]; then
|
||||
echo "BUG: object minio1/test-bucket/mpartobj not found"
|
||||
exit_1
|
||||
fi
|
||||
count3=$(./mc ls minio1/test-bucket/defpartsize --insecure | wc -l)
|
||||
if [ "${count3}" -ne 1 ]; then
|
||||
echo "BUG: object minio1/test-bucket/defpartsize not found"
|
||||
exit_1
|
||||
fi
|
||||
count4=$(./mc ls minio1/test-bucket/custpartsize --insecure | wc -l)
|
||||
if [ "${count4}" -ne 1 ]; then
|
||||
echo "BUG: object minio1/test-bucket/custpartsize not found"
|
||||
exit_1
|
||||
fi
|
||||
|
||||
# List the objects from replicated site
|
||||
echo "Objects from replicated instance"
|
||||
./mc ls minio2/test-bucket --insecure
|
||||
repcount1=$(./mc ls minio2/test-bucket/encrypted --insecure | wc -l)
|
||||
if [ "${repcount1}" -ne 1 ]; then
|
||||
echo "BUG: object test-bucket/encrypted not replicated"
|
||||
exit_1
|
||||
fi
|
||||
repcount2=$(./mc ls minio2/test-bucket/mpartobj --insecure | wc -l)
|
||||
if [ "${repcount2}" -ne 1 ]; then
|
||||
echo "BUG: object test-bucket/mpartobj not replicated"
|
||||
exit_1
|
||||
fi
|
||||
repcount3=$(./mc ls minio2/test-bucket/defpartsize --insecure | wc -l)
|
||||
if [ "${repcount3}" -ne 1 ]; then
|
||||
echo "BUG: object test-bucket/defpartsize not replicated"
|
||||
exit_1
|
||||
fi
|
||||
repcount4=$(./mc ls minio2/test-bucket/custpartsize --insecure | wc -l)
|
||||
if [ "${repcount4}" -ne 1 ]; then
|
||||
echo "BUG: object test-bucket/custpartsize not replicated"
|
||||
exit_1
|
||||
fi
|
||||
|
||||
# Stat the objects from source site
|
||||
echo "Stat minio1/test-bucket/encrypted"
|
||||
./mc stat --no-list minio1/test-bucket/encrypted --insecure --json
|
||||
stat_out1=$(./mc stat --no-list minio1/test-bucket/encrypted --insecure --json)
|
||||
src_obj1_algo=$(echo "${stat_out1}" | jq '.metadata."X-Amz-Server-Side-Encryption"')
|
||||
src_obj1_keyid=$(echo "${stat_out1}" | jq '.metadata."X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id"')
|
||||
echo "Stat minio1/test-bucket/defpartsize"
|
||||
./mc stat --no-list minio1/test-bucket/defpartsize --insecure --json
|
||||
stat_out2=$(./mc stat --no-list minio1/test-bucket/defpartsize --insecure --json)
|
||||
src_obj2_algo=$(echo "${stat_out2}" | jq '.metadata."X-Amz-Server-Side-Encryption"')
|
||||
src_obj2_keyid=$(echo "${stat_out2}" | jq '.metadata."X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id"')
|
||||
echo "Stat minio1/test-bucket/custpartsize"
|
||||
./mc stat --no-list minio1/test-bucket/custpartsize --insecure --json
|
||||
stat_out3=$(./mc stat --no-list minio1/test-bucket/custpartsize --insecure --json)
|
||||
src_obj3_algo=$(echo "${stat_out3}" | jq '.metadata."X-Amz-Server-Side-Encryption"')
|
||||
src_obj3_keyid=$(echo "${stat_out3}" | jq '.metadata."X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id"')
|
||||
echo "Stat minio1/test-bucket/mpartobj"
|
||||
./mc stat --no-list minio1/test-bucket/mpartobj --enc-c "minio1/test-bucket/mpartobj=${TEST_MINIO_ENC_KEY}" --insecure --json
|
||||
stat_out4=$(./mc stat --no-list minio1/test-bucket/mpartobj --enc-c "minio1/test-bucket/mpartobj=${TEST_MINIO_ENC_KEY}" --insecure --json)
|
||||
src_obj4_etag=$(echo "${stat_out4}" | jq '.etag')
|
||||
src_obj4_size=$(echo "${stat_out4}" | jq '.size')
|
||||
src_obj4_md5=$(echo "${stat_out4}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"')
|
||||
|
||||
# Stat the objects from replicated site
|
||||
echo "Stat minio2/test-bucket/encrypted"
|
||||
./mc stat --no-list minio2/test-bucket/encrypted --insecure --json
|
||||
stat_out1_rep=$(./mc stat --no-list minio2/test-bucket/encrypted --insecure --json)
|
||||
rep_obj1_algo=$(echo "${stat_out1_rep}" | jq '.metadata."X-Amz-Server-Side-Encryption"')
|
||||
rep_obj1_keyid=$(echo "${stat_out1_rep}" | jq '.metadata."X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id"')
|
||||
echo "Stat minio2/test-bucket/defpartsize"
|
||||
./mc stat --no-list minio2/test-bucket/defpartsize --insecure --json
|
||||
stat_out2_rep=$(./mc stat --no-list minio2/test-bucket/defpartsize --insecure --json)
|
||||
rep_obj2_algo=$(echo "${stat_out2_rep}" | jq '.metadata."X-Amz-Server-Side-Encryption"')
|
||||
rep_obj2_keyid=$(echo "${stat_out2_rep}" | jq '.metadata."X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id"')
|
||||
echo "Stat minio2/test-bucket/custpartsize"
|
||||
./mc stat --no-list minio2/test-bucket/custpartsize --insecure --json
|
||||
stat_out3_rep=$(./mc stat --no-list minio2/test-bucket/custpartsize --insecure --json)
|
||||
rep_obj3_algo=$(echo "${stat_out3_rep}" | jq '.metadata."X-Amz-Server-Side-Encryption"')
|
||||
rep_obj3_keyid=$(echo "${stat_out3_rep}" | jq '.metadata."X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id"')
|
||||
echo "Stat minio2/test-bucket/mpartobj"
|
||||
./mc stat --no-list minio2/test-bucket/mpartobj --enc-c "minio2/test-bucket/mpartobj=${TEST_MINIO_ENC_KEY}" --insecure --json
|
||||
stat_out4_rep=$(./mc stat --no-list minio2/test-bucket/mpartobj --enc-c "minio2/test-bucket/mpartobj=${TEST_MINIO_ENC_KEY}" --insecure --json)
|
||||
rep_obj4_etag=$(echo "${stat_out4}" | jq '.etag')
|
||||
rep_obj4_size=$(echo "${stat_out4}" | jq '.size')
|
||||
rep_obj4_md5=$(echo "${stat_out4}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"')
|
||||
|
||||
# Check the algo and keyId of replicated objects
|
||||
if [ "${rep_obj1_algo}" != "${src_obj1_algo}" ]; then
|
||||
echo "BUG: Algorithm: '${rep_obj1_algo}' of replicated object: 'minio2/test-bucket/encrypted' doesn't match with source value: '${src_obj1_algo}'"
|
||||
exit_1
|
||||
fi
|
||||
if [ "${rep_obj1_keyid}" != "${src_obj1_keyid}" ]; then
|
||||
echo "BUG: KeyId: '${rep_obj1_keyid}' of replicated object: 'minio2/test-bucket/encrypted' doesn't match with source value: '${src_obj1_keyid}'"
|
||||
exit_1
|
||||
fi
|
||||
if [ "${rep_obj2_algo}" != "${src_obj2_algo}" ]; then
|
||||
echo "BUG: Algorithm: '${rep_obj2_algo}' of replicated object: 'minio2/test-bucket/defpartsize' doesn't match with source value: '${src_obj2_algo}'"
|
||||
exit_1
|
||||
fi
|
||||
if [ "${rep_obj2_keyid}" != "${src_obj2_keyid}" ]; then
|
||||
echo "BUG: KeyId: '${rep_obj2_keyid}' of replicated object: 'minio2/test-bucket/defpartsize' doesn't match with source value: '${src_obj2_keyid}'"
|
||||
exit_1
|
||||
fi
|
||||
if [ "${rep_obj3_algo}" != "${src_obj3_algo}" ]; then
|
||||
echo "BUG: Algorithm: '${rep_obj3_algo}' of replicated object: 'minio2/test-bucket/custpartsize' doesn't match with source value: '${src_obj3_algo}'"
|
||||
exit_1
|
||||
fi
|
||||
if [ "${rep_obj3_keyid}" != "${src_obj3_keyid}" ]; then
|
||||
echo "BUG: KeyId: '${rep_obj3_keyid}' of replicated object: 'minio2/test-bucket/custpartsize' doesn't match with source value: '${src_obj3_keyid}'"
|
||||
exit_1
|
||||
fi
|
||||
|
||||
# Check the etag, size and md5 of replicated SSEC object
|
||||
if [ "${rep_obj4_etag}" != "${src_obj4_etag}" ]; then
|
||||
echo "BUG: Etag: '${rep_obj4_etag}' of replicated object: 'minio2/test-bucket/mpartobj' doesn't match with source value: '${src_obj4_etag}'"
|
||||
exit_1
|
||||
fi
|
||||
if [ "${rep_obj4_size}" != "${src_obj4_size}" ]; then
|
||||
echo "BUG: Size: '${rep_obj4_size}' of replicated object: 'minio2/test-bucket/mpartobj' doesn't match with source value: '${src_obj4_size}'"
|
||||
exit_1
|
||||
fi
|
||||
if [ "${src_obj4_md5}" != "${rep_obj4_md5}" ]; then
|
||||
echo "BUG: MD5 checksum of object 'minio2/test-bucket/mpartobj' doesn't match with source. Expected: '${src_obj4_md5}', Found: '${rep_obj4_md5}'"
|
||||
exit_1
|
||||
fi
|
||||
|
||||
# Check content of replicated objects
|
||||
./mc cat minio2/test-bucket/encrypted --insecure
|
||||
./mc cat minio2/test-bucket/mpartobj --enc-c "minio2/test-bucket/mpartobj=${TEST_MINIO_ENC_KEY}" --insecure >/dev/null || exit_1
|
||||
./mc cat minio2/test-bucket/defpartsize --insecure >/dev/null || exit_1
|
||||
./mc cat minio2/test-bucket/custpartsize --insecure >/dev/null || exit_1
|
||||
|
||||
echo -n "Starting MinIO instances with different kms key ..."
|
||||
CI=on MINIO_KMS_SECRET_KEY=minio3-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=minio123 minio server --certs-dir /tmp/certs --address ":9003" --console-address ":10000" /tmp/minio3/disk{1...4} >/tmp/minio3_1.log 2>&1 &
|
||||
CI=on MINIO_KMS_SECRET_KEY=minio4-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g= MINIO_ROOT_USER=minio MINIO_ROOT_PASSWORD=minio123 minio server --certs-dir /tmp/certs --address ":9004" --console-address ":11000" /tmp/minio4/disk{1...4} >/tmp/minio4_1.log 2>&1 &
|
||||
echo "done"
|
||||
|
||||
export MC_HOST_minio3=https://minio:minio123@localhost:9003
|
||||
export MC_HOST_minio4=https://minio:minio123@localhost:9004
|
||||
|
||||
./mc ready minio3 --insecure
|
||||
./mc ready minio4 --insecure
|
||||
|
||||
./mc admin replicate add minio3 minio4 --insecure
|
||||
./mc mb minio3/bucket --insecure
|
||||
./mc cp --insecure --enc-kms minio3/bucket=minio3-default-key /tmp/data/encrypted minio3/bucket/x
|
||||
sleep 10
|
||||
st=$(./mc stat --json --no-list --insecure minio3/bucket/x | jq -r .replicationStatus)
|
||||
if [ "${st}" != "FAILED" ]; then
|
||||
echo "BUG: Replication succeeded when kms key is different"
|
||||
exit_1
|
||||
fi
|
||||
|
||||
cleanup
|
||||
196
docs/site-replication/run-ssec-object-replication-with-compression.sh
Executable file
196
docs/site-replication/run-ssec-object-replication-with-compression.sh
Executable file
@@ -0,0 +1,196 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
# shellcheck disable=SC2120
|
||||
exit_1() {
|
||||
cleanup
|
||||
|
||||
echo "minio1 ============"
|
||||
cat /tmp/minio1_1.log
|
||||
echo "minio2 ============"
|
||||
cat /tmp/minio2_1.log
|
||||
|
||||
exit 1
|
||||
}
|
||||
|
||||
cleanup() {
|
||||
echo -n "Cleaning up instances of MinIO ..."
|
||||
pkill minio || sudo pkill minio
|
||||
pkill -9 minio || sudo pkill -9 minio
|
||||
rm -rf /tmp/minio{1,2}
|
||||
echo "done"
|
||||
}
|
||||
|
||||
cleanup
|
||||
|
||||
export MINIO_CI_CD=1
|
||||
export MINIO_BROWSER=off
|
||||
export MINIO_ROOT_USER="minio"
|
||||
export MINIO_ROOT_PASSWORD="minio123"
|
||||
TEST_MINIO_ENC_KEY="MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTIzNDU2Nzg5MDA"
|
||||
|
||||
# Create certificates for TLS enabled MinIO
|
||||
echo -n "Setup certs for MinIO instances ..."
|
||||
wget -O certgen https://github.com/minio/certgen/releases/latest/download/certgen-linux-amd64 && chmod +x certgen
|
||||
./certgen --host localhost
|
||||
mkdir -p /tmp/certs
|
||||
mv public.crt /tmp/certs || sudo mv public.crt /tmp/certs
|
||||
mv private.key /tmp/certs || sudo mv private.key /tmp/certs
|
||||
echo "done"
|
||||
|
||||
# Start MinIO instances
|
||||
echo -n "Starting MinIO instances ..."
|
||||
minio server --certs-dir /tmp/certs --address ":9001" --console-address ":10000" /tmp/minio1/{1...4}/disk{1...4} /tmp/minio1/{5...8}/disk{1...4} >/tmp/minio1_1.log 2>&1 &
|
||||
minio server --certs-dir /tmp/certs --address ":9002" --console-address ":11000" /tmp/minio2/{1...4}/disk{1...4} /tmp/minio2/{5...8}/disk{1...4} >/tmp/minio2_1.log 2>&1 &
|
||||
echo "done"
|
||||
|
||||
if [ ! -f ./mc ]; then
|
||||
echo -n "Downloading MinIO client ..."
|
||||
wget -O mc https://dl.min.io/client/mc/release/linux-amd64/mc &&
|
||||
chmod +x mc
|
||||
echo "done"
|
||||
fi
|
||||
|
||||
export MC_HOST_minio1=https://minio:minio123@localhost:9001
|
||||
export MC_HOST_minio2=https://minio:minio123@localhost:9002
|
||||
|
||||
./mc ready minio1 --insecure
|
||||
./mc ready minio2 --insecure
|
||||
|
||||
# Prepare data for tests
|
||||
echo -n "Preparing test data ..."
|
||||
mkdir -p /tmp/data
|
||||
echo "Hello world" >/tmp/data/plainfile
|
||||
echo "Hello from encrypted world" >/tmp/data/encrypted
|
||||
touch /tmp/data/defpartsize
|
||||
shred -s 500M /tmp/data/defpartsize
|
||||
touch /tmp/data/mpartobj.txt
|
||||
shred -s 500M /tmp/data/mpartobj.txt
|
||||
echo "done"
|
||||
|
||||
# Enable compression for site minio1
|
||||
./mc admin config set minio1 compression enable=on extensions=".txt" --insecure
|
||||
./mc admin config set minio1 compression allow_encryption=off --insecure
|
||||
|
||||
# Create bucket in source cluster
|
||||
echo "Create bucket in source MinIO instance"
|
||||
./mc mb minio1/test-bucket --insecure
|
||||
|
||||
# Load objects to source site
|
||||
echo "Loading objects to source MinIO instance"
|
||||
./mc cp /tmp/data/plainfile minio1/test-bucket --insecure
|
||||
./mc cp /tmp/data/encrypted minio1/test-bucket/encrypted --enc-c "minio1/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure
|
||||
./mc cp /tmp/data/defpartsize minio1/test-bucket/defpartsize --enc-c "minio1/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure
|
||||
|
||||
# Below should fail as compression and SSEC used at the same time
|
||||
# DISABLED: We must check the response header to see if compression was actually applied
|
||||
#RESULT=$({ ./mc put /tmp/data/mpartobj.txt minio1/test-bucket/mpartobj.txt --enc-c "minio1/test-bucket/mpartobj.txt=${TEST_MINIO_ENC_KEY}" --insecure; } 2>&1)
|
||||
#if [[ ${RESULT} != *"Server side encryption specified with SSE-C with compression not allowed"* ]]; then
|
||||
# echo "BUG: Loading an SSE-C object to site with compression should fail. Succeeded though."
|
||||
# exit_1
|
||||
#fi
|
||||
|
||||
# Add replication site
|
||||
./mc admin replicate add minio1 minio2 --insecure
|
||||
# sleep for replication to complete
|
||||
sleep 30
|
||||
|
||||
# List the objects from source site
|
||||
echo "Objects from source instance"
|
||||
./mc ls minio1/test-bucket --insecure
|
||||
count1=$(./mc ls minio1/test-bucket/plainfile --insecure | wc -l)
|
||||
if [ "${count1}" -ne 1 ]; then
|
||||
echo "BUG: object minio1/test-bucket/plainfile not found"
|
||||
exit_1
|
||||
fi
|
||||
count2=$(./mc ls minio1/test-bucket/encrypted --insecure | wc -l)
|
||||
if [ "${count2}" -ne 1 ]; then
|
||||
echo "BUG: object minio1/test-bucket/encrypted not found"
|
||||
exit_1
|
||||
fi
|
||||
count3=$(./mc ls minio1/test-bucket/defpartsize --insecure | wc -l)
|
||||
if [ "${count3}" -ne 1 ]; then
|
||||
echo "BUG: object minio1/test-bucket/defpartsize not found"
|
||||
exit_1
|
||||
fi
|
||||
sleep 120
|
||||
|
||||
# List the objects from replicated site
|
||||
echo "Objects from replicated instance"
|
||||
./mc ls minio2/test-bucket --insecure
|
||||
repcount1=$(./mc ls minio2/test-bucket/plainfile --insecure | wc -l)
|
||||
if [ "${repcount1}" -ne 1 ]; then
|
||||
echo "BUG: object test-bucket/plainfile not replicated"
|
||||
exit_1
|
||||
fi
|
||||
repcount2=$(./mc ls minio2/test-bucket/encrypted --insecure | wc -l)
|
||||
if [ "${repcount2}" -ne 1 ]; then
|
||||
echo "BUG: object test-bucket/encrypted not replicated"
|
||||
exit_1
|
||||
fi
|
||||
repcount3=$(./mc ls minio2/test-bucket/defpartsize --insecure | wc -l)
|
||||
if [ "${repcount3}" -ne 1 ]; then
|
||||
echo "BUG: object test-bucket/defpartsize not replicated"
|
||||
exit_1
|
||||
fi
|
||||
|
||||
# Stat the SSEC objects from source site
|
||||
echo "Stat minio1/test-bucket/encrypted"
|
||||
./mc stat --no-list minio1/test-bucket/encrypted --enc-c "minio1/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure --json
|
||||
stat_out1=$(./mc stat --no-list minio1/test-bucket/encrypted --enc-c "minio1/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure --json)
|
||||
src_obj1_etag=$(echo "${stat_out1}" | jq '.etag')
|
||||
src_obj1_size=$(echo "${stat_out1}" | jq '.size')
|
||||
src_obj1_md5=$(echo "${stat_out1}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"')
|
||||
echo "Stat minio1/test-bucket/defpartsize"
|
||||
./mc stat --no-list minio1/test-bucket/defpartsize --enc-c "minio1/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json
|
||||
stat_out2=$(./mc stat --no-list minio1/test-bucket/defpartsize --enc-c "minio1/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json)
|
||||
src_obj2_etag=$(echo "${stat_out2}" | jq '.etag')
|
||||
src_obj2_size=$(echo "${stat_out2}" | jq '.size')
|
||||
src_obj2_md5=$(echo "${stat_out2}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"')
|
||||
|
||||
# Stat the SSEC objects from replicated site
|
||||
echo "Stat minio2/test-bucket/encrypted"
|
||||
./mc stat --no-list minio2/test-bucket/encrypted --enc-c "minio2/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure --json
|
||||
stat_out1_rep=$(./mc stat --no-list minio2/test-bucket/encrypted --enc-c "minio2/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure --json)
|
||||
rep_obj1_etag=$(echo "${stat_out1_rep}" | jq '.etag')
|
||||
rep_obj1_size=$(echo "${stat_out1_rep}" | jq '.size')
|
||||
rep_obj1_md5=$(echo "${stat_out1_rep}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"')
|
||||
echo "Stat minio2/test-bucket/defpartsize"
|
||||
./mc stat --no-list minio2/test-bucket/defpartsize --enc-c "minio2/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json
|
||||
stat_out2_rep=$(./mc stat --no-list minio2/test-bucket/defpartsize --enc-c "minio2/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json)
|
||||
rep_obj2_etag=$(echo "${stat_out2_rep}" | jq '.etag')
|
||||
rep_obj2_size=$(echo "${stat_out2_rep}" | jq '.size')
|
||||
rep_obj2_md5=$(echo "${stat_out2_rep}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"')
|
||||
|
||||
# Check the etag and size of replicated SSEC objects
|
||||
if [ "${rep_obj1_etag}" != "${src_obj1_etag}" ]; then
|
||||
echo "BUG: Etag: '${rep_obj1_etag}' of replicated object: 'minio2/test-bucket/encrypted' doesn't match with source value: '${src_obj1_etag}'"
|
||||
exit_1
|
||||
fi
|
||||
if [ "${rep_obj1_size}" != "${src_obj1_size}" ]; then
|
||||
echo "BUG: Size: '${rep_obj1_size}' of replicated object: 'minio2/test-bucket/encrypted' doesn't match with source value: '${src_obj1_size}'"
|
||||
exit_1
|
||||
fi
|
||||
if [ "${rep_obj2_etag}" != "${src_obj2_etag}" ]; then
|
||||
echo "BUG: Etag: '${rep_obj2_etag}' of replicated object: 'minio2/test-bucket/defpartsize' doesn't match with source value: '${src_obj2_etag}'"
|
||||
exit_1
|
||||
fi
|
||||
if [ "${rep_obj2_size}" != "${src_obj2_size}" ]; then
|
||||
echo "BUG: Size: '${rep_obj2_size}' of replicated object: 'minio2/test-bucket/defpartsize' doesn't match with source value: '${src_obj2_size}'"
|
||||
exit_1
|
||||
fi
|
||||
|
||||
# Check content of replicated SSEC objects
|
||||
./mc cat minio2/test-bucket/encrypted --enc-c "minio2/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure
|
||||
./mc cat minio2/test-bucket/defpartsize --enc-c "minio2/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure >/dev/null || exit_1
|
||||
|
||||
# Check the MD5 checksums of encrypted objects from source and target
|
||||
if [ "${src_obj1_md5}" != "${rep_obj1_md5}" ]; then
|
||||
echo "BUG: MD5 checksum of object 'minio2/test-bucket/encrypted' doesn't match with source. Expected: '${src_obj1_md5}', Found: '${rep_obj1_md5}'"
|
||||
exit_1
|
||||
fi
|
||||
if [ "${src_obj2_md5}" != "${rep_obj2_md5}" ]; then
|
||||
echo "BUG: MD5 checksum of object 'minio2/test-bucket/defpartsize' doesn't match with source. Expected: '${src_obj2_md5}', Found: '${rep_obj2_md5}'"
|
||||
exit_1
|
||||
fi
|
||||
|
||||
cleanup
|
||||
223
docs/site-replication/run-ssec-object-replication.sh
Executable file
223
docs/site-replication/run-ssec-object-replication.sh
Executable file
@@ -0,0 +1,223 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
# shellcheck disable=SC2120
|
||||
exit_1() {
|
||||
cleanup
|
||||
|
||||
echo "minio1 ============"
|
||||
cat /tmp/minio1_1.log
|
||||
echo "minio2 ============"
|
||||
cat /tmp/minio2_1.log
|
||||
|
||||
exit 1
|
||||
}
|
||||
|
||||
cleanup() {
|
||||
echo -n "Cleaning up instances of MinIO ..."
|
||||
pkill minio || sudo pkill minio
|
||||
pkill -9 minio || sudo pkill -9 minio
|
||||
rm -rf /tmp/minio{1,2}
|
||||
echo "done"
|
||||
}
|
||||
|
||||
cleanup
|
||||
|
||||
export MINIO_CI_CD=1
|
||||
export MINIO_BROWSER=off
|
||||
export MINIO_ROOT_USER="minio"
|
||||
export MINIO_ROOT_PASSWORD="minio123"
|
||||
TEST_MINIO_ENC_KEY="MDEyMzQ1Njc4OTAxMjM0NTY3ODkwMTIzNDU2Nzg5MDA"
|
||||
|
||||
# Create certificates for TLS enabled MinIO
|
||||
echo -n "Setup certs for MinIO instances ..."
|
||||
wget -O certgen https://github.com/minio/certgen/releases/latest/download/certgen-linux-amd64 && chmod +x certgen
|
||||
./certgen --host localhost
|
||||
mkdir -p /tmp/certs
|
||||
mv public.crt /tmp/certs || sudo mv public.crt /tmp/certs
|
||||
mv private.key /tmp/certs || sudo mv private.key /tmp/certs
|
||||
echo "done"
|
||||
|
||||
# Start MinIO instances
|
||||
echo -n "Starting MinIO instances ..."
|
||||
minio server --certs-dir /tmp/certs --address ":9001" --console-address ":10000" /tmp/minio1/{1...4}/disk{1...4} /tmp/minio1/{5...8}/disk{1...4} >/tmp/minio1_1.log 2>&1 &
|
||||
minio server --certs-dir /tmp/certs --address ":9002" --console-address ":11000" /tmp/minio2/{1...4}/disk{1...4} /tmp/minio2/{5...8}/disk{1...4} >/tmp/minio2_1.log 2>&1 &
|
||||
echo "done"
|
||||
|
||||
if [ ! -f ./mc ]; then
|
||||
echo -n "Downloading MinIO client ..."
|
||||
wget -O mc https://dl.min.io/client/mc/release/linux-amd64/mc &&
|
||||
chmod +x mc
|
||||
echo "done"
|
||||
fi
|
||||
|
||||
export MC_HOST_minio1=https://minio:minio123@localhost:9001
|
||||
export MC_HOST_minio2=https://minio:minio123@localhost:9002
|
||||
|
||||
./mc ready minio1 --insecure
|
||||
./mc ready minio2 --insecure
|
||||
|
||||
# Prepare data for tests
|
||||
echo -n "Preparing test data ..."
|
||||
mkdir -p /tmp/data
|
||||
echo "Hello world" >/tmp/data/plainfile
|
||||
echo "Hello from encrypted world" >/tmp/data/encrypted
|
||||
touch /tmp/data/defpartsize
|
||||
shred -s 500M /tmp/data/defpartsize
|
||||
touch /tmp/data/custpartsize
|
||||
shred -s 500M /tmp/data/custpartsize
|
||||
echo "done"
|
||||
|
||||
# Add replication site
|
||||
./mc admin replicate add minio1 minio2 --insecure
|
||||
# sleep for replication to complete
|
||||
sleep 30
|
||||
|
||||
# Create bucket in source cluster
|
||||
echo "Create bucket in source MinIO instance"
|
||||
./mc mb minio1/test-bucket --insecure
|
||||
|
||||
# Load objects to source site
|
||||
echo "Loading objects to source MinIO instance"
|
||||
set -x
|
||||
./mc cp /tmp/data/plainfile minio1/test-bucket --insecure
|
||||
./mc cp /tmp/data/encrypted minio1/test-bucket/encrypted --enc-c "minio1/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure
|
||||
./mc cp /tmp/data/defpartsize minio1/test-bucket/defpartsize --enc-c "minio1/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure
|
||||
./mc put /tmp/data/custpartsize minio1/test-bucket/custpartsize --enc-c "minio1/test-bucket/custpartsize=${TEST_MINIO_ENC_KEY}" --insecure --part-size 50MiB
|
||||
set +x
|
||||
sleep 120
|
||||
|
||||
# List the objects from source site
|
||||
echo "Objects from source instance"
|
||||
./mc ls minio1/test-bucket --insecure
|
||||
count1=$(./mc ls minio1/test-bucket/plainfile --insecure | wc -l)
|
||||
if [ "${count1}" -ne 1 ]; then
|
||||
echo "BUG: object minio1/test-bucket/plainfile not found"
|
||||
exit_1
|
||||
fi
|
||||
count2=$(./mc ls minio1/test-bucket/encrypted --insecure | wc -l)
|
||||
if [ "${count2}" -ne 1 ]; then
|
||||
echo "BUG: object minio1/test-bucket/encrypted not found"
|
||||
exit_1
|
||||
fi
|
||||
count3=$(./mc ls minio1/test-bucket/defpartsize --insecure | wc -l)
|
||||
if [ "${count3}" -ne 1 ]; then
|
||||
echo "BUG: object minio1/test-bucket/defpartsize not found"
|
||||
exit_1
|
||||
fi
|
||||
count4=$(./mc ls minio1/test-bucket/custpartsize --insecure | wc -l)
|
||||
if [ "${count4}" -ne 1 ]; then
|
||||
echo "BUG: object minio1/test-bucket/custpartsize not found"
|
||||
exit_1
|
||||
fi
|
||||
|
||||
# List the objects from replicated site
|
||||
echo "Objects from replicated instance"
|
||||
./mc ls minio2/test-bucket --insecure
|
||||
repcount1=$(./mc ls minio2/test-bucket/plainfile --insecure | wc -l)
|
||||
if [ "${repcount1}" -ne 1 ]; then
|
||||
echo "BUG: object test-bucket/plainfile not replicated"
|
||||
exit_1
|
||||
fi
|
||||
repcount2=$(./mc ls minio2/test-bucket/encrypted --insecure | wc -l)
|
||||
if [ "${repcount2}" -ne 1 ]; then
|
||||
echo "BUG: object test-bucket/encrypted not replicated"
|
||||
exit_1
|
||||
fi
|
||||
repcount3=$(./mc ls minio2/test-bucket/defpartsize --insecure | wc -l)
|
||||
if [ "${repcount3}" -ne 1 ]; then
|
||||
echo "BUG: object test-bucket/defpartsize not replicated"
|
||||
exit_1
|
||||
fi
|
||||
|
||||
repcount4=$(./mc ls minio2/test-bucket/custpartsize --insecure | wc -l)
|
||||
if [ "${repcount4}" -ne 1 ]; then
|
||||
echo "BUG: object test-bucket/custpartsize not replicated"
|
||||
exit_1
|
||||
fi
|
||||
|
||||
# Stat the SSEC objects from source site
|
||||
echo "Stat minio1/test-bucket/encrypted"
|
||||
./mc stat --no-list minio1/test-bucket/encrypted --enc-c "minio1/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure --json
|
||||
stat_out1=$(./mc stat --no-list minio1/test-bucket/encrypted --enc-c "minio1/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure --json)
|
||||
src_obj1_etag=$(echo "${stat_out1}" | jq '.etag')
|
||||
src_obj1_size=$(echo "${stat_out1}" | jq '.size')
|
||||
src_obj1_md5=$(echo "${stat_out1}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"')
|
||||
echo "Stat minio1/test-bucket/defpartsize"
|
||||
./mc stat --no-list minio1/test-bucket/defpartsize --enc-c "minio1/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json
|
||||
stat_out2=$(./mc stat --no-list minio1/test-bucket/defpartsize --enc-c "minio1/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json)
|
||||
src_obj2_etag=$(echo "${stat_out2}" | jq '.etag')
|
||||
src_obj2_size=$(echo "${stat_out2}" | jq '.size')
|
||||
src_obj2_md5=$(echo "${stat_out2}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"')
|
||||
echo "Stat minio1/test-bucket/custpartsize"
|
||||
./mc stat --no-list minio1/test-bucket/custpartsize --enc-c "minio1/test-bucket/custpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json
|
||||
stat_out3=$(./mc stat --no-list minio1/test-bucket/custpartsize --enc-c "minio1/test-bucket/custpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json)
|
||||
src_obj3_etag=$(echo "${stat_out3}" | jq '.etag')
|
||||
src_obj3_size=$(echo "${stat_out3}" | jq '.size')
|
||||
src_obj3_md5=$(echo "${stat_out3}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"')
|
||||
|
||||
# Stat the SSEC objects from replicated site
|
||||
echo "Stat minio2/test-bucket/encrypted"
|
||||
./mc stat --no-list minio2/test-bucket/encrypted --enc-c "minio2/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure --json
|
||||
stat_out1_rep=$(./mc stat --no-list minio2/test-bucket/encrypted --enc-c "minio2/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure --json)
|
||||
rep_obj1_etag=$(echo "${stat_out1_rep}" | jq '.etag')
|
||||
rep_obj1_size=$(echo "${stat_out1_rep}" | jq '.size')
|
||||
rep_obj1_md5=$(echo "${stat_out1_rep}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"')
|
||||
echo "Stat minio2/test-bucket/defpartsize"
|
||||
./mc stat --no-list minio2/test-bucket/defpartsize --enc-c "minio2/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json
|
||||
stat_out2_rep=$(./mc stat --no-list minio2/test-bucket/defpartsize --enc-c "minio2/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json)
|
||||
rep_obj2_etag=$(echo "${stat_out2_rep}" | jq '.etag')
|
||||
rep_obj2_size=$(echo "${stat_out2_rep}" | jq '.size')
|
||||
rep_obj2_md5=$(echo "${stat_out2_rep}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"')
|
||||
echo "Stat minio2/test-bucket/custpartsize"
|
||||
./mc stat --no-list minio2/test-bucket/custpartsize --enc-c "minio2/test-bucket/custpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json
|
||||
stat_out3_rep=$(./mc stat --no-list minio2/test-bucket/custpartsize --enc-c "minio2/test-bucket/custpartsize=${TEST_MINIO_ENC_KEY}" --insecure --json)
|
||||
rep_obj3_etag=$(echo "${stat_out3_rep}" | jq '.etag')
|
||||
rep_obj3_size=$(echo "${stat_out3_rep}" | jq '.size')
|
||||
rep_obj3_md5=$(echo "${stat_out3_rep}" | jq '.metadata."X-Amz-Server-Side-Encryption-Customer-Key-Md5"')
|
||||
|
||||
# Check the etag and size of replicated SSEC objects
|
||||
if [ "${rep_obj1_etag}" != "${src_obj1_etag}" ]; then
|
||||
echo "BUG: Etag: '${rep_obj1_etag}' of replicated object: 'minio2/test-bucket/encrypted' doesn't match with source value: '${src_obj1_etag}'"
|
||||
exit_1
|
||||
fi
|
||||
if [ "${rep_obj1_size}" != "${src_obj1_size}" ]; then
|
||||
echo "BUG: Size: '${rep_obj1_size}' of replicated object: 'minio2/test-bucket/encrypted' doesn't match with source value: '${src_obj1_size}'"
|
||||
exit_1
|
||||
fi
|
||||
if [ "${rep_obj2_etag}" != "${src_obj2_etag}" ]; then
|
||||
echo "BUG: Etag: '${rep_obj2_etag}' of replicated object: 'minio2/test-bucket/defpartsize' doesn't match with source value: '${src_obj2_etag}'"
|
||||
exit_1
|
||||
fi
|
||||
if [ "${rep_obj2_size}" != "${src_obj2_size}" ]; then
|
||||
echo "BUG: Size: '${rep_obj2_size}' of replicated object: 'minio2/test-bucket/defpartsize' doesn't match with source value: '${src_obj2_size}'"
|
||||
exit_1
|
||||
fi
|
||||
if [ "${rep_obj3_etag}" != "${src_obj3_etag}" ]; then
|
||||
echo "BUG: Etag: '${rep_obj3_etag}' of replicated object: 'minio2/test-bucket/custpartsize' doesn't match with source value: '${src_obj3_etag}'"
|
||||
exit_1
|
||||
fi
|
||||
if [ "${rep_obj3_size}" != "${src_obj3_size}" ]; then
|
||||
echo "BUG: Size: '${rep_obj3_size}' of replicated object: 'minio2/test-bucket/custpartsize' doesn't match with source value: '${src_obj3_size}'"
|
||||
exit_1
|
||||
fi
|
||||
|
||||
# Check content of replicated SSEC objects
|
||||
./mc cat minio2/test-bucket/encrypted --enc-c "minio2/test-bucket/encrypted=${TEST_MINIO_ENC_KEY}" --insecure
|
||||
./mc cat minio2/test-bucket/defpartsize --enc-c "minio2/test-bucket/defpartsize=${TEST_MINIO_ENC_KEY}" --insecure >/dev/null || exit_1
|
||||
./mc cat minio2/test-bucket/custpartsize --enc-c "minio2/test-bucket/custpartsize=${TEST_MINIO_ENC_KEY}" --insecure >/dev/null || exit_1
|
||||
|
||||
# Check the MD5 checksums of encrypted objects from source and target
|
||||
if [ "${src_obj1_md5}" != "${rep_obj1_md5}" ]; then
|
||||
echo "BUG: MD5 checksum of object 'minio2/test-bucket/encrypted' doesn't match with source. Expected: '${src_obj1_md5}', Found: '${rep_obj1_md5}'"
|
||||
exit_1
|
||||
fi
|
||||
if [ "${src_obj2_md5}" != "${rep_obj2_md5}" ]; then
|
||||
echo "BUG: MD5 checksum of object 'minio2/test-bucket/defpartsize' doesn't match with source. Expected: '${src_obj2_md5}', Found: '${rep_obj2_md5}'"
|
||||
exit_1
|
||||
fi
|
||||
if [ "${src_obj3_md5}" != "${rep_obj3_md5}" ]; then
|
||||
echo "BUG: MD5 checksum of object 'minio2/test-bucket/custpartsize' doesn't match with source. Expected: '${src_obj3_md5}', Found: '${rep_obj3_md5}'"
|
||||
exit_1
|
||||
fi
|
||||
|
||||
cleanup
|
||||
1
docs/site-replication/rw.json
Normal file
1
docs/site-replication/rw.json
Normal file
@@ -0,0 +1 @@
|
||||
{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["admin:*"]},{"Effect":"Allow","Action":["s3:*"],"Resource":["arn:aws:s3:::*"]}]}
|
||||
Reference in New Issue
Block a user